Telegraf & InfluxDB & Grafana

TTN version: TTNv3
System: 64-bit
Last updated: January 5, 2024

For data storage and visualization, we use an open-source time series database InfluxDB and an open-source analytical and interactive visualization web application Grafana. Data from The Things Stack is sent via MQTT to the MQTT consumer Telegraf, which then stores the data in the InfluxDB database.

Dashboard

 

Prepare

  • Raspberry Pi 3/4 or Linux server
  • The Things Stack with active sensor (for example BME280)

 

OS/Server and Docker installation

This section describes two installation processes based on the operating system – Raspberry Pi OS and Ubuntu Server.

 

Raspberry Pi OS

Raspberry Pi OS setup
  1. From https://www.raspberrypi.com/software/ download Raspberry Pi Imager and install it.
  2. Insert the microSD card into the computer.
  3. Run Raspberry Pi Imager.
  4. Click on CHOOSE OS.
  5. Click on Raspberry Pi OS (other).
  6. Click on Raspberry Pi OS Lite (64-bit).
  7. Click on CHOOSE STORAGE.
  8. Select the inserted microSD card on which you want to install Raspberry Pi OS.
  9. Click on Next.
  10. Would you like to apply OS customisation settings? – EDIT SETTINGS.
  11. Set hostname.
  12. Set username and password. (we recommend a 17-digit password containing lower and upper case letters, numbers and symbols)
  13. If you will use WiFi – Configure wireless LAN. (optional)
  14. Set locale settings.
  15. Click on tab SERVICES.
  16. If you will be connecting remotely via SSH – Enable SSH – Use password authentication. (optional)
  17. Click on SAVE.
  18. Click on YES.
  19. Click on YES.
  20. Click on CONTINUE.
  21. Insert the microSD card into the Raspberry Pi.
  22. Turn on the Raspberry Pi.

 

Remote connection via SSH (optional)
  1. Connect to your router and find the IP address of your Raspberry Pi.
  2. From a Windows PC, you can connect using Command Prompt (CMD) or PuTTY.
  3. On Windows PC open Command Prompt (CMD).
  4. Type ssh -p port username@IP_Address (for example: ssh -p 22 loravsb@192.168.1.120).
  5. Type your password.

 

Post-installation steps
  1. Update, upgrade and reboot Raspberry Pi. This will ensure that any security vulnerabilities are patched: 
    $ sudo apt update
$ sudo apt full-upgrade
$ sudo reboot
  2. Connect again via SSH.
  3. Use a different port for SSH and disable root login. First, open the SSH configuration file with a text editor: 
    $ sudo nano /etc/ssh/sshd_config
  4. Locate the line with Port 22 and change it to the desired port number (for example, change the port to 2222). Locate the line with PermitRootLogin and change its value to no (this will disable root login via SSH): 
    Port 2222
PermitRootLogin no
  5. Save the changes and exit the text editor.
  6. Restart the SSH service to apply the changes: 
    $ sudo systemctl restart ssh
  7. Log out and connect again via SSH on the new port.
  8. Install and configure the built-in firewall, ufw, to limit incoming and outgoing network connections. In the sudo ufw limit 2222/tcp rule, change port 2222 to port you have chose for SSH above. 
    $ sudo apt install ufw
$ sudo ufw default deny incoming
$ sudo ufw default deny outgoing
$ sudo ufw limit 2222/tcp
$ sudo ufw allow out 123/udp
$ sudo ufw allow out to any port 53
$ sudo ufw allow out to any port 80
$ sudo ufw allow out to any port 443
$ sudo ufw enable

    Note: Firewall rules will have no effect on ports opened by Docker.

  9. Reboot Raspberry Pi: 
    $ sudo reboot

 

Docker
  1. Connect again via SSH.
  2. Install additional packages: 
    $ sudo apt install ca-certificates curl gnupg lsb-release
  3. Add Docker’s official GPG key: 
    $ sudo mkdir -p /etc/apt/keyrings
$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  4. Use the following command to set up the repository: 
    $ echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  5. Update the apt package index: 
    $ sudo apt update
  6. Install the latest Docker version: 
    $ sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
  7. Verify that the Docker Engine installation is successful by running the hello-world image: 
    $ sudo docker run hello-world
  8. Continue to the InfluxDB setup section below.

For more information on Docker installation, see https://docs.docker.com/engine/install/debian/.

 

Ubuntu Server

  1. From https://ubuntu.com/download/server download ISO image.

 

If the server’s operating system will run directly on the hardware, proceed as follows, otherwise skip these steps:
  1. Download Balena Etcher Portable from https://www.balena.io/etcher#download-etcher and run it.
  2. Click on Flash from file.
  3. Select the downloaded ISO Image.
  4. Insert the USB flash drive into the computer.
  5. Click on Select target.
  6. Select the USB flash drive.
  7. Click on Flash!
  8. Insert the USB flash drive into the server.

 

Ubuntu Server and Docker setup
  1. Boot from the USB flash drive / ISO image.
  2. Select Try or Install Ubuntu Server.
  3. Select Language.
  4. Choose Update to the new installer.
  5. Select keyboard layout.
  6. Choose the base for the installation – Ubuntu Server.
  7. Configure Network connections.
  8. Configure the proxy address or leave it blank.
  9. Configure Ubuntu archive mirror – default.
  10. Storage configuration – default.
  11. Profile setup – Enter username, password, …
  12. SSH Setup – Select Install OpenSSH server if you want to connect remotely. (optional)
  13. Features Server Snaps – Select a docker.
  14. Click on Reboot.

 

Remote connection via SSH (optional)
  1. Log in to the server.
  2. Type to the server’s terminal ip a.
  3. From a Windows PC, you can connect using Command Prompt (CMD) or PuTTY.
  4. On Windows PC open Command Prompt (CMD).
  5. Type ssh -p port username@IP_Address (for example: ssh -p 22 loravsb@192.168.1.120).
  6. Type your password.

 

Post-installation steps
  1. Update, upgrade and reboot server. This will ensure that any security vulnerabilities are patched: 
    $ sudo apt update
$ sudo apt full-upgrade
$ sudo reboot
  2. Connect again via SSH.
  3. Use a different port for SSH and disable root login. First, open the SSH configuration file with a text editor: 
    $ sudo nano /etc/ssh/sshd_config
  4. Locate the line with Port 22 and change it to the desired port number (for example, change the port to 2222). Locate the line with PermitRootLogin and change its value to no (this will disable root login via SSH): 
    Port 2222
PermitRootLogin no
  5. Save the changes and exit the text editor.
  6. Restart the SSH service to apply the changes: 
    $ sudo systemctl restart ssh
  7. Log out and connect again via SSH on the new port.
  8. Install and configure the built-in firewall, ufw, to limit incoming and outgoing network connections.
  9. Reboot server: 
    $ sudo reboot

 

Docker test
  1. Verify that the Docker Engine installation is successful by running the hello-world image: 
    $ sudo docker run hello-world

 

InfluxDB setup 

  1. Create internal Docker networks: 
    $ sudo docker network create myNetworkIandG
$ sudo docker network create myNetworkIandT
  2. Run this commands to run InfluxDB 1.8.10 and connect the container to second network: 
    $ sudo docker run -d --restart always --name=influxdb --network myNetworkIandG -v influxdb:/var/lib/influxdb influxdb:1.8.10
$ sudo docker network connect myNetworkIandT influxdb
  3. Connect to InfluxDB container: 
    $ sudo docker container exec -it influxdb /bin/sh
  4. Connect to InfluxDB using influx CLI, create an admin and two users, create a database and give the first user read permission (for Grafana) and the second user write permission (for Telegraf) to the created database: 
    # influx
> CREATE USER myNewAdmin WITH PASSWORD 'myAdminPassword' WITH ALL PRIVILEGES
> CREATE USER myNewUser WITH PASSWORD 'myUserPassword'
> CREATE USER myNewTelegrafUser WITH PASSWORD 'myUserTelegrafPassword'
> CREATE DATABASE ttsDatabase
> GRANT READ ON "ttsDatabase" TO "myNewUser"
> GRANT WRITE ON "ttsDatabase" TO "myNewTelegrafUser"
> exit
  5. Enable authentication by adding the following [http] section to the influxdb.conf file: 
    # cd etc/influxdb
# cat <<EOT >> influxdb.conf
>
>[http]
>  enabled = true
>  auth-enabled = true
>EOT
# exit
  6. Restart the influxdb container: 
    $ sudo docker restart influxdb

 

TTS MQTT Integration

  1. In TTS -> Applications -> YourAppName -> Integrations -> MQTT click on Generate new API key.
  2. In the Telegram section below, you will use the Connection credentials to connect to the TTS MQTT server.

MQTT Integration

 

Telegraf

  1. Run this command to create and edit the telegraf.conf file: 
    $ nano telegraf.conf
  2. Copy and paste the configuration below into your telegraf.conf file. Then in the [[inputs.mqtt_consumer]] section, enter the username and the generated API key. Next, in the [[outputs.influxdb]] section, edit the database, username and password that you created in the InfluxDB setup section above. 
    [agent]
    interval = "5s"
    round_interval = true
    metric_batch_size = 1000
    metric_buffer_limit = 10000
    collection_jitter = "5s"
    flush_interval = "10s"
    flush_jitter = "5s"

[[inputs.mqtt_consumer]]
    name_override = "tts"
    servers = ["tcp://eu1.cloud.thethings.network:1883"]
    connection_timeout = "30s"
    topics = ["v3/+/devices/#"]
    username = "THING_USERNAME"
    password = "THING_API_KEY"
    data_format = "json"

[[outputs.influxdb]]
    database = "ttsDatabase"
    urls = ["http://influxdb:8086"]
    username = "myNewTelegrafUser"
    password = "myUserTelegrafPassword"
  3. Run this commands to run Telegraf: 
    $ sudo docker run -d --restart always --name=telegraf --network myNetworkIandT -v $PWD/telegraf.conf:/etc/telegraf/telegraf.conf:ro telegraf

For more information on Telegraf installation, see https://hub.docker.com/_/telegraf.

 

Grafana

Grafana setup

  1. Navigate to the /home/username directory.
  2. Create a my-grafana-plugins folder: 
    $ mkdir my-grafana-plugins
  3. Run this command to run Grafana: 
    $ sudo docker run -d --restart always --name=grafana --network myNetworkIandG -p 3000:3000 -v "$(pwd)"/my-grafana-plugins:/var/lib/grafana/plugins grafana/grafana
  4. Now you can visit your Grafana web application from a browser in your internal network at http://IP_AddressOfServer:3000 (for example 192.168.1.120:3000).
  5. Log in with username admin and password admin.
  6. Change your Grafana account password.
  7. Click on Configuration -> Data sources -> Add data source.
  8. Choose InfluxDB.
  9. Query Language = InfluxQL
  10. URL = http://influxdb:8086
  11. Database = ttsDatabase
  12. User = myNewUserFromInfluxDB
  13. Password = myUserPasswordFromInfluxDB
  14. HTTP Method = GET
  15. Click on Save & test

For more information on Grafana installation, see https://hub.docker.com/r/grafana/grafana.

 

Grafana dashboard setup

  1. Click on Dashboards -> Browse -> New -> New Dashboard.
  2. Choose Add a new panel.
  3. In the FROM line click on select measurement and select tts.
  4. Click on + and select topic.
  5. Click on select tag value and select your TTS device.
  6. In the SELECT line click on value and select for example uplink_message_decoded_payload_temperature_1.
  7. Then change mean() to last().
  8. In the GROUP BY line change fill(null) to fill(none).
  9. You can change the Type, Title, Unit, Display name, Color scheme and many more in the menu on the right.
  10. Next, click on Save.
  11. Continue like this for the other panels.

Grafana Dashboard Setup

 

Grafana panel plugins

If you are interested in how to create your own Grafana panel plugin or import an existing one, follow this tutorial: Grafana panel plugin (for sending downlinks)