Monday, November 4th, 2024
Latest updates

Datalogger example using Sense Hat, InfluxDB and Grafana




Introduction

In this article we will make a simple datalogger application running on the Raspberry Pi. It is compatible with Raspberry Pi Zero W, Pi 2 and Pi 3.
A data logger records data from something; usually some form of sensors but could also be I/O ports, switches and so forth. An important feature of any datalogger is the ability to timestamp each sample or measurement. For simplicity, our datalogger example will use the Raspberry Sense Hat add-on board and the companion easy to use Sense Hat python library.
The Sense HAT features an 8×8 RGB LED matrix, a mini joystick and the following sensors :

  • Gyroscope
  • Accelerometer
  • Magnetometer
  • Temperature
  • Humidity
  • Barometric pressure
Raspberry Pi Sense Hat

We will logg data from the Sense Hat temperature, humidity and the barometric pressure sensors.

Influxdb Database:
Our simple datalogger needs some kind of place to organize and store the datalogger measurement samples. We have decided to use the Influxdb database to organize and store our data. InfluxDB is a Time Series Database built from the ground up to handle high write & query loads. InfluxDB is a custom high performance datastore written specifically for timestamped data, including DevOps monitoring, application metrics, IoT sensor data, & real-time analytics. It is possible to conserve space on your machine by configuring InfluxDB to keep data for a defined length of time, automatically expiring and deleting any unwanted data from the system. InfluxDB also offers a SQL-like query language for interacting with data. In our simple datalogger example, the data is stored on the SD card; hence make sure you use a good quality type of card as it will be written to many times. You could store the data another place as in the cloud or on an external hard drive or usb stick. That will be discussed in a future article.

Grafana; Presenting the datalogger results:
The data stored by our datalogger could also been written to a file as comma separated data and then import that file in excel and made some nice graphs and presentations. In this simple datalogger example, we want to try out the Grafana application. Grafana is an open-source, general purpose dashboard and graph composer, which runs as a web application.

What is a datalogger?
A data logger is an electronic instrument that records measurements at set intervals over a period of time. Depending on the particular data logger, such measurements can include:
  • Air temperature
  • Relative humidity
  • AC/DC current and voltage
  • Differential pressure
  • Time-of-use (lights, motors, etc.)
  • Light intensity
  • Water temperature
  • Dissolved oxygen
  • Soil moisture
  • Rainfall
  • Wind speed and direction
  • Leaf wetness
  • Pulse signals
  • Room occupancy
  • Plug load
  • …and many more

Data loggers are typically compact, battery-powered devices equipped with an internal microprocessor, data storage, and one or more sensors. They can be deployed indoors, outdoors, and underwater, and can record data for up to years at a time, unattended. A data logger may be a single-unit, stand-alone device with internal sensors, which fits in the palm of a hand, or it may be a multi-channel data collection instrument equipped with one or more external sensors.
Why use data loggers?
Data loggers are valuable tools for anyone who wants to continuously monitor conditions without having to be on site. Environmental consultants, building managers, energy auditors, research scientists, and many more professionals all rely on data loggers.

Step by step Tutorial

1. Install the Influx database on the Raspberry Pi

a. Installing the Influx database is pretty simple. Please follow the instructions in this article.

b. Do not forget to start the Influxdb service on the Raspberry Pi by running this bash command in the terminal:

sudo service influxdb start

c. With InfluxDB installed, you are ready to start doing some awesome things. We will use the influx command line interface (CLI), which is included in all InfluxDB packages and is a lightweight and simple way to interact with the database. The CLI communicates with InfluxDB directly by making requests to the InfluxDB HTTP API over port 8086 by default.
The influx command should be available via the command line. Executing influx in the Raspberry terminal will start the CLI and automatically connect to the local InfluxDB instance (assuming you have already started the server with service influxdb start or by running influxd directly). The output should look like this:

$ influx -precision rfc3339
Connected to http://localhost:8086 version 1.2.x
InfluxDB shell 1.2.x
>
The command line is now ready to take input in the form of the Influx Query Language (a.k.a InfluxQL) statements. To exit the InfluxQL shell, type exit and hit return. A fresh install of InfluxDB has no databases (apart from the system _internal), so creating one is our first task. You can create a database with the CREATE DATABASE InfluxQL statement, where is the name of the database you wish to create. Names of databases can contain any unicode character as long as the string is double-quoted. Names can also be left unquoted if they contain only ASCII letters, digits, or underscores and do not begin with a digit. Throughout this guide, we will use the database name mydb:
> CREATE DATABASE logger_db
>
Now that the logger_db database is created, we’ll use the SHOW DATABASES statement to display all existing databases:
> SHOW DATABASES
name: databases
---------------
name
_internal
logger_db

>

We have created a database and it is ready to be used in our datalogger application. Exit the Influx shell by typing the command exit.

2. Install the Grafana server on the Raspberry Pi

a. Installing the Grafana realtime graph dashboard is pretty simple. Please follow the instructions in this article.
b. Please remember to start the Grafana server:

sudo service grafana-server start
This will start the grafana-server process as the grafana user, which was created during the package installation. The default HTTP port is 3000 and default user and psw is admin.

To configure the Grafana server to start at boot time:
sudo update-rc.d grafana-server defaults

3. Prepare and Write the datalogger python program

Simple python datalogger program:
A few command arguments have beeen added to make it possible for a user to specify the database name used, to set a measurement name and a run number. The measurement name and the run tag could be used to set different sessions during testing etc. You will need to install the InfluxDB Python module maintained here on github.

Install the python package :
sudo pip install influxdb

On Debian/Ubuntu, you can install it with this command:

sudo apt-get install python-influxdb


# -*- coding: utf-8 -*-
import argparse
import time
import datetime
import sys
from influxdb import InfluxDBClient
from sense_hat import SenseHat

sense=SenseHat()
 
# Set required InfluxDB parameters.
# (this could be added to the program args instead of beeing hard coded...)
host = "localhost" #Could also set local ip address
port = 8086
user = "root"
password = "root"
 
# Sample period (s).
# How frequently we will write sensor data from SenseHat to the database.
sampling_period = 5

def get_args():
    '''This function parses and returns arguments passed in'''
    # Assign description to the help doc
    parser = argparse.ArgumentParser(description='Program writes measurements data from SenseHat to specified influx db.')
    # Add arguments
    parser.add_argument(
        '-db','--database', type=str, help='Database name', required=True)
    parser.add_argument(
        '-sn','--session', type=str, help='Session', required=True)
    now = datetime.datetime.now()
    parser.add_argument(
        '-rn','--run', type=str, help='Run number', required=False,default=now.strftime("%Y%m%d%H%M"))
    
    # Array of all arguments passed to script
    args=parser.parse_args()
    # Assign args to variables
    dbname=args.database
    runNo=args.run
    session=args.session
    return dbname, session,runNo
    
def get_data_points():
    # Get the three measurement values from the SenseHat sensors
    temperature = sense.get_temperature()
    pressure = sense.get_pressure()
    humidity = sense.get_humidity()
    # Get a local timestamp
    timestamp=datetime.datetime.utcnow().isoformat()
    print ("{0} {1} Temperature: {2}{3}C Pressure: {4}mb Humidity: {5}%" .format(session,runNo,
    round(temperature,1),u'u00b0'.encode('utf8'),
    round(pressure,3),round(humidity,1)))
    
    # Create Influxdb datapoints (using lineprotocol as of Influxdb >1.1)
    datapoints = [
            {
                "measurement": session,
                "tags": {"runNum": runNo,
                },
                "time": timestamp,
                "fields": {
                    "temperaturevalue":temperature,"pressurevalue":pressure,"humidityvalue":humidity
                    }
                }
            ]
    return datapoints

# Match return values from get_arguments()
# and assign to their respective variables
dbname, session, runNo =get_args()   
print "Session: ", session
print "Run No: ", runNo
print "DB name: ", dbname

# Initialize the Influxdb client
client = InfluxDBClient(host, port, user, password, dbname)
        
try:
     while True:
        # Write datapoints to InfluxDB
        datapoints=get_data_points()
        bResult=client.write_points(datapoints)
        print("Write points {0} Bresult:{1}".format(datapoints,bResult))
        
        sense.show_message("OK")
            
        # Wait for next sample
        time.sleep(sampling_period)
        
        # Run until keyboard ctrl-c
except KeyboardInterrupt:
    print ("Program stopped by keyboard interrupt [CTRL_C] by user. ")
    

Save the script as ‘dataloggersensehat.py’ and run using python dataloggersensehat.py. To run the script in the background and indefinately as a ssh user use nohup python dataloggersensehat.py &.

Let us understand key points of the code listing above:
We import the InfluxDBClient class at the start of the program code.
We define two constants, one for the InfluxDB Host, which is the Public IP of your Compute Engine instance. Remember to replace it in the code.
The next statement is important. We initialize the InfluxDBClient with the hostname, port (8086 for the API) and provide it with the database name.
Now that we have the client object, we can then either query or insert database records.
To write data to InfluxDB, we use the `write_points` method that takes in a list of measurement points that we want to write to the database. Each measurement point, and which you should be familiar with now, specifies the measure, one or more tags (test1 and runNo), and the three field values. Also notice that we are passing in the timestamp so that we record the timestamp from the board that it was recorded at. We use the `datetime` package in python to help us get the Unix timestamp that we need to send to InfluxDB.
The result of the `write_points` method is of boolean type. If the operation is successful, it returns back a `true`.

4. Start the datalogger program

For simplicity we will run it in the python shell:

python dataloggersensehat.py -db=databasename -sn=test1

If you have forgotten to setup a valid Influxdb database you will get an error.

5. Setup Grafana dashboard and graphs for our datalogger

Unlike InfluxDB, Grafana doesn’t enable it’s service, so do this to enable at boot and start the service now:

sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Navigating your web browser to port 3000, you should be presented with the Grafana web GUI.
http://192.168.1.101:3000 or http://localhost:3000
Grafana login html image
Grafana has plenty of powerful querying tools to make lots of pretty and informative graphs. If you’re just starting out, we recommend taking a look at Grafana well made documentation for Basic Concept and Getting Started guides.

The first thing you will need to do is add the ‘logger’ database as a Datasource. Navigate to Datasource->Add New and fill in as below:

Grafana data source setup
Fill in the database type (Influxdb) and a convinient name, then the name of the database we created “logger_db”. User and psw: “root” and “root”. Click the “Add” button when finished. Grafana will start testing the database and flag error or success on connection.

Next step is to setup a dashboard for our datalogger:
1
Grafana create new dashboard

2.
Grafana edit new graph

3. Enter the name you specified for the datalogger_db and click on add query.

Grafana add database query

4. Set up the measurements metrics as below: The graph will be drawn as soon as the metrics is edited.

Grafana mertics configuration

5.

Here is the final result of the datalogger dashboard:

Sense Hat datalogger grafana dashboard

About Terje

Check Also

How to setup UART on the Raspberry Pi 3 GPIO

How to setup the UART on Raspberry Pi 3

Tips and tricks about configuring the UART on the Raspberry Pi 3 and Pi Zero W. The introduced bluetooth module makes this a little bit different.....

50 Raspberry Pi Linux Bash commands you’ll actually use

So, what is Bash? The terminal (or ‘command-line’) on a computer allows a user a …

How to Setup CLAWS Mail

What is Claws mail? Claws Mail is an email client aiming at being fast, easy-to-use …

10 comments

  1. Hi, Thanks for the example – very good!
    I would be keen for your assistance please. When i run the above code, I get the following error:

    File “dataloggersensehat.py”, line 90, in
    bResult=client.write_points(datapoints) …

    … influxdb.exceptions.InfluxDBClientError: 400: {“error”:”partial write: field type conflict: input field “pressurevalue” on measurement “test1″ is type float, already exists as type integer dropped=1”}

    I have tried various ways of converting pressure to float, but nothing seems to work.

    Any assistance you could provide would be very much appreciated.

    Thanks and best regards,

    Dan

  2. Hello,

    Good job all is working for me with my raspberry 4 and new version of influx and grafana.

    Just one thing please, how to display Temperature C° value instead of “OK”
    ->> “sense.show_message(“OK”)”

    Thanks a lot.

  3. Hi! What i can send to dashboard info x,y,z from magnet compass for magnetometer in Phyton script? Many thanks!

  4. This is useless:

    Session: test
    Run No: 202006220647
    DB name: logger_db
    test 202006220647 Temperature: 30.3u00b0C Pressure: 989.762mb Humidity: 36.1%
    Traceback (most recent call last):
    File “dataloggersensehat.py”, line 82, in
    bResult=client.write_points(datapoints)
    File “/usr/lib/python2.7/dist-packages/influxdb/client.py”, line 170, in write_points
    return self.write_points_with_precision(*args, **kwargs)
    File “/usr/lib/python2.7/dist-packages/influxdb/client.py”, line 194, in write_points_with_precision
    status_code=200
    File “/usr/lib/python2.7/dist-packages/influxdb/client.py”, line 124, in request
    raise InfluxDBClientError(response.content, response.status_code)
    influxdb.client.InfluxDBClientError: 404: 404 page not found

  5. HI

    Many thanks for the tutorial. The whole system is working well. My biggest issue is that the databloggersensehat.py generates a new timestamp every time it starts. Therefore in Graphana, which is now in v7.0, won’t display anything unless the runNum parameter is changed to the latest one. Which means, if the pi needs rebooting all previously recorded data will be lost and and a new one needs to be selected automatically. Anybody has any recommendation to overcome this problem?
    Thanks

  6. Hi,

    How can i specify the database name in the python code?
    Can someone give me an example?

    Thanks!

  7. Works perfect on a Pi 3B+, thanks!

  8. Thank you for this tutorial which works perfectly! I would like to know how we can also introduce the values ​​of the accelerometer ?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.