Introduction
In this article we will make a simple datalogger application running on the Raspberry Pi. It is compatible with Raspberry Pi Zero W, Pi 2 and Pi 3.
A data logger records data from something; usually some form of sensors but could also be I/O ports, switches and so forth. An important feature of any datalogger is the ability to timestamp each sample or measurement. For simplicity, our datalogger example will use the Raspberry Sense Hat add-on board and the companion easy to use Sense Hat python library.
The Sense HAT features an 8×8 RGB LED matrix, a mini joystick and the following sensors :
- Gyroscope
- Accelerometer
- Magnetometer
- Temperature
- Humidity
- Barometric pressure
We will logg data from the Sense Hat temperature, humidity and the barometric pressure sensors.
Influxdb Database:
Our simple datalogger needs some kind of place to organize and store the datalogger measurement samples. We have decided to use the Influxdb database to organize and store our data. InfluxDB is a Time Series Database built from the ground up to handle high write & query loads. InfluxDB is a custom high performance datastore written specifically for timestamped data, including DevOps monitoring, application metrics, IoT sensor data, & real-time analytics. It is possible to conserve space on your machine by configuring InfluxDB to keep data for a defined length of time, automatically expiring and deleting any unwanted data from the system. InfluxDB also offers a SQL-like query language for interacting with data. In our simple datalogger example, the data is stored on the SD card; hence make sure you use a good quality type of card as it will be written to many times. You could store the data another place as in the cloud or on an external hard drive or usb stick. That will be discussed in a future article.
Grafana; Presenting the datalogger results:
The data stored by our datalogger could also been written to a file as comma separated data and then import that file in excel and made some nice graphs and presentations. In this simple datalogger example, we want to try out the Grafana application. Grafana is an open-source, general purpose dashboard and graph composer, which runs as a web application.
Step by step Tutorial
1. Install the Influx database on the Raspberry Pi
a. Installing the Influx database is pretty simple. Please follow the instructions in this article.
b. Do not forget to start the Influxdb service on the Raspberry Pi by running this bash command in the terminal:
sudo service influxdb start
c. With InfluxDB installed, you are ready to start doing some awesome things. We will use the influx command line interface (CLI), which is included in all InfluxDB packages and is a lightweight and simple way to interact with the database. The CLI communicates with InfluxDB directly by making requests to the InfluxDB HTTP API over port 8086 by default.
The influx command should be available via the command line. Executing influx
in the Raspberry terminal will start the CLI and automatically connect to the local InfluxDB instance (assuming you have already started the server with service influxdb start or by running influxd directly). The output should look like this:
$ influx -precision rfc3339
Connected to http://localhost:8086 version 1.2.x
InfluxDB shell 1.2.x
>
exit
and hit return. A fresh install of InfluxDB has no databases (apart from the system _internal), so creating one is our first task. You can create a database with the CREATE DATABASE > CREATE DATABASE logger_db
>
> SHOW DATABASES
name: databases
---------------
name
_internal
logger_db
>
exit
.
2. Install the Grafana server on the Raspberry Pi
a. Installing the Grafana realtime graph dashboard is pretty simple. Please follow the instructions in this article.
b. Please remember to start the Grafana server:
sudo service grafana-server start
To configure the Grafana server to start at boot time:
sudo update-rc.d grafana-server defaults
3. Prepare and Write the datalogger python program
Simple python datalogger program:
A few command arguments have beeen added to make it possible for a user to specify the database name used, to set a measurement name and a run number. The measurement name and the run tag could be used to set different sessions during testing etc. You will need to install the InfluxDB Python module maintained here on github.
Install the python package :
sudo pip install influxdb
On Debian/Ubuntu, you can install it with this command:
sudo apt-get install python-influxdb
# -*- coding: utf-8 -*-
import argparse
import time
import datetime
import sys
from influxdb import InfluxDBClient
from sense_hat import SenseHat
sense=SenseHat()
# Set required InfluxDB parameters.
# (this could be added to the program args instead of beeing hard coded...)
host = "localhost" #Could also set local ip address
port = 8086
user = "root"
password = "root"
# Sample period (s).
# How frequently we will write sensor data from SenseHat to the database.
sampling_period = 5
def get_args():
'''This function parses and returns arguments passed in'''
# Assign description to the help doc
parser = argparse.ArgumentParser(description='Program writes measurements data from SenseHat to specified influx db.')
# Add arguments
parser.add_argument(
'-db','--database', type=str, help='Database name', required=True)
parser.add_argument(
'-sn','--session', type=str, help='Session', required=True)
now = datetime.datetime.now()
parser.add_argument(
'-rn','--run', type=str, help='Run number', required=False,default=now.strftime("%Y%m%d%H%M"))
# Array of all arguments passed to script
args=parser.parse_args()
# Assign args to variables
dbname=args.database
runNo=args.run
session=args.session
return dbname, session,runNo
def get_data_points():
# Get the three measurement values from the SenseHat sensors
temperature = sense.get_temperature()
pressure = sense.get_pressure()
humidity = sense.get_humidity()
# Get a local timestamp
timestamp=datetime.datetime.utcnow().isoformat()
print ("{0} {1} Temperature: {2}{3}C Pressure: {4}mb Humidity: {5}%" .format(session,runNo,
round(temperature,1),u'u00b0'.encode('utf8'),
round(pressure,3),round(humidity,1)))
# Create Influxdb datapoints (using lineprotocol as of Influxdb >1.1)
datapoints = [
{
"measurement": session,
"tags": {"runNum": runNo,
},
"time": timestamp,
"fields": {
"temperaturevalue":temperature,"pressurevalue":pressure,"humidityvalue":humidity
}
}
]
return datapoints
# Match return values from get_arguments()
# and assign to their respective variables
dbname, session, runNo =get_args()
print "Session: ", session
print "Run No: ", runNo
print "DB name: ", dbname
# Initialize the Influxdb client
client = InfluxDBClient(host, port, user, password, dbname)
try:
while True:
# Write datapoints to InfluxDB
datapoints=get_data_points()
bResult=client.write_points(datapoints)
print("Write points {0} Bresult:{1}".format(datapoints,bResult))
sense.show_message("OK")
# Wait for next sample
time.sleep(sampling_period)
# Run until keyboard ctrl-c
except KeyboardInterrupt:
print ("Program stopped by keyboard interrupt [CTRL_C] by user. ")
Save the script as ‘dataloggersensehat.py’ and run using python dataloggersensehat.py
. To run the script in the background and indefinately as a ssh user use nohup python dataloggersensehat.py &
.
Let us understand key points of the code listing above:
We import the InfluxDBClient class at the start of the program code.
We define two constants, one for the InfluxDB Host, which is the Public IP of your Compute Engine instance. Remember to replace it in the code.
The next statement is important. We initialize the InfluxDBClient with the hostname, port (8086 for the API) and provide it with the database name.
Now that we have the client object, we can then either query or insert database records.
To write data to InfluxDB, we use the `write_points` method that takes in a list of measurement points that we want to write to the database. Each measurement point, and which you should be familiar with now, specifies the measure, one or more tags (test1 and runNo), and the three field values. Also notice that we are passing in the timestamp so that we record the timestamp from the board that it was recorded at. We use the `datetime` package in python to help us get the Unix timestamp that we need to send to InfluxDB.
The result of the `write_points` method is of boolean type. If the operation is successful, it returns back a `true`.
4. Start the datalogger program
For simplicity we will run it in the python shell:
python dataloggersensehat.py -db=databasename -sn=test1
If you have forgotten to setup a valid Influxdb database you will get an error.
5. Setup Grafana dashboard and graphs for our datalogger
Unlike InfluxDB, Grafana doesn’t enable it’s service, so do this to enable at boot and start the service now:
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
Navigating your web browser to port 3000, you should be presented with the Grafana web GUI.
http://192.168.1.101:3000 or http://localhost:3000
Grafana has plenty of powerful querying tools to make lots of pretty and informative graphs. If you’re just starting out, we recommend taking a look at Grafana well made documentation for Basic Concept and Getting Started guides.
The first thing you will need to do is add the ‘logger’ database as a Datasource. Navigate to Datasource->Add New and fill in as below:
Fill in the database type (Influxdb) and a convinient name, then the name of the database we created “logger_db”. User and psw: “root” and “root”. Click the “Add” button when finished. Grafana will start testing the database and flag error or success on connection.
Next step is to setup a dashboard for our datalogger:
1
2.
3. Enter the name you specified for the datalogger_db and click on add query.
4. Set up the measurements metrics as below: The graph will be drawn as soon as the metrics is edited.
5.
Hi, Thanks for the example – very good!
I would be keen for your assistance please. When i run the above code, I get the following error:
File “dataloggersensehat.py”, line 90, in
bResult=client.write_points(datapoints) …
… influxdb.exceptions.InfluxDBClientError: 400: {“error”:”partial write: field type conflict: input field “pressurevalue” on measurement “test1″ is type float, already exists as type integer dropped=1”}
I have tried various ways of converting pressure to float, but nothing seems to work.
Any assistance you could provide would be very much appreciated.
Thanks and best regards,
Dan
HI
I am not an expert but had the same issue. Stopping and restarting the script sorted it out for me.
Did you ever sort this out – have the same problem – really want to make this work, but am a Python newb…
Hello,
Good job all is working for me with my raspberry 4 and new version of influx and grafana.
Just one thing please, how to display Temperature C° value instead of “OK”
->> “sense.show_message(“OK”)”
Thanks a lot.
Hi! What i can send to dashboard info x,y,z from magnet compass for magnetometer in Phyton script? Many thanks!
This is useless:
Session: test
Run No: 202006220647
DB name: logger_db
test 202006220647 Temperature: 30.3u00b0C Pressure: 989.762mb Humidity: 36.1%
Traceback (most recent call last):
File “dataloggersensehat.py”, line 82, in
bResult=client.write_points(datapoints)
File “/usr/lib/python2.7/dist-packages/influxdb/client.py”, line 170, in write_points
return self.write_points_with_precision(*args, **kwargs)
File “/usr/lib/python2.7/dist-packages/influxdb/client.py”, line 194, in write_points_with_precision
status_code=200
File “/usr/lib/python2.7/dist-packages/influxdb/client.py”, line 124, in request
raise InfluxDBClientError(response.content, response.status_code)
influxdb.client.InfluxDBClientError: 404: 404 page not found
HI
Many thanks for the tutorial. The whole system is working well. My biggest issue is that the databloggersensehat.py generates a new timestamp every time it starts. Therefore in Graphana, which is now in v7.0, won’t display anything unless the runNum parameter is changed to the latest one. Which means, if the pi needs rebooting all previously recorded data will be lost and and a new one needs to be selected automatically. Anybody has any recommendation to overcome this problem?
Thanks
Hi,
How can i specify the database name in the python code?
Can someone give me an example?
Thanks!
Works perfect on a Pi 3B+, thanks!
Thank you for this tutorial which works perfectly! I would like to know how we can also introduce the values of the accelerometer ?