Project follow-up: Raspberry Pi with 433 MHz radios, mqtt and node-red

Now that my Arduino sensors to Raspberry Pi using 433 MHz radios project is done and I learned a lot about posting to the web and using a db, it was time to learn something new; mqtt is a very hot topic in the IoT world so this was my next target. Using an MQTT broker like mosquitto is not only cool but allows for decoupling of the different parts; for example, instead of writing a monolithic piece of code that does everything (read sensors, post to the web, save to a database, like in my previous project) and which needs a lot of work in order to add some new functionality, one could write a piece of code that for example, only reads sensors and publishes to an MQTT topic; then another piece of code can subscribe to that topic, get the sensor values and post them to the web; and yet another piece can subscribe to the same topic, get the values and save them to a database. And so on, the possibilities are endless and different parts of the system are independent and can be plugged in and out very easily. And since it is very easy to install mosquitto on a Raspberry Pi, I decided to go ahead.

To integrate mqtt in my C code I used mosquitto client library (libmosquitto) with some docs here: not a lot of help for a beginner but I found plenty of articles and example code. I ran into a lot of issues because the library that I installed was older and none of the example code worked; Roger (mosquitto's author) was kind enough to point me to a page explaining how to install the latest library so I installed libmosquitto-dev from this repo and it solved my problems. I also needed to add -lmosquitto to the Makefile.

After this code was done (if you are interested you can find it here), I had to decide what is the next piece so I decided on node-red: a chance for me to dig deeper into this cool technology and add to my experience from my previous projects. As Simen Sommerfeldt so eloquently makes the case in this great article, node-red is perfect as the "glue" between different pieces of code. To start with I decided to create a flow that reads sensor motion data from the MQTT broker and plays a sound via a python script. I already had a script but Simen's was much better so I used his - big thanks goes to him for sharing his work. The node-red flow was done in minutes; I then added 2 exec nodes to start the C code reading sensor data and the python script. As a side note, at this time I am running this exec nodes manually instead of starting them automatically when the flow starts, until I can figure out how to prevent multiple instance from running at the same time. This project is by no means as cool as Simen's moving skull but I will use as my Halloween project to play some random scary sounds when trick-or-treaters come to the door.

While I was working on this project I got an email from the great folks at dweet.io and freeboard about some great features they added recently. When I saw their email I decided to add another branch to my flow to get the sensors data and post it to dweet.io. Node-red is so awesome that I had all this done in a matter of minutes: I found a dweetio node-red node, installed it, added the nodes to post to dweet, and I also had a quick freechart done. The node-red flow is also in my github repo.

By the way, if you haven't yet, you really need to try dweet.io and freeboard: really awesome services with a free tier - big thanks to the guys at Bug Labs that are behind these services.

Arduino sensors to Raspberry Pi using 433 MHz radios

Finally getting to wrap up my first end-to-end project using Arduino and Raspberry Pi talking via inexpensive 433 MHz radios - the distance over which they work is pretty small but this instructables explains how to add an antenna which dramatically improved the distance. The main idea was to have multiple Arduinos with TX radios, each equipped with DHT and PIR sensors; they all send the data to the Raspberry Pi which posts it to the web and also writes it to a local database. In addition to the values from sensors, I am also sending the voltage level, hoping that this way I can monitor the discharge of the batteries (I am not sure if this works or not, I didn't use an analog pin for this, I am simply getting the voltage on the MCU, using code from this great instructables).

The code for both the sender (Arduino) and the receiver (Raspberry Pi) is in my github repo. The comments in the code pretty much explain everything (how it works, how to connect the sensors and radios, and so on). Here are just some quick ideas that may be interesting.

• I wanted to transmit all the data in one go so I combined 4 numbers into one long, as described in both sender and receiver code.
• The transmission between the radios is pretty flaky, that's why I am repeating the send a lot on the Arduino and there is code on the receiver to ignore identical values coming in a 30s interval. This is OK because transmissions from different stations have different station codes so even if the sensor values are the same, the actual overall value is different. Same for a motion sensor: the ON value is different than the OFF value so even if they happen a second apart, both will be received.
• I started by posting the data to SparkFun's data service, then added Dweet.io and in the end settled on ThingSpeak and I'm also saving all the data to a local SQLite database. See some notes about this later in this post.
• The code in github is ready to use: just change the stationCode for each Arduino station and use the right API keys in the Raspberry Pi code.

Notes about posting to the web: there are a lot of great IoT services out there with free tiers, to start with I chose the few mentioned above that have a simple API and that accept GET. Each free service has certain rate limits so I need to consider the data volume I will be posting. With one station only getting the temperature sensor data every 3 minutes, this means 5 posts in 15 min. The motion data is a bit more intensive, roughly 2 posts 5 sec apart (ON and OFF) every time the motion sensor is triggered which at the most (sensor triggered all the time) means 24 posts a minute. Because there is potentially a lot more motion data, I am using 2 different data streams (channels, feeds): one for temperature, humidity and voltage and one for motion; even so with a lot of stations there is the risk of missing data (according to each service rate limits as described below) so I decided to also implement a local SQLite database to store the data and eventually process it later and post it (maybe using node-red).

Rate limits:

1. SparkFun's data service is limited to making 100 log requests every 15 minutes. Given the 5 posts in 15 min, the temperature data stream can accommodate 20 stations without losing data which is more than I need. The motion data though may accommodate only 1 or 2 stations (more if there is not a lot of activity around them) without starting to lose data. I will post all the data I can, the remainder being lost.
2. Dweet.io holds on to the last 500 dweets over a 24 hour period so while there won't be data loss, the motion data will probably cover only a few hours at the most, even for just a couple stations.
3. ThingSpeak has a rate limit of an update per channel every 15 seconds so more than one station per channel will more than likely run into this limit (even 2 temp stations sending data every 15 minutes can happen to send data less than 15 sec apart); I could wait and retry until the post goes through but this would delay the code and risk missing data sent by the stations. Because of this, I decided to tie the local db data to this service: every time I post to ThingSpeak, I check the response: if > 0, the post was successful, I will log the data to the db with a flag of posted=true (1); if = 0, the post was not successful so I will log the data to the db with a flag of posted=false (0). Later I may create a node-red flow to get all the db data with posted=false and repost it to ThingSpeak; each row records the UTC time of the actual insert so I can send it along with the data, the measurements getting recorded at the actual time they took place not the time I post them.

If you find anything useful in this post and the code I am very happy. Most of this stuff is on the web but putting it together in a real project took some work; I am happy I did it and learned so much in the process. When I started I had no idea how to use a radio on either Arduino or Raspberry Pi, how to work with a PIR sensor and how to use interrupts, how to write any real C code, to work with sqlite and post to REST services on Raspberry Pi and much more. I used a lot of info from the web, from too many people and websites to thank individually but a big thank you! goes to everyone that provided help and information in a way or another.

SparkFun's Phant.io; libcurl and sqlite on Raspberry Pi

A few months ago SparkFun announced their new data service: phant.io. There are lots of similar services out there, some really awesome, some simple to use others not so much, some with great charting capabilities, others just a data store but I loved Phant from the beginning, mostly because it doesn't try to do a lot but what it does it does great. Also, its simplicity of use is awesome and is open-source. I was writing a JavaScript application at the time that needed a simple backend storage; I could use almost anything for that storage but when I saw the announcement I quickly decided to give it a try and indeed it was a breeze to post and retrieve data.

Fast forward to now: part of a larger project I am writing with a good friend of mine (in a nutshell an Arduino with DHT and PIR sensors gathers data and sends it using a 433 MHz radio to a Raspberry Pi) I wanted to post the data from a Raspberry Pi to some service on the net so again I chose phant.io. There are plenty of great examples on SparkFun's tutorial page (and other resources) about how to use Phant with Arduino, Python, Electric Imp but since the code running on my Raspberry Pi is written in C (because this was the simplest code I found for the 433 MHz receiver) I had to figure out how to use Phant using C. I tried for a couple hours to use sockets (found great code examples) but no matter what I did I was getting stuck in all kinds of issues. While looking around on stackoverflow for a solution I found a mention of libcurl - I don't remember who mentioned it but I owe him/her big thanks. I was a little bit reluctant to try it at first since I knew nothing about it but then another mention showed up so I decided to give it a try.

First thing I tried, adding #include <curl/curl.h> ended up with an error at compile time: file not found. So I had to install libcurl dev libraries which was really easy:

sudo apt-get install libcurl4-openssl-dev

and quickly verified that curl was now present in /usr/include/. Then I got the code from the very first example on libcurl page: simple.c but this time I got another error, curl_easy_init not defined or something like it. Another quick search and figured out I had to link the new library so I modified Makefile and added -lcurl. Now everything built and the first try with my test code posted data to data.sparkfun.com. The code as it stands today is here part of the repository that will eventually contain all the project code.

libcurl is really awesome: it took only minutes of getting the first version of the code done, in just a few lines (compared to 4 times as much for the socket code), the code is much more readable and the docs are awesome. Thanks to all the authors and contributors for such a great library!

Not completely related to the subject, in the same project I wanted to use a local database on the Raspberry Pi; a quick search got me to sqlite which is very easy to install and there are a lot of articles about it. One thing that most authors don't mention is that same with libcurl there is a library that needs to be installed so sqlite can be used in code and also that is has to be linked for the code to compile. Not a big deal but here are the steps in one place:

sudo apt-get install sqlite3 libsqlite3-dev
compiler link argument: -lsqlite3