Awesome free services from Sense Tecnic: FRED and WoTKit

When Mike from Sense Tecnic announced FRED on the node-red Google group I jumped right away at the opportunity of trying it: and I'm happy I did because it is great! Not only that but the entire team behind it is awesome: the few issues people noticed were resolved in matters of minutes. Of course, having played with node-red for a while now, as you can see in other posts, I have it installed on all my machines, including my Raspberry Pi and I even tried it with Docker. But having an always on node-red instance that I can go to so I can try stuff and learn about node-red is simply awesome; in a matter of days I created a flow that gets my owntracks coords via mqtt, another that monitors a feed of newly released blues albums and highlights the ones I am interested in, one that shows me Top 10 blues and jazz tracks from iTunes and others.

One thing that may keep some from using FRED is the fact that being a hosted environment there are nodes that are not supported like Raspberry Pi or Arduino related. But Sense Tecnic has a solution for this as well: WoTKit. Using this platform one can publish sensor data, aggregate it and display it on dashboards. I have to admit that I've seen WoTKit before but I never sat down to read all the docs. However, when Mike mentioned it to me a couple days ago, I decided to give it a try. Same as FRED, all I can say is that is great! Not only that a lot of work has gone into this platform, what is really great is how responsive and helpful the team is. At first, I started to go through the docs and while a great read, I was stumbling a bit. There are tons of examples on the site for Python and curl but I wanted to get into it faster, preferably using node-red (of course, via FRED) but couldn't find anything. This was solved right away when Roberto from the same team sent me a link to an article exactly about this: node-red and WoTKit. The integration is perfect: as mentioned in the article, WoTKit expects a JSON structure of name:value pairs for sensor data which is so easy to produce from mode-red/FRED.

I won't go into much detail, please read the article I mentioned and all will be clear. Just to see how easy is though, let's go back to my owntracks flow. From owntracks, I get a message payload containing: lat, lon, tst (timestamp in seconds) and a few other fields. WoTKit expects the coordinates in fields named lat and lng and also expects an optional milliseconds timestamp as a long in a field with the same name: timestamp. Given the original payload, all I had to do is add 2 more properties to it, like:

payload.lng = payload.lon;
payload.timestamp = payload.tst*1000;

then added a WoTKit output node to the flow, created the sensor on the WoTKit platform and in a matter of minutes, I had a dashboard showing me the trail of last 20 coordinates posted by owntracks. Simply beautiful!

Another example: I love Scrabble and I play quite a lot on my phone using various apps. One of these is WordFeud for which the great people at Feudia are organizing tournaments. One of the things Feudia keeps track of is the players rating and ranking. When I first started using Feudia I decided to keep track of my rating but couldn't find a way to see the history so once in a while I was saving the values in a Google spreadsheet with a couple charts; FRED and WoTKit gave me an idea of how I can do this automatically.

I created a sensor with 2 fields: rank and rating, then created a simple flow using an http request node to read the Hall Of Fame page then used an html node to extract the info from it; one of reasons I went this route is that for a while now I wanted to figure out how the html node works - it took me a while but after several tries I was able to select the div enclosing the ranking table and extract all the values in its children in an array, that looks like this:

[ "#", "1", "\n \n \t\t", "Deuxchevauxtje", "1956,80", "", "#", "2", "\n \n \t\t", "nte...", "1936,37", "", "#", "3", "\n \n \t\t", "Pix...", "1930,94", "", "#", "4", "\n \n \t\t", "por...", "1922,31", ""....]

In this array, I look for my username and then get the rank 2 items in the array previous to it and the rating 1 item after (since only first 200 players are ranked, if for some reason I will drop lower than 200 and I won't find the username in the list anymore, I simply won't post anything to WoTKit) as you can see in my simple flow below:

[{"id":"b622d78d.49dd28","type":"wotkit-credentials","nickname":"Default","url":"http://wotkit.sensetecnic.com"},{"id":"67bc4260.9843bc","type":"html","name":"","tag":"div.rnk-tm>","ret":"text","as":"single","x":323,"y":265,"z":"8e66ad17.71995","wires":[["e616f09.f19e91"]]},{"id":"2bc264d6.d43d9c","type":"inject","name":"once a day","topic":"","payload":"","payloadType":"date","repeat":"86400","crontab":"","once":false,"x":103,"y":30.090909004211426,"z":"8e66ad17.71995","wires":[["f5e6e137.0a192"]]},{"id":"f5e6e137.0a192","type":"http request","name":"feudia halloffame","method":"GET","ret":"txt","url":"http://www.feudia.com/wordfeud/halloffame.html","x":191,"y":149,"z":"8e66ad17.71995","wires":[["67bc4260.9843bc"]]},{"id":"e616f09.f19e91","type":"function","name":"extract user info","func":"var allInfo = msg.payload;\nvar payload = {};\npayload.found = false;\nfor (i = 0; i < allInfo.length; i++) {\n if(allInfo[i] === 'merlin13') {\n payload.found = true;\n payload.username = \"merlin13\";\n payload.rank = Number(allInfo[i-2]);\n payload.rating = Number(allInfo[i+1].replace(',','.'));\n }\n}\nmsg.payload = payload;\nmsg.headers = {\n \"Content-Type\":\"application/json\"\n};\nreturn msg;","outputs":1,"valid":true,"x":500,"y":187,"z":"8e66ad17.71995","wires":[["902fe43e.6fd018"]]},{"id":"b664ddf4.499b2","type":"wotkit out","name":"feudia sensor","sensor":"claudiuo.feudia","login":"b622d78d.49dd28","x":769,"y":210,"z":"8e66ad17.71995","wires":[]},{"id":"902fe43e.6fd018","type":"switch","name":"","property":"payload.found","rules":[{"t":"true"},{"t":"else"}],"checkall":"true","outputs":2,"x":578,"y":296,"z":"8e66ad17.71995","wires":[["b664ddf4.499b2","33f1575a.cc0ea8"],["33f1575a.cc0ea8"]]},{"id":"33f1575a.cc0ea8","type":"debug","name":"","active":true,"console":"false","complete":"false","x":759,"y":336,"z":"8e66ad17.71995","wires":[]}]

Added the WoTKit output node and all was done! Well, almost. The payload looked great in debug:

{ "found": true, "username": "merlin13", "rank": 150, "rating": 1691.32 }

but no matter what I tried I was getting errors from WoTKit when posting the data. Finally, I decided to change the debug node to print the entire message, not just the payload and noticed the content-type was wrong:

{ "topic": "", "payload": { "username": "merlin13", "rank": 150, "rating": 1691.32 }, "statusCode": 200, "headers": {..."content-type": "text/html; charset=UTF-8" } }

So I added a header to my message as seen in the flow above:

msg.headers = {"Content-Type":"application/json"}

and everything worked like magic. Now my sensor gets data once a day and I also have a dashboard with the 2 charts identical with the original ones in my Google spreadsheet. Disregarding the content-type issue which was my fault, everything was done in less than an hour and now I have a dashboard that updates automatically so I don't have to gather data manually now and then.

Of course, as you see my "sensor" is not really a sensor in the real sense of the word but the concept is the same: instead of using owntracks or scraping the Feudia page, I could as well have a sensor hooked up to my Raspberry Pi or an Arduino, send data to WoTKit directly or to FRED for further processing and then WoTKit and the gap between a physical device and a hosted IoT service is closed, also using a hosted node-red instance in the process. I don't know if I can convey how awesome this is!

If you want to see the beauty of FRED and WoTKit in action, you should definitely try these services: as mentioned in the title, not only they are awesome but also free. Huge thanks to the Sense Tecnic team, especially to Mike and Roberto for all their awesome work, for offering these great services for free and for being so patient and helpful and responsive!

Cloud9, resin.io, Cylon.js - all coming together

As I mentioned in my previous post, I am really happy I discovered Cylon.js and was able to make basic stuff working. This is all cool but I wanted to be able to interact with my robot over the net so I thought it's time to try the API plugins the framework offers. To make things more fun and learn more in the process, I decided to use resin.io for deployment: this way I can update the code and test changes without being close to my Raspberry Pi all the time. I know it is possible but never tried to have a git project with multiple remotes; this is the perfect time for me to learn how this works since resin.io works by pushing code to the resin remote but I also want to be able to push changes to github. And because I don't want to be tied to my local machine, I decided to use Cloud9 for this project and push the code from there directly to both resin and github - which works great as you'll see below. By the way, Cloud9 is similar with Codenvy but the support for node.js is better (at least from what I know at this time) and having access to the entire VM and the command line makes it awesome; it is like working on a local machine but a lot better since it is in the cloud and accessible via a browser from anywhere.

This post is not really about the code itself: it is a work in progress that can be seen in my repo; instead, this post is about all of the tools coming together with a special nod to resin.io.

To start I read a lot of the Cylon.js docs and was able to put together a test robot without an actual device (using loopback instead) to which I plan to send commands using one of the API examples on the site; as a side note, the robot code only has generic commands like cmd1, cmd2 and so on instead of having commands like toggle and turnOn because this setup will let me change the actual code a command is executing while a client may never need to change. Going back to the API idea, I decided to start with the simplest API plugin (HTTP) even if there are no examples for it on the site. Unfortunately because I want to access my RasPi from outside my network, I don't know the IP (which will be assigned dynamically by resin) and the HTTP API needs to be configured with an IP; I am pretty sure there are solutions for this but instead of digging more, I decided to try the MQTT API which is tied only to a broker and doesn't need a definite IP. The client code is also very simple at this time but I hope it will evolve as I find some time; in the end though, I plan to issue the API commands via node-red which integrates very easily with MQTT.

It was very easy to start with Cloud9: I connected it to my github account, then created a new node.js workspace, there are plenty of docs on the site. And since Cloud9 gives access to the underlying OS, it was also easy to install libusb-dev (needed for Digispark as mentioned in my previous post) and also install all the node modules I need to start with; here are the commands for reference (last module is only needed for the client and I used the --save option so all the modules are registered automatically in package.json):

sudo apt-get install libusb-dev
npm install cylon cylon-digispark cylon-api-mqtt mqtt --save

Next thing was to add resin.io as a secondary remote which was pretty easy:

git remote add resin git@git.resin.io:username/application_name.git

Then all works as normal, git add/commit/push. The only special thing I needed to do was figure out how to install libusb-dev in the resin image. After some search on the web, I found out I can add a "preinstall" script to package.json. This was easy but it took me quite a while to figure out how to install this library because the only one found by apt-get was libusb-0.1-4 and not libusb-dev which I needed. After a lot of fiddling, I asked in the resin.io forum and the answer was quite simple: add apt-get update before the apt-get libusb-dev, as seen in the current package.json. A new push to the resin remote built the image without errors this time. Great!

The coolest thing is that when I built this image my Pi was offline but as soon as I plugged it in hours later, the new image was updated automatically - I know this is documented but it was so neat to see it working. This is so awesome! The resin.io team really thought of everything and I can't say how happy I am to be using their service. The small complaints I had in my original post are really minor, resin.io is really a great way to update your Pi code remotely. Again, big thanks to the entire team!

Hopefully now that all pieces are in place, I will find some time to write a robot that actually does something, and command it via MQTT from node-red. Soon...

Cylon.js - an amazing robot and IoT framework

A few days ago on a blog I follow I noticed an article about the release of Cylon.js 1.0. Never before heard about Cylon.js but the article sounded very interesting, mentioning robots and IoT, javascript and also support for 35 platforms so I decided to check it out. I am really happy I did, I have to say from the start that it is an amazing framework with a great design and tons of supported platforms and drivers, to make it really useful for tons of things: not just robots as the name implies but basically anything related to physical computing and the Internet of Things. It makes it incredibly easy to command robots and devices, and the API plugins it already comes with (http, mqtt and socket.io) make it really easy to connect and interact with these devices online. Really great!

Like I said, there are tons of platforms supported (basically anything I can think of is already supported) but since I happened to have a Digispark with an RGB LED shield handy since I last played with it and node-red, I decided to give it a try. It would have been easier probably to start with an Arduino to avoid a few hiccups but in the end I am very happy I gave it a try because it worked really well.

The Digispark documentation is really good but since I ran in a couple stumbling blocks on my Linux Mint machine (quickly clarified on the IRC chat by a very helpful user) I decided to quickly document the steps here, maybe they'll help somebody some day.

As mentioned in the Ubuntu section of the Digispark docs, first thing to do is install the cylon-digispark npm module. Next commands use "gort" and while this may not be an issue for anybody else, it was for me; I am not familiar with it and apt-get didn't find it so I stumbled a bit with the next step. However, when I asked about it on the chat channel I got a reply right away, saying I need to download it from here. Same user also mentioned that after I install it, I should run

gort digispark set-udev-rules

which was a great pointer because the docs where not very clear about what to run next (this one or upload) so this helped me a lot. Next command in the docs though is

gort digispark upload

which didn't work for me no matter what I tried. In the end I looked at the output of the command and decided to try instead

gort digispark install

and this worked right away. Then cd to the examples dir in the cylon-digispark module and first example I tried, blink, worked like a breeze. After trying most of the examples all I can say is that Cylon.js is indeed awesome and in the end pretty easy with just a couple stopping points, mostly due to my lack of Linux experience, I'm sure.

A big thank you to the Hybrid Group team behind this great project!

IBM Bluemix and node-red

I've heard about Bluemix a few months back but only a few days ago after watching a couple YouTube movies about node-red running in Bluemix I decided to give it a try. There is a 30 day free trial after which payment for some services is required; however, there is a free allowance after the trial of 375 GB-hours free for each IBM-provided runtime and 375 GB-hours free for all other runtimes combined. According to the pricing page, 1 GB-hours = Total GB/App x Number of App Instances x Total Hours running so 375 free GB-hours per month basically means one app using up to 512 MB of RAM (or more apps with a total of 512 MB of RAM for all of them) running non-stop which is pretty cool.

Starting with the node-red boilerplate was a bit bumpy at first: the movies I watched showed different steps but the service appears to have been changed a little since then; currently the steps are: after login, click Create an app, select Web, click on Browse Sample Apps and then Explore Samples, choose Node-RED Starter and finally name your app and click Create. In the end though, creating my first app was easy, and using this boilerplate gives you the node.js SDK with node-red preinstalled along with 2 services: Monitoring and Analytics, free not just during the trial period but after that as well and Cloudant NoSQL DB which I believe has a free plan after the trial period, need to look into it if I decide to use it after trial. Also, this app comes with some pretty cool nodes, like ibmiot designed to connect the app with IBM's Internet of Things module, and others.

The coolest thing with this setup is that now I have a node-red app running in the cloud complete with access to the editor so my Raspberry Pi can take a break from running non-stop and no more need for port-forwarding to access my Raspberry Pi node-red from outside my home network. I tried doing the same thing using Weaved but without luck: I was able to use Weaved to connect to port 1880 and the node-red editor came up without issues, however saving flows didn't work, I guess the underlying code didn't like this setup. Another option to run node-red in the cloud is documented by Chris Mobberley on his awesome Hardware_Hacks blog that I never got to try. I assume running node-red in the cloud should be possible using Heroku for example, since Heroku is great for hosting node.js apps but again, never tried.

I have now a flow running non-stop in Bluemix (that I will go over in a future post) and all I can say is that I am very happy I gave Bluemix a try and I am sure I will continue using it beyond the free trial. If you haven't used it, it is definitely worth checking out. Big thanks to IBM for providing such a great service, to Nicholas O'Leary and Dave Conway-Jones and others who I believe are responsible for the node-red boilerplate and provide great support in the Bluemix forum, and also to everyone else that steered me to Bluemix through their videos and comments.

[Update] Since weaved released a new version recently, I decided to give it another try. I uninstalled the previous version 1.2.5 and installed the new 1.2.8 setting up a TCP service on port 1880 (node-red default editor port), connected to it and this time the flows saved and worked as expected. This is really awesome! Huge thanks to the Weaved team for all their great work!

CheerLights node-red edition

A couple days ago I found my DigiPixel so I started thinking what quick project I could use it for and remembered CheerLights. I gave away my Ethernet shield to a friend and never got around to buy another one so I decided to have it connected to my laptop and get data using Serial through USB. I quickly wrote a sketch to do just this but since I didn't want the DigiPixel to stay on all the time I started to modify it to turn off after a while and then back on for a few seconds. While messing with the code node-red came to mind and I realized 2 things: first, I don't need to modify the code on the Arduino (and having to do it again and again if I decided to change the behavior, like different ON/OFF timeouts), I can leave the sketch simple as it was and implement the behavior I wanted in the node-red flow; second, I can run the flow on my RasPi which uses wifi to connect to my home network and for which I also have a battery so my project becomes "wireless".

A quick search revealed this @Cheerlights to various RGB devices example flow (thanks dceejay!) which is exactly what I needed it. Started my RasPi, deployed the flow, replaced the blink device with a serial node connected to my Arduino and I was done! After I "deployed" the project in my living room and my son started to ask me why there is no light I realized that if the CheerLights color doesn't change for a long time, the DigiPixel doesn't show anything so I tweaked the flow a little bit to wake up every 2 minutes and show the last known color and if there is no last known color just use some color to start things off; the result is here, not as good and generic as the original flow but exactly what I needed.

My setup is a bit more complex than it needs to be because I am using a Raspberry Pi A which has only one USB: since I need USB for both the wifi dongle and to communicate with the Arduino, I used a hub (it doesn't need power, the Pi power supply is good enough to power both the dongle and the Arduino).

node-red is indeed amazing: the amount of changes that can be made in short time and the ability to swap out nodes for different ones very easily (I could decide to replace my Arduino+DigiPixel with some other device and probably be up and running in a matter of minutes or an hour at the most), make it an awesome tool that will very likely be used more and more in the IoT world. I'm repeating myself but I simply love it and can't be thanking enough to its authors!

Project follow-up: Raspberry Pi with 433 MHz radios, mqtt and node-red

Now that my Arduino sensors to Raspberry Pi using 433 MHz radios project is done and I learned a lot about posting to the web and using a db, it was time to learn something new; mqtt is a very hot topic in the IoT world so this was my next target. Using an MQTT broker like mosquitto is not only cool but allows for decoupling of the different parts; for example, instead of writing a monolithic piece of code that does everything (read sensors, post to the web, save to a database, like in my previous project) and which needs a lot of work in order to add some new functionality, one could write a piece of code that for example, only reads sensors and publishes to an MQTT topic; then another piece of code can subscribe to that topic, get the sensor values and post them to the web; and yet another piece can subscribe to the same topic, get the values and save them to a database. And so on, the possibilities are endless and different parts of the system are independent and can be plugged in and out very easily. And since it is very easy to install mosquitto on a Raspberry Pi, I decided to go ahead.

To integrate mqtt in my C code I used mosquitto client library (libmosquitto) with some docs here: not a lot of help for a beginner but I found plenty of articles and example code. I ran into a lot of issues because the library that I installed was older and none of the example code worked; Roger (mosquitto's author) was kind enough to point me to a page explaining how to install the latest library so I installed libmosquitto-dev from this repo and it solved my problems. I also needed to add -lmosquitto to the Makefile.

After this code was done (if you are interested you can find it here), I had to decide what is the next piece so I decided on node-red: a chance for me to dig deeper into this cool technology and add to my experience from my previous projects. As Simen Sommerfeldt so eloquently makes the case in this great article, node-red is perfect as the "glue" between different pieces of code. To start with I decided to create a flow that reads sensor motion data from the MQTT broker and plays a sound via a python script. I already had a script but Simen's was much better so I used his - big thanks goes to him for sharing his work. The node-red flow was done in minutes; I then added 2 exec nodes to start the C code reading sensor data and the python script. As a side note, at this time I am running this exec nodes manually instead of starting them automatically when the flow starts, until I can figure out how to prevent multiple instance from running at the same time. This project is by no means as cool as Simen's moving skull but I will use as my Halloween project to play some random scary sounds when trick-or-treaters come to the door.

While I was working on this project I got an email from the great folks at dweet.io and freeboard about some great features they added recently. When I saw their email I decided to add another branch to my flow to get the sensors data and post it to dweet.io. Node-red is so awesome that I had all this done in a matter of minutes: I found a dweetio node-red node, installed it, added the nodes to post to dweet, and I also had a quick freechart done. The node-red flow is also in my github repo.

By the way, if you haven't yet, you really need to try dweet.io and freeboard: really awesome services with a free tier - big thanks to the guys at Bug Labs that are behind these services.

Arduino sensors to Raspberry Pi using 433 MHz radios

Finally getting to wrap up my first end-to-end project using Arduino and Raspberry Pi talking via inexpensive 433 MHz radios - the distance over which they work is pretty small but this instructables explains how to add an antenna which dramatically improved the distance. The main idea was to have multiple Arduinos with TX radios, each equipped with DHT and PIR sensors; they all send the data to the Raspberry Pi which posts it to the web and also writes it to a local database. In addition to the values from sensors, I am also sending the voltage level, hoping that this way I can monitor the discharge of the batteries (I am not sure if this works or not, I didn't use an analog pin for this, I am simply getting the voltage on the MCU, using code from this great instructables).

The code for both the sender (Arduino) and the receiver (Raspberry Pi) is in my github repo. The comments in the code pretty much explain everything (how it works, how to connect the sensors and radios, and so on). Here are just some quick ideas that may be interesting.

• I wanted to transmit all the data in one go so I combined 4 numbers into one long, as described in both sender and receiver code.
• The transmission between the radios is pretty flaky, that's why I am repeating the send a lot on the Arduino and there is code on the receiver to ignore identical values coming in a 30s interval. This is OK because transmissions from different stations have different station codes so even if the sensor values are the same, the actual overall value is different. Same for a motion sensor: the ON value is different than the OFF value so even if they happen a second apart, both will be received.
• I started by posting the data to SparkFun's data service, then added Dweet.io and in the end settled on ThingSpeak and I'm also saving all the data to a local SQLite database. See some notes about this later in this post.
• The code in github is ready to use: just change the stationCode for each Arduino station and use the right API keys in the Raspberry Pi code.

Notes about posting to the web: there are a lot of great IoT services out there with free tiers, to start with I chose the few mentioned above that have a simple API and that accept GET. Each free service has certain rate limits so I need to consider the data volume I will be posting. With one station only getting the temperature sensor data every 3 minutes, this means 5 posts in 15 min. The motion data is a bit more intensive, roughly 2 posts 5 sec apart (ON and OFF) every time the motion sensor is triggered which at the most (sensor triggered all the time) means 24 posts a minute. Because there is potentially a lot more motion data, I am using 2 different data streams (channels, feeds): one for temperature, humidity and voltage and one for motion; even so with a lot of stations there is the risk of missing data (according to each service rate limits as described below) so I decided to also implement a local SQLite database to store the data and eventually process it later and post it (maybe using node-red).

Rate limits:

1. SparkFun's data service is limited to making 100 log requests every 15 minutes. Given the 5 posts in 15 min, the temperature data stream can accommodate 20 stations without losing data which is more than I need. The motion data though may accommodate only 1 or 2 stations (more if there is not a lot of activity around them) without starting to lose data. I will post all the data I can, the remainder being lost.
2. Dweet.io holds on to the last 500 dweets over a 24 hour period so while there won't be data loss, the motion data will probably cover only a few hours at the most, even for just a couple stations.
3. ThingSpeak has a rate limit of an update per channel every 15 seconds so more than one station per channel will more than likely run into this limit (even 2 temp stations sending data every 15 minutes can happen to send data less than 15 sec apart); I could wait and retry until the post goes through but this would delay the code and risk missing data sent by the stations. Because of this, I decided to tie the local db data to this service: every time I post to ThingSpeak, I check the response: if > 0, the post was successful, I will log the data to the db with a flag of posted=true (1); if = 0, the post was not successful so I will log the data to the db with a flag of posted=false (0). Later I may create a node-red flow to get all the db data with posted=false and repost it to ThingSpeak; each row records the UTC time of the actual insert so I can send it along with the data, the measurements getting recorded at the actual time they took place not the time I post them.

If you find anything useful in this post and the code I am very happy. Most of this stuff is on the web but putting it together in a real project took some work; I am happy I did it and learned so much in the process. When I started I had no idea how to use a radio on either Arduino or Raspberry Pi, how to work with a PIR sensor and how to use interrupts, how to write any real C code, to work with sqlite and post to REST services on Raspberry Pi and much more. I used a lot of info from the web, from too many people and websites to thank individually but a big thank you! goes to everyone that provided help and information in a way or another.

SparkFun's Phant.io; libcurl and sqlite on Raspberry Pi

A few months ago SparkFun announced their new data service: phant.io. There are lots of similar services out there, some really awesome, some simple to use others not so much, some with great charting capabilities, others just a data store but I loved Phant from the beginning, mostly because it doesn't try to do a lot but what it does it does great. Also, its simplicity of use is awesome and is open-source. I was writing a JavaScript application at the time that needed a simple backend storage; I could use almost anything for that storage but when I saw the announcement I quickly decided to give it a try and indeed it was a breeze to post and retrieve data.

Fast forward to now: part of a larger project I am writing with a good friend of mine (in a nutshell an Arduino with DHT and PIR sensors gathers data and sends it using a 433 MHz radio to a Raspberry Pi) I wanted to post the data from a Raspberry Pi to some service on the net so again I chose phant.io. There are plenty of great examples on SparkFun's tutorial page (and other resources) about how to use Phant with Arduino, Python, Electric Imp but since the code running on my Raspberry Pi is written in C (because this was the simplest code I found for the 433 MHz receiver) I had to figure out how to use Phant using C. I tried for a couple hours to use sockets (found great code examples) but no matter what I did I was getting stuck in all kinds of issues. While looking around on stackoverflow for a solution I found a mention of libcurl - I don't remember who mentioned it but I owe him/her big thanks. I was a little bit reluctant to try it at first since I knew nothing about it but then another mention showed up so I decided to give it a try.

First thing I tried, adding #include <curl/curl.h> ended up with an error at compile time: file not found. So I had to install libcurl dev libraries which was really easy:

sudo apt-get install libcurl4-openssl-dev

and quickly verified that curl was now present in /usr/include/. Then I got the code from the very first example on libcurl page: simple.c but this time I got another error, curl_easy_init not defined or something like it. Another quick search and figured out I had to link the new library so I modified Makefile and added -lcurl. Now everything built and the first try with my test code posted data to data.sparkfun.com. The code as it stands today is here part of the repository that will eventually contain all the project code.

libcurl is really awesome: it took only minutes of getting the first version of the code done, in just a few lines (compared to 4 times as much for the socket code), the code is much more readable and the docs are awesome. Thanks to all the authors and contributors for such a great library!

Not completely related to the subject, in the same project I wanted to use a local database on the Raspberry Pi; a quick search got me to sqlite which is very easy to install and there are a lot of articles about it. One thing that most authors don't mention is that same with libcurl there is a library that needs to be installed so sqlite can be used in code and also that is has to be linked for the code to compile. Not a big deal but here are the steps in one place:

sudo apt-get install sqlite3 libsqlite3-dev
compiler link argument: -lsqlite3

My first Node-Red node: gcalendar

First thing I found when I was playing with homA was the calendar component that reads an event from a Google calendar - I loved the idea and the implementation. So when I got interested in Node-Red my first thought was: wouldn't it be cool if there was a node like that? I checked the extra nodes repository and couldn't find one so I thought this could be the best opportunity for me to dive deeper into node.js and node-red.

It wasn't all that easy at first since the Node-Red website didn't have any docs at that time about how to write a node - thanks to Nicholas O'Leary a guide is now available, not 100% finished but great enough to help anyone starting on this path. Anyway, looking at existing nodes and experimenting with stuff I was able to do this so here is my first node: gcalendar. While working on this I hit some snags:

• To get authorized to a Google calendar, OAuth2 must be used. I could have used Alexander's code from the homA calendar component but since I have no experience with OAuth2 I was not confident I can figure it out. Alexander is great and he always helped me when I asked him something but didn't want to take his time with minor things. On the other hand, a while back I heard about Temboo and never tried to use it so this looked like the perfect time to do so. And I'm happy I did: following the steps on the GetNextEvent page
I was able in a few minutes to create the app on Google API, get the credentials and get the code to call this Choreo and get the calendar data. There are 2 drawbacks to this approach, none of them a deal breaker for me:
• there are 3 more properties to set now to configure this node, all related to Temboo - since they don't change in time, this is not a big issue;
• Temboo only allows 1000 calls per month for free - since I don't plan to call this node more than 10 times a day, this limitation doesn't really affect me; anyway, I like their service a lot and if I'm going to be using them more, I won't mind paying for the next tier service.
• After the main JavaScript coding was done, I had to figure out the .html file for the node. This was quite challenging without having any docs and it took me a lot of trials to figure out how and what to do; this page was of great help - all of this is now documented on this page which explains everything in great detail.
• The first version of the node (let's call it v0.1) had all the properties set in the node directly which worked fine but it was a pain to have to re-enter them every time when I created the node during testing; plus, they were saved in the flow JSON which is not a good idea at all. Hardcoding them worked fine for testing but not an option for the final code. So now it was time to figure out the credentials part. First pass was to hardcode them in the settings.js file in Node-Red install dir which worked (this is sort of v1.0 of my node, documented in this file I kept for reference) but not a good idea according to the answers I got to my question. Based on the answers I got from Charalampos and Dave and after a lot more messing with the code, I was able to come up with the current version on my gcalendar node, where credentials are stored in a separate file. There is plenty more I can do to improve this version, but this is the one I use now and I am really happy with it.

In my next post, I will try to show how I use this node in a real flow.

Starting with Node-Red

A couple years ago I started to look into connecting devices to the Internet and played with quite a few services, like Pachube then Cosm now Xively and a few others - this all evolved in what is now named the Internet of Things (IoT) and a lot more services like Xively are now available. These services are usually independent of one another and each has a different way of setting up the communication with the device which makes things a bit difficult for beginners; these devices are sometimes hobby oriented like Arduino or Raspberry Pi but in other case they are products that monitor the home and also take actions (like NinjaBlocks, SmartThings, Insteon, TCP Lighting and so many many more). To make working with these devices easier (detection, customization, adding rules, communication with other services) new frameworks and applications are coming up, like TheThingSystem (TTS - node.js based, v1.8 was just released a couple days ago), OpenHab (Java based), Alexander Rust's homA and others.

A few months back, a new application showed up on my radar: Node-Red; built on node.js, Node-Red is another great product started by IBM researchers. Briefly said, Node-Red provides a browser-based flow editor that makes it easy to wire together flows using a wide variety of nodes presented in the palette; flows can be then deployed to the runtime in the same interface. No point in getting into more details about it, you can read more on the Node-Red website and in various articles like this one.

At that time I was digging a lot into the homA app and its components and I loved it; I was just getting started with the rules engine (which is way cool, by the way); but after discovering Node-Red and after seeing Alexander's comment to this google+ post I realized that I can use Node-Red to simplify the interaction between my devices and the outside world, so I started to research it more. There are great resources around it, the ones I use the most are:

• Node-Red docs;
• flows contributed by people working on Node-Red and 3rd parties;
• Node-Red github repo;
• other nodes contributed by people working on Node-Red and 3rd parties;
• the Node-Red google group - Nicholas O'Leary and Dave C-J are awesome, their extremely fast answers helped me a ton.

There are many other interesting resources, like:

• building a Node-Red sample application;
• The ThingBox Project: Raspberry Pi image ready to go containing a MQTT broker (Mosquitto), a javascript HTTP server built on top of Node.Js and Node-Red.

I will try to add more resources as I find them. I hope you will like Node-Red as much as I do.