Docker + node-red = awesome!

When I first heard about Docker a few weeks ago, I realized how cool it was so I started reading about it right away. Because I kept talking about it, I got tasked at work to look into how we can use it and a couple weeks later, I was deploying 2 linked containers for my team that definitely made our development easier, even if all we are using are database containers, at least for now.

But my first thought when reading about Docker was how could I use it on my Raspberry Pi so I don't keep mixing stuff on the same SD card (which sometimes is not a very good idea, like when I messed up my node-red because I installed an IDE that used an older version of node.js). I know most of the software packages can co-exist without issues but I like to keep things separate so I have a bunch of SD cards now, one for Java projects, one for node-red and a couple more. Docker seems to be the answer to this - at least for my Raspberry Pi B, the As I have are a bit too constrained for Docker but they are dedicated to other projects anyway.

So, I started looking around and the first site that popped up was the excellent resin.io blog, specifically this article. It sounded awesome but it required Arch Linux which I am not familiar with so I decided to wait a bit. As I was researching Docker for work I happened to find a new blog article at hypriot.com that talked about a new Docker compatible image created by this awesome team. This was so great that I immediately cleaned up an SD card and installed this image. As advertised, it worked from the first try: I can't tell you how happy I was to see Docker running on my Pi. And this guys didn't stop at creating the main SD card image, they also published several Docker images made for Raspberry Pi - like I said, an awesome team. Thank you so much for all you do!

I started playing right away with Docker and couldn't wait to come back to it the next day. To my disappointment though, after I restarted my Pi, I kept getting errors no matter what docker command I tried. Given my luck of experience, I thought I broke something (because I also noticed that after changing the password, I started to get a warning every time I used sudo but it turns out this was easily fixed according to this post by adding 127.0.0.1 black-pearl to /etc/hosts) but after quite a lot of digging, I found a post mentioning how to restart the docker daemon - very simple, in hindsight I realize that I should've thought of it:

sudo /etc/init.d/docker start

Now that all was well, I started to work on what I really wanted to do from the start: create a node-red image, because there wasn't one when I started looking into Docker. Of course, there are several node-red images, including this one and since Dave C-J is one the creators of node-red I trust his image the most; but this image is not for Raspberry Pi. I started to work on my own image and I was able to create something fast but after that I spent a few long hours trying to make the rpi-gpio nodes to work without success. In the end, I published my image on Docker Hub but the fact that rpi-gpio nodes was bugging me so I ended up deleting it; I kept the Dockerfile in this gist so I can redo it at any time if I ever feel the need. Which I don't think it will happen because this morning doing yet another search on Docker Hub for "rpi nodered" luck was on my side and I found this image from nieleyde; there is no Dockerfile but I pulled the image immediately and it works great! Thank you so much, nieleyde!

Very important to note in the docker run command provided by nieleyde is the --privileged option (some notes here). When I first started the container, I noticed in the log that the userDir is /root/.node-red; I want to have access to the flows files and also to be able to install more nodes easily without messing up with the original image, so I start the container with a volume option (as detailed in the "Overriding Dockerfile image defaults" section of this article):

docker run -it -p 1880:1880 --rm --privileged -v /home/pi/.node-red:/root/.node-red nieleyde/rpi-nodered

This way, everything that happens in the real /root/.node-red user directory is mirrored in my /home/pi/.node-red dir and the other way around, so the flows files, new nodes, library files are shared between these directories. I am not sure if this is the best way but it works for me (well, I still need to check the new added nodes idea but the flows file works as expected so I hope new nodes will as well; also settings.js works fine as I will mention later).

The second thing I did to make it easier: the flows file by default is named flows_<machine_name>.json, for example flows_519c0741e1f0.json. The problem is that the machine name is the actual container short ID and it changes every time when the container restarts so the previous flows are not accessible anymore (the file is still present but is not read because the name doesn't match the machine name anymore). I tried naming the container when running it using --name option, but the name is not used by the flows file, only the container ID is used. To fix this, now that I have access to the user directory via the volume option, I placed a settings.js file in /home/pi/.node-red that changes the flows file name to flows.json. And it worked as I hoped it would: my file overwrites the settings. js file in the node-red install, as described here. Now each time I restart the container the flows file is the same so all my saved flows start immediately; this can be easily seen in the node-red logs: Flows file : /root/.node-red/flows.json.

In conclusion, Docker is really awesome and due to teams like hypriot and users like nieleyde Docker on Raspberry Pi and node-red in Docker are great to use! Thanks to everyone for all the great work!

Reformat Raspberry Pi SD cards

If you are using Windows and ever wanted to write a new image on an SD card previously used with Raspberry Pi, you probably noticed the card looks much smaller than it really is, only a few tens of MB; if I understand correctly this is because we only see the size of the boot partition and not the other Linux partition. When I first ran into this issue, I reformatted the SD card on my Linux Mint machine which worked quite well. Second time though I was away from home and had to use a Windows 7 machine. After some digging on the web I found out I can use diskpart which comes with Windows and works quite well, but there are several steps that need to be done:

C:\temp>diskpart
DISKPART> list disk

This will list all your drives, including the SD card; you need to be very careful to select the SD card and not your hard-drive, usually it is easy to recognize the SD card because its size is only a few GB (depending on the card you use) as compared to the HDD which is usually much larger.

DISKPART> select disk 1
Disk 1 is now the selected disk.
DISKPART> list part
......... list of partitions .........
DISKPART> select part 1
Partition 1 is now the selected partition.
DISKPART> delete part
DiskPart successfully deleted the selected partition.

Now you have to repeat the last 2 steps (select/delete) for as many partitions as you have, the default is 2 partitions so normally you have to do this only once more. After the last partition is deleted, you create a primary one and exit:

DISKPART> create part pri
DiskPart succeeded in creating the specified partition.
DISKPART> exit

At last, you remove the card and re-insert it and windows will prompt you to format it; no need to do a full format, quick format works great. This process works very well for me, I've done it a lot of times but it is quite involved.

Last night I ran into another great post on the excellent Raspberry Pi Spy website about how to format Raspberry Pi SD cards using SD Formatter. I won't detail the steps here, the article I mentioned is really good and I do want to thank Matt for such a great post!

node-red is best for... everything

Like I said in a previous post, node-red is becoming for me more and more the first choice for all kinds of projects I'm doing. I can definitely write code for all these little things but every time I start a project I ask myself first if it can be done in node-red.

Case in point: yesterday I remembered the DigiSparks I got a while back from kickstarter and decided to play with them. As you may know, a Digispark is a tiny Arduino-like device, not 100% compatible (because it uses the Attiny85 controller unlike Arduino's ATmega168/328) but plenty powerful; one of mine has an RGB shield; when I first got it from the kickstarter project I downloaded the example code from github and after quite a lot of fiddling I got it to work, both on my Linux Mint laptop and my Raspberry Pi. But that was a long time ago so now that I wanted to play with it a bit more I decided to see if I can make it work with node-red. First thought was that I could probably use the exec node and issue the same DigiRGB.py command I did last time.

But a quick search pointed me to the digirgb node. I quickly installed it but got make errors related to node-hid. After quite a lot of time spent on the web trying to figure out what may be wrong with my environment and after installing quite a few extra libraries and packages I found mentioned here and there (like libssl-dev and build-essential), I did what I should've done from the start: read the error message more carefully; this is how I noticed it said libusb.h missing. Tried:

$ sudo apt-get install libusb-1.0-0

but it was already up to date. Next I tried:

$ sudo apt-get install libusb-1.0-0-dev

and to my surprise this time npm install finished without errors. I connected my Digispark with the RGB led and checked it was visible: $ lsusb -> shows Bus 001 Device 005: ID 16c0:05df VOTI

Restarted node-red and the digiRGB node was right there. A quick test with an inject node sending a "100,100,100" string turned on the LED from the first try. I know by now I should not be amazed any more that node-red is so great but I still am, every time - it is simply awesome!

node-red static directory

This weekend I updated to node-red 0.10.4 which has a major change: the userDir where all the user files are stored is now outside of the node-red install dir, by default being the $HOME/.node-red dir (you can override it with the --userDir option). The steps to upgrade are documented really well here. Another big change is that the way to update using git pull is not recommended any more, instead the preferred way being

    sudo npm install -g node-red

This installs node-red in /usr/local/lib/node_modules/node-red which is accessible to the root user but it should really not be used to store user files. And there is really no reason to do so: the extra nodes can be installed directly in the userDir location, the flows and .config.js are saved there as well, settings.js is read from this directory if present, also flows exported to the library are stored here too. There is only one thing that seems it needs to go in the node-red install dir: static files.

For example, in my case, I have a flow that responds to URLs like /books/:type/:topic/:genre/:num and in order to not have to fill in the type, topic, genre and num params every time, I created a simple HTML page saved in /public/books/index.html that sends various values for these params. Now that node-red is separating the user content from the actual node-red content, I would like this page to be in userDir as well. So I started going through the configuration docs trying to figure out how to do it. At first glance, at least the way I read those docs the only way to do so is to use the httpStatic property; the problem is the docs say "When this property is used, httpAdminRoot must also be used to make editor UI available at a path other than /." I tried to change both httpStatic and httpAdminRoot and couldn't find a good solution until I decided to leave httpAdminRoot unchanged and just set httpStatic to /home/pi/.node-red/public and it worked. So now I have index.html in .node-red/public/books that loads in the browser as expected at http://<node-red ip>:1880/books/ while the actual <node-red install dir>/public dir is unchanged and the editor still works at http://<node-red ip>:1880

I thought this was a problem in the docs but according to Nick the docs are just trying to prevent possible problems so this worked all along. In any case, this is not a big deal in itself but it took me a while to figure it out so I want to document it, to remember it later.

node-red update, new rpi-gpio node is great

A few days ago a new version of node-red was released: 0.10.1, details on the node-red blog. Tons of new features are in this release as detailed in the article I just mentioned but for me one stands out: the new rpi-gpio node.

A couple months ago I tried to use a PIR sensor with node-red on my Raspberry Pi but using interrupts didn't work as expected, as I mentioned here. The new rpi-gpio node was completely rewritten by Dave C-J as detailed in this thread in the node-red Google group. As mentioned here and also in the node-red release notes, the new node uses the built in RPi.GPIO python library (part of of the newer Raspbian distributions) instead of the wiringpi library; to make sure you have the necessary files do:

sudo apt-get update
sudo apt-get install python-dev python-rpi.gpio
~/node-red/nodes/core/hardware/nrgpio ver 0

Last command should reply 0.5.8 (or better) which is the version of the RPi.GPIO library.

I installed the new node-red version and followed the steps mentioned above (even if I may not have had to since I am using the latest Raspbian, released last year in December) and the last command returned 0.5.9. I connected the PIR sensor to the Pi as mentioned in this ModMyPi article: VCC to +5V [Pin 2], GND to GND [Pin 6] and OUT to GPIO 7 [Pin 26]. Added a new rpi-gpio in node to the editor, configured it to use pin 26, deployed and it worked from the first try. Simply awesome! Even more awesome: in the new node-red version the node status option is on by default so I didn't even need to add a debug node: I can see the rpi-gpio node's status reporting 1 as soon as something moves in front of the PIR sensor and 0 when it resets.

One great thing about the PIR sensor I am not sure I ever mentioned is this: even when the motion sensor is powered with 5V, the output voltage on the data pi is 3.3V (high) and 0V (low); I found this info in several places, in the ModMyPi article mentioned above, on the Learn Adafruit website and a couple more places like this instructable. This makes it perfect to be used with Raspberry Pi without any worries about the voltage applied on the data pin.

To end this short post, there is something I wanted to mention for a while: even if I keep finding new frameworks and services out there (for example, a couple days ago I discovered Lelylan and OpenHAB is on my list of things to study deeper), node-red is the service I keep coming back to every time I need to write an app on my Pi, GPIO related or not: it may not have the fancy charts other services have and it may not have the rule engine others do, but nothing beats node-red when you need to come up with real functionality fast, to connect services easily without having to write new code from scratch each time. I really love it! Give it a try - you will love it, too.

IBM Bluemix and node-red

I've heard about Bluemix a few months back but only a few days ago after watching a couple YouTube movies about node-red running in Bluemix I decided to give it a try. There is a 30 day free trial after which payment for some services is required; however, there is a free allowance after the trial of 375 GB-hours free for each IBM-provided runtime and 375 GB-hours free for all other runtimes combined. According to the pricing page, 1 GB-hours = Total GB/App x Number of App Instances x Total Hours running so 375 free GB-hours per month basically means one app using up to 512 MB of RAM (or more apps with a total of 512 MB of RAM for all of them) running non-stop which is pretty cool.

Starting with the node-red boilerplate was a bit bumpy at first: the movies I watched showed different steps but the service appears to have been changed a little since then; currently the steps are: after login, click Create an app, select Web, click on Browse Sample Apps and then Explore Samples, choose Node-RED Starter and finally name your app and click Create. In the end though, creating my first app was easy, and using this boilerplate gives you the node.js SDK with node-red preinstalled along with 2 services: Monitoring and Analytics, free not just during the trial period but after that as well and Cloudant NoSQL DB which I believe has a free plan after the trial period, need to look into it if I decide to use it after trial. Also, this app comes with some pretty cool nodes, like ibmiot designed to connect the app with IBM's Internet of Things module, and others.

The coolest thing with this setup is that now I have a node-red app running in the cloud complete with access to the editor so my Raspberry Pi can take a break from running non-stop and no more need for port-forwarding to access my Raspberry Pi node-red from outside my home network. I tried doing the same thing using Weaved but without luck: I was able to use Weaved to connect to port 1880 and the node-red editor came up without issues, however saving flows didn't work, I guess the underlying code didn't like this setup. Another option to run node-red in the cloud is documented by Chris Mobberley on his awesome Hardware_Hacks blog that I never got to try. I assume running node-red in the cloud should be possible using Heroku for example, since Heroku is great for hosting node.js apps but again, never tried.

I have now a flow running non-stop in Bluemix (that I will go over in a future post) and all I can say is that I am very happy I gave Bluemix a try and I am sure I will continue using it beyond the free trial. If you haven't used it, it is definitely worth checking out. Big thanks to IBM for providing such a great service, to Nicholas O'Leary and Dave Conway-Jones and others who I believe are responsible for the node-red boilerplate and provide great support in the Bluemix forum, and also to everyone else that steered me to Bluemix through their videos and comments.

[Update] Since weaved released a new version recently, I decided to give it another try. I uninstalled the previous version 1.2.5 and installed the new 1.2.8 setting up a TCP service on port 1880 (node-red default editor port), connected to it and this time the flows saved and worked as expected. This is really awesome! Huge thanks to the Weaved team for all their great work!

CheerLights node-red edition

A couple days ago I found my DigiPixel so I started thinking what quick project I could use it for and remembered CheerLights. I gave away my Ethernet shield to a friend and never got around to buy another one so I decided to have it connected to my laptop and get data using Serial through USB. I quickly wrote a sketch to do just this but since I didn't want the DigiPixel to stay on all the time I started to modify it to turn off after a while and then back on for a few seconds. While messing with the code node-red came to mind and I realized 2 things: first, I don't need to modify the code on the Arduino (and having to do it again and again if I decided to change the behavior, like different ON/OFF timeouts), I can leave the sketch simple as it was and implement the behavior I wanted in the node-red flow; second, I can run the flow on my RasPi which uses wifi to connect to my home network and for which I also have a battery so my project becomes "wireless".

A quick search revealed this @Cheerlights to various RGB devices example flow (thanks dceejay!) which is exactly what I needed it. Started my RasPi, deployed the flow, replaced the blink device with a serial node connected to my Arduino and I was done! After I "deployed" the project in my living room and my son started to ask me why there is no light I realized that if the CheerLights color doesn't change for a long time, the DigiPixel doesn't show anything so I tweaked the flow a little bit to wake up every 2 minutes and show the last known color and if there is no last known color just use some color to start things off; the result is here, not as good and generic as the original flow but exactly what I needed.

My setup is a bit more complex than it needs to be because I am using a Raspberry Pi A which has only one USB: since I need USB for both the wifi dongle and to communicate with the Arduino, I used a hub (it doesn't need power, the Pi power supply is good enough to power both the dongle and the Arduino).

node-red is indeed amazing: the amount of changes that can be made in short time and the ability to swap out nodes for different ones very easily (I could decide to replace my Arduino+DigiPixel with some other device and probably be up and running in a matter of minutes or an hour at the most), make it an awesome tool that will very likely be used more and more in the IoT world. I'm repeating myself but I simply love it and can't be thanking enough to its authors!

resin.io

I recently discovered resin.io and I have to admit it sounded great being able to deploy apps to a running Raspberry Pi so I decided to give it a try. And apart from a few small things I'd like to see changed, it is indeed awesome.

Following the Getting Started guide I loaded the OS on an 8 GB Sandisk micro SD, inserted it in my Raspberry Pi B+, booted it up, waited and... nothing happened (well, the power LED on the RasPi came on but that's about it). I reformatted the card and tried again and this time RasPi's green LED started to blink soon after boot and like it says on the website, in about 6-10 minutes, my RasPi showed in the list of devices. I assume the first try didn't work because of something I did, so this is not in any way a bad mark for resin.io - in fact, the process is really smooth and explained in such detail on the website that setting it up is quite easy - the only reason I mentioned it is if someone else runs into the same issue, don't give up, retry and it will work for sure.

Now the device showed up on the dashboard, I went ahead with the next steps: cloned the text2speech app, added the resin git endpoint, commit the code and waited for the project to build. This step worked great but it took a very long time to install and compile all the packages and their dependencies; it would have taken a lot longer to build all the node.js modules directly on the Pi so this is a definite advantage. The problem though is that every time when I update the code and commit and push again, the install process starts all over again and it takes as long as the first time; for a simpler app, with a lot less dependencies this is not an issue but for the example app I tried this was definitely a nuisance. Normally, if I deployed such an app on a Pi the initial time to setup the node modules and their dependencies would have been much longer; but subsequent updates would find the modules in place and would only take seconds.

Another issue I found and it is probably only an issue for me is the terminal on the web interface didn't start; every time I try I get a message saying terminal works only for recently provisioned devices but my devices was added recently so it should have worked. This combined with the fact that I was not able to ssh into my Pi (maybe because the SSH service is not installed/started [update] because that would be a security risk) made me shutdown the Pi by unplugging it which is not very safe from the SD card point of view, it can become easily corrupted. It is true that if this is the case I can always start over by installing the OS file again; also, I can probably create my own Docker container with SSH installed and started by default so this may not be a real problem but since I am not familiar with Docker, it is a problem for me.

Last small issue is not having the ability to deploy more than one app on a running Pi - maybe this is possible but I haven't seen anywhere the docs talking about multiple apps on the same device; same app on multiple devices is possible but the other way around I am not sure how to do. This reason along with the previous ones I mentioned made me decide to wait until resin.io becomes a bit more mature before trying it again.

I hope these small problems I ran into won't stop you from trying resin.io - the idea is great and it works so it is indeed an awesome app worth checking out. Thank you to resin.io author(s) for such a great idea and app!

[Update] Recently I started to look into Docker which seems to be really awesome; a quick search for Docker on Raspberry Pi turned out this article on resin.io blog. I wanted to bring this up because this was a big task for the resin.io team and a huge success and whoever is going to use Docker on Raspberry Pi will owe the team a big thank you. Thank you, guys!

The almost perfect Raspberry Pi dev setup

I want to learn more coding for Raspberry Pi and Arduino and one problem I have is being able to do so from different places and machines. I use both github and bitbucket and they are awesome but not once it happened that I wrote a bit of code and forgot to commit before leaving that machine. Even if I could be more careful with this, there is the aspect of having an IDE installed everywhere like Eclipse for Java, Arduino IDE, IDLE for Python. The solution for Arduino coding is codebender that I mentioned in a couple of my previous posts. Python I am mostly doing on Raspberry Pi and the solution is Adafruit's WebIDE - it is a great editor and one of the first things I install on every SD card for my Raspberry Pis.

For Java, I started a couple weeks ago to look at online editors like Cloud9 and others. Cloud9 looks great and it is one of the few editors of this kind that handles Java projects. However, recently I found an article (unfortunately, I didn't keep the link) that was reviewing several online editors from a Java coding point of view and the one the author of that article liked the most was Codenvy about which I have to admit I didn't even know until then. There were a lot of pro arguments for it in the article so I decided to give it a try. And I am so happy I did! As I read in the article, creating Java projects is a breeze; I tried both a new project and one cloned from a git repo: both worked like a charm. As a side note, I don't know Maven and Codenvy uses Maven (same as Heroku and other cloud apps) so this was a bit of a hiccup for me but I'm learning.

Codenvy integrates with both github and bitbucket so one can keep online and offline projects in sync this way. Also, there is a very powerful command line utility, Codenvy CLI that can be used to interface with projects, take them offline, or execute critical developer services.

But the thing I absolutely love about Codenvy is the fact that it has an Eclipse plugin that makes Eclipse aware of the Codenvy projects and allows both updating the Eclipse project with the Codenvy changes and the other way around. This is absolutely awesome! Now I can write code from anywhere without having Eclipse installed locally and when I get home, update the Eclipse project with these changes, keep coding and when done pushing the new code back to Codenvy. More, my Eclipse setup at home uses a plugin I found recently, LaunchPi that makes it possible to run and debug Java projects on Raspberry Pi directly from Eclipse. I found another similar plugin but it needed to be configured, with jars to be installed on the Pi and started separately; LaunchPi is so simple, just configure the Pi's IP address, the main class to run and done; and it also allows debugging which is simply great.

And if I want to deploy the code to the Pi, not just test it, I can always use a Maven goal to do this; as an example, look at the pom file used by Robert Savage, the author of the amazing Pi4J - but about this awesome library in another post.

I love this setup and the only reason I said in the title the "almost" perfect setup because there is one thing missing: if I could run and debug the code on the Pi from the Codenvy editor directly, that would be the icing on the cake. Maybe it is possible, I know there are ways to create a Pi emulator using QEMU but I haven't figured out yet how to do it and if it is possible to use it directly from Codenvy; maybe not but I'll try to find a way. Even without that, I am very happy with the current setup. Now, to do some coding!

ModMyPi, node-red, gpio issues, the great WiringPi library

Now that I figured out how to use a PIR motion sensor with Arduino I decided to try the same thing on a Raspberry Pi; pretty easy task since a quick search on google returned a lot of links. Using one of these examples, I put together a quick Python script which worked flawlessly. The only thing I didn't like: the continuous polling of the pin; instead, I wanted to use interrupts, same as I did on my Arduino. A few more searches later I found an article about using the PIR sensor via interrupts; this article is just one among many awesome ones on the ModMyPi blog - I am very happy I discovered this website and very grateful to the authors for all the tutorials, articles and help they offer to the community. Thank you!

Since the Python script in the article mentioned above worked without any issues, I decided to try the same thing using node-red. According to the node-red docs related to RPi, there are 2 ways of interacting with RPi: the gpio command using the rpi-gpio nodes and using the wiring-pi module which is the more complex way. So, I decided to give the rpi-gpio node a try. I created a simple flow with an inject and rpi-gpio nodes; to my surprise, as soon as I deployed the flow, I got an error in the debug window: "Error: Command failed: /bin/sh: 1: gpio: not found". This was very strange because I know I tried the gpio command and it works fine; also, the Python scripts had no issues. After a lot of tries and failures and retries, I think I finally figured out what is happening: this is only an issue when node-red is started automatically on boot using the auto-start script (my current setup is based on these steps including the auto-start script that starts node-red when RPi boots up); if I do a simple sudo service node-red restart then gpio works without a glitch. I guess it may have to do with loading node-red before gpio or something like it but since I am not a Linux wiz, I don't know if this is the case and what to do to fix it but maybe someday I will figure it out.

Because of the issue with gpio I decided to look into using the wiring-pi module which is a wrapper to the WiringPi library - I've heard about it before but never got to use it until now: just looking at the docs and examples I can say that it looks like a great library. Indeed, writing the code to blink an LED in a function node as described here worked great from the first try. The only issue is that I wanted to write code for my PIR sensor using interrupts and while WiringPi has great ways of working with interrupts, I couldn't find anything like that in the wiring-pi wrapper. So, while WiringPi is great and I plan to learn more about it in the future, for my plans it may not work in node-red via the wiring-pi wrapper which is in fact fine because it gives more of an incentive to look into and maybe learn other libraries or even languages.