Docker + node-red = awesome!

When I first heard about Docker a few weeks ago, I realized how cool it was so I started reading about it right away. Because I kept talking about it, I got tasked at work to look into how we can use it and a couple weeks later, I was deploying 2 linked containers for my team that definitely made our development easier, even if all we are using are database containers, at least for now.

But my first thought when reading about Docker was how could I use it on my Raspberry Pi so I don't keep mixing stuff on the same SD card (which sometimes is not a very good idea, like when I messed up my node-red because I installed an IDE that used an older version of node.js). I know most of the software packages can co-exist without issues but I like to keep things separate so I have a bunch of SD cards now, one for Java projects, one for node-red and a couple more. Docker seems to be the answer to this - at least for my Raspberry Pi B, the As I have are a bit too constrained for Docker but they are dedicated to other projects anyway.

So, I started looking around and the first site that popped up was the excellent resin.io blog, specifically this article. It sounded awesome but it required Arch Linux which I am not familiar with so I decided to wait a bit. As I was researching Docker for work I happened to find a new blog article at hypriot.com that talked about a new Docker compatible image created by this awesome team. This was so great that I immediately cleaned up an SD card and installed this image. As advertised, it worked from the first try: I can't tell you how happy I was to see Docker running on my Pi. And this guys didn't stop at creating the main SD card image, they also published several Docker images made for Raspberry Pi - like I said, an awesome team. Thank you so much for all you do!

I started playing right away with Docker and couldn't wait to come back to it the next day. To my disappointment though, after I restarted my Pi, I kept getting errors no matter what docker command I tried. Given my luck of experience, I thought I broke something (because I also noticed that after changing the password, I started to get a warning every time I used sudo but it turns out this was easily fixed according to this post by adding 127.0.0.1 black-pearl to /etc/hosts) but after quite a lot of digging, I found a post mentioning how to restart the docker daemon - very simple, in hindsight I realize that I should've thought of it:

sudo /etc/init.d/docker start

Now that all was well, I started to work on what I really wanted to do from the start: create a node-red image, because there wasn't one when I started looking into Docker. Of course, there are several node-red images, including this one and since Dave C-J is one the creators of node-red I trust his image the most; but this image is not for Raspberry Pi. I started to work on my own image and I was able to create something fast but after that I spent a few long hours trying to make the rpi-gpio nodes to work without success. In the end, I published my image on Docker Hub but the fact that rpi-gpio nodes was bugging me so I ended up deleting it; I kept the Dockerfile in this gist so I can redo it at any time if I ever feel the need. Which I don't think it will happen because this morning doing yet another search on Docker Hub for "rpi nodered" luck was on my side and I found this image from nieleyde; there is no Dockerfile but I pulled the image immediately and it works great! Thank you so much, nieleyde!

Very important to note in the docker run command provided by nieleyde is the --privileged option (some notes here). When I first started the container, I noticed in the log that the userDir is /root/.node-red; I want to have access to the flows files and also to be able to install more nodes easily without messing up with the original image, so I start the container with a volume option (as detailed in the "Overriding Dockerfile image defaults" section of this article):

docker run -it -p 1880:1880 --rm --privileged -v /home/pi/.node-red:/root/.node-red nieleyde/rpi-nodered

This way, everything that happens in the real /root/.node-red user directory is mirrored in my /home/pi/.node-red dir and the other way around, so the flows files, new nodes, library files are shared between these directories. I am not sure if this is the best way but it works for me (well, I still need to check the new added nodes idea but the flows file works as expected so I hope new nodes will as well; also settings.js works fine as I will mention later).

The second thing I did to make it easier: the flows file by default is named flows_<machine_name>.json, for example flows_519c0741e1f0.json. The problem is that the machine name is the actual container short ID and it changes every time when the container restarts so the previous flows are not accessible anymore (the file is still present but is not read because the name doesn't match the machine name anymore). I tried naming the container when running it using --name option, but the name is not used by the flows file, only the container ID is used. To fix this, now that I have access to the user directory via the volume option, I placed a settings.js file in /home/pi/.node-red that changes the flows file name to flows.json. And it worked as I hoped it would: my file overwrites the settings. js file in the node-red install, as described here. Now each time I restart the container the flows file is the same so all my saved flows start immediately; this can be easily seen in the node-red logs: Flows file : /root/.node-red/flows.json.

In conclusion, Docker is really awesome and due to teams like hypriot and users like nieleyde Docker on Raspberry Pi and node-red in Docker are great to use! Thanks to everyone for all the great work!

Reformat Raspberry Pi SD cards

If you are using Windows and ever wanted to write a new image on an SD card previously used with Raspberry Pi, you probably noticed the card looks much smaller than it really is, only a few tens of MB; if I understand correctly this is because we only see the size of the boot partition and not the other Linux partition. When I first ran into this issue, I reformatted the SD card on my Linux Mint machine which worked quite well. Second time though I was away from home and had to use a Windows 7 machine. After some digging on the web I found out I can use diskpart which comes with Windows and works quite well, but there are several steps that need to be done:

C:\temp>diskpart
DISKPART> list disk

This will list all your drives, including the SD card; you need to be very careful to select the SD card and not your hard-drive, usually it is easy to recognize the SD card because its size is only a few GB (depending on the card you use) as compared to the HDD which is usually much larger.

DISKPART> select disk 1
Disk 1 is now the selected disk.
DISKPART> list part
......... list of partitions .........
DISKPART> select part 1
Partition 1 is now the selected partition.
DISKPART> delete part
DiskPart successfully deleted the selected partition.

Now you have to repeat the last 2 steps (select/delete) for as many partitions as you have, the default is 2 partitions so normally you have to do this only once more. After the last partition is deleted, you create a primary one and exit:

DISKPART> create part pri
DiskPart succeeded in creating the specified partition.
DISKPART> exit

At last, you remove the card and re-insert it and windows will prompt you to format it; no need to do a full format, quick format works great. This process works very well for me, I've done it a lot of times but it is quite involved.

Last night I ran into another great post on the excellent Raspberry Pi Spy website about how to format Raspberry Pi SD cards using SD Formatter. I won't detail the steps here, the article I mentioned is really good and I do want to thank Matt for such a great post!

node-red is best for... everything

Like I said in a previous post, node-red is becoming for me more and more the first choice for all kinds of projects I'm doing. I can definitely write code for all these little things but every time I start a project I ask myself first if it can be done in node-red.

Case in point: yesterday I remembered the DigiSparks I got a while back from kickstarter and decided to play with them. As you may know, a Digispark is a tiny Arduino-like device, not 100% compatible (because it uses the Attiny85 controller unlike Arduino's ATmega168/328) but plenty powerful; one of mine has an RGB shield; when I first got it from the kickstarter project I downloaded the example code from github and after quite a lot of fiddling I got it to work, both on my Linux Mint laptop and my Raspberry Pi. But that was a long time ago so now that I wanted to play with it a bit more I decided to see if I can make it work with node-red. First thought was that I could probably use the exec node and issue the same DigiRGB.py command I did last time.

But a quick search pointed me to the digirgb node. I quickly installed it but got make errors related to node-hid. After quite a lot of time spent on the web trying to figure out what may be wrong with my environment and after installing quite a few extra libraries and packages I found mentioned here and there (like libssl-dev and build-essential), I did what I should've done from the start: read the error message more carefully; this is how I noticed it said libusb.h missing. Tried:

$ sudo apt-get install libusb-1.0-0

but it was already up to date. Next I tried:

$ sudo apt-get install libusb-1.0-0-dev

and to my surprise this time npm install finished without errors. I connected my Digispark with the RGB led and checked it was visible: $ lsusb -> shows Bus 001 Device 005: ID 16c0:05df VOTI

Restarted node-red and the digiRGB node was right there. A quick test with an inject node sending a "100,100,100" string turned on the LED from the first try. I know by now I should not be amazed any more that node-red is so great but I still am, every time - it is simply awesome!

node-red static directory

This weekend I updated to node-red 0.10.4 which has a major change: the userDir where all the user files are stored is now outside of the node-red install dir, by default being the $HOME/.node-red dir (you can override it with the --userDir option). The steps to upgrade are documented really well here. Another big change is that the way to update using git pull is not recommended any more, instead the preferred way being

    sudo npm install -g node-red

This installs node-red in /usr/local/lib/node_modules/node-red which is accessible to the root user but it should really not be used to store user files. And there is really no reason to do so: the extra nodes can be installed directly in the userDir location, the flows and .config.js are saved there as well, settings.js is read from this directory if present, also flows exported to the library are stored here too. There is only one thing that seems it needs to go in the node-red install dir: static files.

For example, in my case, I have a flow that responds to URLs like /books/:type/:topic/:genre/:num and in order to not have to fill in the type, topic, genre and num params every time, I created a simple HTML page saved in /public/books/index.html that sends various values for these params. Now that node-red is separating the user content from the actual node-red content, I would like this page to be in userDir as well. So I started going through the configuration docs trying to figure out how to do it. At first glance, at least the way I read those docs the only way to do so is to use the httpStatic property; the problem is the docs say "When this property is used, httpAdminRoot must also be used to make editor UI available at a path other than /." I tried to change both httpStatic and httpAdminRoot and couldn't find a good solution until I decided to leave httpAdminRoot unchanged and just set httpStatic to /home/pi/.node-red/public and it worked. So now I have index.html in .node-red/public/books that loads in the browser as expected at http://<node-red ip>:1880/books/ while the actual <node-red install dir>/public dir is unchanged and the editor still works at http://<node-red ip>:1880

I thought this was a problem in the docs but according to Nick the docs are just trying to prevent possible problems so this worked all along. In any case, this is not a big deal in itself but it took me a while to figure it out so I want to document it, to remember it later.