Like I said a few days ago, I took a break on Discovery of Witches and started reading Gone Girl and I am so happy I did. I was not sure what to expect, this is another book I wasn't planning to read but I found in a local little free library; I've never seen the movie but I remember it was quite a success and I was worried the book is just an easy thriller. I don't know about the movie but the book is great! The writing is amazing, the way the relationship unravels in parallel with the main event and from both hers and his points of view, in stories and diary entries. And the way the lives of all major characters is detailed and analyzed in so much detail but in simple words and ideas is simply awesome. Gillian Flynn is another author that I will follow and try to read more. When I'll finish this book I'll be dropping it back in another little free library in the area and buy my own copy, something I like to do as a small thanks to the authors I like.

A new (for me) Neil Gaiman book

My son got a new book from the library and I was thrilled to see it was by Neil Gaiman, one of my favorite authors, but also surprised, to my shame I've never heard about Fortunately, the Milk before. After he was done, he told me I have to read it because it has time travel; I did and I'm happy I did, it was indeed great. The best part is that it is the first book that exposes him to time travel - every time when we watched some time travel action in a movie, he told me he is very confused, I really want to have a talk with him to see if this book helped or not. Now heading online to get my own copy...

Started reading A Discovery of Witches by Deborah Harkness and while I love the writing and how the book started, pretty soon the romance aspect started to take over and be a bit too much for me. I will keep the audiobook around and listen to it now and then and I am sure I will finish it someday but the paperback I will drop in a little free library in the area so someone else can find it and enjoy it (as a cool side note, I was amazed to see how many people are wishing for this book: according to LibraryThing there 599 people wishing for it right now.

In the meantime I have a lot more attractive books waiting for me, like N.K.Jemisin's The Broken Earth Trilogy (The Fifth Season, The Obelisk Gate, The Stone Sky). I started reading The Fifth Season a while back and fell in love with it, bought the entire trilogy and now is waiting for me to read it all.

A China Miéville ebook on sale

I just bought Un Lun Dun on sale in the Kindle store. I love everything I read until now by China Miéville so I am sure this won't be a disappointment either. And it seems to be age appropriate for my son to be able to read it as well; if this is true, getting him to know China Miéville would be awesome.

A new writer I just discovered but I already love: Lori Rader-Day

I just finished Lori Rader-Day's The Day I Died and I fell in love with it. Like one of the blurbs said, this book is "much more than a psychological thriller". I got this book from a local Little Free Library and intended to keep it but I loved it so much that I released back in the same library, hoping other readers will love it as well. And I will buy my own copy, along with her other books: Little Pretty Things, Under a Dark Sky (got it!) and Black Hour -- a simple way to say "thank you!" to Lori.

Mistborn saga continues

A couple months ago I finished Mistborn book 1 and decided to wait a bit before starting the second book in the series. Well, that time has come and Well of Ascension was well worth the wait. Hopefully I will finish it this year to add to my Goodreads book challenge.

I like Andrea Camilleri more with each book

I just finished another Montalbano book: The Potter's Field by Andrea Camilleri and I liked it more than the previous two: The Terracotta Dog and The Snack Thief. This is book 20 in my Goodreads reading challenge so I'm doubly happy. I think I will treat myself with the collection of first 10 Montalbano books.

Awesome free services from Sense Tecnic: FRED and WoTKit

When Mike from Sense Tecnic announced FRED on the node-red Google group I jumped right away at the opportunity of trying it: and I'm happy I did because it is great! Not only that but the entire team behind it is awesome: the few issues people noticed were resolved in matters of minutes. Of course, having played with node-red for a while now, as you can see in other posts, I have it installed on all my machines, including my Raspberry Pi and I even tried it with Docker. But having an always on node-red instance that I can go to so I can try stuff and learn about node-red is simply awesome; in a matter of days I created a flow that gets my owntracks coords via mqtt, another that monitors a feed of newly released blues albums and highlights the ones I am interested in, one that shows me Top 10 blues and jazz tracks from iTunes and others.

One thing that may keep some from using FRED is the fact that being a hosted environment there are nodes that are not supported like Raspberry Pi or Arduino related. But Sense Tecnic has a solution for this as well: WoTKit. Using this platform one can publish sensor data, aggregate it and display it on dashboards. I have to admit that I've seen WoTKit before but I never sat down to read all the docs. However, when Mike mentioned it to me a couple days ago, I decided to give it a try. Same as FRED, all I can say is that is great! Not only that a lot of work has gone into this platform, what is really great is how responsive and helpful the team is. At first, I started to go through the docs and while a great read, I was stumbling a bit. There are tons of examples on the site for Python and curl but I wanted to get into it faster, preferably using node-red (of course, via FRED) but couldn't find anything. This was solved right away when Roberto from the same team sent me a link to an article exactly about this: node-red and WoTKit. The integration is perfect: as mentioned in the article, WoTKit expects a JSON structure of name:value pairs for sensor data which is so easy to produce from mode-red/FRED.

I won't go into much detail, please read the article I mentioned and all will be clear. Just to see how easy is though, let's go back to my owntracks flow. From owntracks, I get a message payload containing: lat, lon, tst (timestamp in seconds) and a few other fields. WoTKit expects the coordinates in fields named lat and lng and also expects an optional milliseconds timestamp as a long in a field with the same name: timestamp. Given the original payload, all I had to do is add 2 more properties to it, like:

payload.lng = payload.lon;
payload.timestamp = payload.tst*1000;

then added a WoTKit output node to the flow, created the sensor on the WoTKit platform and in a matter of minutes, I had a dashboard showing me the trail of last 20 coordinates posted by owntracks. Simply beautiful!

Another example: I love Scrabble and I play quite a lot on my phone using various apps. One of these is WordFeud for which the great people at Feudia are organizing tournaments. One of the things Feudia keeps track of is the players rating and ranking. When I first started using Feudia I decided to keep track of my rating but couldn't find a way to see the history so once in a while I was saving the values in a Google spreadsheet with a couple charts; FRED and WoTKit gave me an idea of how I can do this automatically.

I created a sensor with 2 fields: rank and rating, then created a simple flow using an http request node to read the Hall Of Fame page then used an html node to extract the info from it; one of reasons I went this route is that for a while now I wanted to figure out how the html node works - it took me a while but after several tries I was able to select the div enclosing the ranking table and extract all the values in its children in an array, that looks like this:

[ "#", "1", "\n \n \t\t", "Deuxchevauxtje", "1956,80", "", "#", "2", "\n \n \t\t", "nte...", "1936,37", "", "#", "3", "\n \n \t\t", "Pix...", "1930,94", "", "#", "4", "\n \n \t\t", "por...", "1922,31", ""....]

In this array, I look for my username and then get the rank 2 items in the array previous to it and the rating 1 item after (since only first 200 players are ranked, if for some reason I will drop lower than 200 and I won't find the username in the list anymore, I simply won't post anything to WoTKit) as you can see in my simple flow below:

[{"id":"b622d78d.49dd28","type":"wotkit-credentials","nickname":"Default","url":"http://wotkit.sensetecnic.com"},{"id":"67bc4260.9843bc","type":"html","name":"","tag":"div.rnk-tm>","ret":"text","as":"single","x":323,"y":265,"z":"8e66ad17.71995","wires":[["e616f09.f19e91"]]},{"id":"2bc264d6.d43d9c","type":"inject","name":"once a day","topic":"","payload":"","payloadType":"date","repeat":"86400","crontab":"","once":false,"x":103,"y":30.090909004211426,"z":"8e66ad17.71995","wires":[["f5e6e137.0a192"]]},{"id":"f5e6e137.0a192","type":"http request","name":"feudia halloffame","method":"GET","ret":"txt","url":"http://www.feudia.com/wordfeud/halloffame.html","x":191,"y":149,"z":"8e66ad17.71995","wires":[["67bc4260.9843bc"]]},{"id":"e616f09.f19e91","type":"function","name":"extract user info","func":"var allInfo = msg.payload;\nvar payload = {};\npayload.found = false;\nfor (i = 0; i < allInfo.length; i++) {\n if(allInfo[i] === 'merlin13') {\n payload.found = true;\n payload.username = \"merlin13\";\n payload.rank = Number(allInfo[i-2]);\n payload.rating = Number(allInfo[i+1].replace(',','.'));\n }\n}\nmsg.payload = payload;\nmsg.headers = {\n \"Content-Type\":\"application/json\"\n};\nreturn msg;","outputs":1,"valid":true,"x":500,"y":187,"z":"8e66ad17.71995","wires":[["902fe43e.6fd018"]]},{"id":"b664ddf4.499b2","type":"wotkit out","name":"feudia sensor","sensor":"claudiuo.feudia","login":"b622d78d.49dd28","x":769,"y":210,"z":"8e66ad17.71995","wires":[]},{"id":"902fe43e.6fd018","type":"switch","name":"","property":"payload.found","rules":[{"t":"true"},{"t":"else"}],"checkall":"true","outputs":2,"x":578,"y":296,"z":"8e66ad17.71995","wires":[["b664ddf4.499b2","33f1575a.cc0ea8"],["33f1575a.cc0ea8"]]},{"id":"33f1575a.cc0ea8","type":"debug","name":"","active":true,"console":"false","complete":"false","x":759,"y":336,"z":"8e66ad17.71995","wires":[]}]

Added the WoTKit output node and all was done! Well, almost. The payload looked great in debug:

{ "found": true, "username": "merlin13", "rank": 150, "rating": 1691.32 }

but no matter what I tried I was getting errors from WoTKit when posting the data. Finally, I decided to change the debug node to print the entire message, not just the payload and noticed the content-type was wrong:

{ "topic": "", "payload": { "username": "merlin13", "rank": 150, "rating": 1691.32 }, "statusCode": 200, "headers": {..."content-type": "text/html; charset=UTF-8" } }

So I added a header to my message as seen in the flow above:

msg.headers = {"Content-Type":"application/json"}

and everything worked like magic. Now my sensor gets data once a day and I also have a dashboard with the 2 charts identical with the original ones in my Google spreadsheet. Disregarding the content-type issue which was my fault, everything was done in less than an hour and now I have a dashboard that updates automatically so I don't have to gather data manually now and then.

Of course, as you see my "sensor" is not really a sensor in the real sense of the word but the concept is the same: instead of using owntracks or scraping the Feudia page, I could as well have a sensor hooked up to my Raspberry Pi or an Arduino, send data to WoTKit directly or to FRED for further processing and then WoTKit and the gap between a physical device and a hosted IoT service is closed, also using a hosted node-red instance in the process. I don't know if I can convey how awesome this is!

If you want to see the beauty of FRED and WoTKit in action, you should definitely try these services: as mentioned in the title, not only they are awesome but also free. Huge thanks to the Sense Tecnic team, especially to Mike and Roberto for all their awesome work, for offering these great services for free and for being so patient and helpful and responsive!

Johnny-five and node-red in a Docker container

Picking up where I've left off in my previous post, I tried to get the simple Blink 2 flow (bottom of the page) working inside a Docker container. And in the end I did, but I had to get over a couple issues first.

My first tries, nothing happened: every time when node-red would start, I see in the console "looking for connected device..." and that was it. Seeing that johnny-five would not connect and getting errors running the flow (cannot read property 'type' of null), I exited the container without stopping it as described in my previous post (Ctrl+PQ followed by Ctrl+C and docker exec -it mynodered /bin/bash), and tried to run the johnny-five board.js example app in node_modules/node-red/eg with:

node board.js

Again, "looking for connected device..." was displayed and nothing else happened no matter how long I waited. Wondering what could be wrong, I checked /sys/class/tty and indeed there was no ttyUSB0 there; I remembered then the --privileged docker run option mentioned in my previous post and in the Docker docs so I restarted the container using this command:

docker run -it -p 1880:1880 --privileged -v ~/my-node-red:/root/.node-red --name mynodered --rm claudiuo/node-red

Unfortunately this didn't change anything, same message showed up both when node-red and board.js started. ttyUSB0 was now present so I knew I was on the right track but still had no idea how to make things work. In a last attempt, I decided to modify board.js and specify the port explicitly as var board = new five.Board({ port: "/dev/ttyUSB0" }); as mentioned somewhere in the johnny-five docs and this time board.js connected to my Arduino and the LED on pin 13 started blinking. This was an awesome moment!

Next step was to modify settings.js and do the same thing, changing the default global context entry:

j5board:require("johnny-five").Board({repl:false})
to:
j5board:require("johnny-five").Board({port: "/dev/ttyUSB0", repl:false})

and restarting the Docker container this time I saw johnny-five connecting and the blink flow worked right away. I can't say I like this solution very much because ttyUSB0 may change on a different machine (or maybe even if I plug in my Arduino in a different USB port) but the fact that it works is awesome. (As a side note, not setting the port explicitly in settings.js works great with node-red outside Docker; not sure why this is the case). Now I need to take the next step and figure out how to use callbacks in a flow (callbacks are key to some of johnny-five functionality but luckily the awesome node-red team added them in node-red 0.10.6 as described here).

One cool thing that needs mentioned is that while I was searching the web for solutions to my issues, I found out a book about johnny-five was just published a few days ago on May 8: Make: JavaScript Robotics: Building NodeBots with Johnny-Five, Raspberry Pi, Arduino, and BeagleBone - I'm sure it is great and I will be getting it very soon.

Node-red and Docker update

This is sort of a follow up to my earlier post related to running node-red inside a Docker container. I said "sort of" a follow up because that post was about a Docker container for Raspberry Pi; this one will talk about a container to run on my laptop. A lot of the stuff here applies to Raspberry Pi directly, some needs changes (like the base image, for example). Since that post I learned some things that make building and customizing a Docker container for node-red a lot easier, all thanks to one of the main contributors to node-red, Dave C-J.

All this started because I wanted to build a Docker container with custom packages installed (case in point, I am talking about johnny-five) and I wanted to start with theceejay's master node-red image thinking I will create my own Dockerfile to add the new packages and build a custom image. While looking on Docker hub I noticed theceejay's write up to his other node-red image which is really awesome: it explains things I didn't know about (like using package.json along with Dockerfile) which makes installing new npm packages very easy; also, it talks about overriding the settings.js file when building the custom image and not at runtime as I was doing it previously. This write up is linked to a github repo which I forked and cloned on my local machine, then I modified a little bit by adding johnny-five to package.json. I then built my own custom node-red Docker image (which is basically identical with theceejay's one with the addition of johnny-five package) and started the container mapping a local directory to /root/.node-red so I get a copy of the flow.json and all flows saved in the library:

docker build -t claudiuo/node-red .
docker run -it -p 1880:1880 -v ~/my-node-red:/root/.node-red --name mynodered --rm claudiuo/node-red

Once the container started, Ctrl+PQ followed by Ctrl+C exists the container without shutting it down after which I was able to connect to it using:

docker exec -it mynodered /bin/bash

and confirmed johnny-five was installed. More, looking at the node_modules/node-red/settings.js file, I noticed that jfive and j5Board were already added to the global context, commented out. This was a surprise to me, I had no idea the latest version of node-red comes with these modules already added; this is really cool.

In my previous post, I mentioned that I was also placing a settings.js in my local dir to change the name of the flow file: this is not needed because package.json specifies flow.json as the filename. However, the write up also mentions: "This also copies any files in the same directory as the Dockerfile to the /usr/src/app directory in the container… this means you can also add other node_modules or pre-configured libraries - or indeed overwrite the node_modules/node-red/settings.js file if you wish." So I made a copy of the settings.js in the dir mentioned above, uncommented the 2 johnny-five entries in the global context, placed the file alongside Dockerfile as seen in my repo clone and rebuilt the image. Started the container again and checked it and indeed, the new settings.js is in place.

I was thinking at some point to publish my new custom node-red image to Docker hub but since building it from scratch is as easy as cloning the repo and running docker build, I won't do it. In fact, rebuilding the image will take care of upgrading node-red as well so I rather not publish an image which will be out of date when next release comes out.

Now I need to figure out how to use johnny-five with node-red and build some cool little robot; at this point I have no idea how but I will start with the notes in the second half of this page and go from there. Until then, again, big thanks to Dave C-J for all his awesome work and help.