Awesome free services from Sense Tecnic: FRED and WoTKit

When Mike from Sense Tecnic announced FRED on the node-red Google group I jumped right away at the opportunity of trying it: and I'm happy I did because it is great! Not only that but the entire team behind it is awesome: the few issues people noticed were resolved in matters of minutes. Of course, having played with node-red for a while now, as you can see in other posts, I have it installed on all my machines, including my Raspberry Pi and I even tried it with Docker. But having an always on node-red instance that I can go to so I can try stuff and learn about node-red is simply awesome; in a matter of days I created a flow that gets my owntracks coords via mqtt, another that monitors a feed of newly released blues albums and highlights the ones I am interested in, one that shows me Top 10 blues and jazz tracks from iTunes and others.

One thing that may keep some from using FRED is the fact that being a hosted environment there are nodes that are not supported like Raspberry Pi or Arduino related. But Sense Tecnic has a solution for this as well: WoTKit. Using this platform one can publish sensor data, aggregate it and display it on dashboards. I have to admit that I've seen WoTKit before but I never sat down to read all the docs. However, when Mike mentioned it to me a couple days ago, I decided to give it a try. Same as FRED, all I can say is that is great! Not only that a lot of work has gone into this platform, what is really great is how responsive and helpful the team is. At first, I started to go through the docs and while a great read, I was stumbling a bit. There are tons of examples on the site for Python and curl but I wanted to get into it faster, preferably using node-red (of course, via FRED) but couldn't find anything. This was solved right away when Roberto from the same team sent me a link to an article exactly about this: node-red and WoTKit. The integration is perfect: as mentioned in the article, WoTKit expects a JSON structure of name:value pairs for sensor data which is so easy to produce from mode-red/FRED.

I won't go into much detail, please read the article I mentioned and all will be clear. Just to see how easy is though, let's go back to my owntracks flow. From owntracks, I get a message payload containing: lat, lon, tst (timestamp in seconds) and a few other fields. WoTKit expects the coordinates in fields named lat and lng and also expects an optional milliseconds timestamp as a long in a field with the same name: timestamp. Given the original payload, all I had to do is add 2 more properties to it, like:

payload.lng = payload.lon;
payload.timestamp = payload.tst*1000;

then added a WoTKit output node to the flow, created the sensor on the WoTKit platform and in a matter of minutes, I had a dashboard showing me the trail of last 20 coordinates posted by owntracks. Simply beautiful!

Another example: I love Scrabble and I play quite a lot on my phone using various apps. One of these is WordFeud for which the great people at Feudia are organizing tournaments. One of the things Feudia keeps track of is the players rating and ranking. When I first started using Feudia I decided to keep track of my rating but couldn't find a way to see the history so once in a while I was saving the values in a Google spreadsheet with a couple charts; FRED and WoTKit gave me an idea of how I can do this automatically.

I created a sensor with 2 fields: rank and rating, then created a simple flow using an http request node to read the Hall Of Fame page then used an html node to extract the info from it; one of reasons I went this route is that for a while now I wanted to figure out how the html node works - it took me a while but after several tries I was able to select the div enclosing the ranking table and extract all the values in its children in an array, that looks like this:

[ "#", "1", "\n \n \t\t", "Deuxchevauxtje", "1956,80", "", "#", "2", "\n \n \t\t", "nte...", "1936,37", "", "#", "3", "\n \n \t\t", "Pix...", "1930,94", "", "#", "4", "\n \n \t\t", "por...", "1922,31", ""....]

In this array, I look for my username and then get the rank 2 items in the array previous to it and the rating 1 item after (since only first 200 players are ranked, if for some reason I will drop lower than 200 and I won't find the username in the list anymore, I simply won't post anything to WoTKit) as you can see in my simple flow below:

[{"id":"b622d78d.49dd28","type":"wotkit-credentials","nickname":"Default","url":"http://wotkit.sensetecnic.com"},{"id":"67bc4260.9843bc","type":"html","name":"","tag":"div.rnk-tm>","ret":"text","as":"single","x":323,"y":265,"z":"8e66ad17.71995","wires":[["e616f09.f19e91"]]},{"id":"2bc264d6.d43d9c","type":"inject","name":"once a day","topic":"","payload":"","payloadType":"date","repeat":"86400","crontab":"","once":false,"x":103,"y":30.090909004211426,"z":"8e66ad17.71995","wires":[["f5e6e137.0a192"]]},{"id":"f5e6e137.0a192","type":"http request","name":"feudia halloffame","method":"GET","ret":"txt","url":"http://www.feudia.com/wordfeud/halloffame.html","x":191,"y":149,"z":"8e66ad17.71995","wires":[["67bc4260.9843bc"]]},{"id":"e616f09.f19e91","type":"function","name":"extract user info","func":"var allInfo = msg.payload;\nvar payload = {};\npayload.found = false;\nfor (i = 0; i < allInfo.length; i++) {\n if(allInfo[i] === 'merlin13') {\n payload.found = true;\n payload.username = \"merlin13\";\n payload.rank = Number(allInfo[i-2]);\n payload.rating = Number(allInfo[i+1].replace(',','.'));\n }\n}\nmsg.payload = payload;\nmsg.headers = {\n \"Content-Type\":\"application/json\"\n};\nreturn msg;","outputs":1,"valid":true,"x":500,"y":187,"z":"8e66ad17.71995","wires":[["902fe43e.6fd018"]]},{"id":"b664ddf4.499b2","type":"wotkit out","name":"feudia sensor","sensor":"claudiuo.feudia","login":"b622d78d.49dd28","x":769,"y":210,"z":"8e66ad17.71995","wires":[]},{"id":"902fe43e.6fd018","type":"switch","name":"","property":"payload.found","rules":[{"t":"true"},{"t":"else"}],"checkall":"true","outputs":2,"x":578,"y":296,"z":"8e66ad17.71995","wires":[["b664ddf4.499b2","33f1575a.cc0ea8"],["33f1575a.cc0ea8"]]},{"id":"33f1575a.cc0ea8","type":"debug","name":"","active":true,"console":"false","complete":"false","x":759,"y":336,"z":"8e66ad17.71995","wires":[]}]

Added the WoTKit output node and all was done! Well, almost. The payload looked great in debug:

{ "found": true, "username": "merlin13", "rank": 150, "rating": 1691.32 }

but no matter what I tried I was getting errors from WoTKit when posting the data. Finally, I decided to change the debug node to print the entire message, not just the payload and noticed the content-type was wrong:

{ "topic": "", "payload": { "username": "merlin13", "rank": 150, "rating": 1691.32 }, "statusCode": 200, "headers": {..."content-type": "text/html; charset=UTF-8" } }

So I added a header to my message as seen in the flow above:

msg.headers = {"Content-Type":"application/json"}

and everything worked like magic. Now my sensor gets data once a day and I also have a dashboard with the 2 charts identical with the original ones in my Google spreadsheet. Disregarding the content-type issue which was my fault, everything was done in less than an hour and now I have a dashboard that updates automatically so I don't have to gather data manually now and then.

Of course, as you see my "sensor" is not really a sensor in the real sense of the word but the concept is the same: instead of using owntracks or scraping the Feudia page, I could as well have a sensor hooked up to my Raspberry Pi or an Arduino, send data to WoTKit directly or to FRED for further processing and then WoTKit and the gap between a physical device and a hosted IoT service is closed, also using a hosted node-red instance in the process. I don't know if I can convey how awesome this is!

If you want to see the beauty of FRED and WoTKit in action, you should definitely try these services: as mentioned in the title, not only they are awesome but also free. Huge thanks to the Sense Tecnic team, especially to Mike and Roberto for all their awesome work, for offering these great services for free and for being so patient and helpful and responsive!