Wednesday, November 12, 2014

Evolution of a DIY PVR

Almost 10 years ago, I wrote an article for Make magazine about building a DIY digital video recorder (DVR).  At the time, there weren't any devices available, outside of subscription-based devices like Tivo, to help record TV.

My DIY device served me well for several years, but I get occasional questions on how or if I've updated my recorder, since much of the software and hardware I used at the time is now obsolete.

The answer is that I have been steadily upgrading that recorder during the intervening years, since there still isn't a commercial DVR on the market that meets my needs.

My original rig




The original article was here, but it appears to have been taken down, although the comments are still viewable. There is still a re-print of the article here.  Here is a brief summary of the components of that system:
  • Hardware: Dell Dimension 4500 with 512MB running Windows 2000 and a Hauppauge WinTV-PVR-250 TV card
  • BeyondTV: For TV recording, scheduling, and live TV overlays
  • WinDVD, Winamp, SlimServer: For media playback and streaming
  • Various console game emulators
  • Girder automation software: To create on-screen launcher menus
  • Cygwin and server software: For Linux-like ftp and ssh servers

Switch from BeyondTV to NPVR

Some time went by and I was getting annoyed with the amount of maintenance required to keep the machine up and running.  Various software components were always crashing and otherwise costing me time.  I started by removing the cygwin software because I wasn't using them enough to justify the time commitment.  At this point I also upgrade the PC I was using to Windows XP.

I also switched from using BeyondTV to NPVR (now called NextPVR).  NPVR had more features, such as DVD playback, and a good plugin system which allowed me to get rid of the extra media playback software like WinDVD.  I was also able to absorb the emulators into the NPVR interface using a plugin.  And best of all, NPVR was free!

The dominoes kept falling and since all of the functionality I needed was handled by NPVR, I got rid of the Girder-based menu system.

Streaming video to Roku
Up until this point I had my PC directly connected to my TV, but around this time, streaming set-top boxes were becoming inexpensive.  So I grabbed a Roku box, and found an obscure way to play media files from my local network, called the MyMedia channel.

While the MyMedia channel worked well, it needed a very particular video format, so I wrote a python script to transcode the video that NPVR produces automatically for me.  The script monitors a directory location for new files and then transcodes them using Handbrake and moves the resulting files to a location where the MyMedia server will pick them up.

I later added some more advanced features like also making the script monitor incoming video from bittorrent, and also making it intelligently rename the files and sort them into my existing video library.  I also made the script act as a simple system monitor and restart troublesome processes automatically to avoid crashes, hangs, and poor performance, which were a continual problem since I was still running on an old Windows XP box.

I also had a few more service running on the box, such as a Subsonic server for music streaming, a Ventrilo server for voice chat while playing games with friends,  an apache server to serve up my electronic parts database, a uTorrent client with a remote web interface.

Somewhere around this time I also invested in a Chromecast, and found I could use an app, Localcast, to stream media files to my TV.  This acts as a good backup to the Roku.

At this point I had a distributed video streaming network, with a central media server, and a remote web administration interface provided by NPVR.  This served me well for about 5 years.

Transition to Linux
About a year ago the coming demise of Windows XP and my growing fascination with Linux converged to make me take the plunge and re-implement my PVR on a Linux box.  I chose to go with Linux Mint 17 as my distro.

I had to port over all of the services that had been on my Windows box:
  • MyMedia server - cross-platform python script, so it just works
  • Subsonic - debian packages available
  • uTorrent - Switched to Deluge, since, at the time of this writing, the Linux version of uTorrent isn't robust
  • Ventrilo -  Linux versions available
  • transcoder script - python script, required a little bit of porting for paths and the like
  • Handbrake - needed by transcoder script, Linux versions available
  • smb file system support for streaming via LocalCast
  • various server software - apache, php, mysql, ssh, etc.
Most of those packages didn't have proper init scripts to make them run at system startup, so I wrote scripts for those that didn't have them.

My server is now much more stable and trouble-free, owing to the fact that Linux is better designed for running servers.  This also gave me a useful platform for software development, such as Node-Red,  and a private git repo for source code control.

Future Plans
Even though my current server box (can it even be called a PVR any more?), I still have some improvements I would like to make:
  1. Install mythtv so that I can record TV again.  You may have noticed that I didn't mention porting over to a replacement for NPVR.  Mythtv is that replacement, but I don't record that much TV anymore so I haven't yet bothered to take the time to set it up.
  2. Try out the Plex media server as a possible replacement to the MyMedia channel.  I've had multiple people recommend Plex to me for local media streaming and I'd like to give it a try and see if it's worth the fuss.

Wednesday, October 29, 2014

First steps with Node-RED

In my ongoing quest to find a good home automation software framework to run the hardware I built, I discovered Node-Red.  Node-Red is the self-professed "visual tool for wiring the Internet of Things".  Sounds promising.

Node-Red is a graphical programming environment based on node.js.  (Node.js is basically server-side javascript). Since it's built on node.js, Node-Red can run anywhere node.js runs.  It's all open source, and was started by IBM.



The interface to Node-Red runs completely in a web-browser, so there's no need to install anything on your development machine, and it is even usable from a mobile browser. It works by connecting wires between different functional nodes, kind of like the picture above.  There are lots of different nodes that do various things, from interfacing to a WeMo light switch to accessing emails from a Gmail account.

There is no built-in UI in Node-Red, but I think that might actually be an advantage.  It has numerous methods of connecting to a custom UI, and this way, you don't restrict users to a default UI that might not suit their application.

I got interested in Node-Red because I saw several different people using it to do home automation with JeeNodes.

Installation
I mostly followed the instructions here to install Node-Red to an old laptop running Xubuntu.

First I installed the Node.js package manager and Node.js itself:
sudo apt-get install npm
sudo apt-get install node

Then I downloaded the latest release zip file from nodered.org and unzipped to ~/dev
cded into that dir and ran 'npm install --production'.

Next up I launched Node-Red it by doing 'nodejs red.js'.  Running 'node red.js' doesn't work since the debian packages install it as nodejs to avoid a conflict. Eventually I ended up symlinking node to nodejs to make things easier:
ln -s /usr/bin/nodejs /usr/bin/node

And with that Node-Red seems to be up and running.

Node-Red ships with a pretty good set of basic nodes, but there is also a git repo that has even more nodes, so I thought I'd try that out.  I installed the extra nodes by doing 'cd nodes; git clone https://github.com/node-red/node-red-nodes.git'.

It should be noted that many of these additional nodes rely on node.js libraries and until those libraries are installed, the new node will not appear in the Node-Red palette.  As an example, if I want to use the suncalc node to generate an event at sunrise every day, I would need to install the suncalc library by doing 'npm install suncalc'.

Hello World
I ran through the initial tutorial and was successful.  Programs in Node-Red are called "flows". The basic design pattern in Node-Red is: drop and config nodes, wire them together, hit deploy.  Pretty simple, and even better, it actually works.

The debugging is a little simplistic in that you can print values to a debug log.  But again, it's simple, and it works, so I can't complain too much.

Flows can be exported as a json string, and imported by the same mechanism.  For the rest of this post I'll post the exported json for the flows I've used.

Controlling  execution
The Node-Red documentation has a good "hello world" tutorial, and has a tutorial on creating new plugin nodes, but it's a little sparse beyond that.  I did some experiments to figure out how the nodes in Node-Red actually execute.  Here's what I've figured out.

First off, Node-Red is completely event based.  Nothing executes unless an external event triggered it.  The "delay" and "trigger" nodes also allow timing based events to be generated.

The wires between the nodes are there to symbolize a data connection to the previous node.  More specifically the wires are really a graphical representation of how javascript Objects get passed to and from nodes.  These are usually referred to as msg objects and their contents can be somewhat arbitrary but most often contain a data member called payload.

So how can you control whether a node executes, like you can with an "if" statement in most text based languages?  If you return a "null" for any of the outputs then any of the subsequent nodes attached to that output don't execute.  A way to demonstrate that is by creating a function node that has multiple outputs which means you return an array of msg objects.  Any element in the array that has a value of null will prevent the connected node from executing.  By the way, this is exactly how a "switch" node works.  Check out the "demo switch" node in the example flow below.



Here's the example flow I'll be using for the rest of the post:
[{"id":"f64d219e.53a208","type":"inject","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":439,"y":454,"z":"d5a6cdcf.6203b","wires":[["5863bf74.888a3"]]},{"id":"5863bf74.888a3","type":"function","name":"demo switch","func":"msg.payload = \"hello\";\n//return [null, msg];\nreturn [msg, msg];","outputs":"2","x":628,"y":452,"z":"d5a6cdcf.6203b","wires":[["6d0fe6ba.2d2e88"],["d6a0f7c1.e949e8"]]},{"id":"6d0fe6ba.2d2e88","type":"debug","name":"demo out 1","active":true,"console":"false","complete":"false","x":821,"y":407,"z":"d5a6cdcf.6203b","wires":[]},{"id":"d6a0f7c1.e949e8","type":"debug","name":"demo out 2","active":true,"console":"false","complete":"false","x":820,"y":486,"z":"d5a6cdcf.6203b","wires":[]},{"id":"1ec662d2.3b7c25","type":"http in","name":"","url":"/test","method":"get","x":450,"y":277,"z":"d5a6cdcf.6203b","wires":[["dd8cd3d3.17b08"]]},{"id":"9c30b932.4da1e","type":"debug","name":"test URL","active":true,"console":"false","complete":"false","x":850,"y":189,"z":"d5a6cdcf.6203b","wires":[]},{"id":"6b3281a7.0a29c8","type":"http response","name":"","x":841,"y":276,"z":"d5a6cdcf.6203b","wires":[]},{"id":"dd8cd3d3.17b08","type":"function","name":"Gen response","func":"var resp = \"Hello \" + msg.payload.name;\nmsg.payload = resp;\nreturn msg;","outputs":1,"x":665,"y":276,"z":"d5a6cdcf.6203b","wires":[["6b3281a7.0a29c8","9c30b932.4da1e"]]},{"id":"7a4aeee7.85b51","type":"inject","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":424.5182189941406,"y":573.5229034423828,"z":"d5a6cdcf.6203b","wires":[["fbd25db9.042da"]]},{"id":"fbd25db9.042da","type":"function","name":"count","func":"if (context.hasOwnProperty(\"counter\"))\n\tcontext.counter += 1;\nelse\n\tcontext.counter = 0;\n\ncontext.global.counter = context.counter + 1;\n\nmsg.payload = \"local counter = \" + parseInt(context.counter, 10);\n\nreturn msg;","outputs":1,"x":602.5182647705078,"y":572.522876739502,"z":"d5a6cdcf.6203b","wires":[["caaa07a5.3555f8"]]},{"id":"caaa07a5.3555f8","type":"debug","name":"","active":true,"console":false,"complete":false,"x":782.5182189941406,"y":577.5229034423828,"z":"d5a6cdcf.6203b","wires":[]},{"id":"36a0278c.c95fd8","type":"inject","name":"","topic":"","payload":"","payloadType":"date","repeat":"","crontab":"","once":false,"x":424.5182189941406,"y":638.5228900909424,"z":"d5a6cdcf.6203b","wires":[["4b22fcbc.b4dd04"]]},{"id":"4b22fcbc.b4dd04","type":"function","name":"get global count","func":"msg.payload = \"global counter = \" + parseInt(context.global.counter, 10);\n\nreturn msg;","outputs":1,"x":603.5182189941406,"y":634.5229034423828,"z":"d5a6cdcf.6203b","wires":[["f51d7f6a.0ae28"]]},{"id":"f51d7f6a.0ae28","type":"debug","name":"","active":true,"console":false,"complete":false,"x":782.5182762145996,"y":633.5228900909424,"z":"d5a6cdcf.6203b","wires":[]}]

Local and Global variables
Within a function node there is a way to persist data between calls to that node that is both local to that node and global to all nodes.  It's documented pretty well here.  I created a quick demonstration of globals using a couple of function nodes.



First web service
Just to try out what I'd learned I used some techniques from here, and got a working test URL using http in and http response nodes!  Just pass a "name" parameter to the URL and it will print a Hello message.  Something like this: "http://site/test?name=Ken"




Conclusions
I really liked working with Node-Red, and I think I'll probably continue using it.

Pros
  • Open source and actively developed by IBM
  • Web-based development
  • Graphical programming/rule development
  • https and basic http authentication is supported.  See here for more info.
  • Mobile-friendly interface
Cons
  • No built-in UI
  • No built-in database for sensor readings, though there are nodes to interface with various DB formats

Friday, October 24, 2014

Trying out HouseAgent

For the last couple of years I've been looking for some Home Automation software to run the hardware that I'm slowly building to automate my house. There are so many choices that I'm not sure where to start.  I discovered HouseAgent mostly because it's written in Python, and I really like Python, so I thought I'd give it a test drive.

HouseAgent is a python-based home automation tool, with a promising architecture.  It appears that plugins are separate processes that communicate via AMQP, which means they can be written in (almost) any programming language.  It has a web interface, and a simple rules engine.

I installed the latest daily build of House Agent on a Windows 7 machine and ran into some problems.  The main problem was that it wouldn't launch.

After poking around for a while I figured out that you could run the House Agent daemon at the command line to get more info.  Just run 'houseagent.exe debug' to display debug messages.  I quickly discovered that there were files missing, namely houseagent.conf.  I grabbed these files from the git repo and put them in the places the error messages referred to.  The next error I had was that the port it was trying to use was already in use (8080) so I changed that in houseagent.conf.

At this point House Agent successfully launched, and I could get to the main web interface.  Then I went to the git repo for the JeeLabs plugin and grabbed all of the files there.  There wasn't any documentation so I made a directory called JeeLabs in the plugins folder in the House Agent install directory and copied all the files from the git repo there.  I then restarted the HouseAgent daemon and was able to successfully add a JeeLabs device.

I stopped at this point since I'll need to actually tie in some JeeNode hardware to continue my investigation.  There is a .conf file in the JeeLabs plugin directory that can be used to set the COM port to use and various other things.

There is also a fork of the JeeLabs which appears to be a little more up-to-date here. Even with those changes, when compared to another more up-to-date plugin, like ZWave, it looks like the JeeLabs plugin has been somewhat abandoned.

The web UI is also pretty plain and if I decide to use HouseAgent I'd like to modify it to look a little more snazzy. Right now it looks like an "admin" interface, rather than one that a user would really appreciate.  There does seem to be a branch where someone is working on mobile interface here.

At this point I kind of gave up since it seemed like getting HouseAgent to run was just fixing one bug after another.  HouseAgent seems like a great architecture but would probably take too much time and effort to make usable.

Pros:
  • Written in python (yeah!)
  • Good plugin architecture
  • Web-based UI
Cons: 
  • It doesn't look like HouseAgent has been under active developement in several years.  Has the project been abandoned?
  • In general, it appears to be a little rough around the edges
  • Web UI is a little ugly and not mobile friendly

Wednesday, October 22, 2014

embeddedWorld 2014

I was lucky enough to get chosen to speak at embeddedWorld 2014 in Nuremburg, Germany.  I gave a short presentation on using Yocto to develop a commercial product.  If you're not sure what Yocto is, don't worry, most people don't; it's a tool that helps Linux run on embedded hardware (like your WiFi router, for instance).  Rather than talk about a bunch of technical stuff I'll just stick mostly to just how the trip itself, impressions of Germany, etc.

General Observations
As I was flying from Dusseldorf to Nuremberg, I could see at least 4 nuclear reactors, quite a few windmills, and lots of solar panel installations (many on the roofs of houses and businesses and some empty fields with lots of solar panels in rows).  I also noticed lots of large-scale greenhouses in the farm fields, I couldn't tell if these were permanent installations or just temporary structures built over top of farm fields.  At least from the air, Germany looks very "green".

Once I got to Germany, the language barrier was not a problem at all.  I'd been told that there were alot of English speakers in Germany, but it really seemed that everyone spoke English.  It was funny too because most people would say "my English is not very good" and then they would go on to speak nearly flawless English.

Germany just felt very comfortable to me for some reason that's hard for me to articulate.  And I guess the Germans thought so too, because they almost always would initially speak German to me, until I asked "Sprechen Sie Englisch?".  The other people that came over with me from the States would usually get English right off the bat from the Germans they met.

We had several people also attending the conference from NI's Munich Office, and they were outstanding hosts, taking us to dinner each night and generally helping us out in any way.

I also found that business etiquette was a bit different:
  1. It's expected to wear a suit and tie, the US standard of a button up shirt and khakis doesn't cut it.
  2. Don't be late.  You'll get some very disapproving stares if you aren't punctual.
And this should surprise no one: Germans know beer.  Every beer I had there was excellent (and I had quite a few).  Don't worry about getting something you won't like; just drink it and it will be good.

I didn't have any time for sight-seeing so this is a crappy pic I took in downtown Nuremburg


Embedded World Exhibition Floor
The Exhibition part of Embedded World was crazy-huge, with 856 exhibitors and 26,000+ visitors.   It took me a long time just to do a relatively quick walkthrough of the floor.

I kept my eyes open for any LabVIEW front panels and I saw quite a few.  One of particular interest was a cool looking demo from Intel.  It's a slightly creepy looking insectile robot that plays the Chinese mandolin (not sure what the real name of the instrument is).  You can see the LabVIEW front panel on the monitor in the picture below.  The Intel guy running the demo couldn't say enough good things about developing in LabVIEW.



Presentation
My presentation went really well.  Almost every seat was filled; probably around 50 people.  I got some good questions and some compliments after it was over.  I also talked for a while with a couple of Intel guys that are heavily involved in the Yocto project.

Flight Back
The flight back was the only negative part of the whole trip. Coming in through Customs Stateside is no fun.  'Nuf said.

Wednesday, October 15, 2014

First open source patch accepted

I was about to post about my first patch accepted to an upstream open source project, but then I started digging around and discovered this one was actually my first!

I'm not trying to toot my own horn (ok, I am a little), I was just proud that I actually have some small, insignificant bits of code contributed to the open source community.

If anyone is remotely interested the first patch is to the Busybox project to fix a bug whereby a linked local network interface will keep getting a different IP between reboots when it should be getting the same IP on subsequent boots.  I can see your eyes glazing over...

The second patch is to the Yocto project and has to do with the generated entropy on an embedded Linux system.  Basically, if there's not enough entropy in the system then random numbers become much less random which can be a security risk if those random numbers are used for something related to cryptography.

And I guess to add to that I have lots of commits submitted to repos here and here, but those are repos owned by the company I work for so I wasn't counting those.




Wednesday, October 30, 2013

Wolverine claws for Halloween

A friend is wearing a Wolverine costume for Halloween, and he asked me to help him make some wooden claws.  He wanted them made out of thin (brittle) wood so no one would mistake them as actual weapons.  Although we haven't tested them, we're pretty sure the claws would break very easily if anyone tried to use them as a weapon.

I didn't think to take any pictures while we were making them but here are the final results:



Sunday, February 03, 2013

Scale results, round 2

I made a second run on my scale tester.  This time I used a 10KOhm thermistor as my temperature sensor.  This gave me around 0.15 degrees (F) of resolution, compared with the 1 degree resolution of the previous temperature sensor I used for the last test run.



Figure 1: Weight and Temperature vs. Time

The first graph this time is the usual weight and temperature over time.  You can see that the weight and temperature appear to be highly correlated just like the first test run. The other thing is that the temperature appears much less noisy then in the previous run.

Figure 2: Nominal weight minus weight vs. Average temperature
In Figure 2, I graphed the delta in weight vs nominal against a moving average of temperature.  I also threw down a best-fit linear trend line.  I can immediately see that the point cloud is much tighter than in the previous test run, which seems good.

Figure 3: Adjusted weight

The last graph is a plot of the weight reading adjusted by the trend line equation from Figure 2.  The ideal line is a straight line at 55 lb. Clearly the line is not straight but it certainly improves the data.  The weight reading varies by +/- 0.2 lb instead of +/- 0.75 lb for the unadjusted weight readings.

Conclusions

The adjusted weight did not seem much different from the previous test run, but it still seems useful.  I think I can now go on to implement the data logger for this project which will convert weight and temperature sensor readings to specific gravity.

There are still some unknowns here.  One is that the test runs have only been run on a constant 55 lb test weight.  It's certainly possible that the numbers will change if a different weight is used.  It might be worth re-testing with weights of 45, 65, and 75 lb weights.

Additionally, there seems to sometimes be a lag between temperature changes and weight changes.  This could be attributable to the temperature sensor and scale having different thermal mass, which is almost certainly true.  I could account for this by attaching the temperature sensor to the scale with thermal paste to ensure that they are at relatively the same temperature.