Computing stuff tied to the physical world

Posts Tagged ‘JeeMon’

,

FritzBox call log

In Software on Mar 12, 2010 at 00:01

I’ve been using the FRITZ!Box 7170 as ADSL modem and telephone interface for some time now:

Screen shot 2010-03-08 at 12.50.41.png

It has a nifty feature: when you enable the built-in call monitor by typing in “#96*5*”, you can connect to TCP/IP port 1012 to get real-time messages about all phone calls.

This is easy to hook up to JeeMon. I added a new “FritzBox.tcl” file with the following code:

Screen shot 2010-03-08 at 13.15.39.png

(Some experimental code was omitted from the above listing)

To enable this, the following line was added to application.tcl:

Screen shot 2010-03-08 at 13.14.42.png

Here’s an example of an incoming call, as shown in the JeeMon log:

Screen shot 2010-03-09 at 13.51.12.png

And here’s an outgoing call:

Screen shot 2010-03-08 at 12.55.09.png

Neat! I’ll hook this up as events once the JeeMon event system is ready.

These experiments are very useful to see how the current design is working out. The above code depends on the “MessageStream” class in the “Serial” rig. It doesn’t really add much value in this case, but some class hierarchies seem to be emerging. This socket connection is an example of a similar-but-not-quite-the same case as serial connections, requiring its own class. A socket connection is very similar to a serial connection once initialized – but this doesn’t show as clearly in the code as I would like and there’s already a bit of code duplication.

Will probably need some re-shuffling later, to better match concepts / design and implementation.


Outbound X10

In Software on Mar 11, 2010 at 00:01

I’ve added support for controlling X10 devices via the CTX15 module described in the past two days:

Screen shot 2010-03-09 at 20.31.20.png

First thing was to extend the ookRemote GUI, by adding these definitions to ookRemote.tcl:

Screen shot 2010-03-09 at 20.34.10.png

With this change, the configuration section in “config.txt” supports CTX15 buttons:

Screen shot 2010-03-09 at 20.34.56.png

These new buttons will send X10 commands to the CTX15 module. But this needs a bit of wrapping to turn them into frames, including the proper checksums. So I added two methods to “sketches/ctx15try/host.tcl”:

Screen shot 2010-03-09 at 20.37.16.png

The “send” method overrides the existing one and sets up a frame before calling the original method with it.

That’s it. I can now make the Marmitek AM12 switch “clunk” with my mouse – audible feedback! :)


JeeMon interface for X10

In Software on Mar 10, 2010 at 00:01

Now that the CTX15 has been connected via a JeeNode, it’s fairly straightforward to tie it into JeeMon.

First I created a new folder called “sketches/ctx15try/” with a file “host.tcl” in it, using this code as quick test:

Screen shot 2010-03-08 at 14.37.36.png

The reason there is nothing more, is that I changed the ctx15try.pde sketch a little to return lines starting with “CTX15″ – since these are the easiest to tie into JeeMon. The change affects these lines in yesterday’s demo:

Screen shot 2010-03-08 at 14.33.08.png

I’ve also wrapped the received frame in {}’s, because that simplifies processing in JeeMon (Tcl interprets $’s and []’s unless escaped).

To activate everything, I added an extra line to the config.txt configuration file, so that the output from the JeeNode USB (serial# A900adwo) automatically gets picked up as sketch:

Screen shot 2010-03-08 at 15.01.22.png

Here is an example of the above code running:

Screen shot 2010-03-08 at 14.36.50.png

Good. CTX15 events are coming in. They are easily recognizable as (unit#, command) pairs.

But as you can see, these pairs don’t always arrive in the same frame. So we have to add a bit of logic to get things right. Here’s an updated version of host.tcl:

Screen shot 2010-03-08 at 14.51.50.png

The solution used above is as follows: if it looks like a unit code (A..P + 1..16), then save that as last one seen. If it’s anything else, use the last saved location. I’m not sure this covers all possible scenarios, but for this very simple test it seems to work. Here’s a screen shot of the real-time status window:

Screen shot 2010-03-08 at 14.50.48.png

As you can see, the CTX15 event was turned into an “AOFF” command for the “ctx_A01″ device, which was created on-the-fly by this code.

Hmm, looking further, I think the “A” prefix needs to be stripped from “AOFF”. Oh well, it’s all scripted, I’ll adjust that when I get to using this stuff for real.

So that’s it. A few dozen lines of C code in the JeeNode to act as pass-through and do the polling, and a few dozen lines of Tcl code to tell JeeMon how to interpret the incoming data. Now we get notifications whenever any X10 activity is detected on the mains power line.

Onwards!


JeeMon goes embedded

In Hardware, Software on Mar 5, 2010 at 00:01

Testing out a new setup … I started by launching the JeeMon runtime executable as follows:

Screen shot 2010-03-02 at 15.41.09.png

Then I went back to my Mac and opened a browser window:

Screen shot 2010-03-02 at 15.56.12.png

It works! You’re looking at a little web server running on a tiny Technologic TS-7500 board:

DSC_1213.jpg

ARM based, 64 Mb RAM, µSD card socket on board, running Debian (lenny). It’s not very high powered, but in return it only uses 400 mA @ 5V, i.e. it can run off USB power.

I haven’t yet figured out how to get the FTDI USB serial driver on there, but once that’s solved this little unit will be able to act as host for USB-connected JeeNodes or JeeLinks.

The RAM and storage is more than sufficient to run even a very elaborate JeeMon setup, I expect.

This might be an interesting low-end always-on home sensor & automation system!

,

Pushing data as web client

In Software on Mar 3, 2010 at 00:01

Another way to get data to another place is via push HTTP requests, i.e. acting as a client for another web server somewhere on internet.

The Pachube site is one place to send measurement results. I’ve set up a test feed to use with JeeMon, with two data streams. One of the nice features is that you can easily produce and embed graphs from that system:

Screen shot 2010-02-26 at 02.45.13.png

Here’s how I implemented it in JeeMon. First, I created a configuration section:

Screen shot 2010-02-25 at 22.37.12.png

(I’ve omitted most of my private API key, but anyone can sign up and get their own …)

Next, the code, in the form of a “pachube.tcl” file / module / rig:

Screen shot 2010-03-02 at 03.11.12.png

Note how the configuration file in fact contains “settings” which are simply evaluated as a script: the two lines starting with “Fetch …” are handled by the Fetch proc in pachube.tcl, using the JeeMon notification mechanism to get called back whenever either of the “meter1″ or “meter2″ readings changes. The Update proc then updates the “latest” variable accordingly. Lastly, SendToWebSite periodically does an HTTP PUT request in the background, supplying all the info needed to successfully submit to the Pachube website.

Btw, this simple example is flawed, in that it does not calculate averages – it just sends the last reading. But things like averaging require a sense of history, and persistence. Haven’t added that to JeeMon yet…

The missing link, as usual, is a line in the application file to start the ball rolling:

Screen shot 2010-03-02 at 03.03.46.png

There’s a lot of flexibility at the protocol and network levels. Such as creating an XML request via templates, if the remote server needs XML. This isn’t limited to HTTP requests, or to using port 80 – send emails, FTP files, etc.

There are quite a few details in the above code which I won’t go into. Again, I’m doing this mostly to show how little code it takes to initiate periodic HTTP client requests to an outside website.

Apart from sites such as Pachube, this could also be used from a tiny embedded Linux running JeeMon, to submit incoming data to a different setup elsewhere on the LAN, for example. IOW, JeeMon can be a front-end for other systems. It’s not about lock-in. It’s a switchboard. It can glue systems together. It bridges gaps.

,

JeeMon as web server

In Software on Mar 2, 2010 at 00:01

JeeMon comes with a nifty built-in HTTP/1.1 web server (using coroutines to handle requests in parallel).

So let’s create another real-time status display, but as web application this time:

Screen shot 2010-02-25 at 17.56.25.png

As before, I’m leaving out the lipstick – just showing the core functionality.

Here’s what I did to generate this page. First of all, I’m re-using some functionality which is already in the GUI version, to dynamically construct a matrix with the proper columns and rows, so the “statusWindow” code has to be running for the above to work. For real use, that code should probably be reorganized into its own rig.

The main task was to create a HTML template in a file called “statusPage.tmpl”:

Screen shot 2010-02-26 at 00.41.23.png

This uses a simple templating mechanism based on Tcl’s “subst” command. The meta tag sets up the page to self-refresh every 5 seconds. The last line tells the Wibble web server to deliver the page as HTML.

To launch the server, this line was added to the main application file (port 8080, current dir is doc root):

Screen shot 2010-02-25 at 18.00.15.png

That’s it – there’s very little to activating a web server in JeeMon, as you can see. Which is important, because on tiny embedded Linux systems, an HTTP server will probably be the only option to present information.

Creating a full-blown site with CSS, JavaScript, and Ajax is a matter of adding more files – the usual stuff…

Did I mention that it’s all 100% open source, so you can browse / extend / change all of this? – I did? Oh, ok ;)

,

Node discovery

In Software on Mar 1, 2010 at 00:01

This post isn’t about GUIs, but about the dynamic behavior behind them.

And about wireless. Let’s take the voltmeter setup I described recently, and make it send its readings over the air – it’s a JeeNode after all, so it has wireless connectivity built in.

I extended the original sketch to send readings out roughly once per second:

Screen shot 2010-02-25 at 15.57.03.png

Extending the host side to also work from received packets is trivial:

Screen shot 2010-02-25 at 16.49.26.png

The problem now, is that this node will start sending out packets, but JeeMon has no idea what to do with them. One day, automatic node type registration would be great to add, but for now I’ll just use the configuration file to associate nodes with sketches. Here’s an extract of my current “config.txt” file:

Screen shot 2010-02-25 at 16.00.30.png

For this example, I added the “3/ { type analogPlug }” line to the “868:5/” entry (i.e. 868 MHz, netgroup 5). And sure enough, JeeMon will now launch the GUI for this node, displaying results coming in over the air in real-time:

Screen shot 2010-02-25 at 16.03.00.png

Here’s the relevant part of the log output:

Screen shot 2010-02-25 at 16.04.34.png

The fun part is that a timeout mechanism has also been implemented. If no packets come in from the analog node for 5 seconds, the connection is torn down and the window is closed again. The timeout value is sketch-specific and is included in the above configuration info.

In other words: node is ON => window appears, node is OFF => window goes away. Look ma, no hands!

Note: it’s not good UI practice to open and close windows like this. In a more refined setup, these windows would be panels in a larger window, or they could just be grayed-out to signal their disconnected state, for example.

The analog node is surprisingly convenient this way. I can use it anywhere, turn it on, look on the screen, and turn it off when I’m done. It wouldn’t be hard to add another wireless node to show the results on an LCD screen, btw. If this is extended with a way to assign node IDs ad-hoc, and a way for a sketch to identify itself over the air, then a whole family of wireless “instruments” could be created – using nothing but a JeeNode and some plugs.

This illustrates a guiding principle in JeeMon: don’t ask, just do it! – JeeMon tries to be as dynamic as possible, adapting to changes in status and configuration by adjusting everything it does accordingly. In the case of nodes, new ones are discovered and stale ones are weeded out, in real-time. With objects getting created and destroyed automatically. What these objects do, as side-effect, is up to each one of them,

Right now, this dynamism in JeeMon is still fairly messy. There’s too much state in too many places which needs to be managed carefully. But with a bit of luck, this can eventually be simplified further to create a very consistent and predictable behavior.


A real-time status GUI

In Software on Feb 28, 2010 at 00:01

One more post about setting up a GUI with JeeMon…

I wanted to have a basic real-time display of all the readings coming in, sorted by source. Like this:

Screen shot 2010-02-25 at 15.48.08.png

(the size of this screen shot was reduced to fit this post)

Everything in this window is dynamic: rows get added when new sources start reporting their data, and columns get added when new parameter types are reported. I’m not storing any information persistently yet, which is why this status window looks as follows right after starting up JeeMon:

Screen shot 2010-02-23 at 21.31.15.png

Not very exciting, but all you have to do is wait …

Let’s see what it took to create this GUI. First, I added a generic “Notifier” rig to JeeMon, and a “Reading” method for all sketches to use. The rooms sketch, for example, now reports its incoming results as follows:

Screen shot 2010-02-23 at 22.06.04.png

Similar changes were made to the ookRelay and RF12demo “host.tcl” sources.

The rest of the GUI code is in “statusWindow.tcl”:

Screen shot 2010-02-24 at 01.16.42.png

The tabular display is based on a very nice Tcl/Tk open source package called Tablelist by Csaba Nemethi, which has been added to the JeeMon core library.

The coupling with the rest of the system is accomplished by the “Notifier attach readings * …” call at the end of the setup code. This registers a callback which will be invoked when anything in the “readings” notification group changes. Notifiers are going to be used a lot in JeeMon because they provide such a manageable way to glue independent parts of an application together.

And lastly, adding the following line to the main application starts the ball rolling:

Screen shot 2010-02-23 at 21.42.16.png

There’s quite a bit going on here which I won’t go into. I just wanted to illustrate how little code it takes to create a basic, fully dynamic, cross-platform, self-updating status display window. There – how’s that for adjectives!

,

Voltmeter with GUI

In Software on Feb 27, 2010 at 00:01

Let’s make a 5-digit voltmeter with a JeeNode, an Analog plug, and JeeMon. But instead of a readout on an LCD screen, I’m going to show the results on the desktop. To start with that last part, here’s my little display on a Mac:

Screen shot 2010-02-23 at 20.23.25.png

You’re looking at my feeble attempt to create a good-looking / souped-up GUI, btw :)

Anyway. Here’s the sketch I’m running on the JeeNode, with an Analog plug inserted into port 3:

Screen shot 2010-02-23 at 11.01.16.png

Trivial stuff. It sends out lines with “VOLT <value>” readings twice per second.

The important part is that this sketch starts up with “[analogPlug]” as greeting. To hook up to this sketch, all we need to do is create a directory called “analogPlug”, with a file called “host.tcl” in it.

This new “analogPlug” directory is a sub-directory of “JeeMon-sketches”. This directory, in turn, has been registered as being used for sketches – with this command in my main application code:

Screen shot 2010-02-23 at 12.21.51.png

Back to the “JeeMon-sketches/analogPlug/host.tcl” file. Here’s its contents:

Screen shot 2010-02-23 at 11.40.34.png

That’s the whole kaboodle.

Now, whenever I plug in a JeeNode running the above “analogPlug.pde” sketch, a window pops up showing readings in real-time. When I unplug the JeeNode, the window is closed again. It’s all automatic, as long as JeeMon is kept running.

This mechanism is not limited to JeeNodes or Analog plugs, of course.


OOK remote control

In Software on Feb 26, 2010 at 00:01

The JeeMon story continues. Here’s a fun little panel for controlling some switches around the house:

Screen shot 2010-02-22 at 21.42.14.png Screen shot 2010-02-22 at 23.31.35.png

Screen shot 2010-02-23 at 20.19.28.png Screen shot 2010-02-22 at 21.51.23.png

These screen shots show Windows (7 and 2K), Mac OS X, and Linux Ubuntu, respectively – long live diversity!

This GUI was created with these configuration settings:

Screen shot 2010-02-26 at 02.09.33.png

Those settings were loaded by adding this line to the main application code:

Screen shot 2010-02-26 at 02.09.16.png

Which in turn loaded the following “ookRemote.tcl” source file I wrote, as module / rig:

Screen shot 2010-02-26 at 02.08.48.png

That’s the entire app. Adding lipstick (i.e. a fancier visual design) is left as exercise for the reader.

Note how the design is split in a configuration section and a source code file – I’m probably going to use this approach often in JeeMon. The idea is that people who don’t care about the code behind all this stuff can just adjust the configuration file (using a visual interface, hopefully, one day). While those who like to tinker can view the full source code, and explore and extend it at will, and more importantly: ad infinitum.

It’s called open source for a reason!

PS. There’s another major reason for the split between configuration settings and source code files: turnkey deployment. You can put all the source code files in a ZIP archive called “JeeMon-rigs.zip”, so that all you need for a new installation with JeeMon, is: 1) your archive, 2) the proper JeeMon runtime, and 3) a config file to match the target setup. The config file is optional, btw – more on that another time…

,

Declarative configuration

In Software on Feb 22, 2010 at 00:01

Things are starting to come together quite nicely with JeeMon – all in less than 1000 lines of Tcl code so far. That’s not to say that it’s perfect yet – I’m still massively rearranging some of these code structures – but there is already some functionality to play with, which helps expose the strengths and weaknesses so far.

The basic approach is declarative in nature: specifying what should happen, but not necessarily in the same place as when or how all the work should be done.

Here’s my first tentative configuration file:

Screen shot 2010-02-21 at 14.31.44.png

For each serial interface, an action is specified when that interface comes on-line. For example I can now plug in my A8007Up6 device (a USB-BUB with a JeeNode) and JeeMon will open a terminal window showing the output stream. Unplugging closes the window again (and automatically cleans everything up).

Right now, I’m editing this config file by hand, but at some point that could be augmented with a set of GUI or web-based configuration panels (even remotely).

There are a lot of challenges ahead. One of them is decoupling things so that code and data end up organized in the most convenient, logical, and flexible way. I’m trying very hard to avoid any calls to the “Config” module from generic JeeMon code. This allows you to set up the configuration file structure and hierarchy in any way you like.

My application file currently looks as follows:

Screen shot 2010-02-21 at 15.42.58.png

As you can see, I’m setting up a periodic port scanner and supplying a “SerialEvent” callback to decide what to do when serial ports get added or removed. In those decisions, most of the logic is driven from the “config.txt” configuration file. I’m about to implement a similar callback for radio events, i.e. RF12 nodes coming online and dropping off again.

In a way, the above two files are the entire application.

The rest is utility code to make the above convenient and concise. That rest might well be 99% of the code, but it’s all structured as loosely-coupled “rigs” (modules), which can be used / overridden / ignored at will.

The way I see it, there can be three sources for such utility code: your own code, plug-ins from others, and a couple of rigs included in the JeeMon core itself. The above two files require just the core, using the built-in rigs (Config, Gui, JeeSketch, Log, Serial, and SysDep).

,

Serial USB devices

In Software on Feb 21, 2010 at 00:01

For JeeMon, I’d like to be able to auto-detect USB device insertions and to identify FTDI serial numbers (with names like “FTE54GKC” and “A9007CNE”).

Cross-platform… i.e. on Windows, Mac, and Linux!

It looks like it can be done.

On Windows, we can browse the good ol’ registry, as follows:

Screen shot 2010-02-20 at 15.24.16.png

Sample output:

Screen shot 2010-02-20 at 15.24.41.png

This was tested on Win2000 and Windows 7, in the hope that everything in between works the same.

On Mac OS X, it’s easiest (for a change):

Screen shot 2010-02-20 at 15.26.02.png

Sample output:

Screen shot 2010-02-20 at 15.26.37.png

On Linux, it’s a bit messy because there are so many different distributions:

Screen shot 2010-02-20 at 15.27.43.png

Sample output:

Screen shot 2010-02-20 at 15.28.01.png

Tested on Debian 5 (Lenny), Ubuntu 9.10 (Karmic), and Gentoo.

The above code can be found in the subversion repository, i.e. here.

I’m testing this all at once on a single machine btw, courtesy of VMware Fusion. And using JeeMon’s built-in self-update mechanism to quickly get new versions across while debugging and tweaking things.

By calling “SysDep listSerialPorts” periodically, we can automatically detect a change and see which plug was inserted (FTDI only for now). Without depending on any external libs or executables.

Onwards!

,

RF transport independence

In Software on Feb 20, 2010 at 00:01

With the basic serial interface access and dispatch in place in JeeMon, it’s time to move on to JeeNode / JeeLink packet processing.

What I want is to be able to forget all about how readings got to JeeMon. It shouldn’t make a difference whether I’m using a directly connected Room Node and grabbing the readings off its serial USB connection, or whether they came in through the air via the RF12 wireless driver – or by any other means. Take a snapshot with your cell phone, send the picture to EverNote, have it OCR’d, and then grab the readings off the web … whatever!

The way I’ve tied this into JeeMon, is to let the interface to RF12DEMO act as de-multiplexer. This is purely a decision at the RF12DEMO listener level. Each incoming packet is examined to determine which node it came from. Then we need a way to map from nodes to listener class – i.e. find out what sketch is running on the remote node. This is hard-coded for now:

Screen shot 2010-02-18 at 22.57.41.png

What this does is actually a bit more elaborate: a RF12DEMO listener will set up and manage a bunch of extra “RF12_PacketStream” listeners, one for each node. When packets come in, they will simply be dispatched to the corresponding stream. Each packet stream can process its packets in a different way.

The fun part is that these packet streams can use the same listener classes as with direct serial interfaces. All they need is an extra “decode_RF12″ method:

Screen shot 2010-02-18 at 23.02.04.png

The task of decode_RF12 is to re-cast the incoming packet as messages of the same structure as what would come in over a serial connection.

Here’s the “rooms” listener as example:

Screen shot 2010-02-18 at 23.04.34.png

This one class encapsulates all the protocol details of room nodes, both serial and via RF12. When a 4-byte data packet comes in via RF12 (as $a..$d), the bits are decoded and an “onMessage” call is generated with a “ROOM” message id and the 5 decoded readings.

Here is a log of this code in action, one message per line:

Screen shot 2010-02-18 at 22.37.15.png

The way to read these messages is as key-value pairs, i.e. id = OK, type = RF12, name = usb-A900ad5m, etc.

The first two lines show an incoming OK message from node 21 (53=21+32), which is then turned into a ROOM message, tagged as coming from the “rf12-868.5.23″ packet listener.

The next 3 lines are more involved: first an EM10 message came in over USB, then an OK message came in which got dispatched again, as the same EM10 message. That’s because I’m running JeeMon with a direct connection to the ookRelay board, even though it transmits all its information over RF12. So everything from the ookRelay comes in twice (great for debugging).

The point is that the two EM10 messages have the same info. It no longer matters how the message got here (but it is traceable, using the remaining details). And all the code to accomplish this is in a single source file, right next to the sketch running on the ookRelay board.

This design makes it possible to develop an application using only the serial USB connection, and then later add logic to send the information via RF12 (or not). Infrared, XBee, Twitter, anything: transport independence!

Note that nothing is done with these decoded messages at this stage. This messaging framework is independent of higher-level application decisions, such as where to store / send / show msgs, or even when to process them.

,

Serial port encapsulation

In Software on Feb 19, 2010 at 00:01

This post continues to look a bit into the new JeeMon design.

Let’s focus on serial interfaces first, mostly USB. There’s a “Serial” module which does all the basics. On the Mac, if I want to open device /dev/tty.usbserial-A900ad5l, then the following call with do everything:

Serial connect usb-A900ad5l 57600

The name of the device would be different on Windows and Linux (COM5, or USB1), but that’s all.

By default, this creates a new Serial object, which logs all incoming text to stdout. To send a command out, we need to keep a handle to this object:

set conn [Serial connect usb-A900ad5l 57600]
$conn send "some text"

Nice, but not very exciting…

Let’s take it one step further. The “JeeSketch” module does the automation described in the previous post, i.e. detect the running sketch, associate it with a class, instantiate an object for it, and call the methods of that object whenever text lines come in over that serial port. Here’s a complete JeeMon custom “application.tcl” program:

Screen shot 2010-02-18 at 14.44.59.png

First, all the sketch drivers are made available with one or more “register” calls. This lets the appropriate classes take over for each new serial connection – depending on what sketch is running. That’s all it takes. Servicing such serial ports now becomes an event-driven background activity.

The listeners are defined in separate files, one for each type of sketch:

Screen shot 2010-02-18 at 15.05.58.png

The ookRelay/host.tcl file looks like this, for example:

Screen shot 2010-02-18 at 14.51.58.png

This structure makes it easy to manage stuff that belongs together. Projects can be exchanged (or archived, or revision-controlled), with all the pieces needed to use them in one place. And as far as I’m concerned, it won’t be limited to JeeNodes etc, or Tcl, or a specific platform. This has to remain general enough to hook up to any hardware and use with any language (via networking, files, and direct launching of executables/scripts). My goal for JeeMon is not to limit anyone’s options, but to create a simple switchboard between whatever is needed.

(I’m still mucking around with the organization and naming of code and files, as you can see)

Nice, but still not very practical…

The problem with the above is that it doesn’t deal with devices getting plugged in or unplugged. Well, unplugging is the easy bit – the above code will automatically clean up after itself on connection loss, so that part is covered.

Wouldn’t it be nice if we could just plug in new devices and get them to automatically start doing something?

I implemented such a mechanism in a recent revision of the code, but I’m hesitant to add it again – because it was Mac OS X specific, where USB devices connected via the FTDI driver include a unique code. On Windows and Linux, you just get COM<n> and USB<n> devices, where “n” seems to be related to the order and number of device insertions.

I haven’t looked into “libusb” yet. Should I? Will it help for Windows too? Do I need to start learning about USB enumeration? What OSS-compatible tools and libs are there?

Update – on Linux, it looks like /sys/bus/usb/devices/* has all the info needed to identify USB devices. So that only leaves Windows – good: at most one lib or dll to deal with.

,

In the kitchen

In Software on Feb 18, 2010 at 00:01

Ok, so maybe it’s time to start describing some of the new stuff cooking in the kitchen lab.

I’ve been exploring software designs to use as basis for JeeMon, that new switchboard-type application for physical computing devices. The idea is to treat the system as one big message-passing machine – a bit like Message-Oriented-Middleware (MOM), which has been around for ages, but without getting sucked into any heavy-weight conventions.

Messages can be passed around, queued, stored, duplicated, filtered, transmitted, returned, ignored, sorted, etc. After all, living organisms do nothing but send chemical messages around, so why not do the same for an infra-structure focused on physical computing (sensors, displays, actuators) and home automation?

If you’ve been following this weblog for a while, you’ll have noticed that almost all the output I generate in sketches is of the form “identifier arg1 arg2 …”. So for example, packets received by RF12demo look like:

OK 61 9 2 8 79 243 87 13 0 15 0

A barometric reading from the BMP085 sensor on the pressure plug may get reported as:

BMP 233 10019

And so on. One or more upper case letters, optionally followed by digits. Then the payload (which may also include floating point and string values, not just ints).

Another convention I’ve been sticking with is to report the name of the sketch on the serial port during startup:

[ookRelay]

Or an identifier plus the current configuration settings:

[RF12DEMO] W i23 g5 @ 868 MHz

These two conventions can be used for an object-oriented design. With a bit of preparation, the name of such sketches can be automatically associated with a class, and each line treated as a method call.

Here’s the skeleton of the first level of code I use for decoding RF12demo.pde output:

Screen shot 2010-02-16 at 00.15.27.png

Another class definition, for the ookRelay.pde sketch:

Screen shot 2010-02-16 at 00.16.11.png

Or to put it differently: with JeeMon running, you can hook up a device (JeeLink, JeeNode, Arduino, etc) containing some sketch and the matching class will automatically be associated with that serial port, once the sketch has been identified. A new object is then created, and its methods will be called whenever the device sends new output (messages?) over the serial line.

Simple!

What I would like to do, is manage the sketch (C/C++) and the class (Tcl) together, perhaps as two files in a common development directory for that project. That way the interface between the two pieces of hardware essentially vanishes into the background. The point being that each class can then do all sorts of things, such as storing results in a database, sending it to another system over the network, updating web server pages, popping up a GUI window to show incoming data in real-time, etc.

This mechanism is very simple, even under the hood. This matters, because it has to work even when JeeMon is running on low-end embedded Linux hardware. But at the same time, such a MOM + OO design will allow considerable flexibility for abstract / high-end use later.

PS. If you’re familiar with Tcl, you might be surprised to see all this “oo” stuff in here. That’s because JeeMon uses Tcl 8.6 with built-in object system. Multiple inheritance, mixins, filters, dynamic class change support, delegation, prototypes / per-object methods, it’s all in there. I’m also using ensembles and there’s an interesting coroutine-based web server waiting in the wings (called wibble).

For the record: nothing ever gets added just to be buzz-word compliant. If a feature simplifies application-level concepts and leads to a cleaner implementation in JeeMon, it’ll be used, else I’ll pass. Life’s too short to jump on bandwagons.

,

Temperature plots

In Software on Jan 22, 2010 at 00:01

With over a dozen nodes now active, JeeMon is finally starting to collect a bit more useful data to work with:

Screen shot 2010-01-22 at 12.03.23.png

Looks like some temperatures have a granularity of 1°C – might be a few cheap DS1820 units I had…

The Mason-like templating code in “try-temps.html” to generate this plot is still far from obvious:

Screen shot 2010-01-20 at 01.52.42.png

Once I get to spend some real time on all this, it’ll look totally different. JeeMon as a whole is going to change completely in fact – until then, you can find some technical background information in an older post.

Let’s just say that it gets the job done right now…


JeeMon – one year later

In Software on Jan 4, 2010 at 00:01

JeeMon is a little web server / database / reporting tool I wrote a while back. It has been in use at Jee Labs for over a year now. Here is the electricity use for 2009:

Screen shot 2009-12-31 at 17.27.15.png

Couple of glitches, such as incorrect readouts when power failed and the sending node got reset to zero counts.

Here’s the gas consumption for 2009:

Screen shot 2009-12-31 at 17.28.06.png

Gas consumption (heating and hot water) is relatively high for this house, which is an open split-level design with lots of windows. Well isolated, but there’s simply no way to keep heat from rising up through the open stairways.

Here are the temperature and humidity readings for the past month:

Screen shot 2009-12-31 at 17.38.39.png

These are commercial S300 and KS300 sensors, and they seem to have frequent glitches. It looks like those could easily be filtered out, though.

Still, we managed to get €600 back on the 2009 energy bill, and with about 3000 kWh and 2000 m3 we’re 30% below the average consumption in this residential neighborhood, for both electricity and gas.

Being aware of what’s going on makes a difference, IMO. It has become a habit to turn off all the lights when I leave a room, and closing curtains right away when it gets dark. When we go out, we turn down the thermostat. Who needs high tech, when common sense is all you need? It’s so obvious, yet so effective …

The JeeMon database for all of 2009 is 26 Mb, i.e. tiny when considering that this includes every reading and some aggregated series as well. In fact, it contains a lot more data than what’s shown in the above graphs.

I’ve got several ideas and plans for JeeMon in 2010. I want to make it far more modular, so that nearly all its current functionality becomes available as easy-to-extend plug-ins. And it needs to be fullly configurable – as it is, JeeMon is still little more than a one-off implementation. But it has served its purpose very nicely already.


More efficient computing

In Hardware on Nov 4, 2009 at 00:01

A few days ago, I replaced my (3y old) MacBook Pro + (8y old) Cinema Display with a brand new 27″ iMac. Here’s the result:

Screen shot 2009-10-31 at 17.59.38.png

That’s the total energy consumption of my work desk – i.e. computer, lights, and a some USB-powered stuff. The scale is in Watt-per-5-minutes, so if you multiply all figures by 12 you’ll get the normal Watt ratings.

There are many different operating modes here – night-time near off, full sleep, display sleep, display on, a few computing peaks, and in the evening also the lights on.

On Oct 30 both machines were active, while all the data from the old setup was being migrated to the new one.

The interesting parts are the low periods at night which more or less dropped to zero, even though the iMac is simply sleeping (with BlueTooth and USB still on), not turned off.

It’s also nice to see that even with this new dual-core 3.3 GHz monster, energy consumption is noticeably lower most of the time – not just in idle mode!

These readings were obtained with the ES-1 sensor described a while back + a central JeeNode + JeeMon.


Monitoring failure?

In Hardware, Software on Oct 23, 2009 at 00:01

Yesterday, the Jee Labs energy monitoring system stopped working:

Picture 2.png

A few hours went by before I found out, because I’m not constantly checking these graphs nowadays.

Couldn’t figure out why, so I reset the sending node next to the electricty/gas meters, and shrugged it off as a fluke because it all started working again.

A few hours later, after dinner, we find out that there’s water dripping down right next to that JeeNode, but more importantly the whole meter closet is soaking wet in some places. Whoops… not good!

Turns out that there was a leak in the kitchen drain right above that meter closet. Fortunately, the problem was easy to identify. And didn’t lead to fireworks.

That was around 8 PM. Only this time the JeeNode is also soaking wet – free-hanging constructions are not such a good idea after all, I guess. Unplug. Wipe off the water. Let it dry for a few hours. Plug it back in.

OK, the energy monitoring setup is working again. Tomorrow, I’ll fix the drain pipe – properly.

The drawback of tinkering is that you always blame your own stuff when there is a failure. But in this case, it was trying to tell me about a problem coming from elsewhere…

,

OOK packet decoding

In Software on Oct 22, 2009 at 00:01

The OOK packets are now collected and stored in the JeeMon database.

Here are the first few readings from the barometric pressure sensor inside the OOK relay, as received by the central node and graphed by JeeMon:

Picture 3.png

Here is the outside temperature from a WS300 weather station on the roof:

Picture 1.png

JeeMon is written in Tcl – the following new code was added to process packets from the OOK relay and get the different parameters stored:

Picture 2.png

The RELAY decoder processes incoming data, separating the bytes into one or more messages and calling the corresponding type-specifc decoders. The RF868 decoder merely reformats incoming data to the format used by another decoder called “ALT” and which was already present in JeeMon. The RF434 decoder is just a stub for now. Note also that the DCF decoder logs the current time – it may seem odd, but storing the received time along with the current system time can be used later to detect local clock drift and compensate for it.

It’s perhaps all mumbo-jumbo for now, I just wanted to illustrate how messages get routed inside JeeMon, and that it takes relatively little logic to deal with new data sources.

This would really benefit from a flexible plug-in system, so that this sort of logic does not have to be added in JeeMon itself but can be picked up on-the-fly. Then it would be easy to add new nodes such as the OOK relay and have JeeMon automatically incorporate the corresponding logic to decode, process, and store their data packets.

, ,

Parallel WSN choices

In Hardware on May 20, 2009 at 00:01

Just came across the eKo range of commercial products by Crossbow. Interesting how the eKo and Jee-Node/-Hub/-Mon designs almost exactly overlap – well, seen from a large distance anyway.

Their “JeeNode”:

Picture 2.png

Their “JeeHub”:

Picture 3.png

Their Linux server hardware and software:

Picture 4.png

Of course the eKo products are a million times more mature and advanced than the Jee-stuff which isn’t even a product, but still…


Bubba JeeMon back up

In Hardware, Software on May 12, 2009 at 00:01

The title of this post can be interpreted in two ways, both accurate: 1) the Bubba Two NAS from Excito was my back-up server for the MMnet1001 module, and 2) that Bubba box is now my main JeeMon server and back in the air with all the electricity / gas / weather monitoring working again, as well as the Pachube and Bwired feeds.

I will probably stick to the Bubba from now on. It’s running Debian Etch Linux, and there’s a JeeMon runtime for it so the switch-over was trivial.

bubba-two.jpg.jpeg

This NAS box consumes about 4 watt, which is less than the specs because it has been modded to use a 2.5″ SATA disk instead of the standard 3.5″ ones they normally ship. It’s also a 300+ Gb file server which has been running various periodic tasks for almost a year now. There’s in fact much more disk storage in the form of 3 additional USB disks, but these are normally powered down (via a KAKU switch).


Added bwired.nl data feed

In Software on Apr 27, 2009 at 00:01

The bwired.nl website has been promoting domotics “by example” for a long time now. Pieter Knuvers, the guy behind it all, added a new service to submit and display data from other fellow energy geeks like me…

The bwired upload service was announced (in Dutch) on Pieter’s weblog.

So I added the bwired.tcl Tcl code to JeeMon to generate the XML input he wants:

Picture 2.png

It’s trivial but way too messy stuff, mainly because I don’t have the proper abstractions in place yet to easily generate all sorts of aggregated data. In this case hourly averages and daily totals.

Results can be viewed here, name “jcw”.

Oh well, good exercise. Pretty much the same logic as for the Pachube feed.


JeeMon self-update mechanism

In Software on Apr 26, 2009 at 00:01

As I’ve mentioned before, JeeMon is a self-contained executable which requires no installation and runs on Windows, Mac OS X, and various Linux systems (including the ARM chip on the MMnet1001 modules used in the JeeHub). There are no special requirements or dependencies, no Java runtime, no .NET, no nothing (note: on Linux you have to check that the standard C++ runtime libraries are present).

The JeeMon builds are here – you only need to fetch one and launch it to use it.

This is based on a generic runtime called Tclkit which contains a full implementation of the Tcl programming language as well as the Metakit transaction-based embedded database. It’s hard to overestimate the amount of functionality included in these 1 .. 2 Mb executables.

The only specific code in JeeMon is the startup logic to turn it into a web-server with home monitoring functionality. It’s all inside one main.tcl file, the essence of which is shown here:

Picture 3.png

The code starts off with a few definitions and then loads the rest of the application. From file if present, else from a file loaded from the internet. The “fetchLatestLibrary” code by default also updates to the latest version, this can be disabled by deleting the “Jee-library.update” file.

By changing one URL shown above, JeeMon can be turned into any other type of Tcl application, because everything the application actually does comes from the downloaded file.

For those who don’t like this automatic mode, there is always the option to download all the pieces and run them as is. Both JeeMon and the Jee-library are based 100% on open source software. JeeMon is implemented in C/C++ and can be compiled from scratch using the standard gcc/automake tools, and Jee-library can be unwrapped to inspect every single bit of it – it is nothing but a packaged collection of plain text files and images. I’d be happy to document this if needed.

I suspect that most people won’t quite appreciate the implications of the way JeeMon is built and packaged until they try it out. And the reality is that most people won’t ever try out this stuff – fine with me. The technology underneath JeeMon is just a set of choices. In the end, the degree to which these choices become invisible in day-to-day use of this system is all that matters. And for a platform-independent modern programming environment, it doesn’t get much more general purpose and deployable than this…


Tracking JeeMon status

In Software on Apr 25, 2009 at 00:01

This is what I see on my screen these days:

Picture 1.png

This text is not inside a window but on the desktop, i.e. it gets covered up by all regular windows (there is always Exposé’s F11 shortcut to uncover it).

The text is barely readable, not only using a tiny font but also using 50% transparency. This is intentional: the text is not there to distract but to offer me a quick way to check on the status of my permanently-running setup on the JeeHub with embedded MMnet1001 Linux module.

It’s all done through a little utility called GeekTool, there are no doubt similar ones for Windows and Linux. Geektool periodically executes some shell commands and shows the output on the desktop.

The first listing displays the last 5 lines of the log output:

ssh propie tail -5 jee_out

The second listing is plain-text output generated by a new web page in JeeMon:

wget -q -O- http://propie/data/current

Adding that page to JeeMon was trivial (data collection details omitted):

Picture 2.png

Now, I can always see the latest readings and the status of the JeeMon server.

P.S. I will be away until May 3rd and won’t be able to respond to emails and comments until then. This weblog will be on auto-pilot for a week, enjoy!


Nine times faster

In Software on Apr 17, 2009 at 00:01

The graph pages in JeeMon are a lot faster now. Here are the “before” …

Picture 1.png

… and “after” timing statistics:

Picture 2.png

A nine-fold speed increase. Great.

The change is that hourly and daily values have been re-calculated and stored in the database, instead of constantly recalculating these on-the-fly. The last value is not stored, as it might be incomplete and could change when more readings come in.

Keep in mind that these measurements are made with the MMnet100 Linux module, which is a very low-cost / low-power ARM-based system. On a modern PC, the graph pages usually come out instantly, even without storing any condensed hourly or daily datasets.

The other thing to keep in mind is that all JeeMon processing takes place in scripted languages: Tcl on the server and JavaScript in the browser. And that this is handling non-trivial amounts of data, since JeeMon is basing all these graphs on 5-minute values collected for up to an entire year.

In short: 9x faster is good enough, the graph pages now finish loading within two seconds.

It’s nice to see hunches work out in reality…

Update – on the MMnet1001, the UBIFS flash filesystem has compression turned on as default for all files. It would seem to make more sense to keep Metakit database files uncompressed, so I did a “chattr -c Jee-database” and made sure to rewrite the file from scratch with decompressed data. However, it looks like this only shaves another 5% off the graph page times, bringing times down to from 2.0 to 1.9 seconds. Oh well, it was worth a try.


Inside JeeMon

In Software on Apr 14, 2009 at 00:01

I thought I’d share a bit here how JeeMon works and what it’s made of.

One of the main goals for JeeMon, is that it should support two modes of operation:

  • running on a personal computer as one of many processes, launched and stopped at will
  • running on a dedicated system which is always running, possibly with quite limited resources

There is not that much difference between these two modes, but it does mean JeeMon should present itself as a webserver and that its processing demands have to be low. Being portable to different platforms also helps.

JeeMon is essentially a webserver plus database, both embedded and combined into a single process. It also contains all the logic to collect readings from – and send commands to – a JeeHub via a USB or serial port.

I picked the technology I know well, some of which I have helped develop or developed myself over the past years:

  • the code is written in two languages: Tcl on the server and JavaScript in the browser
  • the Tcl runtime uses a highly portable and self-contained system called Tclkit
  • the web server is my own design, using a highly modular Mason-like system called Mavrig
  • the embedded database is Metakit, for very compact and efficient storage of data
  • the JavaScript code uses jQuery as library to simplify many tasks
  • interactive plots are generated in the browser using the Flot package
  • the JeeMon application is packaged as a Starpack into a single installation-free runtime
  • similarly, all Tcl code, Javascript code, and website content is wrapped up into a Starkit

[more details below...]

Read the rest of this entry »


Graph navigation

In Software on Apr 10, 2009 at 00:01

JeeMon lets you zoom in (“drill down”) on the electricity / gas / water consumption data. Here’s how:

Picture 1.png

That’s 4 graphs: 1-day details with 5-minute resolution, 2 weekly summaries, and an entire year.

Picture 2.png

The weekly graphs only differ in the level of detail: on the left is the daily change each hour, whereas the graph on the right shows daily totals.

Clicking on a day in either of the weekly graphs causes the daily graph to display that day. And a click on any week in the year overview switches both weekly graphs to that week.

With two mouse clicks on this page you can display the details of any day in the past year. To easily go back to the initial display, a “Go to Today” button appears once a different graph has been selected (same as a refresh).

Note also that selecting a time range in the daily graph calculates the total consumption in that period, and hovering over a bar in the bar graphs displays the exact value as a tooltip.

No other zooming or panning is available, just these fixed choices. Simple and effective, IMO.


JeeMon demo (alpha)

In News, Software on Apr 9, 2009 at 00:01

Today I’m releasing a first version of the open source JeeMon home monitoring application, along with a demo dataset. Warning: this is an early alpha version – all features and bugs are still likely to change, a lot!

There are builds for Windows (x86), Mac OS X (x86 and ppc), and Linux (x86 + x86_64 + some embedded systems, such as the JeeHub’s MMnet1001 module).

It takes 3 steps to run this demo – there is no installation:

  • Get a runtime for your computer from this area
  • On Unix-like systems, run this: chmod +x JeeMon-*
  • Launch JeeMon

Here is the debug output this generates on my laptop:

Picture 1.png

And here’s a Windows screenshot with similar output:

Picture 1.png

Now point your browser to http://localhost:8888/ to see what JeeMon has to offer.

When started for the first time, JeeMon downloads a few extra files from the internet: “Jee-demodata”, “Jee-library”, and “Jee-library.update” (under 2 Mb total). These files are obtained from public servers at Dropbox, I do not track downloads or statistics.

On every subsequent startup, JeeMon looks for updates and refreshes the “Jee-library” file if there is a new version, so by simply starting JeeMon again a few days later you can track its development progress. To disable automatic updates, delete the “Jee-library.update” file.

I’m releasing this code to give you an early glimpse into the Jee Labs kitchen – and to gauge the interest and figure out how to improve JeeMon. If the demo works for you, great. If it doesn’t, you can either try to figure out what the problem is, or delete all the Jee-* files and try again at a later date. The new mailing list announced yesterday is open to anyone wishing to comment and make suggestions.

Let me reiterate that this JeeMon alpha release is for infinitely curious and technically interested people, not for those looking for a finished home energy monitoring solution. Running this JeeMon demo is bound to raise more questions than I can deal with – but it’s all open source, so feel free to explore this as much as you like.

Whatever. Enjoy …

Update – the demo dataset now contains data since 2009-01-01, i.e. a total of over 3 months of readings.

, ,

New "jeelab" mailing list

In News on Apr 8, 2009 at 00:01

For those who want to share their interest / questions / ideas w.r.t. any of the Jee Labs projects, I’ve set up a mailing list at Google Groups:

Picture 2.png

See http://groups.google.com/group/jeelab/about – if you don’t want to sign in to Google to join this list, let me know and I’ll add your email address manually.

Update – the mailing list has been replaced by the Jee Labs discussion forums at http://talk.jeelabs.net/.


It gets a lot worse…

In Software on Apr 7, 2009 at 00:01

Yesterday’s post was about web server performance. Here’s the latest build of JeeMon with the same page:

Picture 1.png

That’s nearly 17 seconds to render the page now, versus 5 yesterday… whoops!

Here’s why I call this major progress anyway:

  • The “electricity.html” page no longer contains plot data values and is half its previous size.
  • All plot values are obtained via Ajax calls, running (mostly) in parallel.
  • This opens up the path to unobtrusive refreshes and automatic plot updates.
  • All plots now contain meaningful data, the previous version used some bogus values.

As you can see in the above graph, over 15 seconds is spent in waiting for the server to return all plot data.

And the reason this now takes so long is very simple: all values are calculated on-the-fly by the server from stored 5-minute values. That’s over 100,000 aggregated values to generate a simple 52-week bar graph!

All this should sort itself out once I start caching hourly and daily results in the database. For now, only the raw readings and the 5-minute aggregated counts get stored. The finecky bit is to make all aggregations / calculations consistent across arbitrary JeeMon restarts.

Here’s the latest version of the generated page:

Picture 2.png


Website debugging tool

In Software on Apr 6, 2009 at 00:01

Just to prove that JeeMon is far from ready for real-world use…

The Safari web browser has excellent tools to figure out where bottlenecks are. Here is an overview of where the loading time is going for the main electricity consumption page, as produced by JeeMon running on the MMnet1001 Linux module. That web page has several detailed graphs on it – here are the stats:

Picture 1.png

As you can see, all the time is consumed by the initial page generation on the server. That’s 4.64 seconds to generate the page – which is totally unacceptable. JeeMon serves that same page 50 times faster when running locally on my Mac laptop (!).

Here are the amounts of data transferred to serve that page:

Picture 2.png

The main page including the plot data is about 5 % of the total size – the other files are cached as static content, which is why they do not increase the transfer time.

There is a lot more information to be gleaned from these measurements, btw.

I have a hunch on where the bottlenecks in JeeMon are, but haven’t looked into it yet. Which is exactly how it should be: aim for solid / correct behavior first and use the proper instruments to measure performance and track the effects of optimizations later (I happen to use Safari, but similar tools exist for other browsers).

Stay tuned for updates, once I get to tweaking this…


JeeMon sneak preview

In Software on Apr 5, 2009 at 00:01

I’ve started work on a basic website design for JeeMon:

pastedGraphic-1.png

Here’s the gas consumption page:

pastedGraphic.png

The same JeeMon code is currently running on the “production” (ahem) JeeHub and on my development Mac.

These are real live webpages, with real (but not yet 100% correct) datasets. Still, this is just a tiny tip of a massive iceberg. Lots and lots of things left to do.

Well, at least it’s a pretty iceberg IMO ;)


Real-time tracking

In Software on Apr 3, 2009 at 00:01

Here’s a real-time graph of data collected by JeeMon:

And a snapshot, in case the above real-time graph isn’t working right now:

Picture 2.png

It’s generated via a new site called pachube – I’ve extended JeeMon to feed that site through simple REST-style HTTP requests. Since the graph only has 15-minute resolution, that’s how often it gets updated.

Here’s another real-time feed, showing our gas consumption in l/hr:

Pachube is still in beta. I’ve only just started looking into it. You can define any number of feeds, each having any number of data values. Not sure how it deals with missing values, nor what to do about data which is collected at different rates. The graphs are somewhat configurable, as you can see - this will no doubt still evolve further.

Pachube can poll a server you designate (“automatic” mode), or you can feed it new data at your own pace (“manual” mode), as is being done here. The advantage of the latter is that it gets through firewalls and avoids the security issues of running a public server.

It’s an interesting approach, apparently pachube aims to become some sort of free exchange for producers and consumers of location-specific real-time data.

I’ll leave this setup running for now, but don’t be surprised if this post stops showing an up-to-date graph, since I might stop this feed one day.


The logic of data logging

In Software on Mar 22, 2009 at 00:01

The point of several projects here is to monitor energy consumption, and probably also some weather data because it’s so easy to add. Where “monitoring” means logging as well as keeping measurement data in a database for quick access. But how do you make sure that all “readings” get logged, yet still retain the ability to adjust and extend the underlying database and software? It would be awful if each non-trivial change required conversion.

Here are a few basic design decisions I took for JeeMon:

  • Readings come in as text from the attached JeeNode, one line per reading.
  • All readings are time-stamped and appended to a logfile, in text format.
  • To keep things manageable, logfiles roll over to a new one on 0:00 GMT each day.
  • The first entry in a new logfile is the name and size of the previous one.
  • The database is treated as a cache: it can be reconstructed from the logs at any time.

And here’s a requirement I want to meet:

  • It must be possible to restart JeeMon at any time without losing readings.

The idea then is to “catch up” with the logs on each restart where possible, and to clear the cache db and reload it from scratch when the software changes are more substantial.

So what does it take to not lose any readings? Keeping in mind that JeeMon will run either on a little embedded Linux module right next to the central JeeNode, or on a personal Mac, Windows, or Linux computer. Well, first of all, this is why the central JeeNode has a backup battery and dataflash memory: it stays on at all times, and continues to receive and collect packets from all the other JeeNodes even when there is no JeeMon running. The duration of this autonomous operation will be fairly modest: a few hours or perhaps one day. Enough to handle brief power loss, to mess around with configurations, and to re-connect things occasionally.

This means there are now three places where data gets added and must be kept in sync with the rest:

Picture 1.png

(Four places if you also count the final destination: dynamically updating browser windows)

  • The dataflash memory gets a copy of each reading collected by the central JeeNode.
  • All readings must end up in the log files when JeeMon is running.
  • The cache database stores decoded values in an efficient internal format.
  • And lastly, the browser(s) present more or less detailed results, summaries, and derived info.

JeeMon will start up in logfile “catch up” mode: the current content of the dataflash memory will be checked against the last logfile entry. The first task is then to request old data from the JeeNode to bring the log files up to date. During this time, the JeeNode continues to add new incoming data to the dataflash.

Once all missed data has been transferred, the JeeNode switches to “real-time” mode, saving new readings to dataflash and sending them out to JeeMon at the same time. In real-time mode, the log files will track all readings as soon as they come in.

That’s just part of the story, though. Now JeeMon enters “db sync” mode. With information from the cache db, new entries and new log files are scanned and processed, with new readings added to the database. Once that is done, JeeMon switches to normal real-time operation.

All log files are kept online. Rebuilding a database from scratch is as simple as deleting the current one and restarting JeeMon. Since a full rebuild might take quite some time, the internal JeeMon webserver always starts up in a “please-wait” mode and switches to the real server after real-time operation has been enabled.

Part of the above logic has now been implemented. JeeMon resumes its logs, syncs / rebuilds its database as needed, and processes new readings while running. The firmware in the JeeNode to save and replay readings to and from dataflash is still work-in-progress, however.

So while the JeeHub / JeeNode needs to stay powered up, I can now restart JeeMon at will. Which is great, because that streamlines development.

, ,

Let's pick a few more names

In Hardware, Software on Mar 20, 2009 at 00:01

Not to worry: I’m not going to go crazy on the Jee<blah> naming used so far…

But I do need to give the different pieces some name. To be able to refer to them in these posts, but more importantly as names for the software for all this. I also need to introduce a basic structure (and some limits).

Picture 1.png

Here goes:

  • A JeeNode is this ATmega-with-4-ports-and-RFM12B thing. There may be lots of JeeNodes for various purposes. They can communicate with each other via wireless.
  • I use the Ports and RF12 libraries with JeeNodes, although this is not a hard requirement – both the JeeNodes and the libraries can be used in numerous other ways.
  • Each JeeNode has a letter ‘A’ to ‘Z’ assigned to it as node ID. Nodes are usually given a unique ID to avoid mixups, but this is not strictly required.
  • There can be up to 250 separate groups of JeeNodes. Nodes can only communicate with other nodes in the same group. Gateways between groups could be implemented later, if needed.
  • Most nodes will be pulse JeeNodes, i.e. running a specific piece of software called – you guessed it – “pulse”, which continuously monitors some attached sensors and reports the readings via wireless.
  • The JeeHub is either a JeeNode by itself connected to a Mac, Windows, or Linux PC via USB, or a JeeNode connected to Ethernet via a small dedicated Linux module.
  • This JeeNode-as-part-of-a-JeeHub is called the central node from now on. It runs a specific software configuration, also called “central”. There should always be exactly one central node.
  • The software running on the Mac, Windows, or Linux machine(s) is called the server from now on (how original, eh?). It consists of a system-dependent executable runtime called JeeMon plus the code and data for the application itself.

It probably doesn’t hurt to reiterate that “JeeNode” and “JeeHub” are hardware, whereas “JeeMon” and the Ports / RF12 libraries are software.

The above names are also used in the source tree repository now.

There are no doubt still infinitely many ways to lead you astray, but I hope that these definitions will help me place everything into an increasingly coherent context.

, ,

Collecting data with JeeMon

In Software on Mar 17, 2009 at 00:01

This is the information I’m currently tracking real-time:

Picture 1.png

(it’s in Dutch, but you can probably guess most of these)

And this is the setup I’m using for it:

Picture 2.png

The NAS in there is used as gateway to pass 868 MHz OOK data from a CUL receiver, which decodes signals from my KS300 weather station plus a couple of S300 temperature/humidity sensors. This will one day be replaced by on-board reception on the JeeHub, so that only the JeeHub needs to stay powered up at all times. All other sensors are hooked up to a couple of JeeNodes which transmit the readings wirelessly.

The Mac, Windows, Linux, and the JeeHub all run identical copies of the software, which is called “JeeMon”. It has a built-in web server, an embedded database, and a flexible set of network functions.

Each JeeMon instance will automatically self-update to the latest version on startup. During development on the Mac the JeeHub acts as transparent proxy, as if the different sensors were connected directly to the Mac (through a little Tcl-based system called “Tequila”). Once ready, the latest JeeMon release is wrapped into one file and placed on the internet. Finally, a restart of the JeeHub completes the upgrade.

That first screen dump above is a small test app on my Mac which bypasses JeeMon and connects directly to the JeeHub as Tequila client. I keep it open to check that data is coming in and gets saved on the JeeHub.

So this is the big picture for collecting energy/gas/water and environmental data in the house. The software can run on practically anything, can be accessed with a browser anywhere, and with proper security in place the various pieces can be connected and used across any network topology.

All of the above is working this very moment. The major task ahead is the full-scale processing, presentation, and interaction of it all. But that can now conveniently be done on my development machine, with HTML, CSS, JavaScript, etc.

Anyway, IMO this is a very flexible foundation for a 1-watt home monitoring server.


Ignoring imprecise data

In Software on Feb 9, 2009 at 12:05

For some time, I’ve had a second set of calculations running which produces much more accurate and regular data than some of what’s on these past charts:

Picture 1.png

The difference with the graphs is that only readings which come in exactly at the time the counter changes are considered here. All retransmission (i.e. readings tagged with a late time stamp) are simply ignored. Only one retransmission happens to be in this example (the one marked “6+2″), but it still produces consistent results.

It looks like the gas measurements could be improved by tagging all readings in the sensor node instead of at the receiving end of the packets. That way retransmissions would still be tagged properly, even when readings arrive a bit late.

Hmm, this requires running a clock with milli-second accuracy on the sensor board for best results… not entirely trivial over extended periods of time. The sensor board will need to be synchronized to some other clock – either an on-board DCF77 radio or some data received from another node. And I’ll need to use a crystal instead of a resonator (0.005% vs 0.5% drift).


Still not right

In Software on Feb 8, 2009 at 23:55

The gas measurements are still giving me trouble:

Picture 3.png

I’ve changed scales and plotted all values to better see what’s going on over a small time frame. The electricity consumption looks ok now (these are night-time values). But the gas measurements keep jumping up and down by a factor of around 5 all the time!

Maybe there’s a problem with the sensor readout – there are about 4 readings coming in per minute. Or perhaps the automatic idle transmission of packets every 10 seconds is interfering with this cycle?

I don’t know what’s going on… needs more work!


Measurement anomalies

In Software on Feb 7, 2009 at 02:38

Here’s a more detailed view of the measurement quirks related to gas consumption:

Picture 1.png

The yellowish line is electricity usage, which hovers around 200 watt, while the blue line is the calculated gas consumption. What happens is that the heater cycles on and off approximately once every 15 minutes. When off, no gas is consumed so the meter just stops providing pulses. Then, after roughly 10 minutes it starts again at our heater’s standard rate of around 1.5 m3/hr. If you look very closely, you can see that the “zero” values are actually just above zero. That’s because whatever gas flowed until the rotation stops and whatever gas flows right after it starts all gets averaged-out over the entire period.

What needs to be done, I think, is to re-interpret the long averaged-out period as a period of continued flow at the original full rate, followed by zero flow:

Picture 3.png

I.e. the incoming data was treated as the single-hatched “B”, whereas it should be treated as the cross-hatched bar “A”. Both areas are the same.

Actually, this is not 100% accurate since we don’t know whether the meter stopped near the beginning or near the end of its revolution. But there is simply not enough information in the gas measurement data to disambiguate these cases.

However… as you can see, the electricity consumption also has some surprisingly regular small changes. Those measurements suffer from the same problem, but they have a much smaller impact due the the higher measurement rate. My hunch is that these small wattage changes are actually the electricity consumption of the heater, as it turns on and off!

But hey, who cares for now – these are just instant-by-instant consumption rates. The integral of these rates is the total consumption, and those are by definition “accurate” since I’m measuring the very same metering revolutions which the electricity and gas company use to charge me later on.

Update – Here are the results with the above adjustments:

Picture 1.png


Energy plots

In Software on Feb 6, 2009 at 00:03

And here’s the electricity and gas consumption for (part of) last Wednesday:

Picture 1.png

There are some quirks due to the way things are measured. The meter cannot accurately detect when gas consumption drops to zero, since that simply produces no readings. This can be resolved by adding fake readings when a very low consumption is reported, estimating the times when the gas flow stopped and then started again.

The electricity peaks at 19:45 are from the induction stove, the one around 20:15 is probably the close-in boiler.

This Tcl code generates the necessary HTML/JavaScript snippet to produce the plot:

Picture 3.png

All measurement data selection is handled by the “dataset loop” command, which takes a measurement parameter name, the selection range, and the names of the time and value variables to set during the loop.


Temperature graphs

In Software on Feb 5, 2009 at 13:30

Some weather & energy data has been coming in over here for over a month now, logged to text files, and patiently awaiting further processing. Here are the results of a few days of hacking around on the software side of things:

Picture 1.png

This is a test page served by a custom web server, with all data now stored in a database. I’ve used several bits of software which are dear and near to me:

  • the database used is my own trusty Metakit
  • the custom software is written in Tcl
  • the web server is based on my Rig modules
  • the plots are generated in the browser by Flot
  • it’s all in two files using using Tclkit / Starkits

The code is portable and works on Mac OS X, Windows, and Linux. The system requirements are minimal, so it runs just fine on my NSLU2 and Bubba NAS boxes.

Things are starting to come together nicely…


More readings

In Software on Dec 19, 2008 at 15:55

Here is the latest display showing on my screen:

7474AFF5-C9D3-4DA7-B5E3-492EE65C15DB.jpg

Updated in real time, with new readings coming in every few seconds for electricity and gas and every few minutes from the weather station (still sitting here in the office for now, so the values are bogus).