Computing stuff tied to the physical world

Archive for the ‘Software’ Category

,

FritzBox call log

In Software on Mar 12, 2010 at 00:01

I’ve been using the FRITZ!Box 7170 as ADSL modem and telephone interface for some time now:

Screen shot 2010-03-08 at 12.50.41.png

It has a nifty feature: when you enable the built-in call monitor by typing in “#96*5*”, you can connect to TCP/IP port 1012 to get real-time messages about all phone calls.

This is easy to hook up to JeeMon. I added a new “FritzBox.tcl” file with the following code:

Screen shot 2010-03-08 at 13.15.39.png

(Some experimental code was omitted from the above listing)

To enable this, the following line was added to application.tcl:

Screen shot 2010-03-08 at 13.14.42.png

Here’s an example of an incoming call, as shown in the JeeMon log:

Screen shot 2010-03-09 at 13.51.12.png

And here’s an outgoing call:

Screen shot 2010-03-08 at 12.55.09.png

Neat! I’ll hook this up as events once the JeeMon event system is ready.

These experiments are very useful to see how the current design is working out. The above code depends on the “MessageStream” class in the “Serial” rig. It doesn’t really add much value in this case, but some class hierarchies seem to be emerging. This socket connection is an example of a similar-but-not-quite-the same case as serial connections, requiring its own class. A socket connection is very similar to a serial connection once initialized – but this doesn’t show as clearly in the code as I would like and there’s already a bit of code duplication.

Will probably need some re-shuffling later, to better match concepts / design and implementation.


Outbound X10

In Software on Mar 11, 2010 at 00:01

I’ve added support for controlling X10 devices via the CTX15 module described in the past two days:

Screen shot 2010-03-09 at 20.31.20.png

First thing was to extend the ookRemote GUI, by adding these definitions to ookRemote.tcl:

Screen shot 2010-03-09 at 20.34.10.png

With this change, the configuration section in “config.txt” supports CTX15 buttons:

Screen shot 2010-03-09 at 20.34.56.png

These new buttons will send X10 commands to the CTX15 module. But this needs a bit of wrapping to turn them into frames, including the proper checksums. So I added two methods to “sketches/ctx15try/host.tcl”:

Screen shot 2010-03-09 at 20.37.16.png

The “send” method overrides the existing one and sets up a frame before calling the original method with it.

That’s it. I can now make the Marmitek AM12 switch “clunk” with my mouse – audible feedback! :)


JeeMon interface for X10

In Software on Mar 10, 2010 at 00:01

Now that the CTX15 has been connected via a JeeNode, it’s fairly straightforward to tie it into JeeMon.

First I created a new folder called “sketches/ctx15try/” with a file “host.tcl” in it, using this code as quick test:

Screen shot 2010-03-08 at 14.37.36.png

The reason there is nothing more, is that I changed the ctx15try.pde sketch a little to return lines starting with “CTX15″ – since these are the easiest to tie into JeeMon. The change affects these lines in yesterday’s demo:

Screen shot 2010-03-08 at 14.33.08.png

I’ve also wrapped the received frame in {}’s, because that simplifies processing in JeeMon (Tcl interprets $’s and []’s unless escaped).

To activate everything, I added an extra line to the config.txt configuration file, so that the output from the JeeNode USB (serial# A900adwo) automatically gets picked up as sketch:

Screen shot 2010-03-08 at 15.01.22.png

Here is an example of the above code running:

Screen shot 2010-03-08 at 14.36.50.png

Good. CTX15 events are coming in. They are easily recognizable as (unit#, command) pairs.

But as you can see, these pairs don’t always arrive in the same frame. So we have to add a bit of logic to get things right. Here’s an updated version of host.tcl:

Screen shot 2010-03-08 at 14.51.50.png

The solution used above is as follows: if it looks like a unit code (A..P + 1..16), then save that as last one seen. If it’s anything else, use the last saved location. I’m not sure this covers all possible scenarios, but for this very simple test it seems to work. Here’s a screen shot of the real-time status window:

Screen shot 2010-03-08 at 14.50.48.png

As you can see, the CTX15 event was turned into an “AOFF” command for the “ctx_A01″ device, which was created on-the-fly by this code.

Hmm, looking further, I think the “A” prefix needs to be stripped from “AOFF”. Oh well, it’s all scripted, I’ll adjust that when I get to using this stuff for real.

So that’s it. A few dozen lines of C code in the JeeNode to act as pass-through and do the polling, and a few dozen lines of Tcl code to tell JeeMon how to interpret the incoming data. Now we get notifications whenever any X10 activity is detected on the mains power line.

Onwards!

,

Xanura CTX15

In Hardware, Software on Mar 9, 2010 at 00:01

Another popular home automation module is the Xanura CTX15:

DSC_1228.jpg

That’s a live 220V power-line connection in the top right corner!

I’ve connected the CTX15 module through a UART Plug, with a 4.7 kΩ resistor in series with the RX signal (yellow from CTX15). This is because the signal swings up to 5V, whereas the UART plug only accepts 3.3V voltage levels. The module is powered from the PWR pin, which on a JeeNode USB carries 5V.

The CTX15 is a bi-directional interface, it can send as well as receive A10-type power-line commands (a superset of X10). The trouble is that it needs to be polled to read out what has been received and buffered so far.

Here’s a sketch which takes care of that:

Screen shot 2010-03-08 at 12.08.53.png

And here’s some sample output:

Screen shot 2010-03-08 at 12.08.40.png

As a test, I powered up yesterday’s XM10E test setup as well, to send out on and off commands to unit A.1 every 3 seconds. As you can see with the CTX being read out every 5 seconds, multiple received packets will sometimes be combined and returned as one reply.


JeeMon goes embedded

In Hardware, Software on Mar 5, 2010 at 00:01

Testing out a new setup … I started by launching the JeeMon runtime executable as follows:

Screen shot 2010-03-02 at 15.41.09.png

Then I went back to my Mac and opened a browser window:

Screen shot 2010-03-02 at 15.56.12.png

It works! You’re looking at a little web server running on a tiny Technologic TS-7500 board:

DSC_1213.jpg

ARM based, 64 Mb RAM, µSD card socket on board, running Debian (lenny). It’s not very high powered, but in return it only uses 400 mA @ 5V, i.e. it can run off USB power.

I haven’t yet figured out how to get the FTDI USB serial driver on there, but once that’s solved this little unit will be able to act as host for USB-connected JeeNodes or JeeLinks.

The RAM and storage is more than sufficient to run even a very elaborate JeeMon setup, I expect.

This might be an interesting low-end always-on home sensor & automation system!

,

Pushing data as web client

In Software on Mar 3, 2010 at 00:01

Another way to get data to another place is via push HTTP requests, i.e. acting as a client for another web server somewhere on internet.

The Pachube site is one place to send measurement results. I’ve set up a test feed to use with JeeMon, with two data streams. One of the nice features is that you can easily produce and embed graphs from that system:

Screen shot 2010-02-26 at 02.45.13.png

Here’s how I implemented it in JeeMon. First, I created a configuration section:

Screen shot 2010-02-25 at 22.37.12.png

(I’ve omitted most of my private API key, but anyone can sign up and get their own …)

Next, the code, in the form of a “pachube.tcl” file / module / rig:

Screen shot 2010-03-02 at 03.11.12.png

Note how the configuration file in fact contains “settings” which are simply evaluated as a script: the two lines starting with “Fetch …” are handled by the Fetch proc in pachube.tcl, using the JeeMon notification mechanism to get called back whenever either of the “meter1″ or “meter2″ readings changes. The Update proc then updates the “latest” variable accordingly. Lastly, SendToWebSite periodically does an HTTP PUT request in the background, supplying all the info needed to successfully submit to the Pachube website.

Btw, this simple example is flawed, in that it does not calculate averages – it just sends the last reading. But things like averaging require a sense of history, and persistence. Haven’t added that to JeeMon yet…

The missing link, as usual, is a line in the application file to start the ball rolling:

Screen shot 2010-03-02 at 03.03.46.png

There’s a lot of flexibility at the protocol and network levels. Such as creating an XML request via templates, if the remote server needs XML. This isn’t limited to HTTP requests, or to using port 80 – send emails, FTP files, etc.

There are quite a few details in the above code which I won’t go into. Again, I’m doing this mostly to show how little code it takes to initiate periodic HTTP client requests to an outside website.

Apart from sites such as Pachube, this could also be used from a tiny embedded Linux running JeeMon, to submit incoming data to a different setup elsewhere on the LAN, for example. IOW, JeeMon can be a front-end for other systems. It’s not about lock-in. It’s a switchboard. It can glue systems together. It bridges gaps.

,

JeeMon as web server

In Software on Mar 2, 2010 at 00:01

JeeMon comes with a nifty built-in HTTP/1.1 web server (using coroutines to handle requests in parallel).

So let’s create another real-time status display, but as web application this time:

Screen shot 2010-02-25 at 17.56.25.png

As before, I’m leaving out the lipstick – just showing the core functionality.

Here’s what I did to generate this page. First of all, I’m re-using some functionality which is already in the GUI version, to dynamically construct a matrix with the proper columns and rows, so the “statusWindow” code has to be running for the above to work. For real use, that code should probably be reorganized into its own rig.

The main task was to create a HTML template in a file called “statusPage.tmpl”:

Screen shot 2010-02-26 at 00.41.23.png

This uses a simple templating mechanism based on Tcl’s “subst” command. The meta tag sets up the page to self-refresh every 5 seconds. The last line tells the Wibble web server to deliver the page as HTML.

To launch the server, this line was added to the main application file (port 8080, current dir is doc root):

Screen shot 2010-02-25 at 18.00.15.png

That’s it – there’s very little to activating a web server in JeeMon, as you can see. Which is important, because on tiny embedded Linux systems, an HTTP server will probably be the only option to present information.

Creating a full-blown site with CSS, JavaScript, and Ajax is a matter of adding more files – the usual stuff…

Did I mention that it’s all 100% open source, so you can browse / extend / change all of this? – I did? Oh, ok ;)

,

Node discovery

In Software on Mar 1, 2010 at 00:01

This post isn’t about GUIs, but about the dynamic behavior behind them.

And about wireless. Let’s take the voltmeter setup I described recently, and make it send its readings over the air – it’s a JeeNode after all, so it has wireless connectivity built in.

I extended the original sketch to send readings out roughly once per second:

Screen shot 2010-02-25 at 15.57.03.png

Extending the host side to also work from received packets is trivial:

Screen shot 2010-02-25 at 16.49.26.png

The problem now, is that this node will start sending out packets, but JeeMon has no idea what to do with them. One day, automatic node type registration would be great to add, but for now I’ll just use the configuration file to associate nodes with sketches. Here’s an extract of my current “config.txt” file:

Screen shot 2010-02-25 at 16.00.30.png

For this example, I added the “3/ { type analogPlug }” line to the “868:5/” entry (i.e. 868 MHz, netgroup 5). And sure enough, JeeMon will now launch the GUI for this node, displaying results coming in over the air in real-time:

Screen shot 2010-02-25 at 16.03.00.png

Here’s the relevant part of the log output:

Screen shot 2010-02-25 at 16.04.34.png

The fun part is that a timeout mechanism has also been implemented. If no packets come in from the analog node for 5 seconds, the connection is torn down and the window is closed again. The timeout value is sketch-specific and is included in the above configuration info.

In other words: node is ON => window appears, node is OFF => window goes away. Look ma, no hands!

Note: it’s not good UI practice to open and close windows like this. In a more refined setup, these windows would be panels in a larger window, or they could just be grayed-out to signal their disconnected state, for example.

The analog node is surprisingly convenient this way. I can use it anywhere, turn it on, look on the screen, and turn it off when I’m done. It wouldn’t be hard to add another wireless node to show the results on an LCD screen, btw. If this is extended with a way to assign node IDs ad-hoc, and a way for a sketch to identify itself over the air, then a whole family of wireless “instruments” could be created – using nothing but a JeeNode and some plugs.

This illustrates a guiding principle in JeeMon: don’t ask, just do it! – JeeMon tries to be as dynamic as possible, adapting to changes in status and configuration by adjusting everything it does accordingly. In the case of nodes, new ones are discovered and stale ones are weeded out, in real-time. With objects getting created and destroyed automatically. What these objects do, as side-effect, is up to each one of them,

Right now, this dynamism in JeeMon is still fairly messy. There’s too much state in too many places which needs to be managed carefully. But with a bit of luck, this can eventually be simplified further to create a very consistent and predictable behavior.


A real-time status GUI

In Software on Feb 28, 2010 at 00:01

One more post about setting up a GUI with JeeMon…

I wanted to have a basic real-time display of all the readings coming in, sorted by source. Like this:

Screen shot 2010-02-25 at 15.48.08.png

(the size of this screen shot was reduced to fit this post)

Everything in this window is dynamic: rows get added when new sources start reporting their data, and columns get added when new parameter types are reported. I’m not storing any information persistently yet, which is why this status window looks as follows right after starting up JeeMon:

Screen shot 2010-02-23 at 21.31.15.png

Not very exciting, but all you have to do is wait …

Let’s see what it took to create this GUI. First, I added a generic “Notifier” rig to JeeMon, and a “Reading” method for all sketches to use. The rooms sketch, for example, now reports its incoming results as follows:

Screen shot 2010-02-23 at 22.06.04.png

Similar changes were made to the ookRelay and RF12demo “host.tcl” sources.

The rest of the GUI code is in “statusWindow.tcl”:

Screen shot 2010-02-24 at 01.16.42.png

The tabular display is based on a very nice Tcl/Tk open source package called Tablelist by Csaba Nemethi, which has been added to the JeeMon core library.

The coupling with the rest of the system is accomplished by the “Notifier attach readings * …” call at the end of the setup code. This registers a callback which will be invoked when anything in the “readings” notification group changes. Notifiers are going to be used a lot in JeeMon because they provide such a manageable way to glue independent parts of an application together.

And lastly, adding the following line to the main application starts the ball rolling:

Screen shot 2010-02-23 at 21.42.16.png

There’s quite a bit going on here which I won’t go into. I just wanted to illustrate how little code it takes to create a basic, fully dynamic, cross-platform, self-updating status display window. There – how’s that for adjectives!

,

Voltmeter with GUI

In Software on Feb 27, 2010 at 00:01

Let’s make a 5-digit voltmeter with a JeeNode, an Analog plug, and JeeMon. But instead of a readout on an LCD screen, I’m going to show the results on the desktop. To start with that last part, here’s my little display on a Mac:

Screen shot 2010-02-23 at 20.23.25.png

You’re looking at my feeble attempt to create a good-looking / souped-up GUI, btw :)

Anyway. Here’s the sketch I’m running on the JeeNode, with an Analog plug inserted into port 3:

Screen shot 2010-02-23 at 11.01.16.png

Trivial stuff. It sends out lines with “VOLT <value>” readings twice per second.

The important part is that this sketch starts up with “[analogPlug]” as greeting. To hook up to this sketch, all we need to do is create a directory called “analogPlug”, with a file called “host.tcl” in it.

This new “analogPlug” directory is a sub-directory of “JeeMon-sketches”. This directory, in turn, has been registered as being used for sketches – with this command in my main application code:

Screen shot 2010-02-23 at 12.21.51.png

Back to the “JeeMon-sketches/analogPlug/host.tcl” file. Here’s its contents:

Screen shot 2010-02-23 at 11.40.34.png

That’s the whole kaboodle.

Now, whenever I plug in a JeeNode running the above “analogPlug.pde” sketch, a window pops up showing readings in real-time. When I unplug the JeeNode, the window is closed again. It’s all automatic, as long as JeeMon is kept running.

This mechanism is not limited to JeeNodes or Analog plugs, of course.


OOK remote control

In Software on Feb 26, 2010 at 00:01

The JeeMon story continues. Here’s a fun little panel for controlling some switches around the house:

Screen shot 2010-02-22 at 21.42.14.png Screen shot 2010-02-22 at 23.31.35.png

Screen shot 2010-02-23 at 20.19.28.png Screen shot 2010-02-22 at 21.51.23.png

These screen shots show Windows (7 and 2K), Mac OS X, and Linux Ubuntu, respectively – long live diversity!

This GUI was created with these configuration settings:

Screen shot 2010-02-26 at 02.09.33.png

Those settings were loaded by adding this line to the main application code:

Screen shot 2010-02-26 at 02.09.16.png

Which in turn loaded the following “ookRemote.tcl” source file I wrote, as module / rig:

Screen shot 2010-02-26 at 02.08.48.png

That’s the entire app. Adding lipstick (i.e. a fancier visual design) is left as exercise for the reader.

Note how the design is split in a configuration section and a source code file – I’m probably going to use this approach often in JeeMon. The idea is that people who don’t care about the code behind all this stuff can just adjust the configuration file (using a visual interface, hopefully, one day). While those who like to tinker can view the full source code, and explore and extend it at will, and more importantly: ad infinitum.

It’s called open source for a reason!

PS. There’s another major reason for the split between configuration settings and source code files: turnkey deployment. You can put all the source code files in a ZIP archive called “JeeMon-rigs.zip”, so that all you need for a new installation with JeeMon, is: 1) your archive, 2) the proper JeeMon runtime, and 3) a config file to match the target setup. The config file is optional, btw – more on that another time…


Output and Expander plug fixes

In Software on Feb 25, 2010 at 00:01

While testing an Output Plug, I found a little mess-up: the documentation pinout is wrong, all even and odd I/O pins on the 2×6 output connector were swapped. Here is the correct pinout:

op1-pinout.png

I’ve updated the documentation.

The “expander” sketch is also very confusing, it was still for the PCA8574A, whereas the current Expander Plug uses an MCP23008.

Here’s an improved sketch, with a little running-light demo:

Screen shot 2010-02-22 at 12.38.18.png

This code has been checked in to replace the “expander.pde” sketch in the Ports examples.

,

Secure transmissions

In Software on Feb 23, 2010 at 00:01

For some time, I’ve been thinking about adding optional encryption to the RF12 wireless driver. I’ll leave it to the cryptography experts to give hard guarantees about the resulting security of such a system…

The basic goal is to provide a mechanism which lets me get messages across with a high level of confidence that an outsider cannot successfully do the same.

The main weakness which most home automation systems such as FS20 and KAKU completely fail to address is message replay. The “house code” used by FS20 has 16 bits and the address has 8 bits, with the naive conclusion being that it takes millions of attempts to send out messages to which my devices respond. Unfortunately, that’s a huge fallacy: all you have to do is sit outside the house for a while, eaves-dropping on the radio packets, and once you’ve got them, you’ve figured out the house code(s) and adresses in use…

I don’t care too much about people picking up signals which turn the lights on or close the curtains. You don’t need technology to see those events from outside the house anyway. I do care about controlling more important things, such as a server or a door opener.

Here are my design choices for optional encryption in the RF12 driver:

  • The cipher used is David Wheeler’s XXTEA, which takes under 2 Kb of code.
  • The keys are 128 bits, they have to be stored in EEPROM on all nodes involved.
  • All nodes in the same net group will use either no encryption or a shared encryption key.
  • A sequence number of 6, 14, 22, or 30 bits is added to each packet.

To start with the latter: XXTEA requires padding of packets to a multiple of 4 bytes. What I’ve done is add the sequence number at the end, using as many bytes as needed to pad to the proper length, with 2 bits to indicate the sequence number size. Encrypted packets must be 4..62 bytes long. It’s up to the sender to decide what size packets to send out, and implicitly how many bits of the sequence number to include. Each new transmission bumps the sequence number.

To enable encryption, call the new rf12_encrypt() function with a pointer to the 16-byte key (in EEPROM):

Screen shot 2010-02-21 at 18.37.36.png

Encryption will then be transparently applied to both sending and receiving sides. This mechanism also works in combination with the easy transmission functions. To disable encryption, pass a null pointer instead.

The received sequence number is available as a new “rf12_seq” global variable. It is up to the receiver (or in the case of acks: the originator) to ascertain that the sequence number is in the proper range. Bogus transmissions will decrypt to an inappropriate sequence number. To make absolutely certain that the packet is from a trusted source, include some known / fixed bytes – these will only be correct if the proper encryption key was used.

This new functionality has been implemented in such a way that the extra code is only included in your sketch if you actually have a call to rf12_encrypt(). Without it, the RF12 driver still adds less than 3 Kb overhead.

I’ve added two sample sketches called “crypSend” and “crypRecv” to the RF12 library. The test code sends packets with 4..14 bytes of data, containing “ABC 0123456789″ (truncated to the proper length). The receiving end alternates between receiving in encrypted mode for 10 packets, then plaintext for another 10, etc:

Screen shot 2010-02-21 at 22.42.27.png

As expected, the encrypted packets look like gibberish and are always padded to multiples of 4 bytes. Note also that the received sequence number is only 6 bits on every 4th packet, when the packet size allows for only one byte padding. The strongest protection against replay attacks will be obtained by sending packets which are precisely a multiple of 4 bytes (with a 30-bit sequence number included in the 4 bytes added for padding).

So this should provide a fair amount of protection for scenarios that need it. Onwards!

,

Declarative configuration

In Software on Feb 22, 2010 at 00:01

Things are starting to come together quite nicely with JeeMon – all in less than 1000 lines of Tcl code so far. That’s not to say that it’s perfect yet – I’m still massively rearranging some of these code structures – but there is already some functionality to play with, which helps expose the strengths and weaknesses so far.

The basic approach is declarative in nature: specifying what should happen, but not necessarily in the same place as when or how all the work should be done.

Here’s my first tentative configuration file:

Screen shot 2010-02-21 at 14.31.44.png

For each serial interface, an action is specified when that interface comes on-line. For example I can now plug in my A8007Up6 device (a USB-BUB with a JeeNode) and JeeMon will open a terminal window showing the output stream. Unplugging closes the window again (and automatically cleans everything up).

Right now, I’m editing this config file by hand, but at some point that could be augmented with a set of GUI or web-based configuration panels (even remotely).

There are a lot of challenges ahead. One of them is decoupling things so that code and data end up organized in the most convenient, logical, and flexible way. I’m trying very hard to avoid any calls to the “Config” module from generic JeeMon code. This allows you to set up the configuration file structure and hierarchy in any way you like.

My application file currently looks as follows:

Screen shot 2010-02-21 at 15.42.58.png

As you can see, I’m setting up a periodic port scanner and supplying a “SerialEvent” callback to decide what to do when serial ports get added or removed. In those decisions, most of the logic is driven from the “config.txt” configuration file. I’m about to implement a similar callback for radio events, i.e. RF12 nodes coming online and dropping off again.

In a way, the above two files are the entire application.

The rest is utility code to make the above convenient and concise. That rest might well be 99% of the code, but it’s all structured as loosely-coupled “rigs” (modules), which can be used / overridden / ignored at will.

The way I see it, there can be three sources for such utility code: your own code, plug-ins from others, and a couple of rigs included in the JeeMon core itself. The above two files require just the core, using the built-in rigs (Config, Gui, JeeSketch, Log, Serial, and SysDep).

,

Serial USB devices

In Software on Feb 21, 2010 at 00:01

For JeeMon, I’d like to be able to auto-detect USB device insertions and to identify FTDI serial numbers (with names like “FTE54GKC” and “A9007CNE”).

Cross-platform… i.e. on Windows, Mac, and Linux!

It looks like it can be done.

On Windows, we can browse the good ol’ registry, as follows:

Screen shot 2010-02-20 at 15.24.16.png

Sample output:

Screen shot 2010-02-20 at 15.24.41.png

This was tested on Win2000 and Windows 7, in the hope that everything in between works the same.

On Mac OS X, it’s easiest (for a change):

Screen shot 2010-02-20 at 15.26.02.png

Sample output:

Screen shot 2010-02-20 at 15.26.37.png

On Linux, it’s a bit messy because there are so many different distributions:

Screen shot 2010-02-20 at 15.27.43.png

Sample output:

Screen shot 2010-02-20 at 15.28.01.png

Tested on Debian 5 (Lenny), Ubuntu 9.10 (Karmic), and Gentoo.

The above code can be found in the subversion repository, i.e. here.

I’m testing this all at once on a single machine btw, courtesy of VMware Fusion. And using JeeMon’s built-in self-update mechanism to quickly get new versions across while debugging and tweaking things.

By calling “SysDep listSerialPorts” periodically, we can automatically detect a change and see which plug was inserted (FTDI only for now). Without depending on any external libs or executables.

Onwards!

,

RF transport independence

In Software on Feb 20, 2010 at 00:01

With the basic serial interface access and dispatch in place in JeeMon, it’s time to move on to JeeNode / JeeLink packet processing.

What I want is to be able to forget all about how readings got to JeeMon. It shouldn’t make a difference whether I’m using a directly connected Room Node and grabbing the readings off its serial USB connection, or whether they came in through the air via the RF12 wireless driver – or by any other means. Take a snapshot with your cell phone, send the picture to EverNote, have it OCR’d, and then grab the readings off the web … whatever!

The way I’ve tied this into JeeMon, is to let the interface to RF12DEMO act as de-multiplexer. This is purely a decision at the RF12DEMO listener level. Each incoming packet is examined to determine which node it came from. Then we need a way to map from nodes to listener class – i.e. find out what sketch is running on the remote node. This is hard-coded for now:

Screen shot 2010-02-18 at 22.57.41.png

What this does is actually a bit more elaborate: a RF12DEMO listener will set up and manage a bunch of extra “RF12_PacketStream” listeners, one for each node. When packets come in, they will simply be dispatched to the corresponding stream. Each packet stream can process its packets in a different way.

The fun part is that these packet streams can use the same listener classes as with direct serial interfaces. All they need is an extra “decode_RF12″ method:

Screen shot 2010-02-18 at 23.02.04.png

The task of decode_RF12 is to re-cast the incoming packet as messages of the same structure as what would come in over a serial connection.

Here’s the “rooms” listener as example:

Screen shot 2010-02-18 at 23.04.34.png

This one class encapsulates all the protocol details of room nodes, both serial and via RF12. When a 4-byte data packet comes in via RF12 (as $a..$d), the bits are decoded and an “onMessage” call is generated with a “ROOM” message id and the 5 decoded readings.

Here is a log of this code in action, one message per line:

Screen shot 2010-02-18 at 22.37.15.png

The way to read these messages is as key-value pairs, i.e. id = OK, type = RF12, name = usb-A900ad5m, etc.

The first two lines show an incoming OK message from node 21 (53=21+32), which is then turned into a ROOM message, tagged as coming from the “rf12-868.5.23″ packet listener.

The next 3 lines are more involved: first an EM10 message came in over USB, then an OK message came in which got dispatched again, as the same EM10 message. That’s because I’m running JeeMon with a direct connection to the ookRelay board, even though it transmits all its information over RF12. So everything from the ookRelay comes in twice (great for debugging).

The point is that the two EM10 messages have the same info. It no longer matters how the message got here (but it is traceable, using the remaining details). And all the code to accomplish this is in a single source file, right next to the sketch running on the ookRelay board.

This design makes it possible to develop an application using only the serial USB connection, and then later add logic to send the information via RF12 (or not). Infrared, XBee, Twitter, anything: transport independence!

Note that nothing is done with these decoded messages at this stage. This messaging framework is independent of higher-level application decisions, such as where to store / send / show msgs, or even when to process them.

,

Serial port encapsulation

In Software on Feb 19, 2010 at 00:01

This post continues to look a bit into the new JeeMon design.

Let’s focus on serial interfaces first, mostly USB. There’s a “Serial” module which does all the basics. On the Mac, if I want to open device /dev/tty.usbserial-A900ad5l, then the following call with do everything:

Serial connect usb-A900ad5l 57600

The name of the device would be different on Windows and Linux (COM5, or USB1), but that’s all.

By default, this creates a new Serial object, which logs all incoming text to stdout. To send a command out, we need to keep a handle to this object:

set conn [Serial connect usb-A900ad5l 57600]
$conn send "some text"

Nice, but not very exciting…

Let’s take it one step further. The “JeeSketch” module does the automation described in the previous post, i.e. detect the running sketch, associate it with a class, instantiate an object for it, and call the methods of that object whenever text lines come in over that serial port. Here’s a complete JeeMon custom “application.tcl” program:

Screen shot 2010-02-18 at 14.44.59.png

First, all the sketch drivers are made available with one or more “register” calls. This lets the appropriate classes take over for each new serial connection – depending on what sketch is running. That’s all it takes. Servicing such serial ports now becomes an event-driven background activity.

The listeners are defined in separate files, one for each type of sketch:

Screen shot 2010-02-18 at 15.05.58.png

The ookRelay/host.tcl file looks like this, for example:

Screen shot 2010-02-18 at 14.51.58.png

This structure makes it easy to manage stuff that belongs together. Projects can be exchanged (or archived, or revision-controlled), with all the pieces needed to use them in one place. And as far as I’m concerned, it won’t be limited to JeeNodes etc, or Tcl, or a specific platform. This has to remain general enough to hook up to any hardware and use with any language (via networking, files, and direct launching of executables/scripts). My goal for JeeMon is not to limit anyone’s options, but to create a simple switchboard between whatever is needed.

(I’m still mucking around with the organization and naming of code and files, as you can see)

Nice, but still not very practical…

The problem with the above is that it doesn’t deal with devices getting plugged in or unplugged. Well, unplugging is the easy bit – the above code will automatically clean up after itself on connection loss, so that part is covered.

Wouldn’t it be nice if we could just plug in new devices and get them to automatically start doing something?

I implemented such a mechanism in a recent revision of the code, but I’m hesitant to add it again – because it was Mac OS X specific, where USB devices connected via the FTDI driver include a unique code. On Windows and Linux, you just get COM<n> and USB<n> devices, where “n” seems to be related to the order and number of device insertions.

I haven’t looked into “libusb” yet. Should I? Will it help for Windows too? Do I need to start learning about USB enumeration? What OSS-compatible tools and libs are there?

Update – on Linux, it looks like /sys/bus/usb/devices/* has all the info needed to identify USB devices. So that only leaves Windows – good: at most one lib or dll to deal with.

,

In the kitchen

In Software on Feb 18, 2010 at 00:01

Ok, so maybe it’s time to start describing some of the new stuff cooking in the kitchen lab.

I’ve been exploring software designs to use as basis for JeeMon, that new switchboard-type application for physical computing devices. The idea is to treat the system as one big message-passing machine – a bit like Message-Oriented-Middleware (MOM), which has been around for ages, but without getting sucked into any heavy-weight conventions.

Messages can be passed around, queued, stored, duplicated, filtered, transmitted, returned, ignored, sorted, etc. After all, living organisms do nothing but send chemical messages around, so why not do the same for an infra-structure focused on physical computing (sensors, displays, actuators) and home automation?

If you’ve been following this weblog for a while, you’ll have noticed that almost all the output I generate in sketches is of the form “identifier arg1 arg2 …”. So for example, packets received by RF12demo look like:

OK 61 9 2 8 79 243 87 13 0 15 0

A barometric reading from the BMP085 sensor on the pressure plug may get reported as:

BMP 233 10019

And so on. One or more upper case letters, optionally followed by digits. Then the payload (which may also include floating point and string values, not just ints).

Another convention I’ve been sticking with is to report the name of the sketch on the serial port during startup:

[ookRelay]

Or an identifier plus the current configuration settings:

[RF12DEMO] W i23 g5 @ 868 MHz

These two conventions can be used for an object-oriented design. With a bit of preparation, the name of such sketches can be automatically associated with a class, and each line treated as a method call.

Here’s the skeleton of the first level of code I use for decoding RF12demo.pde output:

Screen shot 2010-02-16 at 00.15.27.png

Another class definition, for the ookRelay.pde sketch:

Screen shot 2010-02-16 at 00.16.11.png

Or to put it differently: with JeeMon running, you can hook up a device (JeeLink, JeeNode, Arduino, etc) containing some sketch and the matching class will automatically be associated with that serial port, once the sketch has been identified. A new object is then created, and its methods will be called whenever the device sends new output (messages?) over the serial line.

Simple!

What I would like to do, is manage the sketch (C/C++) and the class (Tcl) together, perhaps as two files in a common development directory for that project. That way the interface between the two pieces of hardware essentially vanishes into the background. The point being that each class can then do all sorts of things, such as storing results in a database, sending it to another system over the network, updating web server pages, popping up a GUI window to show incoming data in real-time, etc.

This mechanism is very simple, even under the hood. This matters, because it has to work even when JeeMon is running on low-end embedded Linux hardware. But at the same time, such a MOM + OO design will allow considerable flexibility for abstract / high-end use later.

PS. If you’re familiar with Tcl, you might be surprised to see all this “oo” stuff in here. That’s because JeeMon uses Tcl 8.6 with built-in object system. Multiple inheritance, mixins, filters, dynamic class change support, delegation, prototypes / per-object methods, it’s all in there. I’m also using ensembles and there’s an interesting coroutine-based web server waiting in the wings (called wibble).

For the record: nothing ever gets added just to be buzz-word compliant. If a feature simplifies application-level concepts and leads to a cleaner implementation in JeeMon, it’ll be used, else I’ll pass. Life’s too short to jump on bandwagons.

,

Wireless works, sort of…

In Software on Feb 10, 2010 at 00:01

Ah, now we’re getting somewhere!

This is my current test setup:

Screen shot 2010-02-07 at 21.23.55.png

The JeeNode on the printer side implements a packet pass-through system, receiving command packets from the JeeLink and sending back response packets. Here is the sketch which does all the work:

Screen shot 2010-02-08 at 01.02.16.png

On the Mac/PC side, some Tcl code was added to go through the RF12demo text-mode protocol to send and receive arbitrary data, using the “a” command. Still work in progress, but the basic transport encapsulation works.

The tricky part is timing … it always is with this sort of real-time control stuff. Unfortunately, the current G3 software on the CupCake isn’t quite as responsive as defined in the specs. Some responses take way over 80 milliseconds to come back from the motherboard. This is the case when scanning the SD card, as well as when stopping the extruder motor.

So what this sketch does is wait up to 500 ms for a reply to come in. Even if there isn’t one, an acknowledgement packet will be sent back. The new code on the Mac in turn waits up to 1 second for that ack to come back.

If no ack came back, then there was an error in the wireless connection (this can happen either during the request or during the ack, there is no way to tell!). Probably best thing to do would be to resend the command.

If an empty ack came back, then the response packet did not arrive within 500 ms. In this case, we could send an empty command and wait for its ack. This hasn’t been implemented yet, but it will allow dealing with even the slowest responses, simply by polling a few more times with an “empty command”.

But hey – it works, and the output is the same as before:

Screen shot 2010-02-07 at 21.26.39.png

This is probably the first wirelessly controllable CupCake in the world :)

What I should mention though is that this doesn’t yet work reliably due to those very loose timing behaviors and the fact that packet errors are not yet dealt with. Test runs fail occasionally – mostly in the SD card access code, i.e. while grabbing all the filenames with NEXT_FILENAME.

,

Connecting to a CupCake

In Software on Feb 9, 2010 at 00:01

To follow up on yesterday’s post, I wrote some test code to request the extruder nozzle temperature from the JeeCake and send the results out over wireless. Here is the full “jeeStatus.pde” sketch:

Screen shot 2010-02-07 at 16.19.43.png

And sure enough, it works:

Screen shot 2010-02-07 at 16.08.55.png

You’re seeing the nozzle cool down, after heating it up via ReplicatorG. The connected JeeNode has been given node ID 19, and it’s transmitting in group 5 of the 868 MHz band, so I can simply track these incoming packets through the JeeLink which is already collecting all sorts of data anyway.

So much for the easy part – the real software will be more work!


New date / time / RTC library

In AVR, Software on Feb 5, 2010 at 00:01

Not so long ago, I had the opportunity to work a bit on something which has bugged me for a long time – the lack of date and time handling in connection with RTC chips. There are a few libraries out there, but I think I could do better – i.e. make it simpler, smaller, yet sufficiently powerful for real day-to-day use.

Seeing where this was going on the Arduino developer mailing list (and disagreeing with just about everything that happens over there), I decided to put my money time where my mouth is, and build my own library.

Here’s the header file of the new RTClib Arduino-compatible library:

Screen shot 2010-02-04 at 13.52.13.png

This lets you do date / time calculations, and it provides two different ways to implement a clock: via a hardware chip or using the built-in millis() timer.

RTClib has been checked into subversion, see the CODE page for details on how to get it.

It includes four example sketches:

  • datecalc illustrates how to do calculate with dates and times
  • ds1307 interfaces with a DS1307 RTC chip, connected via the Wire library
  • plugrtc interfaces with the RTC Plug, connected via the Ports library
  • softrtc demonstrates how to do the same with just software

One fun trick I added, inspired by a comment from Limor Fried, is to allow initializing a DateTime object from the __DATE__ and __TIME__ strings generated by the C compiler. That means you can run that “softrtc” sketch without hardware support, and it’ll automatically have its clock set to the compilation date of the sketch, i.e. fairly close to correct. Not good enough for general use, but great during quick debug cycles when you’re re-compiling your sketch all the time anyway.

Note that to use RTClib, you need to include the “Wire.h” library – even if you don’t use it!

The inability to properly deal with libraries, particularly in a resource-constrained embedded processor context, is one of the aspects of the current Arduino direction which irritates me – see an older post for more details.

,

Fascinating concurrency

In AVR, Software on Feb 4, 2010 at 00:01

There is a new language for the Arduino / JeeNode / ATmega328, called Occam-π.

I found out about it yesterday, at http://concurrency.cc/ – it’s high level, and it supports parallel programming. The current development environment release is for Mac OS X, with Windows and Linux coming soon.

Here is a complete program with 4 blinking LED’s, one on each DIO pin of the JeeNode ports:

Screen shot 2010-02-03 at 01.13.19.png

That’s it. Compiles to roughly 2 Kb. Each extra blink adds just 20 bytes, btw.

And yes, it really makes four LEDs blink at an independent rate:

DSC_1167.jpg

There is slightly more to it than that, but this is mind-blowing stuff. The “parallelism” is simulated, of course. Looks like the ATmega can do around 6000 context switches per second (i.e. parallel task switches).

There is a roughly 20 Kb interpreter part that needs to be uploaded once (which is why this requires at least an ATmega328). After that, the IDE will upload just the bytecode for your program, i.e. 2 Kb in the above case.

B R I L L I A N T .

Imagine hooking up the RF12 driver to this – there’s plenty of room for the extra 3 Kb or so. And for doing all sorts of things… in parallel! My earlier complaint post about how awful it is to do several things at once on an ATmega board might just have been wiped off the table.

Looks like I’ve got some very serious learning ahead of me to try and get to grips with all this.

,

Temperature plots

In Software on Jan 22, 2010 at 00:01

With over a dozen nodes now active, JeeMon is finally starting to collect a bit more useful data to work with:

Screen shot 2010-01-22 at 12.03.23.png

Looks like some temperatures have a granularity of 1°C – might be a few cheap DS1820 units I had…

The Mason-like templating code in “try-temps.html” to generate this plot is still far from obvious:

Screen shot 2010-01-20 at 01.52.42.png

Once I get to spend some real time on all this, it’ll look totally different. JeeMon as a whole is going to change completely in fact – until then, you can find some technical background information in an older post.

Let’s just say that it gets the job done right now…

,

Sleep mode fix

In AVR, Software on Jan 20, 2010 at 00:01

The Rooms sketch (latest code here) had problems with the sleep mode added about a month ago:

Screen shot 2010-01-17 at 14.05.55.png

The first value is the current consumption in µA.

For some reason or other, the node would stop working and enter a permanent-on mode, drawing over 7 mA and draining the battery in a matter of days. Not good.

It seems to be related to the way the power down mode was implemented. To get absolutely rock bottom power draw, I was using the RF12’s watchdog timer. The ATmega watchdog time draws slightly more current and isn’t quite as configurable.

I’ve now reverted to using the ATmega watchdog anyway. Here is the modified logic:

Screen shot 2010-01-18 at 12.09.39.png

The watchdog is set to interrupt every 16 milliseconds, constantly. When the node is powered down, this will wake it up again. What the new loseSomeTime() code does, is simply to power down a couple of times, until the target delay time has been reached. There is some inaccuracy in these timings since the watchdog timer is free-running, but this should not matter too much when waiting for a second or so.

The new code has been running fine for over a day on six nodes. Here’s a sample from my power tracker:

Screen shot 2010-01-19 at 16.56.22.png

That’s 110 µA right now, 265 µA in the last minute, and 230 µA in the last hour.

At around 250 µA average, the power consumption is a bit higher than before, but my main concern is first to get the nodes running reliably. Even @ 250 µA, the AA batteries should last several months.

The average power down current draw is 110 µA/sec, 40 µA of which is due to the PIR sensor. The total current draw while transmitting is around 29 mA – during reception it’s about half, i.e. 14 mA. Still, due to the very brief on-times, this current consumption averages to only about 400 µA during 1 second for a normal send + ack sequence. Under optimal RF conditions, the long-term average consumption of a node will be under 150 µA. I’m confident that further optimizations could reduce this to well under 100 µA.

But there is one known flaw, which can be observed to happen once in a while: the nodes always wait for an ack in full-power mode, i.e. with the receiver on. This will normally be within milliseconds, but if the connection is flakey or if the central node is unresponsive, then nodes can spend a great deal of time in full-power mode. This needs to be fixed one day.

I’ve started re-flashing all the room nodes and replacing their dead batteries, here at Jee Labs – let’s see how it goes this time around.


More C++ template trials

In AVR, Software on Jan 18, 2010 at 00:01

Here are some more results in trying to use templates for embedded software on JeeNodes. This time, it’s about using Jee Plugs with bit-banged I2C on one of the 4 ports and with built-in TWI hardware on pins A4/A5.

Let’s start with an extract of the Latest JeeLib.h header:

Screen shot 2010-01-17 at 11.50.27.png

I’ve omitted all the actual code, for brevity in this example. The full code is here. The previous version of JeeLib has been moved to the avr-jeelib-first branch in subversion.

The Port<N> class is essentially the same as in an earlier post. It generates efficient code for raw pin access, given fixed pin assignments known at compile time.

The above code adds a templatized PortI2C<N,W> class, where N is again the port number (1..4) and W is a constant which determines the bus speed. As with Port<N> class, this leads to a class definition which requires no state and which can therefore consist entirely of inlined static members.

A HardwareI2C<W> is also defined, with the same API as PortI2C<N,W>, but using the hardware TWI logic in the ATmega. The point is that in use, PortI2C<N,W> and HardwareI2C<W> objects are interchangeable.

You can see how all this templating stuff complicates the naming of even simple classes such as these.

The final class implemented above is DeviceI2C<P,A> – it does the same as DeviceI2C in the original Ports library, but again using templates to “bring in” the underlying port classes and the device I2C address.

Here is an example sketch built with all this new code:

Screen shot 2010-01-17 at 12.48.05.png

It supports two bit-banged I2C devices on ports 1 and 2, respectively, as well as a third I2C device driven by the built-in TWI hardware.

This compiles to 980 bytes (using Arduino IDE 0017).

The good news is that this generates pretty efficient code. It’s not 100% inlined – but quite a bit is, especially at the lower levels, so the result looks like a pretty good implementation of a high-level I2C driver which can be used for both bit-banged and hardware-supported I2C, all by changing one declaration at the top of the sketch.

But there are quite a few inconveniences with this approach…

First of all, note that the declarations at the top are fairly obscure. I did my best to simplify, but all this template mumbo-jumbo means you have to understand pretty well how to declare a port, and how to declare an I2C device object for that port. The “typeof” keyword in there is a GCC extension, without which these declarations would have looked even more complex.

The major trade-off is that the above example essentially generates separate code for each of these three I2C devices. There is virtually no code sharing. This can lead to code bloat, despite the fact that each version generates pretty good code. In practice this might not be so important – it is not likely after all that you’ll need all three types of I2C buses in the same sketch. Just keep in mind that you’re trading off efficient hard-wired coding against the need to generate such code for each type of I2C access you might need.

So would this be a good foundation to build on? I don’t know yet…

C++ templates do seem to get a lot more of the logic “into” the compiler. Instead of passing say an I2C device address as constant to an object, we generate a highly customized class which is special-cased to implement just one device at just one address. With the result that the compiler can perform quite impressive optimizations. In the above example, there are lots of cbi and sbi instructions in the generated code, just as if you were to dig in and hand-craft an optimized implementation for exactly what you need. All from a small general-purpose library!

But it comes at a price. There is no (non-template) “DeviceI2C” class anymore. Writing a class on top to implement access to the UART Plug for example, means this class has to use templates as well. It’s a bit like “const” – once you start on the path of templates, they will start to permeate all your your code. Yikes!

The other “cost” is that all templates have to end up in header files. The size and complexity of the “JeeLib.h” header file is going to increase immensely. Not so great if you want to get to grips with it and just use the darn stuff. On the plus side, I was pleasantly surprised that error messages are fairly good now.

These drawbacks may be acceptable if all the template code can indeed remain inside the library – I sure wouldn’t want to impose the need for all library users to learn the intricacies of templates. So maybe it’s all doable – but this approach has major implications.

Is it all worth it? Hm, big decision. I do not like big decisions, not at this stage…


New library experiments

In AVR, Software on Jan 14, 2010 at 00:01

Encouraged by the previous post, I started writing a bit more code based on C++ features such as templates and namespaces, in the form of an Arduino library (compatible with it, not depending on it):

Screen shot 2010-01-09 at 02.08.27.png

Things to point out:

Less #defines (none so far), I turned bitRead() and bitWrite() into template functions (getBit and setBit), so that they can be used with 1..4 byte ints, just as the macros.

The Port class is inside a new “Jee” namespace, so there is no conflict with the existing Ports library.

Atomic access hasn’t been solved. Some of this won’t work if I/O pins get changed by interrupt code. I’m hoping to solve that at a higher level.

There are compatibility definitions for pinMode(), digitalRead(), and digitalWrite() using the normal Arduino pin numbering conventions (only Duemilanove so far). These are in their own namespace, but can also be used without any qualifiers by adding “using namespace Jee::Arduino;” to the sketch.

Totally incomplete stuff, untested, and not 100% compatible with Arduino in fact.

The other thing I want to explore, is to treat the choice of what a pin does as a pin initialization mode. Hence the enum with Mode_IN .. Mode_PWM definitions. The underlying code is PortBase::modeSetter(), but it hasn’t been written yet. It’s all steamy hot vapor for now.

Update – I’ve placed the code in subversion, but the API is going to be in flux for a long time.

Update #2 – Atomic access is actually better than I thought. With constant pin numbers, setBit() will use the cbi/sbi instructions, which are atomic.


C++ templates

In AVR, Software on Jan 12, 2010 at 00:01

A recent post described the performance loss in the Arduino’s digitalRead() and digitalWrite() functions, compared to raw pin access.

Can we do better – i.e. hide the details, yet still get the benefits of raw I/O? Sure.

If you’ve used JeeNodes and in particular the “Ports” library, you’ll have noticed that there is a C++ class which hides the details of each port (i.e. JeeNode port, not ATmega port). Let’s look at that first:

Screen shot 2010-01-06 at 12.40.09.png

I’ve omitted the implementation, but there are still lots of secondary details.

The main point is that this is now C++, and uses a “Port” object as central mechanism. Each object has one byte of data, containing the port number (1..4).

Due to heavy inlining, there is almost no additional overhead for using the Port class over using digitalRead() and digitalWrite(), on which they are based. I verified it by running similar tests as in the recent post about pin I/O:

Screen shot 2010-01-06 at 12.48.50.png

Using the definition “Port orig (1);” – and sure enough the results are nearly the same.

There are two issues which make this approach sub-optimal: using the slow digital read/write calls, and storing the port number in a memory location which needs to be accessed at run time. There is no way for the compiler to optimize such calls, even “orig.digiRead()” should be the same as writing “bitRead(PORTD, 4)” in this example.

That’s where C++ templates come in. Check out this definition of a new “XPort” class (named that way to avoid a name conflict) and an example of use for port 1:

Screen shot 2010-01-06 at 12.54.02.png

(As you can see, I’m switching to a different, and hopefully clearer, API along the way)

There’s some funky <…> stuff going on. We’re in fact not declaring one class, but a whole family of classes, parametrized by the integer included in the <…> notation on the last line.

The big difference, is that each class now has that integer value “built-in”, so to speak. So we can define member functions which directly pass that value on to the corresponding bitRead() and bitWrite() macros. And then all of a sudden, all the overhead vanishes: since the member needs no access to object state, it can be made static, and since all the info is known in the header, it can be made inline as well.

So the above template is C++’s modern way of doing far more at compile time, allowing the optimizer to generate much better code.

Note that templates come with some pitfalls: first of all, it’s very easy to inadvertently generate huge amounts of code, so very careful inlining and base class derivation is essential. The second problem is that templates tend to be “instantiated” as late as possible by the compiler, which can lead to confusing error messages when the templates are wrong or used wrongly.

I’m still just exploring this approach for embedded use. The potential performance gains are substantial enough to give it a serious try. My hope is that the hard work can be done in a library, so that everyone else can just use it and benefit from these gains without having to think much about templates, let alone implement new ones. The “one” object declared above acts like any other C++ object, so using it will be just as easy as non-template objects.

Does the above lead to fast code? You bet. Here’s a test sketch:

Screen shot 2010-01-06 at 13.05.29.png

And here’s some sample output:

Screen shot 2010-01-06 at 13.06.14.png

As you can see, values 5 and 6 are virtually the same as values 7 and 8. We’ve obtained the performance of direct pin access while using a high-level port-style notation to access those pins. This is why templates are so attractive for embedded use.

The timings are different from the previous post because the loops are coded differently. In this case, only the relative comparisons are relevant.

,

Pin I/O performance

In AVR, Hardware, Software on Jan 6, 2010 at 00:01

There was a discussion on the Arduino developer’s mailing list about the impact of a small change to the digitalWrite() function, and for some time I’ve been hearing that digitalWrite() has a huge amount of overhead.

Time to find out.

Here is the sketch I used to measure how often a pin I/O command can be issued using various mechanisms:

Screen shot 2010-01-05 at 11.42.53.png

The logic is that I’m counting how often the same command can be called between timer overflows, i.e. every 1024 µs (one byte, incrementing @ 16 MHz / 64), before the timer tick count changes again.

And here’s the sample output:

Screen shot 2010-01-05 at 11.42.14.png

There’s a small amount of jitter, which tells me the loops are syncing up almost exactly on the timer ticks. Interrupts have not been disabled, so the timer interrupt is indeed being serviced – once for each loop.

What these values tell me, is that we can do about:

  • 10 analog 10-bit readings per millisecond with analogRead()
  • 128 pwm settings per millisecond with analogWrite()
  • 220 pin reads per millisecond with digitalRead()
  • 224 pin writes per millisecond with digitalWrite()
  • 1056 pin reads per millisecond with direct port reads
  • 1059 pin writes per millisecond with direct port writes

(I’ve corrected the counts by 1000/1024 to arrive at these millisecond values)

So the Arduino’s digital I/O in IDE version 0017 can do roughly 1/5th the speed of direct port access on a 16 MHz ATmega328.

But WAIT! – There’s a large systematic error in the above calculations, due to the loop overhead. It looks like the loop takes 1024000/1251 = 819 ns overhead, so the actual values are quite different: digitalRead() -> 3712 ns, direct port read -> 151 ns. Now the values are more like 1/25th!

So let’s redo this with more I/O in each loop iteration (all 4 ports):

Screen shot 2010-01-05 at 11.55.31.png

The sample output now becomes:

Screen shot 2010-01-05 at 11.56.59.png

With these results we get: one digitalRead() takes 4134 ns, one direct port read takes 83 ns (again correcting for 819 ns loop overhead). The conclusion being that digitalRead() is 50x as slow as direct port reads.

Which one is correct? I don’t know for sure. I retried the direct port read with 16 entries per loop, and got 67 ns, which seems to indicate that a direct port read takes one processor cycle (62.5 ns), as I would indeed expect.

Conclusion: if performance is the goal, then we may need to ditch the Arduino approach.

Update – Based on JimS’s timing code (see comments): digitalRead() = 58 cycles and direct pin read = 1 cycle.

Update #2 – The “1 cycle” mentioned above is indeed what I measured, but incorrect. The bit extraction was probably optimized away. So it looks like direct pin access can’t be more than 29x faster than digitalRead(). As pointed out by WestfW in the comments, digitalRead() and digitalWrite() have predictable performance across all use cases, including when the pin number is variable. In some cases that may matter more than raw speed.

Update #3 – Another caveat – Lies, damn lies, and statistics! – is that the register allocations for the above loops make it extremely difficult to draw exact conclusions. Let me just conclude with: there are order-of-magnitude performance implications, depending on how you do things. As long as you keep that in mind, you’ll be fine.


JeeMon – one year later

In Software on Jan 4, 2010 at 00:01

JeeMon is a little web server / database / reporting tool I wrote a while back. It has been in use at Jee Labs for over a year now. Here is the electricity use for 2009:

Screen shot 2009-12-31 at 17.27.15.png

Couple of glitches, such as incorrect readouts when power failed and the sending node got reset to zero counts.

Here’s the gas consumption for 2009:

Screen shot 2009-12-31 at 17.28.06.png

Gas consumption (heating and hot water) is relatively high for this house, which is an open split-level design with lots of windows. Well isolated, but there’s simply no way to keep heat from rising up through the open stairways.

Here are the temperature and humidity readings for the past month:

Screen shot 2009-12-31 at 17.38.39.png

These are commercial S300 and KS300 sensors, and they seem to have frequent glitches. It looks like those could easily be filtered out, though.

Still, we managed to get €600 back on the 2009 energy bill, and with about 3000 kWh and 2000 m3 we’re 30% below the average consumption in this residential neighborhood, for both electricity and gas.

Being aware of what’s going on makes a difference, IMO. It has become a habit to turn off all the lights when I leave a room, and closing curtains right away when it gets dark. When we go out, we turn down the thermostat. Who needs high tech, when common sense is all you need? It’s so obvious, yet so effective …

The JeeMon database for all of 2009 is 26 Mb, i.e. tiny when considering that this includes every reading and some aggregated series as well. In fact, it contains a lot more data than what’s shown in the above graphs.

I’ve got several ideas and plans for JeeMon in 2010. I want to make it far more modular, so that nearly all its current functionality becomes available as easy-to-extend plug-ins. And it needs to be fullly configurable – as it is, JeeMon is still little more than a one-off implementation. But it has served its purpose very nicely already.


Better UART code and GPS

In Software on Jan 3, 2010 at 00:01

The UartPlug class which was recently implemented used per-byte access via I2C to retrieve each received character.

This isn’t terribly efficient, since it requires sending several bytes back and forth for each received character: first we address the UART to check its status, then we access it again to get one character from the FIFO. That’s six bytes of data for each received character.

It didn’t work well enough at 38400 baud, so I added a small buffer as optimization. The UartClass API hasn’t changed. The code which does this is:

Screen shot 2009-12-29 at 15.36.49.png

Nothing fancy, just a small buffer (10 bytes currently) to speed things up. The essence is that when lots of data is coming in, we get 10 bytes at a time, for a total I2C overhead of 15 bytes. That’s a 4-fold reduction in bandwidth.

The setup I’m experimenting with is this FV-M8 GPS receiver:

DSC_0928.jpg

The hookup looks daunting, but that’s because the connector is a bit small so I made a little breakout board for it:

Screen shot 2009-12-29 at 15.44.56.png

Here’s some sample output:

Screen shot 2009-12-29 at 16.05.09.png

The “gpsdemo” sketch is simply a serial pass-through, converting 38400 baud to 57600 along the way:

Screen shot 2009-12-29 at 15.46.40.png

But it still loses character along the way, as seen in this snippet:

Screen shot 2009-12-29 at 15.17.26.png

The RMC line has a partial line at the end, after the “*7B” checksum.

My guess is that this isn’t the UARTs fault, nor the I2C bus. What seems to be happening is that the Serial class does not buffer its output, so all output causes the sketch to become unresponsive for input. In theory the 57600 baud link should be able to get data out fast enough, but I suspect that in practice there is an occasional hickup.

If this is indeed the cause, then it’s another example of how wasting time can lead to problems.

Back to the GPS. After a few minutes it finds its position in 3D. The following info comes in 5 x per second:

Screen shot 2009-12-30 at 01.54.08.png

Some additional details are reported once a second:

Screen shot 2009-12-30 at 01.53.35.png

(ignore the last few garbled lines, I really need to lower that baud rate a bit)

Here’s an even more accurate fix I found in the data stream:

$GPGSA,A,3,12,22,09,28,18,26,17,27,,,,,1.32,1.02,0.85*02

So that’s up to 8 satellites – indoors, with stone/concrete walls and coated double-glazing windows. Impressive… older units I tried never even got a fix indoors!

To interpret this GPS data, check out Mikal Hart’s TinyGPS library, it can easily be hooked up to the UART plug – just use this code at the top of his test_with_gps example:

Screen shot 2009-12-31 at 00.58.36.png

… and init nss to match the baud rate of your GPS unit.


Wasting time

In Software on Jan 2, 2010 at 00:01

No, not your time or mine… the topic I want to go into here is how to let a micro-controller deal with things that happen over time.

The way an Arduino sketch works is as follows:

Screen shot 2009-12-28 at 14.59.53.png

The little slashes represent real code, where “stuff happens” so to speak. You can see that there is a clear flow of control, from the start of the code to the end, with delays and calls to other bits of code.

The trouble with this is that it can’t deal with multiple events. You can’t count pulses on an input pin and keep a LED blinking at the same time, for example.

In the old days, interactive computer interfaces were written in this same way. You’d enter a couple of values and then you got some results back. If you made a mistake halfway in the sequence, you had to restart and enter everything all over again.

Then “command menus” were invented: elaborate decision trees of the form “do you want to do X, Y, or Z?” – now at least it was possible to go back just one step. Today, user interfaces are event driven, responding to whatever interaction the user initiates instead of presenting choices. The user is in control, not the computer.

The traditional event-driven logic uses a dispatch / switchboard approach:

Screen shot 2009-12-28 at 15.00.03.png

The more modern approach is to use callbacks and closures. Fancier still is to use coroutines or continuations.

Could we use something similar for micro-controllers? Sure.

Callbacks are not that convenient in C, since it does not support the closures concept which would make it convenient (nor coroutines or continuations). Also, callbacks would have to be represented by pointers to C functions (2 bytes), whereas dispatch codes can probably be represented as a single byte (as long as we don’t have more than 256 of them). Efficiency and memory space matters on small 8-bit chips such as the ATmega.

This requires a change in mindset when writing sketches. A good way to get into such an event-oriented style is to imagine that there is no delay() function. In event-oriented code, you’re not allowed to wait for time to pass.

So how can we blink a LED if we can’t wait to switch it on and off? Well, instead of delays, we have timers. We’re still not allowed to loop until the timer expires, but we can tell this new type of timer to generate an event when its time has come.

For a low-end implementation, an event can simply be a small integer dispatch code.

So instead of waiting until we can turn the LED on or off again, we tell the system to wake us up again at the right time. This approach is also a perfect match for low-power nodes, by the way: sleep and wake-up on events can be taken literally to enter power-down mode and start up again.

The code might look something like this:

Screen shot 2009-12-28 at 16.00.55.png

Sure, it takes some more work than this delay-based code:

Screen shot 2009-12-28 at 16.03.06.png

… but it has the huge benefit that it’s now fairly simple to deal with other activities at the same time. What you need to do is turn each of those activities into one ore more events as well. They can then be added right next to the blink event code, as additional case(s) in the above “switch” statement.

An event-based dispatch mechanism adds a lot of flexibility. Tasks such as counting pulses, blinking LEDs, serial communication, wireless packet reception and transmission – these can all be processed as they occur. With as nice bonus that low-power sleep modes can become fully automatic: when there are no events, go to sleep!

The trick is to ban all uses of “delay()” and “delayMicroseconds()” and to never use “busy loops” to wait for something to happen. This has far-reaching implications, because all libraries must also play by these rules.

I’m going to explore this approach further. Maybe one of the existing nanokernels could be used as foundation. To qualify, it must be truly minimal (and I’m picky!) – i.e. it has to fit into an ATmega328 without claiming too much flash or RAM memory space.

Update – here’s a web page by “inthebitz” which illustrates the problems described above. It’s an absolutely genuine attempt to help people get started, and it no doubt does a very fine job at that, but it also shows how everything gets serialized time-wise. For slow things, it’ll be just fine of course.

,

JeeNodes and complexity

In Hardware, Software on Dec 29, 2009 at 00:01

In yesterday’s post, I commented on a couple of aspects of the Arduino world which are giving me some trouble.

Now it’s time to do the same for all this “JeeNode” and “JeePlug” stuff I’ve been working on in 2009. Because there too, there are some ugly edges, uncomfortable trade-offs, and unsolved issues.

Let me start by pointing out the obvious: the Arduino world and Jee Labs are totally and utterly incomparable in scale. In terms of number of people involved, money involved, business interests, variety, maturity, promotion, reach, ambition, … they really differ in every aspect you can think of.

Except technology: Arduino’s and JeeNodes both live in the space of Physical Computing. And JeeNodes are compatible to Arduino’s in terms of software and PC interfacing. That’s not so surprising, since I designed JN’s to get going as fast as possible with as few key changes as possible.

These key changes were:

  • Voltage levels – run at 3.3V to support more sensors and wireless
  • Hardware ports – multiple identical interface connectors
  • Hardware extension – I2C buses which allow daisy-chaining
  • Software – connector-independence through the “Ports” library
  • Physical shape – a much smaller RBBB-like form factor

Two key additions driving some of this were:

  • Wireless – built-in bi-directional wireless link
  • Low power – selecting all hardware so it can run off batteries

Because one major use for all this is remote battery-powered operation, a minimal PC interface was chosen (3.3V signals for use with an FTDI adapter).

So is all this Jee stuff merely about creating wireless nodes? Not really. That just happens to be my area of interest. Staying focused is good, and I definitely intend to stick to this for quite some time.

So far so good. I thought I had it all figured out.

Then reality – and complexity – set in. Here are a number of issues:

  • Connector choices – ports use 6-pin 0.1″ headers as connectors: cheap and widely available. Not polarized, so you can plug things in the wrong way (and you will) – it took a while, but it has been standardized, and the POF approach solves polarization where it matters most: during experimentation.
  • Library dependencies – the “Ports” and “RF12″ (wireless) libraries have become inter-dependent, so you now have to include both of them all the time. As far as I can tell, this is not a software modularity issue but an Arduino IDE problem. This is starting to become a major inconvenience for small projects.
  • Class dependencies – classes such as “MilliTimer” are very small and self-contained but the more they get used, the more they lead to the above library dependency troubles. If adding modular and useful code leads to headaches, then something is definitely wrong.
  • Wireless functionality – a small driver takes care of packet reception and transmission in the background. It’s very low level but an easy transmission layer on top now adds robust communication for sensor networks. More convenience functions like these need to be added.
  • Multi-node development – it’s tedious to deal with multiple interfaces when more than one JN is plugged in, because USB interface names are meaningless. There is currently no way to auto-detect when a JN is plugged in or removed, let alone do something specific depending on what sketch that JN is running.
  • I2C ≠ I2C – ports support a software (bit-banged) I2C bus, but the ATmega also has pins which support hardware I2C. The underlying driver code is completely different. There is no generalized layer yet to hide these differences, so right now code written to use I2C plugs via the Ports library needs to be changed to work with hardware I2C and the Arduino’s “Wire” library. This must be fixed.
  • Ports vs. Plug Shield – due to the I2C differences described in the previous item, code written for ports on a JN needs to be adapted to work with the Plug Shield on an Arduino, and vice-versa. Tedious.
  • External power – the various power pins on a JN are all tied together, because the voltage drop of diodes would hamper ultra-low power use. That means you can’t hook up a JN to FTDI while a battery is tied to one of the other power pins (unless the battery has an on/off switch and you’re careful).

Some of the above issues have already been sorted out. Others can be addressed in future hardware designs. Some, such as “external power” are likely to remain as is for now. And some just need more or better software.

Adding or modifying software is easy. The hard part is avoiding restrictive decisions which have long-term consequences. Here’s the most challenging one: should I maintain 100% software compatibility with the Arduino world and benefit from all the shared knowledge already built up, or should I start off on a new journey and redo whatever is needed from scratch to reach a workable level again? I haven’t decided.

Screen shot 2009-12-28 at 00.33.52.png

When you hit a wall, do you look for a way around it or do you tear it down? Depends on the wall, I s’pose.


Arduino and complexity

In Hardware, Software on Dec 28, 2009 at 00:01

This is – in a nutshell – the Arduino world:

Screen shot 2009-12-27 at 18.53.04.png

In prose: an ATmega board, some hardware peripherals, an environment for embedded software development, a front-end for visualization, and a website to bring all these facets together for a growing worldwide community.

For each of these, there are alternatives and variations. These add variety and increase the number of choices for everyone interested in this low-cost physical computing world. The “Arduino” name is conquering the world, and it sticks … even as “duino” suffix. Names are a great way to create a brand.

But what is Arduino, really? Is it an ATmega? Is it a board that fits in the palm of your hand? Is it a standard for connecting other boards? Is it a standard way of structuring software? Is it defined by a set of standard libraries? Is it the dual aspect of the IDE, i.e. Wiring vs. Processing? Is it the main web site or the discussion forum?

The answer depends no doubt on who you ask. And if this is a healthy ecosystem, then all of these will evolve and improve over time. In my opinion, the current state of an ecosystem is far less important than its ability to evolve.

That’s where complexity comes in.

It’s not hard to construct a great system, even an elaborate one. But what is extremely hard, is to come up with a system which supports evolution of all the pieces involved. Because with multiple pieces, you have to make decisions. You have to interconnect the pieces, and that requires making choices. And once you do, you reduce the number of future paths, including many you don’t even know about yet.

Making choices can be good. Hooking up an Arduino to a PC via a serial connection is a great low-cost solution for uploading, debugging, and interacting. The transition from an RS232 connection + a separate power supply to a USB connection with “built-in” 5V power is an excellent example of how evolution can lead to substantial progress.

Technological evolution can take years. And as in life, that’s where the really interesting stuff happens.

Unfortunately in the Arduino world, I’m hitting some nasty edges which tell me we need more generality or flexibility. I’ll describe a few, in terms of interfaces between the different pieces shown above:

Voltage levels – the most widely used Arduinos are based on 5V, meaning that all their I/O pins are also 5V-based. More and more sensors can only be used with a supply voltage up to 3.6V or so. Interfacing them to a standard Arduino requires the use of level converters. Often, resistors will do – but you can’t simply ignore the issue. This is an area where I2C can help.

Hardware modules – the Arduino “shield” concept doesn’t extend very far, because each shield determines which pins it needs. Pins are a scarce resource, and each pin location on the connector is fixed. It’s hard to design shields which will actually stack with others. Even more so now that there is a new Arduino Mega with slightly different pin allocations.

Software modules – libraries are a great concept: code which is there when you need it, but doesn’t get in the way (i.e. included) if you don’t. Right now, adding a library which depends on say the “Wire” library means that you have to include the Wire library in your sketch even if you don’t ever use it. This is almost a show-stopper.

C/C++ code – on the surface, programming for the Arduino board is like programming in C and C++, with a bunch of very common issues resolved and a wide variety of useful functions available, ready to use. Newbies can get going fast and old hands can find their way around in no time. Except… the IDE does some funky things with headers and declarations. And it breaks a couple of fairly basic C/C++ conventions.

Automation – the IDE works with a specific scenario in mind. As of the latest 0017 release, multiple sketches and consoles are supported. This solves some of the issues, but it’s hard to automate this further: after uploading, I still have to manually launch the serial console. If I have some other app talking to the hardware, there is no mechanism to upload a new sketch and then resume that app because the serial USB connection is dedicated.

Visualization – it’s interesting to see how the Arduino / Wiring / Processing approach is bridging a huge range of hardware in an integrated manner. But there are several choices involved which reduce flexibility towards entirely different approaches. Some visualizations (as well as other types of data processing) would benefit from using a more internet-/web-centric approach. Lower-end hardware cannot support the Java environment used by Processing, so that code can’t be leveraged. There are alternatives. Scripting language support would be useful.

Don’t get me wrong: all these issues are tough ones! It’s probably impossible to solve them all and to make everyone happy. These issues are definitely not things open source software is obliged to address from day one. But my point is that this is not about where we are today, but about having the mechanisms to evolve to where we want to be tomorrow.

If you’ve followed this weblog for a while then you can probably guess that I’ve got some ideas and suggestions on how to move forward on all this. I would really like to help take the existing Arduino platform further – but I’m not sure at this point that it can be done within the current constraints. I don’t even know whether anyone else considers the issues listed above to be important. All I know is that as I continue to add more software and design new hardware, the “push-back” from the current Arduino design choices is increasing… a lot.

Do I want to change the Arduino world (i.e. make a case, convince people, find compromises) – or do I want to sidestep some of its decisions by starting all the way from scratch? I haven’t decided.


Carrier detection

In AVR, Software on Dec 26, 2009 at 00:01

Wireless communication is a complex process. The ISM bands used by the RFM12B module get used in various ways – some quite simplistic. For “serious” use, what you want is to avoid transmissions interfering with each other – which means at most one transmitter should be active at any given time for a specific frequency band.

Easier said than done. This is what CSMA/CA is all about. It’s quite similar to ethernet: you listen to the “ether” and wait until there is no carrier before starting to send yourself. If everyone does so, then there will be fewer “collisions” – i.e. messed up packets.

CSMA/CA isn’t perfect. It doesn’t scale all that well if there are lots of nodes, all trying to find a free slot to send their packets out. There is always a non-zero probability of collision, when two nodes decide at nearly the same time that there is no-one else transmitting. And then, BOOM! … as they say.

One solution for that is another acronym: TDMA. Basic idea: have all the nodes agree to only send in specific time slots allocated to them. This requires a central coordinator, as well as pretty accurately keeping track of time (which is non-trivial with RC-based watchdog timers for sleep modes used with low-power nodes!).

In the ISM band, at least the 868 MHz one used in Europe, a simpler solution is used: keep the air-waves more or less free. The rule is that each transmitting node should send no more than 1% of the time, on average. With one or two dozen nodes and a bit of randomness in timing, the idea is that collisions will be relatively rare.

I recently added the 1% rule to the easy transmission mechanism in the RF12 driver (for the 868 MHz band only). So with the rf12_easy…() calls, adhering to the 1% rule is now automatic.

The 1% rule is a very simple system, yet it works surprisingly well. I’ve got over a dozen nodes around the house, sending out packets whenever they feel like it. The off-the-shelf commercial weather station nodes I use are very simplistic – they just send, ignore collisions, and send new data again a minute or two later. Those nodes probably don’t even have receive capability.

The RFM12B transceiver module in the JeeNode is slightly more sophisticated, in that communication can take place in both directions. So the receiver can send back an “ack” packet, and the originating node will have some idea of whether its last data packet ever made it to the destination (note that ack’s can get lost as well – but that’s another story).

Still – the RF12 driver used in JeeNodes is just as careless as the other nodes: it starts sending the moment it feels like it – except if a packet for its own net group is currently being received. Sending packets with the RF12 driver can still easily mess up whatever is currently going on in the air.

Well, as of today, things will improve a bit further. I’ve extended the RF12 driver code to look at the RSSI status bit before starting to transmit. If a carrier is detected, even one that isn’t being recognized by the RFM12B, then transmission will be delayed a bit. Here is the latest code in the RF12.cpp driver:

Screen shot 2009-12-21 at 02.16.28.png

This isn’t perfect – nothing ever is – because most nodes will be polling very frequently and then start to send right after the carrier drops. So the chance increases that nodes two and three will both try sending when node one finishes. But let’s assume that the RSSI signal doesn’t drop to 0 for all nodes at exactly the same time. If that turns out to be insufficient, a timer-based exponential back-off mechanism can be added later.

And there is a substantial benefit: nodes will no longer mess up packets which are currently “on the air”. As more nodes are being added around here, I expect this change to cause less degradation due to collisions.

In summary: the RF12 driver is a very simple system. No CSMA/CA, no TDMA. There are definitely limits as to what it can be used for. There are some severe limits on how much data it can send, given that (on 868 MHz) each node can only send up to 1% of the time, but this is also why such a simple approach is actually quite effective. And why the RF12 driver is still just a few Kb on an 8-bit ATmega, leaving lots of room for application specific code. I happen to think that the RF12 approach strikes a pretty decent balance between simplicity and effectiveness – and I’m always open for suggestions on how to take this further.


OOK reception with RFM12B ?

In AVR, Software on Dec 25, 2009 at 00:01

Yesterday’s post described a setup to see the RSSI and DQD status bit reported by the RF12 driver in real time.

One of the interesting results is that I can see the RSSI light come on when pressing a button on the FS20 remote transmitter – even though that’s an OOK signal, not FSK!

When adjusted to run at 433 MHz, the RSSI indicator also lights up with the KAKU remote.

In both cases, the DQD signal appears useless – it just shimmers all the time.

The RSSI signal is encouraging, though. It turns out that getting it to blink reliably did depend on setting the threshold right. At -103 and -97 dBm, it was on all the time – only the -91 dBm value produced a usable signal. I hope that’s the case with all units.

Could this be used to receive FS20 or KAKU?

Well, I just had to try. My idea was to continuously poll the RSSI status bit and then “mirror” its value to a DIO output pin. Then use a second JeeNode to treat this as a normal OOK pulse train.

Here’s the “rssiMirror” sketch I used:

Screen shot 2009-12-20 at 17.07.34.png

Does it work? Unfortunately … no :(

Time to hook up the Logic Analyzer to see what’s going on. I connected the above digital output to the first channel, and a real OOK receiver on the second channel:

uuu.png

Guess what… the RSSI signal is indeed detecting the presence of a transmitted signal, but it’s way too slow!

Here’s the same sample, zoomed in on the real OOK pulse train:

ooo.png

As you can see, there’s a pretty good sequence of transitions, 400 µs and 600 µs apart. Oh well, so much for the RSSI status bit – it’s nice to detect the presence of a carrier, but not more than that.

Next thing I tried was the DQD signal. After tweaking the DQD threshold to 3, this is what came out:

eee.png

Yeah, sure, it seems to track the signal, but not reliably, and with a huge number of extra transitions. Note how the top timings are all multiples of 25 µs apart – that’s because it takes 25 µs to read out the DQD status bit. Coarse, but fine enough in principle to track 400 / 600 µs pulses from an FS20 remote.

So, again: nice, but no dice. Neither the RSSI nor the DQD status bits are fast and accurate enough to decode a slow OOK pulse train with.

Next attempt was to try and pick up the ARSSI signal, direct off the RFM12B module – as mentioned in this forum discussion. There’s a German forum which describes where to pick up that signal:

rfm01.JPG.jpeg

And sure enough, here’s a scope capture of an FS20 transmission:

www.png

Yeah, it’s there alright. But the signal is a bit weak. I’d rather not dedicate the analog comparator or ADC to it, and besides – that still leaves the need to compare against the average level – there’s a nasty 0.4V bias in that signal.

Here’s the same signal, AC coupled:

qqq.png

And here’s a zoomed-in area, showing what looks like pretty decent 400 µs and 600 µs pulses:

hhh.png

So yes, a small self-adjusting comparator can proabably turn this into a nice digital pulse train – but it’ll require some extra components, and I’m a bit out of my league on designing such a circuit.

Oh well – perhaps this information will help someone else further along. It’s been a good learning experience for me, even if the result is not quite what I had hoped…

Tomorrow, I’ll describe another – successful! – outcome from this RSSI / DQD exploration.


RF12 status lights

In Hardware, Software on Dec 24, 2009 at 00:01

Here is a little setup to see what’s going on in the ether, wirelessly speaking, that is:

FlySketchExport.png

Four LEDs, blinking according to the following status signals:

  • RECV – blinks briefly for each received packet
  • RSSI – shows the value of the RSSI bit in the RF12 status
  • DQD – shows the value of the DQD bit in the RF12 status
  • ALIVE – blinks at 1 Hz, just to show that this node is alive

I changed some thresholds to get better results: RSSI threshold -91 dBm and DQD set to 7 (normally -103 and 4, in the current RF12 driver).

Here is the sketch which drives the LEDs accordingly:

Screen shot 2009-12-20 at 16.05.18.png

In this example, I’ve set up the RF12 parameters to receive group 5 in the 868 MHz band.

The result is quite interesting: DQD is flashing constantly, in an irregular pattern. RSSI blinks only once in a while, and some of these cause RECV to light up – as expected: only packets with the proper FSK format are filtered out, and only some of those are for this specific net group.

The reason for doing this will become clear tomorrow…

, ,

Power tracker – software

In AVR, Software on Dec 22, 2009 at 00:01

Yesterday’s post described a small circuit to track power consumption of JeeNodes to help optimize sketches for minimal power consumption.

Let’s put that circuit to use, with a bit of software to measure actual power consumption. The basic idea is to continuously measure current and then integrate these measurements to determine the sum of all power consumption intervals, regardless of levels.

The reason for this is that we’re not really interested in current draw but in the amount of charge consumed by the JeeNode. As far as the battery is concerned, drawing 1 mA for 1 hour is the same as drawing 100 µA for 10 hours (in the ideal case, anyway). The fancy way to say this is that we need to measure Coulombs, not Amps. Or rather micro-Coulombs, i.e. µC. That’s really easy once you realize that 1 µA is the same as 1 µC per second. Or to put it differently again: 1 mAh = 3600 mC. So a 1000 mAh battery is really nothing but a 3600 C charge.

Ok, back to the problem at hand: measuring average current draw per second.

Here is a simple sketch which does all the auto-ranging and integration:

Screen shot 2009-12-19 at 19.34.25.png

It reports averaged power drain in µA/sec (the second value is the number of samples per second). The 10% correction which I had to apply in my setup could be due to a number of factors – most likely it’s due to resistor tolerances (they are all 5%).

Here’s an interesting case with the latest rooms node:

Screen shot 2009-12-19 at 20.01.21.png

As you can see, the baseline power drain is a fantastically low 56 µA/sec in this case, but once or twice a minute it goes up to 14 mA/sec for several seconds. Not sure what’s going in here – need to investigate (now that I can!).

It would be nice to automatically detect the baseline, i.e. the average low-level sleep consumption, and things like the peak current and the percentage of the total consumption caused by such peaks. Extending the software to handle this is more work.

With slightly more elaborate software, it will be possible to place the power measurement plug between the measuring JeeNode and the JeeNode under test, and then leave it alone. A 1-day or 1-week average should give an excellent estimate of battery lifetimes.


UartPlug class

In AVR, Hardware, Software on Dec 20, 2009 at 00:01

Here is a utility class for the UART Plug:

Screen shot 2009-12-18 at 13.42.26.png

The interface is exactly the same as the Serial class, so it can be used interchangeably. Here is the updated “uart_demo” example in the Ports library:

Screen shot 2009-12-18 at 13.38.44.png

Here’s a test setup with a second JeeNode running RF12demo plugged in:

DSC_0866.jpg

Sample output:

Screen shot 2009-12-18 at 13.38.29.png

Both input and output are supported by this UartPlug class – this demo is essentially a serial pass-through.

The UART supports accurate baud rates all the way up to 230400, which is in fact beyond the current I2C rates of the Ports library. Even at 57600 baud, I’ve seen several serious overruns with the above demo. One reason is that it’s only reading out one byte at the time, going through a multi-byte I2C bus sequence for each one (!). Note also that the Serial class does not buffer its output, so it can easily bog down everything else.

The UART hardware can support both hardware handshaking and XON/XOFF throttling, so this would be another way to avoid buffer overruns.

Up to say 9600 baud the UartPlug class should work fine, even with several UART plugs on the same I2C bus.

, ,

Battery life – refinement

In AVR, Hardware, Software on Dec 19, 2009 at 00:01

Yesterday’s post described how to estimate the battery life of a JeeNode running the “rooms” sketch with the SHT11, ELV PIR, and LDR sensors. To summarize:

  • the code started out using 370 µA average current, i.e. roughly 7 months on 3 AA cells
  • of these, 200 µA were caused by the 1-second periodic wakeup “blip”
  • another 120 µA were due to the actual measurements and packets sent every 30 seconds
  • and finally, the remaining 50 µA come from the PIR + JeeNode current draw in sleep mode

Yesterday’s post was also about reducing that 200 µA blip consumption to somewhere around 20 µA.

Today, let’s tackle the other power “hog”: the 300 ms @ 12 mA spike. Here is that pattern again:

b1.png

The high peak at the end is the RF transmission of a packet, followed by a “knee” during which the node is waiting for the ack packet in RF receive mode.

Note that the main power drain is NOT caused by wireless communication!

This period of over 300 milliseconds is when the ATmega is polling the SHT11, waiting for a fresh reading. Twice in fact: once for the temperature and once for the humidity measurement.

So the explanation is very simple: we’re polling and waiting in full-power mode. Quelle horreur!

The fix requires a small modification to the SHT11 driver in the Ports library. Instead of a fixed delay, it needs to be extended to allow using an arbitrary function to waste some time. Here’s the modified code:

Screen shot 2009-12-18 at 01.00.53.png

A new second arg has been added, specifying an optional function to call instead of delay(). The code remains backward compatible, because this argument defaults to zero in Ports.h:

Screen shot 2009-12-18 at 01.02.45.png

So now all we need to do is define a delay function which goes into real power down mode for a little while:

Screen shot 2009-12-18 at 01.52.23.png

… and then adjust the two measurement calls to use this function:

Screen shot 2009-12-18 at 01.05.08.png

Does all this make a difference? You betcha:

b2.png

That’s a substantial fraction of a second in which the ATmega will draw considerably less than 12 mA. How much less? Let’s expand the vertical scale:

b3.png

Most of the time, the voltage is around 50 mV, i.e. 1 mA over 47 Ω. That’s the SHT11 current draw while active. There are two measurements – so everything behaves exactly as expected!

A couple of quick wake-ups remain, to check whether the SH11 measurement is ready. And so does the wireless TX/RX peak, of course. Here is an isolated snapshot of that RF activity (200 mV/div and 4 ms/div):

b4.png

Approximate current draw: TX = 35 mA, RX = 20 mA. Total time is about 10 ms.

Looks like we’ve reduced the power consumption of this once-per-30-second spike by perhaps 90%. As a result, the node now consumes about 20 (blip) + 20 (spike) + 50 (sleep) = 90 µA on average. Even with much smaller 800 mAh AAA cells, the battery life of these low-power nodes should now be over a year.

There are several conclusions to take home from this story, IMO:

  1. The biggest drain determines battery lifetimes.
  2. Measuring actual current profiles always beats guessing.
  3. A simple USB storage scope is plenty to perform such measurements.

If I had followed my hunches, I’d no doubt have spent all my time on getting the current draw of packet transmissions down – but as these experiments show, their effect on power drain is minimal.

There are more optimizations one could explore. There always are. But the gains will be limited, given that the ELV PIR sensor consumes 30..40 µA, and that it needs to be on at all times anyway, to be able to detect motion.

Sooo… end of story – for now :)

All source changes checked in. The entire rooms sketch still compiles to under 8 Kb of code.

,

Battery life estimation

In AVR, Hardware, Software on Dec 18, 2009 at 00:01

To get an indication of battery power drain, I measured the voltage drop over a 47 Ω resistor in series with the 5V supply, using a JeeNode with Rooms Board and the latest version of the “rooms” sketch.

Here’s the blip I see, once a second (100 mV/div and 20 msec/div):

on-time-1.png

That’s a 40 msec pulse of about 5 mA, in other words: 5 mA during 4 % of the time, which averages out to around 200 µA current draw continuously. Not bad, but not stellar either: it’s 4 times the sleep mode current.

Every once in a while, a much longer and bigger spike shows up:

occasional-peak.png

Which looks like roughly 350 msec @ 12 mA.

Let’s assume the big one is a real transmission. It sort of fits: the spike at the end is a brief transmission at ≈ 35 mA total, followed by a 20-ish mA period of reception (waiting for the ack to come in).

Very roughly speaking, the area of that extra spike at the end is about the same as the 5-to-10 mA step at the start of this period. So as an estimate, we’re consuming about 12 mA during 350 msec – let’s round down to 300 msec.

Let’s also assume these bigger current patterns happen every 30 seconds, when the node is reporting changed values (everything other than motion gets reported at that rate in the latest “rooms” sketch).

So 1% of the time (300 ms every 30s), power consumption is 12 mA. This averages out to 120 µA continuous current consumption.

In other words: a JeeNode running this latest rooms sketch with the SHT11 and ELV-PIR sensors, is consuming roughly 200 (blip) + 120 (spike) + 50 (sleep) = 370 µA.

Using a 2000 mAH 3-cell AA battery, this should lead to a 225-day lifetime – over 7 months.

Can we do better? Sure.

It’s basically a matter of figuring out what’s going on during those 40 and 350 msecs, respectively. Interestingly, more can be gained by improving non-transmitting “blips” than twice-a-minute high-power RF packet exchanges.

Do those 40 ms @ 5 mA every second look a bit suspicious? Yep – that’s the “idling” power level. What happens is that I was a bit too pessimistic in the time spent in sleep mode. This was the code:

Screen shot 2009-12-17 at 00.51.26.png

Looks like this is about 40 ms off, and so the code ends up waiting for the 1 sec timer to expire… in idle mode!

Let’s change this to end up closer to the desired time:

Screen shot 2009-12-17 at 02.59.16.png

Here’s the new blip (different scales):

better-blips.png

We’re down from 40 to 10 msec blips – tada!

That translates to an average 50 µA current draw from the blips, bringing the total down to 220 µA. Which translates to a 375-day battery life: over a year!

Now we’re cookin’ … but could we do even better? Sure.

Note that only the 2 ms spike at the end of the 5 mA blip is the actual active period. The time up to then we’re just waiting in idle mode – and wasting power.

We could shorten the sleep timer to 994 ms, since we don’t care whether readings are taken exactly 1 second apart. Now the RFM12B-based watchdog timer will wake us up just 2 ms short of the target time. And sure enough, the 5 mA blip is down to around 3 ms – shown here with an even further expanded time scale:

final-blip.png

But that’s silly. We’re tweaking a millisecond timer, and we’re not even interested in an “exact” 1000 ms cycle in the first place! It makes much more sense to just use the RFM12B wakeup timer to get us close to that 1 second cycle, and then immediately take a measurement. Here’s the corresponding code change in periodicSleep():

Screen shot 2009-12-17 at 02.35.36.png

Does this make a difference? Definitely:

best-blip.png

One final remark: the above battery lifetime estimates do not take into account the increased power consumption when motion is detected and more packets are sent (up to once every 5 seconds). On the plus side, when no light / temperature / humidity changes happen, the packet frequency will drop further, to once-a-minute.

The above changes have been checked into the source code repository.

Update – I just found out that the DSO-2090 scope has a high-pass low-pass filter option:

smooth.png

Sure wish I’d found out about that feature sooner… it’s so much more informative: the initial ramp is probably the clock starting up, and the little peak could well be the LDR pull-up during ADC conversion!

, ,

Rooms sketch, reloaded

In AVR, Hardware, Software on Dec 16, 2009 at 00:01

With the new easy transmission mechanism and the low power logic implemented, it’s time to revisit the “rooms” sketch, which I use for all my house monitoring nodes based on the Room Board.

I’ve wrapped the code used in POF 71 a bit further, with these two extra functions:

Screen shot 2009-12-15 at 22.25.44.png

With this, the main loop becomes very simple – even though it will now power down the RFM12B and the ATmega328 whenever there’s nothing to do for a while:

Screen shot 2009-12-15 at 22.25.57.png

The lowPower() and loseSomeTime() code is still the same as in POF 71 – this is where all the hardware low-power trickery really takes place:

Screen shot 2009-12-15 at 22.24.57.png

Note that these need an “#include <avr/sleep.h>” at the top of the sketch to compile properly.

I’ve also disabled the pull-up resistor on the LDR while not measuring its value. This drops power consumption by over 100 µA, depending on actual light levels.

A quick measurement indicates that power consumption went down from 20 mA to some 50 µA (much of that is probably the PIR sensor). These are only approximate figures, because my simplistic multi-meter setup isn’t really measuring the charge (i.e. integrated current draw), just the current draw while in sleep mode.

These changes have been checked into the repository as “rooms.pde”.

This code isn’t perfect, but since “perfection is the enemy of done” I’ll go with it anyway, for now. One difference with the original rooms sketch is that the motion sensor is not read out as often so brief motion events might be missed. Another issue with this code is that if the central node is off, a lot of re-transmissions will take place – without the node going into sleep mode in between! IOW, a missing or broken central node will cause all remote nodes to drain their batteries much faster than when things are properly ack’ed all the time. Oh well, let’s assume this is a perfect world for now.

With these levels of power consumption, it’s finally possible to run room nodes on battery power. I’ll use some 3x AAA packs, to see what sort of lifetime this leads to – hopefully at least a couple of months.

Will report on this weblog when the batteries run out … don’t hold your breath for it ;)

Update – I just fixed a power-down race condition, so this code really goes back to sleep at all times.


Better FS20 transmissions

In Software on Dec 14, 2009 at 00:01

The RFM12B can be tricked into sending OOK (on-off-keying) signals – which is also called ASK (amplitude shift keying), by doing exactly what the term stands for: turning the transmitter on and off.

This has been used in several examples to control FS20 and KAKU remote power switches – just search for these terms on this weblog in the box at the bottom right of this page and you’ll get all the related posts.

The code I’ve been using for FS20 so far is:

Screen shot 2009-12-13 at 16.29.17.png

As it turns out, the timing is not quite up to scratch. JGJ Veken drew my attention to this in the forum and by sending in a couple of pictures, including this one:

03 jee-node transmitter 0 en 1 not ok.JPG

(Wow – great instrument, a 100 MHz Tektronix 2232 storage scope!)

A 0 bit comes out as 250/468 µS, and a 1 bit as 428/722 µS – pretty far off the 400/400 + 600/600 µS specs.

Here’s what we came up with after a few trials:

Screen shot 2009-12-13 at 17.20.28.png

The end result is within a few percent of the target, well within spec – yippie!

Jeenode Transmitter FS20 bitstream.JPG

I’ve updated the code in the RF12 examples (including RF12demo) in the code repository and the ZIP file.

A similar tweak could probably be used for the KAKU signals, but these use a lower rate of bit signaling, so the jitter is probably somewhat less important.

Update – the KAKU tweak has also been checked in, and the code has been simplified a bit further.


Serial datalogger

In Software on Dec 13, 2009 at 00:01

Two new classes have been added to the Ports library:

  • MemoryPlug – provides access to the EEPROM(s) via PortI2C
  • MemoryStream – access one or more pages as a byte stream

Here is a little serial port datalogger, using a Memory Plug on port 4 and a Blink Plug on port 3:

Screen shot 2009-12-11 at 17.00.58.png

This code is now included as “memory_demo” example in the Ports Library.

It collects all incoming data from the serial / USB port, as much as will fit on the Memory Plug’s EEPROM chips that is (512 Kb if all four M24M01’s have been installed).

Pressing button 1 (green LED on my setup) replays all stored data back to that same serial port (this process is interrupted if anything is received).

Pressing button 2 (red LED for me) resets the stream and clears all data stored so far.

Here is some sample output:

Screen shot 2009-12-11 at 16.25.03.png

What I did was: 1) send “aaa”, 2) press button 2, 3) send “bbb”, 4) send “ccc”, and 5) press button 1. As you can see, everything sent after pressing button 2 is echoed back when button 1 is pressed.

The first line of the sample output illustrates random access to the Memory Plug without using streams. Multiple memory chips will be addressed as if they were one, i.e. the MemoryPlug class automatically selects the proper 0×50..0×70 I2C device address.

The public interface to these two classes is as follows:

Screen shot 2009-12-11 at 17.05.36.png

You can define multiple streams at the same time on a single memory plug, by picking different starting pages, and they can fill additional pages in increasing (dir=1) or decreasing (dir=-1) order.


BlinkPlug class

In Software on Dec 12, 2009 at 00:01

A new “BlinkPlug” class has been added to the Ports library. The public interface is:

Screen shot 2009-12-10 at 20.36.51.png

Here is the new “blink_demo” sketch, using a Blink Plug attached to port 3:

Screen shot 2009-12-10 at 20.34.08.png

This code demonstrates how to turn LEDs 1 and 2 on or off, and how to access buttons 1 and 2 – pushed() will detect a transition from released to pressed (with proper de-bouncing), whereas state() returns the current button state (no de-bouncing logic).

There is some special logic in this class, because each LED/button pair on the Blink Plug is connected to the same I/O line. Reading out the button requires briefly turning the LED off, and the I/O line should never be set to drive the output high. These details are conveniently hidden in the above class.

An updated Ports library has been checked into the source code repository.

Note: the repository location has changed, see the CODE page for details.

, ,

Low power mode again

In AVR, Software on Dec 9, 2009 at 00:01

After yesterday’s Wireless Light Sensor was announced, I wanted to push a bit more on the low-power front.

The POF described two simple tricks to get the power consumption from 19 to 3 mA, roughly. It turns out that a single extra step will get the idle consumption down to some 20 µA. That doesn’t mean we’re getting a 15-fold battery lifetime increase, because I’m measuring the current at idle time but not accounting for the brief periods of high-current activity which also occur. But it’s a major reduction in power consumption.

Here’s how … but this requires going a bit deeper into some low-level AVR/ATmega chip features.

First, here are some utility functions we’re going to need:

Screen shot 2009-12-08 at 10.04.31.png

The lowPower() routine disables the ADC subsystem and then enters the specified low-power mode in the ATmega. Once it resumes, the ADC subsystem setup will be restored to its previous state.

The loseSomeTime() does just what is says on the box: go into comatose mode for a more-or-less controlled amount of time. The trick is to activate the RFM12B watchdog just before passing out. This leads to larger power savings than we would have with the ATmega’s watchdog, btw – and it’s easier to implement.

The complication is that we risk losing all track of time. It’s a bit hard for an ATmega to count heartbeats when its heart has stopped beating – not only are there no beats, it’s also stripped of its counting abilities while in coma…

So instead, we estimate just how long we’ve been away from the watchdog time chosen for the RFM12B, and correct the milliseconds timer built into the Arduino. That’s what the “timer0_millis” stuff above is about. It will not be quite as accurate as before, but that’s probably acceptable for a sensor node like this one.

The last issue is that we need an indication about how long we can go comatose. I’ve added a “remaining()” member to the MilliTimer class to obtain this information.

Now, all the pieces are in place to change the code in the Wireless Light Sensor from this:

Screen shot 2009-12-08 at 10.14.23.png

.. to this:

Screen shot 2009-12-08 at 10.15.53.png

So there you have it. Most of the time between each measurement once a second, the node will now go into a very low-power mode of around 20 µA. My current measurement tools are inadequate to measure exactly what amount of charge is being consumed, which is what this is really about. So accurate battery lifetime calculations are not yet possible – but I expect it to be in the order of months now.

I’ve updated the Wireless Light sensor POF to point to this post and include this trick.

, ,

Wireless Light Sensor – POF 71

In AVR, Hardware, Software on Dec 8, 2009 at 00:01

After last week’s Hello World POF to get started, here is a new Project On Foam:

DSC_0824.jpg

A battery-powered wireless light sensor node. This is POF 71, and it’s fully documented on the wiki.

This project goes through setting up the Ports and RF12 libraries, setting up a central JeeNode or JeeLink, and constructing the light sensor node.

It also describes how to keep the node configuration in EEPROM, how to make a sensor node more responsive, and how to get power consumption down for battery use.

The POF includes code examples and uses the easy transmission mechanism, with the final responsive / low-power sketch requiring just a few dozen lines of code, including comments. The sketch compiles to under 5 Kbyte, leaving lots and lots of room to extend it for your own use.

All suggestions welcome. Anyone who wants to participate in these POFs, or in the wiki in general, just send me an email with the user name you’d like to use. I’m only restricting edit access to the wiki to prevent spamming.

,

JeeNode clock

In Hardware, Software on Dec 4, 2009 at 00:01

To follow up on a recent post, here is the same real time clock demo using a JeeNode + USB BUB instead of an Arduino + Plug Shield:

DSC_0816.jpg

(instead of JeeNode + USB BUB, the new JeeNode NoRF or a JeeNode USB could also have been used)

I chose to use two different ports for RTC and LCD, but these could just as easily have been daisy-chained on a single port.

The changes to the code are minimal, here is the complete sketch:

Screen shot 2009-11-29 at 15.44.07.png

The changes affect a few declarations at the top, and the fact that the use of the Wire library was hard-wired into the getDate() function.


Easy transmissions

In Software on Dec 2, 2009 at 00:01

I’ve extended the RF12 driver with three “rf12_easy…()” functions, which should make it easier to implement nodes which need to periodically report some data to a central JeeNode or JeeLink.

Here’s a complete sketch which sends the current value of the millisecond counter over wireless:

Screen shot 2009-12-01 at 21.50.44.png

The above sketch compiles to roughly 3 Kb of code. Some extra notes:

  • The “rf12_initialize(…)” call sets this node up as node “W” (23), using 868 MHz, and using group 5.
  • The “rf12_easyInit(5)” call sets up the easy transmission system, with data to be sent every 5 seconds.
  • The “rf12_easyPoll()” call needs to be called often – at least once per millisecond or so would be best.
  • The “rf12_easySend(…)” call passes the data to be sent, i.e. pointer to that data and the number of bytes.

There’s quite a bit of additional logic hidden behind these three calls:

  • Even if called often, new data will only be sent at most every 5 seconds, as requested.
  • Lost and damaged packets will automatically be resent, with up to 8 retries.
  • Packets are only sent if the data has changed.
  • To force data to be re-sent even if unchanged, call “rf12_easySend(0,0)”.

Here is some output from a capture utility I use:

Screen shot 2009-12-01 at 23.18.15

As you can see, the 4-byte millisecond value is only sent once, every 5 seconds.

The argument to rf12_easyInit() is in seconds, and can be anything from 1 to 255 seconds. In addition, using zero as argument means: send as often as possible. For the 815 MHz band this depends on the number of data bytes, to enforce the “1% max bandwidth rule”. The actual rate will range from about 7 packets per second for 1 byte of data, to about 1 packet per second for 66 bytes of data. For the 433 and 915 MHz frequency bands, the maximum rate is fixed at 10 packets per second, regardless of the data size.

I’m currently using a simple algorithm which tries to resend at 1-second interfals when packets are lost or damaged. This can be extended to “exponential back-off with randomization” later on. My first concern is a simple-but-robust first implementation.

There is still an issue with the current RF12 driver, which can cause duplicate packets to arrive at the destination (an extra sequence bit needs to be added to each packet to detect loss of acknowledgement packets – this change affects the protocol and requires re-flashing all the current nodes, which I don’t want to do just yet). Such duplicate packets can easily be filtered out at the destination. There is also an issue with picking up the wrong ack, this probably only affects nodes at the limit of their range (i.e. when not all the nodes receive the same things).

No changes were made to the RF12 driver core, these calls are purely a layer on top. They use broadcasting to get data from different nodes to a single central node, and currently there is no way to combine this with reception of data coming towards a node. But as the above example shows, for the specific scenario of sending out data these calls should simplify the development of remote nodes a fair bit.

The new code is now in the subversion repository, as well as in the RF12.zip archive.

Update – documentation for these new calls has been added to the DOCS area.


Hello World – POF 52

In AVR, Hardware, Software on Dec 1, 2009 at 00:01

The first Project On Foam is inevitably a blinking LED:

DSC_0814.jpg

Not very exciting, but it’ll allow me to go through the steps needed to set up the development environment, the first-time USB hookup, and getting a first sign of life out of a JeeNode in Physical Computer terms…

There is a new section on the Jee Labs wiki which I’m going to use for POFs. For rather obscure reasons, this POF is #52 – it’s at http://wiki.jeelabs.net/pof/52 (new POF numbers always increase).

Note that as Project On Foam, this Hello World example is rather silly – because it doesn’t really need a foam base at all. But bear with me – this step is intended to help people through the first (and sometimes daunting) steps of getting started.

The challenge for me right now is to provide the proper information as concisely as possible. These POFs are being placed on the wiki so they can be updated and improved. There will be a few more people involved in this, which again makes the wiki much more suitable to collect and maintain all the POFs than say this weblog. New POFs will be announced on this weblog, but updates are an ongoing process on the wiki pages themselves.

There have been a lot of experiments and projects at Jee Labs over the past year, in various stages of completion. From hooking up all sorts of sensors to the house-monitoring network currently running here at Jee Labs. It is my intention to redo a number of these as POF, in more detail and with more background information. Other POFs will be completely new, though – the list of fun stuff one can do with Physical Computing is endless!

Other news: since assembly and reflow soldering of the JeeLink and the JeeNode USB has been going a lot better lately, their price has been reduced to €29.50 (incl. VAT and shipping) – see the shop for details.

,

Plug Shield clock

In Hardware, Software on Nov 30, 2009 at 00:01

As a reminder that not everything here at Jee Labs is about JeeNodes, here’s a clock for the Arduino (which keeps track of time, even when not plugged in):

DSC_0815.jpg

This was built with an Arduino Duemilanove, a Plug Shield, an RTC Plug, and an LCD Plug piggy-backed onto a 2×16 character display.

Here’s the sketch:

Screen shot 2009-11-29 at 13.12.46.png

And here are links to the PortsLCD.h and PortsLCD.cpp source files.

But there could be some serious trouble ahead…

This code depends on an extended version of the LiquidCrystal library that comes with the Arduino IDE version 0017. Since I don’t want to modify that code, I had to use different names, so the PortsLCD.h/.cpp files use the following class names:

  • LiquidCrystalBase – an abstract class containing all the generic LCD code from the original class
  • LiquidCrystalPins – this is for use with plain pin connections, as before
  • LiquidCrystalPort – this uses bit-banged I2C with one of 4 JeeNode ports
  • LiquidCrystalI2C – this is for use with hardware I2C, using the Wire library

These names are slightly different from the previous ports-only version, btw.

So PortsLCD is a library which does everything the original did, and more. The flexibility is that you can write your sketches with this and then use any type of LCD you like with it, by just changing a single line of code. And if you’ve got a different hardware hookup, a new class can be added for it based on this same code, so that again your sketch only needs to change a single line of code to use it.

So far, so good. This is the benefit of object-oriented code and polymorphism.

But there is a price, due to the way the Arduino IDE does things: even if you don’t use the Wire library, you’ll need to include it in your sketch! For similar reasons, I’ve been forced to include the RF12 driver in all my demo sketches, even those that don’t use it. Leaving it out leads to build errors and prevents uploading.

This is not a C/C++ issue, the gcc compiler is actually quite smart about leaving out things which are never used. No – this complication is caused by the way in which the Arduino IDE tries to do some clever things to simplify naive use of libraries. I’ll go into this in another post – it is not a show stopper yet, but it may become one as I start combining more features and code, since memory on an ATmega is quite limited.

To put it bluntly: the way the Arduino IDE currently deals with libraries is a ticking time bomb…


JeeLink flash memory

In Hardware, Software on Nov 29, 2009 at 00:01

The recently updated version of RF12demo automatically logs all incoming data to the 1 Mbyte flash memory in a JeeLink. It looks like this is starting to work fairly well – I’ve been leaving a JeeLink on for over a week now, picking up packets from a number of different sources around here. It’s still only 80% full:

Screen shot 2009-11-28 at 12.34.27.png

So the JeeLink is now at page 3102 of 3840. Here are the first and last lines from the dump command:

Screen shot 2009-11-28 at 12.36.37.png

…

Screen shot 2009-11-28 at 12.38.30.png

The first two numbers after the colon are the reboot count and the time in seconds since reboot. There have been a bunch of reboots (most of them caused by me connecting to the JeeLink via a terminal program to see how it’s doing). The last run has been going for almost 24 hours.

Here are the first and last lines of the actual stored data replay:

Screen shot 2009-11-28 at 12.42.30.png

…

Screen shot 2009-11-28 at 12.45.23.png

Again, the first two numbers on each replayed packet are reset count and seconds since reboot. The rest is the packet data itself.

That’s quite a bit of stored data – almost 84,000 lines of text. It took over 7 minutes to get it all across USB at 57600 baud. Each received packet of data uses about 10 bytes of flash memory on average, in this example.

These results are better than I expected. Even with much more data coming in, it should be possible to leave the PC turned off over say the weekend, with the JeeLink collecting all incoming data and then replaying it when the PC comes back on. The only requirement is a powered USB socket.

Good. My home monitoring network won’t need a PC or server to be constantly on, since I don’t intend to set up around-the-clock internet access to this info – unless perhaps while away from home.


Arduino?

In AVR, Hardware, Software on Nov 23, 2009 at 00:01

What is it? Hype? Hobby? Hacker stuff? Here is the best overall introduction I’ve seen so far – by Dave Jones. It’s 19 minutes – plenty of time to get used to his accent :)

(view directly on YouTube)

I think he really touches on all the important aspects and inevitable trade-offs.

Me, I use the Arduino bootloader for JeeNodes and JeeLinks all the time, and the Arduino IDE to compile and upload stuff to them, as well as the serial console in many cases. I do use my own editor environment – which is easy to do once you disable Arduino’s built-in one (this is not well documented, Google for “Arduino external editor”). So for me the Arduino is really the bootloader, plus the IDE just as compile/upload system.

As Dave points out, the Arduino is a wrapper around the avr-gcc compiler + avrdude in combination with a convenient USB-based upload/console/power hookup. The rest is libraries, conventions, a Java based IDE (based on Wiring), and optionally a Java-based PC-side front end called Processing.

On the embedded software side, it’s really standard full-scale C and C++.

Which is great, IMO. I can keep going with the Arduino-compatible JeeNodes and JeeLinks, and their built-in wireless, port conventions, 3.3V operation, and all the add-on plugs – knowing that much of this will work fine with as well as without all the stuff going on in the Arduino ecosystem right now.


OOK unit

In AVR, Hardware, Software on Nov 21, 2009 at 00:01

Here’s an ELV 868 MHz OOK receiver, connected to the new JeeNode USB:

DSC_0771.jpg

Note that this plug can be used for any of the four ports, simply by plugging this thing in one of the four possible orientations. In this case, it’s hooked up to port 1.

The receiver is mounted on a JeePlug for stability:

DSC_0768.jpg

There really are only three wires: GND, +3V, and the received bit stream, which is tied to the AIO pin:

DSC_0769.jpg

You can clearly see the built-in pcb antenna on these last two pictures.

The code for this is similar to the one used in an earlier post and basically runs a couple of bit decoders in parallel to recognize FS20, (K)S300, and EM10 commands. It’s available here. Sample output:

Screen shot 2009-11-19 at 15.02.31.png

Or have a look at the OOK relay, with which this plug will also work.

I really need to clean up and merge the different OOK decoding sketches – they all have slightly different capabilities…

One thing which would be very nice to do with this unit, is a general sketch which reports incoming OOK packets but which also uses the OOK sending capabilities of the RF12 driver to send out such packets, to control FS20 devices for example. That mould make this a general-purpose bi-directional 868 MHz OOK unit, connected via USB and controllable via simple serial-port commands.

I’ll punt for now. Don’ have time to go into this. Or perhaps I should say… exercise left for the reader :)

And maybe one day we’ll get OOK input going without extra hardware ?

PS. Good news: everything is now in for the JeeNode USB, so I’ve started shipping them.


New RF12demo

In AVR, Software on Nov 19, 2009 at 00:01

An updated version of the RF12demo sketch has been checked in. Also available as ZIP.

This adds preliminary support for the 1 Mbyte dataflash memory present in the JeeLink v2.

The idea is very simple: the RF12demo app reports all incoming packets as before, but now also stores all valid packets in flash memory, if present. This is done by collecting packets into a 248-byte RAM buffer, and saving it to flash whenever it fills up. Small portions at the high and low end of flash memory are reserved for future use, the remainder is treated as a circular buffer of 256-byte pages – 3808 to be exact (i.e. 952 Kbyte).

The RF12demo commands have been extended accordingly, here is the startup screen on a JeeLink v2:

Screen shot 2009-11-18 at 18.33.06.png

There is a “DF” line at the top which shows that dataflash memory was found, with page #41 written to last, and that we’re in the 3rd reboot since the flash was wiped. This unit picks up incoming data in group 5 @ 868 MHz.

The new “d” (dump) command reports which pages contain data:

Screen shot 2009-11-18 at 18.31.34.png

At this point, I did a reset of the JeeLink, and then left it on to collect some more data. Here are the last few lines of a subsequent dump:

Screen shot 2009-11-18 at 18.45.14.png

The new entries were added starting on page 42, and the clock started at zero again (there is no RTC). Some 248 seconds later, new entries were logged on page 43. The last number on these lines is a page checksum, btw.

The new “e” (erase) and “w” (wipe) commands require specific constant values as input arguments so that these destructive commands are not activated by a mere typo.

So far so good. The JeeLink can now keep track of what is coming in. But that’s just half the story. We also need to be able to get data out again, “replaying” the log from a certain point. This takes a bit more work.

Firstly, whenever a page gets saved to flash, something like this is sent out to USB:

Screen shot 2009-11-18 at 18.45.52.png

This means that page 44 with seqnum 3 was saved with some entries starting at time 505 (seconds since the JeeLink was powered up). Here are the last few lines of a dump made right after that save:

Screen shot 2009-11-18 at 18.46.18.png

The sequence number is an integer which increments on each reboot, and whenever the flash memory buffer wraps back to the first page. So the combination (seqnum,time) is unique and monotonically increasing, over time and across reboots.

On the PC side, these DF “packets” should be treated as readings and stored like any other incoming packet.

To get the saved entries out of the JeeLink, the PC has to send a “replay” command when it connects to it. This replay command must contain the last sequence number and time stored on the PC for the above DF packets. The JeeLink will look for the location of this page in flash memory (or the earliest one thereafter) and will then replay everything stored from there on.

This code is still work-in-progress, but the plan is to replay these packets as lines starting with “R seqnum time” instead of the usual “OK” prefix. Replay takes place at full speed, but depending on the amount of data stored this can still take a substantial amount of time.

After the requested data has been replayed, the JeeLink resumes its normal mode of reporting “OK” packets, “?” checksum errors, and “DF S” save commands.

I haven’t finished implementing the replay command yet, but storage of incoming packets seems to be working, so an updated version will be able to get at that stored data later.

One last point to note is that this storage implementation has built-in automatic “wear leveling” and hence creates no “hot spots”: there is no page in flash memory which gets written to more often than others, so this memory will be usable for at least its full 100,000 rated cycles. This is done by scanning the entire flash memory on startup to determine the last page written to, and by erasing 4 Kbyte blocks only when needed, ahead of the write pointer.

Total size of the new RF12demo is under 10 Kb, so there’s still lots of room left.

Update – several fixes and replay code added. Not 100% right yet, but all functions are implemented now.


JeeLink flash works

In Hardware, Software on Nov 9, 2009 at 00:01

Ok, the last hurdle has been passed. The new 1 Mbyte flash memory on the new JeeLink v2 works – I’ve been able to erase the chip, write data to it, and read it back.

Here is the main test code I used:

Screen shot 2009-11-05 at 00.26.33.png

And here’s the output:

Screen shot 2009-11-05 at 00.26.00.png

Some timing results can be gleaned from that output:

  • Erasing the entire chip takes 5.7 seconds (!)
  • Reading takes about 1.1 mSec per 256-byte page
  • Writing takes about 1.4 mSec per 256-byte page
  • Reading 8 bytes takes about 90 µSec

Good enough, but definitely not instant in the ATmega’s time scale.

This code needs to block-out interrupts while accessing flash memory because the radio can generate lots of interrupts which need to be serviced quickly over that same SPI bus. Some care is needed to give enough attention to the radio during long memory transfers – it looks doable.

Anyway, I’ve started assembling a bunch of these JLv2’s. Here’s the very first hand-soldered batch:

DSC_0741.jpg

At last!

,

Build and pinout errors on older posts

In AVR, Software on Nov 8, 2009 at 00:01

There has been an unfortunate dependency between the Ports library and the RF12 library for some time now, causing the Arduino IDE to generate errors such as these:

ide-errors.png

(etc…)

The workaround right now is to include both in your sketch, even if you only need one:

Screen shot 2009-11-04 at 10.50.42.png

I’ve been planning to do a major overhaul of the library and software in general, but until then this is the way to avoid those pesky errors. It probably affects quite a few sample sketches on this weblog.

Another thing to watch out for (thanks, Ian!) is that some older examples on this weblog use the JeeNode v2 or even v1, which have a different pinout. To uses those examples with the latest JeeNodes, you have to swap pins 4 and 5, i.e. +3V and AIO.

Please let me know when examples from earlier weblog posts don’t work as described. A small fix and note added to these posts might be all that is needed!


LCD Plug

In AVR, Hardware, Software on Nov 1, 2009 at 00:01

Here’s the new LCD Plug:

DSC_0722.jpg

It’s a little piggy-back board for standard LCD modules with 16-pin connectors. The middle 4 pins are not connected, so that two 6-pin male headers are in fact enough to hook this board up to the LCD.

Here is a setup using a 2×16 character display:

DSC_0721.jpg

The interface uses I2C, so this “plug” can be daisy-chained like all the others. The plug uses an I2C expander chip with a fixed I2C address of 0×24. There are two jumpers to select the voltage level for driving the logic supply and the backlight, respectively. The backlight can be turned on and off under software control.

A generalized version of the LiquidCrystal library included with the Ardiuno IDE 0017 is used to redirect the actual interface through a bit-banged I2C bus, using the Ports library. This was described in an earlier post. Details about how this code works are also available, see this post.

The latest Ports library now includes the new code, including the “lcd_demo” example as shown above:

Screen shot 2009-10-31 at 16.06.14.png

The only difference with the LiquidCrystal example is the definition of port 1 as I2C bus, and defining the “lcd” object to connect to the display via this I2C bus.

,

Updated RF12demo

In AVR, Software on Oct 29, 2009 at 00:01

The RF12demo software which comes pre-loaded on all JeeNodes and JeeLinks has been extended a bit:

Picture 1.png

The new commands are:

  • l – to turn the activity LED on or off, if present
  • f – send a FS20 command on the 868 MHz band
  • k – send a KAKU command on the 434 MHz band

The new FS20 and KAKU commands were added so that a JeeLink can be used out-of-the-box to control the commercially available remote switches of these types. The updated demo is included with all new JeeNodes and JeeLinks from now on.

For example, to turn on channel 1 of a FS20 unit with housecode 0×1234, you can type in “18,52,1,17f” – i.e. the house code in bytes, the channel, and 17, which is the “on” command. Likewise, to turn on channel 1 of group B on KAKU, type “2,1,1k”.

The KAKU transmission range is not very high because the RFM12B radio used is tuned for the 868 MHz band, but even more so because the attached wire antenna is completely wrong for use at 434 MHz. If you want to use this for controlling KAKU devices and don’t get enough range, try extending the antenna wire to around 17 cm, i.e. double its current length. You can just attach an extra piece of wire to the end.

Which – unfortunately – is going to substantially reduce the range at 868 Mhz… so if you really want to use both frequency bands, you’ll have to use two JeeNodes or JeeLinks, each with their own properly sized wire antennas.

For more info about FS20, see these posts on the weblog. For KAKU, see these.


Monitoring failure?

In Hardware, Software on Oct 23, 2009 at 00:01

Yesterday, the Jee Labs energy monitoring system stopped working:

Picture 2.png

A few hours went by before I found out, because I’m not constantly checking these graphs nowadays.

Couldn’t figure out why, so I reset the sending node next to the electricty/gas meters, and shrugged it off as a fluke because it all started working again.

A few hours later, after dinner, we find out that there’s water dripping down right next to that JeeNode, but more importantly the whole meter closet is soaking wet in some places. Whoops… not good!

Turns out that there was a leak in the kitchen drain right above that meter closet. Fortunately, the problem was easy to identify. And didn’t lead to fireworks.

That was around 8 PM. Only this time the JeeNode is also soaking wet – free-hanging constructions are not such a good idea after all, I guess. Unplug. Wipe off the water. Let it dry for a few hours. Plug it back in.

OK, the energy monitoring setup is working again. Tomorrow, I’ll fix the drain pipe – properly.

The drawback of tinkering is that you always blame your own stuff when there is a failure. But in this case, it was trying to tell me about a problem coming from elsewhere…

,

OOK packet decoding

In Software on Oct 22, 2009 at 00:01

The OOK packets are now collected and stored in the JeeMon database.

Here are the first few readings from the barometric pressure sensor inside the OOK relay, as received by the central node and graphed by JeeMon:

Picture 3.png

Here is the outside temperature from a WS300 weather station on the roof:

Picture 1.png

JeeMon is written in Tcl – the following new code was added to process packets from the OOK relay and get the different parameters stored:

Picture 2.png

The RELAY decoder processes incoming data, separating the bytes into one or more messages and calling the corresponding type-specifc decoders. The RF868 decoder merely reformats incoming data to the format used by another decoder called “ALT” and which was already present in JeeMon. The RF434 decoder is just a stub for now. Note also that the DCF decoder logs the current time – it may seem odd, but storing the received time along with the current system time can be used later to detect local clock drift and compensate for it.

It’s perhaps all mumbo-jumbo for now, I just wanted to illustrate how messages get routed inside JeeMon, and that it takes relatively little logic to deal with new data sources.

This would really benefit from a flexible plug-in system, so that this sort of logic does not have to be added in JeeMon itself but can be picked up on-the-fly. Then it would be easy to add new nodes such as the OOK relay and have JeeMon automatically incorporate the corresponding logic to decode, process, and store their data packets.

,

OOK relay, revisited

In AVR, Software on Oct 21, 2009 at 00:01

The OOK relay to pass on 433 and 868 MHz signals from other devices has been extended:

DSC_0665.jpg

Spaghetti!

The following functions are now included:

  • 433 MHz signals from KAKU transmitters
  • 868 MHz signals from FS20 and WS300 devices
  • the time signal from the DCF77 atomic clock
  • pressure + temperature from a BMP085 sensor

Each of these functions is optional – if the hardware is not attached, it simply won’t be reported.

The BMP085 sensor in the top of the breadboard could have been plugged directly into port 3, but it’s mounted on a tiny board with the old JeeNode v2 pinout so I had to re-wire those pins.

Sample output:

Picture 1.png

There are still a few unused I/O lines, and additional I2C devices can be connected to port 3. Room to expand.

The source code for this setup is available here. It compiles to 10706 bytes, so there is plenty of room for adding more functionality.

This code includes logic for sending out the collected data to another JeeNode or JeeLink with acknowledgements, but I’m working on a refinement so it gracefully stops retransmitting if no receiver is sending back any acks.

As it is, this code will forever resend its data every 3 seconds until an ack is received, but this doesn’t scale well: all the “rooms” nodes do the same, so when the central receiver is unreachable every node will keep pumping out packets while trying to get an ack. It would be better for nodes to only re-send up to 8 times – and if no ack ever comes in, to disable retransmission so that only the initial sends remain.

I’m also still looking for a way to properly mount all this into an enclosure. There is some interference between the different radio’s, maybe all in one very long thin strip to maximize the distances between them?

,

Analog Plug

In AVR, Hardware, Software on Sep 30, 2009 at 00:01

The second plug panel has arrived. I’ll document my test results in the coming days.

First, the Analog Plug – an 8-channel 12-bit ADC connected via I2C:

DSC_0545.jpg

It’s based on the ADS7828 chip. Here’s a demo sketch to read it out:

Picture 3.png

I hooked it up using various gimmicks lying around here – this ties a trim pot to channel 4, with range 0 .. 3.3V on the wiper:

DSC_0544.jpg

As you can see, the Breadboard Connector can be used to hook up to 8 of the 12 pins of this plug.

Here’s some sample output:

Picture 1.png

I slowly turned the wiper as you can see. It stops at 4095 counts, which represents the 2.5V of its internal reference. Appears to work fine in this test setup at the maximum I2C rate, somewhere over 1 MHz. The readings didn’t change by more than one count when touching various parts of the circuit, so as first impression it looks like it’s pretty stable.

This plug has two solder jumpers, to configure its I2C address to 0×48 .. 0×4B, which in hindsight is a bit overkill. Only minor nit is that I mis-labeled the A0 jumper – as shown in the picture, the address ends up being 0×49 i.s.o. 0×48. Oh well, a small silkscreen fix will resolve that later.

So there you have it – eight 3.5 digit voltmeters on one tiny blue-and-gold plug :)


LCD display via I2C

In AVR, Software on Sep 28, 2009 at 00:01

Ok, it’s working – always takes longer than expected, especially if you mess up in both hardware and software!

DSC_0534.jpg

There’s a miniature trim pot on the left to set the contrast level. I haven’t yet added the transistor to turn the backlight on and off.

Note that with an MCP23017 chip there are 9 I/O lines available for other purposes. On the upcoming LCD Plug, I’ll probably use the smaller MCP23008 instead.

This particular setup works at the maximum I2C bus speed supported by bit-banging, which should be well over 1 MHz – I haven’t measured it. If the bus is long or is shared with slower peripherals it can be lowered to 400 KHz or 100 KHz by specifying an extra argument in the PortI2C constructor.

The sample code is available here. It will be integrated with the Ports library later.

PS. There are a few days left for the special offer in the shop. If you’ve ordered before or have participated on this weblog or in the forum, you can get 20% off until the end of September. Email me for a discount code.

,

Generalized LiquidCrystal library

In AVR, Software on Sep 26, 2009 at 00:01

Let’s dive into some C++ code for a change…

The Arduino version 0017 IDE has a nice library for driving standard character LCD’s, called… “LiquidCrystal”. The nice thing is that it mixes in the print() / println() functionality which is also used in the Serial library.

That makes it very easy to change a sketch to use either the serial port or the LCD screen to display results. Here’s how – let’s say you used this code in your sketch to print a value to the serial port:

Picture 2.png

The first thing to do is to generalize this slightly:

Picture 3.png

Functionally, nothing has changed. That’s the whole point, because you can develop and extend the sketch while testing everything via the serial port, as usual. Just use “Output” everywhere.

To change the output to the LCD, replace that “#define” line by the following lines of code:

Picture 4.png

This adds a bit of logic to easily switch between serial and LCD, without having to change a thing in the rest of the sketch. The “(2, 3, 4, 5, 6, 7)” parameters have to be set to the pin numbers actually used for your specific setup, of course.

So far, so good, although none of this applies to the I2C-connected LCD I’m working on. But by extending and re-organizing it a bit, it is possible to re-use most of the code of the original LiquidCrystal library.

The first thing I had to do was to rip apart the general and the pin-specific logic. That’s where C++’s abstract base classes and virtual member functions come in. I created a new “LiquidCrystalBase” class which has all the original code except for the pin-specific parts:

FlySketchExport.png

The “virtual … =0″ is C++’s funny way of saying it doesn’t need definitions for these three functions yet. The result is an incomplete class – you can’t create objects of type LiquidCrystalBase, you can only derive other subclasses from it. Tricky stuff, but bear with me…

Note that all the pin-specific member variables have been removed as well.

The next step is to re-define the original LiquidCrystal class in terms of LiquidCrystalBase:

Picture 5.png

If you compare this to the original code, you’ll see that most functions member definitions are missing. That’s because they have already been defined in what is now LiquidCrystalBase, and the new LiquidCrystal inherits everything from it.

What LiquidCrystal does add, is a definition and implementation for each of the virtual config(), send(), and write4bits() members. It is sort of “filling in” the missing functionality. So the result is… a LiquidCrystal class which does exactly the same as the original.

That seems pretty useless, but it isn’t – what we now have is a generic part and a specialized part. Which means it is now very easy to implement a variant which works exactly like the original, but going through the I2C port interface instead:

Picture 6.png

Nice. Only a little bit of new code was needed to create a completely different hardware interface which is fully compatible with the original LiquidCrystal class.

Here’s the original example, using this new LCD-via-I2C library:

Picture 8.png

Note that the I2C port is available as “myI2C” and can be shared with other devices by defining them here as well. You could even daisy-chain multiple LCD displays on the same port if you wanted to.

That’s all there is to it. The sketch code itself remains the same. Welcome to the world of abstraction, à la C++ !

Update – the final version and source code are described in this followup post.

,

Boot loader problems

In AVR, Software on Sep 11, 2009 at 00:01

Whoops – I messed up with several packages I’ve sent out to people. Ouch.

The gory details are in the discussion forum, but here’s the short version:

  • I’m shipping ATmega’s pre-programmed with the Arduino bootstrap and the RF12demo sketch. Mighty convenient, since it can get you going a lot quicker.
  • And to simplify my work I, ehm… sort of tried to automate it a bit.
  • So I burn the bootloader with the Arduino IDE and the burn the RF12demo sketch using “avrdude” from the command line.
  • Cool. Except that the second step undid the work of the first step… doh!
  • So I ended with ATmega328’s which have the RF12demo but no bootstrap loader.

It turns out that avrdude erases a chip by default when re-programming it. So the bootloader, which is loaded in high-mem, got cleared when storing the demo in low-mem. And of course that doesn’t show in a quick test: the demo on the chip, which I always check, works just fine. Which is why I sent out a few bad JeeNode kits and JeeLinks.

All it took to fix this was one lousy little “-D” command option for avrdude!

So now I’ve decided to automate all the way, and burn the fuses, the bootloader, and the demo sketch all in one go – from the command line.

Here’s what it takes – I have these files in one directory:

Picture 1.png

The hex files were copied from the Arduino “bootstrap” area and the RF12demo “applet” build area, respectively.

The makefile contains these commands:

Picture 2.png

So all I need to do now is to plug in the ISP programmer and put a fresh ATmega chip in its socket, then type “make”. This works for both the JeeNode and the JeeLink chips.

Easy… once done right!

,

Sensor data coming in

In Software on Aug 25, 2009 at 00:01

There are now a few sensors installed around the house. The only inconvenience is having to run a low-voltage power line to these units. I’ve reduced the number of wall plugs (and the number of wall outlets “consumed” by them) by putting two sensors on a single power supply and placing them close to each other where possible, i.e. on two sides of a wall, overlooking adjacent rooms.

Also added a new graph page to the JeeMon web server to view all readings added to the database:

Picture 4.png

To give you an idea of what’s involved in producing this graph, here’s the “try-rooms.html” definition file for it:

Picture 3.png

It’s written in the Tcl language, using my own “rig” template mechanism to generate HTML pages. The actual graph is drawn in JavaScript by the “Flot” package.

,

Reflow controller

In AVR, Software on Jul 14, 2009 at 00:01

Yesterday, I presented my new reflow system. Here’s the temperature graph from a sample run:

graph.png

The purple steps are the different phases:

  • Preheat (50) is whatever time it takes to reach 140°C.
  • Soak (100) is a 30-second temperature ramp up to 170°C.
  • Dwell (150) is defined as whatever time it takes to reach 220°C.
  • Reflow (200) is defined as 20 seconds at 220°C.
  • Cool down (250) is where the buzzer goes off and I open the lid.

The green bands indicate when the heater is on. The blue line is the target temperature which the system is trying to reach. The red line is the actual temperature. As you can see, the heater stops well before target temperatures are reached, and does a pretty good job of ending up in the desired range.

All in all, this appears to be ok. The 700-watt heater isn’t quite hot enough to ramp up 3°C/sec, more like 2°C/sec at best. Since the grill isn’t heating up quickly enough, the soak phase ends up taking 60 seconds instead of the planned 30, so the target stays pinned at 170°C a bit longer. This could probably be shortened by aiming directly for 170°C, but I don’t think this phase is critical. My only concerns are that it took about 80 sec to “dwell” from 170°C to 220°C and that the system is above 200°C for some 60 sec. This was shorter in other trials, but as you can see the heater is turning on a few times to nudge the system towards 220°C as things slow down a bit.

This plot was produced by a really nice free package for the Mac, called Plot (how original). The readings were obtained by logging the sketch output from the USB port and reformatting it to tab-separated numeric values.

,

Reflow temperature

In Hardware, Software on Jul 12, 2009 at 00:01

For my upcoming reflow experiments, I’d like to use a JeeNode to track the temperature and perhaps also control the grill. As the saying goes… when you have a hammer JeeNode, everything looks like a nail sketch.

So I hooked up a 10 KΩ NTC resistor rated up to 300°C as voltage divider with a 1 KΩ resistor pull-up. Bought a cheap and probably not-so-accurate multi-meter with thermocouple-based temperature readout. And then the fun starts – turned my grill on and watched it heat up and then cool down, while jotting down the “readings” from this “experiment”. I felt like a high-school kid in physics class again :)

The results:

Picture 1.png

Not bad. The readout is a bit coarse at the higher end, but I think it’s accurate enough for reflow soldering in the 170° .. 240° range. The grill goes up to 260°C from what I can tell.

But I didn’t want a lookup table, so I went looking for a polynomial least-squares regression fitter on the web and found this one. Great: copy and paste the 18 heat-up readings, then try out a few polynomes – turns out that a 7th degree polynome fits well, matching the measured data within a few tenths of a degree.

And then I found an even better site called ZunZun – it lets you fit polynomes and graph the results and generate the C++ code. No need to copy lots of coefficients!

Here’s what I ended up with, using a “Lowest sum of orthogonal distance (ODR)” fit:

Picture 1.png

Corresponding C++ code:

Picture 4.png

To be able to configure this thing, it would be nice to be able to enter degrees. That’s easy to do by also fitting the inverse function:

Picture 7.png

Corresponding C++ code for the inverse:

Picture 6.png

Great. Onwards!

,

IO Expander

In AVR, Hardware, Software on Jul 7, 2009 at 00:01

Here’s a way to expand the number of digital I/O lines on a port:

3686066291_f856efcb63_o
This is a JeePlug filled to the rim with tiny components and connectors. It’s based on a PCA8574A I2C 8-bit I/O expander. Each of 8 pins can be used either as inputs or as “mostly” open collector outputs. See the datasheet for details.

Here’s the bottom side of the plug:

3686871488_b2366a7459_o

Crowded indeed!

And here’s a demo sketch called “expander” which blinks the eight LEDs connected between PWR and the respective I/O pins:

Picture 3.png

As with yesterday’s example, this is running the I2C at maximum bit-banging speed, and seems to work fine. It’s used to make 8 leds blink:

3686871410_0e325c3a2b_o

The connector block on this plug has 8 sets of three connections: GND, PWR, I/O. Note that PWR is connected, not +3.3V. The reason for this is that I’d like to try driving a bunch of servos from this plug one day – it’ll load the JeeNode down a bit but it should be feasible.

The “expander” sketch has been added to the Ports library and is available as ZIP file and in subversion.

,

External memory

In AVR, Hardware, Software on Jul 6, 2009 at 00:01

For one of the projects in the Jee Labs, I needed a bit more “permanent” memory than the ATmega’s EEPROM or even flash could provide. Here’s what I came up with:

3683813851_d524fedae2_o

That’s a 64 Kbyte AT24C512B serial EEPROM memory which talks via I2C.

It takes only 4 wires to hook this thing up:

3684626986_c2c8f1119c_o

And this is the mess you get in when you don’t have the right connectors and need to deal with older boards:

3683813971_e00e924cb0_o

That’s the “regret plug” mentioned a few days back, to turn the downward pointing male header into an upward pointing female header, plus a “cross-over plug” to let me wire this thing up for the new JeeNode v3 pinout, yet use it on a (minimally populated) v2 board. Yuck!

Lesson: think about headers before deciding how to solder them on …

All future JeeNodes will be supplied with 6-pin female headers for the ports. Soldering them on pointing up is probably most convenient: it lets you stick little LEDs and wires in there, and JeePlugs can be placed pointing upright – such as this memory.

Back to the memory expansion. Here’s a test sketch:

Picture 5.png

Sample output:

Picture 4.png

As you can see, the count did not start with 255, as it would have if the EEPROM had still been empty. The JeeNode was powered off in between to demonstrate data retention.

This example code runs at full bit-banging speed – seems to work just fine.

The “eemem” demo sketch has been added to the Ports library and is available in this ZIP file as well as here.

There you go … a whopping half million extra bits of permanent storage :)

, ,

Ligthy power save

In AVR, Software on Jul 5, 2009 at 00:01

Getting the power consumption down of yesterday’s “Lighty” example turns out to be quite a challenge.

One thing to do is to separate out the logic for enabling the different sensors, and extending it to also support disabling all of them:

Picture 3.png

Powering down completely works best when all internal peripherals are also turned off, as implemented in the following code:

Picture 4.png

Now the trick is to enable some interrupt source to take us out of this deep sleep phase again. This could be the ATmega watchdog, but the radio watchdog uses even less power, so here’s how to stay off for about 4 seconds:

Picture 5.png

So far so good. This disables all power consuming sensors and internal circuits, preps the radio to wake us up in 4 seconds, powers off, and then reverses the whole process.

Bug there is a bug in all this – somewhere… From some earlier experiments, I would have expected to see a power draw of a few microAmps with this code. But for some reason, it never drops below 2.7 mA, i.e. still “burning” 1/10th of full power!

I haven’t been able to figure out yet where these milliamps are going :(

For the sake of argument, let’s assume this works properly. Then the next problem will come up, which is that measuring and sending packets every 4 seconds drains more power than I’d like to. It takes several milliseconds to measure all readings and send out a packet. But who needs readings every 4 seconds?

So the solution to this is to just sleep a bit longer, using the 4-sec wakeups to quickly read-out some sensors, and calculate their averages. Here’s is the final loop of the power-saving “LightySave” sketch:

Picture 8.png

This will integrate readings for 75x 4 seconds, i.e. 5 minutes, and then send out a single packet. Note how the power-hungry radio module is only enabled at the very last moment. All we have to do is make sure it’s ready to send, then send one packet, then wait again until the send is complete.

Then the loop restarts, sleeping in low-power mode, etc.

Only issue is to find out where the 2.6 mA are going! I’ll try to figure this out, and will post here once fixed …

,

Lighty

In AVR, Software on Jul 4, 2009 at 00:01

Let’s call yesterday’s mystery JeeNode “Lighty” from now on. As promised, here is the first sketch:

Picture 3.png

LightySerial reads out several sensor values each second and dumps the results to the serial port. Plain and simple – using the Ports library for most pins and Arduino’s analogRead() for reading out voltage levels on A4 and A5.

Sample output:

Picture 4.png

FYI, all the Lighty sketches are available for download as a ZIP file.

Now let’s turn that into a wireless version using two JeeNodes. The first step is to separate the measurement and reporting sides of things.

Here is the main code of LightyTransmit, causing it to send out its readings once a second:

Picture 8.png

It puts all values into a structure defined as follows:

Picture 6.png

… and then sends that off to the broadcast address, i.e. whoever is listening. No acknowledgements are used. If the packet arrives, great, if not then the next one will.

The receiver then takes it from there, here is the entire LightyReceive sketch:

Picture 7.png

Note that the Ports library is not needed here since the receiving JeeNode does not use ports, it merely picks up packets and reports them on the serial line.

That’s basically it. Each of these sketches compiles to some 3 .. 4 Kb code.

We’ve got ourselves a Wireless Sensor Network!

LightyTransmit uses about 24 mA. Tomorrow, I’ll go into reducing the power drain, because Lighty’s tiny 20 mAh LiPo battery won’t even last an hour this way.


A tale of stand-alone builds

In AVR, Software on Jun 10, 2009 at 00:01

The JeeBot Duo runs off a Baby Orangutan controller by Pololu, which isn’t entirely compatible with the Arduino IDE. Well, it can be made to work with it but in this case it looks like just using avr-gcc and avrdude directly from a Makefile would be just as simple.

Boy, was I wrong.

I’ve spent nearly two days trying to understand why this thing wasn’t working. Or rather, it was. Sometimes. But with resets all the time, and the watchdog being tripped even though I never enabled it.

One problem is that the BOC has no other signaling on board than a single LED. So first thing was to set up serial communication. Well, that worked. But then it didn’t. And then things got really totally erratic…

The problem turned out to be in the highlighted portion of this Makefile:

Picture 2.png

Without the $(CFLAGS) on the linker line, everything   s o r t   o f   works.

Until you start using RAM for data, such as statics/externals and strings. Or C++ classes inside the Pololu libraries for example.

Then the system goes berzerk. Because the linker wrongly assumes that RAM starts at 0×0060. So strings and data variables end up being written into the extended I/O area. Want to know what’s at address 0×0060? The watchdog control register.

Anyway. It took me many hours of complete despair to figure this one out. Salvation came from “avr-objdump -h”, which clearly showed that the “.data” segment was in the wrong place compared to an Arduino sketch. With many thanks to Ben @ Pololu for nudging me towards the solution and for patiently responding to my emails, some of which must have been pretty weird…

I think that what baffled me most of all was the fact that the simplest bits of code worked just fine. You can get a lot of little tests going without ever using strings or allocating static variables. And the failures being random resets (from the watchdog) kept me looking for hardware problems – silly me.

At one point, I had figured out the incorrect starting RAM address and added this linker option: “-Wl,-Tdata,0×800100″. Unsure why it had to be specified but it got past the most erratic behavior.

But then interrupts wouldn’t work. Disassembly (with “avr-objdump -D”) showed that the generated code was using an incorrect set of vectors. And then it dawned on me: the $(CFLAGS) includes “-mmcu=atmega328p” which the linker needs to put things in the right place and to use the right vectors.

Doh. Totally obvious in hindsight.

Now I can finally generate and upload code without having to go through the Arduino IDE.


Reliable RF12 communication – draft

In Software on Jun 5, 2009 at 00:01

I’m currently adding a reliable communications option to the RF12 driver. This will deal with two important communication errors: lost packets (the receiver did not get the packet) and lost acknowledgements (the sender doesn’t know the packet has arrived). Both involve timers and re-transmissions, something which the current driver doesn’t provide for.

The current driver API is not affected, these changes only add functions. Here is how they should be called:

Picture 3.png

There are two parts, reception and transmission. Reception occurs at unpredictable times, which is why rf12_recvDone() has to be called fairly often. It’s very quick when there is nothing to do, so calling it inside loops is ok and cheap. The receiver code also handles all acknowledgements and other events, so it must be called even if you only intend to send data out. Also, you must always end with a call to rf12_sendReply() when accepting the data, even if the reply is empty.

Transmission can be started by you whenever you want, but you can’t flood the driver with sends, you have to wait until it has finished the previous one. This is why there is a check on “rf12_todo”. If it is ≥ 0, you have to either wait and re-check until it becomes negative, or give up this send and try sending new data later on. Note that sends can slow down indefinitely, for example when the receiver is out of range or even turned off completely.

There is a lot to say about how to design your communications structure, all way beyond the scope of this post. It becomes considerably simpler if you can set up things such that dropping packets is no big deal. By sending all values each time, for example. Or by adding a sequence number such that lost packets are easily spotted. The fancy term for this is to make requests “idem-potent” – the HTTP protocol and web browsers normally work that way too, BTW.

To send data reliably, set the “ack” argument of rf12_sendData() to 1 and be prepared to accept occasional slow-downs. To send data on a best-effort basis, set it to 0.

The last important comment to make here is about buffer management. The RF12 driver only does a limited amount of buffering – this has two important implications:

With received data, you may only access the incoming packet in rf12_buf between the rf12_recvDone() and rf12_sendReply() calls. At all other times, rf12_buf, rf12_crc, and rf12_len are indeterminate.

The other implication is that you have to provide a pointer to your own buffer with data to rf12_sendData(), and this buffer must remain available and unchanged after rf12_sendData() returns. The buffer is given back, i.e. available for your re-use, when rf12_todo becomes negative again. You can use multiple buffers and send a different one each time, as long as the current buffer being sent adheres to these rules.

Limited-bandwidth wireless communication such as with the RFM12B on the 433/868/915 MHz bands is more effective at best-effort packet delivery than at getting each and every packet across with 100% reliability. The RF12 driver is small and uses few resources, but it does come with some trade-offs. With the additions described above, you’ll be able to set up a reliable data stream when needed. I’ll report here when these additions are ready for use.

,

Telemetry

In AVR, Software on Jun 4, 2009 at 00:01

The JeeBot now has telemetry functionality, continuously sending real-time process data by wireless for analysis and display on a desktop PC.

It was quite simple to add this: send a buffer with the latest data every 200 msec, using the RF12 driver. No acks, no checking – the receiver grabs as much as it can off the air, and simply ignores invalid and lost packets.

Here is the essence of the code added to the JeeBot loop, which runs every 20 ms – in lock-step with the pulses it generates to control the servos:

Picture 2.png

A second JeeNode acts as receiver to pick up these data packets and transfer them to the serial port – here’s the complete sketch:

Picture 3.png

Sample output:

Picture 1.png

These values show what the JeeBot is doing while running on its own internal batteries … i.e. wireless telemetry in practice!

The next step will be to analyze and visualize this data to help me properly tune the PID control parameters. That’s more work.

,

Gyro sensor

In AVR, Hardware, Software on May 25, 2009 at 00:01

Here’s the LISY300AL gyro sensor, mounted on a Sparkfun breakout board:

Gyro sensor

The sensor is not extremely sensitive, but works at 3.3 V. It has a single analog output, so readout is trivial:

Picture 1.png

Sample output:

Picture 2.png

The readout changes as this setup is rotated around the horizontal plane. Looks like the self-test (pin 5) to ground can be omitted, btw.

One (tiny!) step closer to a self-balancing bot…

Update – the Segwii website is a fantastic resource on how to accomplish self-balancing in a fairly similar setup.

,

New RF12 sleep support

In AVR, Software on May 23, 2009 at 00:01

The RF12 driver has been extended to support power down as well as wake-up.

This example uses the RFM12B wake-up timer to generate an interrupt in about 1 second, then powers down, then leaves the radio off once power is back up:

Picture 3.png

That last call is important, because otherwise if the system happens to power up due to some other trigger, then the wake-up interrupt may happen later and cause problems. This way, regardless of why the system came back up, we make sure that the RFM12B wake-up interrupt stays off. The radio is still off at this point.

The argument to rf12_sleep() is a time value, in units of 32 ms. The maximum value is 127, i.e. 4 about seconds between wake-ups. If you need longer sleep times, change the logic to go back to sleep right away until the proper amount of time has elapsed.

To turn the radio back on, use “rf12_sleep(-1)” to start listening to packets again after the next rf12_recvDone() poll. Since the RFM12B consumes a fair amount of power, it’s best to wait with this right until you’re ready to send out a packet.

Here is the new code, which is now included in the updated RF12 driver:

Picture 1.png

The RF12 driver software is available over here or as ZIP, this is an Arduino-compatible library.


Trying to receive OOK data

In AVR, Software on May 18, 2009 at 00:01

Here is an experiment to try and (ab)use the RFM12B module for OOK reception, such as sent out by a FS20 remote control. First the complete sketch:

Picture 2.png

The interesting bit is that this appears to work – the LED flashes when I press a button on the FS20 remote:

OOK receive trial

Since the signal is made to come out on the AIO port 1 pin (A0), this whole unit can be plugged into a second JeeNode running the OOK decoding code from an earlier post. Like this:

OOK receive trial

The reason for using two JeeNodes is that the one receiving the OOK signal is 100% tied up in a loop polling the status register. No problem – we simply upgrade to a dual-core setup :)

Unfortunately, the combination isn’t working yet. I’ll need to use the scope or logic analyzer to figure out what’s going on. Perhaps the RFM12B’s RSSI circuit is not fast enough to faithfully reproduce OOK transitions.

Oh, well.

,

Power consumption – more savings

In AVR, Hardware, Software on May 16, 2009 at 00:01

Let’s get that JeeNode power consumption down …

The no-savings baseline from two days back was 10 mA, with a few simple measures getting it down to 1.31 mA, i.e. waiting in idle mode with the pre-scaler set to 64 and turning the RFM12B clock off.

It turns out that the ATmega328 is quite a bit more power efficient in that mode, the same sketch consumes 0.67 mA, i.e. half that of an ATmega168. Great – that’s an easy gain. Unfortunately it doesn’t carry over as much to power down mode: 125 µA versus 137 µA. There is still some unexplained power use there – maybe some of the circuits need to be turned off explicitly, or some more output pins floated.

Anyway, the next idea to go beyond 1.31 mA is to use the watchdog timer for idling away in power down mode. The basic code structure will be as follows:

Picture 1.png

Ooh, wait – turns out that there is a simpler way, which doesn’t require changing the main application logic after all. We can let the watchdog generate an interrupt without resetting the system. Here’s the code I ended up with after some experimentation:

Picture 2.png

Note the added dummy watchdog interrupt routine. Without it, the code apparently enters a nasty infinite loop when the watchdog fires, which is hard to get out of – even after a hardware reset.

Usage: 139 µA while waiting – 99% of the time. Great.

The code can actually be streamlined a bit – simply leave the watchdog running to pull the processor out of power down mode once a second:

Picture 4.png

But there’s more. The ADC can be disabled to reduce the power drain even further:

Picture 5.png

Now usage drops to 24 µA while waiting. Cool.

One last change is to disable the brown-out detection, which in turn disables the internal voltage reference. Changing the high fuse from 0xDD to 0xDF drops total power usage to a diminutive 6.8 µA. Perfect.

So there you have it – 1500 times less power usage!

Update – see the discussion in this post for a change which gets this down to 3.7 µA …

Update 2 – changed the low fuse from 0xFF to 0xEE, i.e. 64 µs start-up i.s.o. 1+64 ms (ok for resonator, not ok for a crystal). This will help get the power up duration down once I start looking into it.

,

Power consumption – subtle bugs

In AVR, Software on May 15, 2009 at 00:01

Yesterday’s code has several bugs:

Picture 2.png

As already mentioned, a delay is needed to let the serial data leave the USART before the pre-scaler change messes up the clock and baudrate. But there are also two other serious problems:

  • the “limit = millis() + …” expression will overflow and fail (after about 49 days)
  • the “millis() == limit” condition can “miss a beat”, leading to an almost-infinite loop

Both bugs can be fixed by carefully rewriting the code:

Picture 1.png

The differences are subtle, but essential. Without it, the loop will get stuck within just a few seconds of starting the sketch. With the above fixes it’ll run properly forever… I think.

,

Power consumption – baseline

In AVR, Software on May 14, 2009 at 00:01

Ok, first step in lowering JeeNode power consumption is to establish a baseline.

Here is the original code, the first few lines of the loop that is – the rest is the actual sensor readout and calculations, which take about 10 milliseconds:

Picture 1.png

Usage: 9.80 mA (slightly different from yesterday’s post, hooked up to USB from now on).

First change is to make the delay loop explicit:

Picture 2.png

Usage: 10.05 mA – no idea why it’s slightly higher. Anyway 10 mA will be used as baseline – this is the current drawn before any power-saving measures are introduced.

First step: lower power use by putting the processor in idle mode while it’s waiting for the next readout time. Included “avr/sleep.h” and changed the loop to be as follows:

Picture 4.png

Usage: 5.58 mA – it works, cool!

Next idea is to lower the system frequency, but only while waiting – so that readout, processing, and serial I/O remain unaffected:

Picture 5.png

Usage: 1.99 mA – whee! But there is a small gotcha – the system now goes into slow mode before the serial port has finished transmitting. A “delay(3)” at the end (or start!) of the loop is needed to fix this.

For comparison: with just the pre-scaler and no idle mode, usage is 2.15 mA – so the bulk of the savings comes from slowing down the system clock. Idle mode saves roughly 10% at lower frequencies.

One more important saving is possible due to the onboard RFM12B radio module, which powers up with its 10 MHz crystal clock running. Adding a bit more code to access the radio and disable its clock lowers power usage from 1.99 mA to 1.31 mA.

So these three simple measures combined already lead to a 7.5-fold power reduction: “waiting” a bit more carefully and turning off the RFM12B. All without affecting the main application code in any way.

As for the bottom line: putting the processor in power down (from which this code won’t ever wake up) uses 137 µA – and removing the MPU chip brings it to 17 µA. Note that with 140 µA average usage, a JeeNode can run for two years on 3 standard AA batteries.

It ’s possible to reduce the power consumption from 1.31 mA to the µA range by powering down the processor and using the watchdog timer to wake it up – but that requires another set of changes to code and fuses – stay tuned…

Update – see this more recent post for additional details.


Bubba JeeMon back up

In Hardware, Software on May 12, 2009 at 00:01

The title of this post can be interpreted in two ways, both accurate: 1) the Bubba Two NAS from Excito was my back-up server for the MMnet1001 module, and 2) that Bubba box is now my main JeeMon server and back in the air with all the electricity / gas / weather monitoring working again, as well as the Pachube and Bwired feeds.

I will probably stick to the Bubba from now on. It’s running Debian Etch Linux, and there’s a JeeMon runtime for it so the switch-over was trivial.

bubba-two.jpg.jpeg

This NAS box consumes about 4 watt, which is less than the specs because it has been modded to use a 2.5″ SATA disk instead of the standard 3.5″ ones they normally ship. It’s also a 300+ Gb file server which has been running various periodic tasks for almost a year now. There’s in fact much more disk storage in the form of 3 additional USB disks, but these are normally powered down (via a KAKU switch).


New RF12 data packet I/O

In AVR, Software on May 10, 2009 at 00:01

Following up on yesterday’s new serial I/O layer, implemented on top of the RF12 driver, here’s another demo using the “<<” and “>>” operators to send data across. First the sample code:

Picture 5.png

Then the sample output:

Picture 3.png

Picture 4.png

(these are slightly older screenshots, not yet using “rf12_config()”)

The output is a bit mixed up because the nodes were not reset at the same time, but you can see the data coming in on both nodes. These are independent bi-directional transmissions, even though in this case similar data is being sent since the same demo is running on both nodes.

As with the “rf12serial” demo, there are still some serious limitations with this – I still need to debug and add more flow-/congestion-control logic in this new implementation (the code is in “RF12sio.cpp” inside the RF12 library).

Much of this sort of logic relies on timers: the “RF12sio.h” header defines a simple millisecond timer which is also available for general use.


New RF12 serial I/O layer

In AVR, Software on May 9, 2009 at 00:01

The new RF12 code is starting to work. I’ve added it to the RF12 library, along with a demo which sends characters across the wireless link in both directions. Here’s the code:

Picture 6.png

And here’s some sample output on two terminal screens:

Picture 1.png

Other side (the “transmittee” text is my typo):

Picture 2.png

There are still major flaws in the code:

  • it doesn’t properly pack/unpack small packets sent back-to-back
  • in fact, it chokes when too much data comes in
  • it seems to have trouble with the very first character sent

But it’s a start!


RF12 configuration in EEPROM

In AVR, Software on May 7, 2009 at 00:01

It’s becoming a bit tedious to set up and manage JeeNodes, each with their own RF12 configuration. Prompted by a recent request, I’ve created what I think is a much more practical configuration system using some of the ATmega’s built-in EEPROM.

The basic idea is to split the configuration and the setup code. The “RF12demo” sketch which is part of the RF12 library has been extended into a general EEPROM configuration utility. Here’s its new startup screen:

Picture 2.png

So now you can type commands on the serial port to adjust settings. These are stored in EEPROM along with a description string and a 16-bit CRC checksum. When launched later on, the last settings get re-used so this becomes a set-and-forget thing.

What’s more important though, is that the RF12 library has been extended with a new “rf12_config()” call which you can use instead of “rf12_initialize(…)”. It does a couple of things:

  • retrieve the settings from EEPROM and check that they are valid
  • if so, configure the RF12 hardware accordingly
  • display the current settings on the serial port

As a result, it’s now a lot simpler to set up a sketch with RF12 communication. All you need to do is upload and run the RF12demo once to configure things, then upload your own sketch and use “rf12_config()” to re-use the same configuration.

Here’s an example / skeleton:

Picture 4.png

The output from this sketch is:

Picture 3.png

It’s a bit cryptic to keep the code overhead small, but you can see that I configured this unit to act as node “Z” (i.e. 26), using network group 212, and operating in the 433 MHz band.

The RF12 configuration data is stored in EEPROM addresses 0×20..0×3F (this can be changed in “RF12.h” if it interferes with your own EEPROM use).

,

Yet another PIR

In AVR, Software on May 1, 2009 at 00:01

Found yet another passive infrared module for around €10, by ELV:

PIR by ELV

Hooked up to port 1 of a JeeNode, and shown here with the USB “BUB” interface by Modern Device. Note that this uses a JeeNode with 6-pin port connectors, since the PIR module requires at least 5V. The output is open-collector, and therefore compatible with the 3.3V levels expected by the JeeNode (using the internal pull-up resistor).

Sample output, from a slightly adjusted version of the “pir_demo” example sketch which is in the Ports library:

Picture 1.png

Range and sensitivity seem to be ok. The signal is more jittery that with the ePIR and Parallax modules, it will have to be de-bounced it a bit in real-world use.

,

RF woes – solved!

In Software on Apr 28, 2009 at 00:01

Turns out my troubles with using the 868 MHz OOK radio next to the RFM12B module were caused by a silly software mistake (I mixed up the port assignments). No hardware or RF issues after all.

Latest sample output:

Picture 2.png

(the VOLT and BARO readings are bogus because the hardware is not connected)

As you can see, it’s receiving both packet types now. Still some trickiness with allocating the port signals properly, since some lines do require specific pins: the OOK receiver uses the analog comparator, but the ADC gets used as well, so I’ll need to adjust things a bit to use a pin change interrupt instead (used a crude workaround for now).

So now the basics are there to receive all types of signals with a single JeeHub: packets from other JeeNodes using the RFM12B, an 868 MHz OOK receiver for weather data and the FS20 remotes, and a 433 MHz OOK receiver for picking up KAKU remote commands (and possibly some other cheap weather sensors later).


Added bwired.nl data feed

In Software on Apr 27, 2009 at 00:01

The bwired.nl website has been promoting domotics “by example” for a long time now. Pieter Knuvers, the guy behind it all, added a new service to submit and display data from other fellow energy geeks like me…

The bwired upload service was announced (in Dutch) on Pieter’s weblog.

So I added the bwired.tcl Tcl code to JeeMon to generate the XML input he wants:

Picture 2.png

It’s trivial but way too messy stuff, mainly because I don’t have the proper abstractions in place yet to easily generate all sorts of aggregated data. In this case hourly averages and daily totals.

Results can be viewed here, name “jcw”.

Oh well, good exercise. Pretty much the same logic as for the Pachube feed.


JeeMon self-update mechanism

In Software on Apr 26, 2009 at 00:01

As I’ve mentioned before, JeeMon is a self-contained executable which requires no installation and runs on Windows, Mac OS X, and various Linux systems (including the ARM chip on the MMnet1001 modules used in the JeeHub). There are no special requirements or dependencies, no Java runtime, no .NET, no nothing (note: on Linux you have to check that the standard C++ runtime libraries are present).

The JeeMon builds are here – you only need to fetch one and launch it to use it.

This is based on a generic runtime called Tclkit which contains a full implementation of the Tcl programming language as well as the Metakit transaction-based embedded database. It’s hard to overestimate the amount of functionality included in these 1 .. 2 Mb executables.

The only specific code in JeeMon is the startup logic to turn it into a web-server with home monitoring functionality. It’s all inside one main.tcl file, the essence of which is shown here:

Picture 3.png

The code starts off with a few definitions and then loads the rest of the application. From file if present, else from a file loaded from the internet. The “fetchLatestLibrary” code by default also updates to the latest version, this can be disabled by deleting the “Jee-library.update” file.

By changing one URL shown above, JeeMon can be turned into any other type of Tcl application, because everything the application actually does comes from the downloaded file.

For those who don’t like this automatic mode, there is always the option to download all the pieces and run them as is. Both JeeMon and the Jee-library are based 100% on open source software. JeeMon is implemented in C/C++ and can be compiled from scratch using the standard gcc/automake tools, and Jee-library can be unwrapped to inspect every single bit of it – it is nothing but a packaged collection of plain text files and images. I’d be happy to document this if needed.

I suspect that most people won’t quite appreciate the implications of the way JeeMon is built and packaged until they try it out. And the reality is that most people won’t ever try out this stuff – fine with me. The technology underneath JeeMon is just a set of choices. In the end, the degree to which these choices become invisible in day-to-day use of this system is all that matters. And for a platform-independent modern programming environment, it doesn’t get much more general purpose and deployable than this…


Tracking JeeMon status

In Software on Apr 25, 2009 at 00:01

This is what I see on my screen these days:

Picture 1.png

This text is not inside a window but on the desktop, i.e. it gets covered up by all regular windows (there is always Exposé’s F11 shortcut to uncover it).

The text is barely readable, not only using a tiny font but also using 50% transparency. This is intentional: the text is not there to distract but to offer me a quick way to check on the status of my permanently-running setup on the JeeHub with embedded MMnet1001 Linux module.

It’s all done through a little utility called GeekTool, there are no doubt similar ones for Windows and Linux. Geektool periodically executes some shell commands and shows the output on the desktop.

The first listing displays the last 5 lines of the log output:

ssh propie tail -5 jee_out

The second listing is plain-text output generated by a new web page in JeeMon:

wget -q -O- http://propie/data/current

Adding that page to JeeMon was trivial (data collection details omitted):

Picture 2.png

Now, I can always see the latest readings and the status of the JeeMon server.

P.S. I will be away until May 3rd and won’t be able to respond to emails and comments until then. This weblog will be on auto-pilot for a week, enjoy!


RF12 protocol improvement

In AVR, Software on Apr 24, 2009 at 00:01

A small change was recently made to the RF12 wireless packet driver. This is the structure of a version 1 packet:

Picture 1.png

And this is the new structure in version 2:

Picture 2.png

Small change, big implications: the 16-bit CRC now includes the SYNC2 byte.

The SYNC2 byte is actually abused a bit by the RF12 driver, because it is set to the network group. Since RFM12B modules can look for a two-byte sync pattern with a configurable second byte, this is actually quite useful. By filtering on the network group, the receiver will completely skip packets which do not start with the proper group. This greatly reduces processor load when packets for different groups are being sent around in the same area.

There may still be some false syncs, since the RFM12B continues to look for the 2-byte pattern during the entire transmission, but it reduces overhead nevertheless.

The problem with version 1 packets is that a bit error in the group byte will generate a valid packet for another group. Extending the CRC tail to include the group fixes the problem, since the CRC will no longer be valid (almost never, that is).

This change is incompatible, i.e. packets sent with one version of the driver cannot be received by the other version. To make this less of an issue, the default group has been changed from 0×50 in v1 to 0×51 in v2.

With the new v2 driver, many network groups can coexist in the same area with no interference, other than having to share the total channel bandwidth.


New RF12 driver progress

In Software on Apr 23, 2009 at 00:01

The current RF12 is a bit simplistic, which means you may have to do a lot of work to get things across reliably. I’ve been working on a more sophisticated layer on top of the current RF12 driver code, which supports a reliable streaming mode.

Here’s a basic example which lets two nodes treat the connection as a serial link:

Picture 1.png

Everything typed on one node ends up on the other, in both directions. This is actually pretty hard to get 100% right, because it has to deal with packet loss as well as congestion (when you try to send data faster than the link can currently handle).

A far more interesting use of the streaming mode is to get all sorts of data across, grouped as a command with extra arguments. For example:

Picture 2.png

The “<<” operator pushes arguments into the stream. The send() call then starts the transmission, including an arbitrary 1-byte command code. Note how different data types can be specified for each argument.

On the receiving end, unwrapping is handled with the “>>” operator and a number of variables. The first value pulled out is actually the command code. Again, all conversions are automatically taken care of.

A basic implementation of the packetizing code is now working. But it needs more work to take care of lost and damaged packets and to make sure this gracefully slows down when the channel capacity is reached. Which could be quite low at times due to range or interference issues.

This example also illustrates how tedious it can be to format data as text using the Serial implementation: a 1-line “RF12 >> …” statement does the equivalent of a dozen “Serial.print(…)” calls.

P.S. The MuxShield and front-end are extremely convenient for debugging this sort of code.

,

More OOK signal decoding

In AVR, Software on Apr 21, 2009 at 00:01

The ES-1 energy sensor described in yesterday’s post is now also recognized by the OOK reception code (the touch panel too, since it uses the FS20 protocol):

Picture 2.png

The main state machine code is still essentially the same:

Picture 1.png

Note that three completely different and independent pulse patterns are being recognized. These recognizers run in parallel, although evidently only at most one will end up with a successful reception at any point in time.

An updated sketch for the Arduino IDE can be found here.


Nine times faster

In Software on Apr 17, 2009 at 00:01

The graph pages in JeeMon are a lot faster now. Here are the “before” …

Picture 1.png

… and “after” timing statistics:

Picture 2.png

A nine-fold speed increase. Great.

The change is that hourly and daily values have been re-calculated and stored in the database, instead of constantly recalculating these on-the-fly. The last value is not stored, as it might be incomplete and could change when more readings come in.

Keep in mind that these measurements are made with the MMnet100 Linux module, which is a very low-cost / low-power ARM-based system. On a modern PC, the graph pages usually come out instantly, even without storing any condensed hourly or daily datasets.

The other thing to keep in mind is that all JeeMon processing takes place in scripted languages: Tcl on the server and JavaScript in the browser. And that this is handling non-trivial amounts of data, since JeeMon is basing all these graphs on 5-minute values collected for up to an entire year.

In short: 9x faster is good enough, the graph pages now finish loading within two seconds.

It’s nice to see hunches work out in reality…

Update – on the MMnet1001, the UBIFS flash filesystem has compression turned on as default for all files. It would seem to make more sense to keep Metakit database files uncompressed, so I did a “chattr -c Jee-database” and made sure to rewrite the file from scratch with decompressed data. However, it looks like this only shaves another 5% off the graph page times, bringing times down to from 2.0 to 1.9 seconds. Oh well, it was worth a try.


Inside JeeMon

In Software on Apr 14, 2009 at 00:01

I thought I’d share a bit here how JeeMon works and what it’s made of.

One of the main goals for JeeMon, is that it should support two modes of operation:

  • running on a personal computer as one of many processes, launched and stopped at will
  • running on a dedicated system which is always running, possibly with quite limited resources

There is not that much difference between these two modes, but it does mean JeeMon should present itself as a webserver and that its processing demands have to be low. Being portable to different platforms also helps.

JeeMon is essentially a webserver plus database, both embedded and combined into a single process. It also contains all the logic to collect readings from – and send commands to – a JeeHub via a USB or serial port.

I picked the technology I know well, some of which I have helped develop or developed myself over the past years:

  • the code is written in two languages: Tcl on the server and JavaScript in the browser
  • the Tcl runtime uses a highly portable and self-contained system called Tclkit
  • the web server is my own design, using a highly modular Mason-like system called Mavrig
  • the embedded database is Metakit, for very compact and efficient storage of data
  • the JavaScript code uses jQuery as library to simplify many tasks
  • interactive plots are generated in the browser using the Flot package
  • the JeeMon application is packaged as a Starpack into a single installation-free runtime
  • similarly, all Tcl code, Javascript code, and website content is wrapped up into a Starkit

[more details below...]

Read the rest of this entry »


Decoding multiple OOK signals

In AVR, Software on Apr 12, 2009 at 00:01

It is possible to decode multiple signals with a single 868 MHz OOK bit-by-bit receiver, like this JeeNode setup:

868 MHz reception

Here’s the output from a sample run, receiving data from an FS20 remote, a KS300 weather station, and two S300 weather sensors:

Picture 1.png

The trick is to maintain independent state for all the signal patterns. The pulses need not have the same lengths, as long as each recognizer only picks up what it wants and stops decoding when anything unexpected comes in.

Here is an interrupt-driven implementation, using the analog comparator to pick up bit transitions and timer 2 to measure pulse widths:

Picture 2.png

If no transition is seen within 1024 µsec, timer 2 overflows and resets all recognizers to their UNKNOWN state.

A sketch for the Arduino IDE can be found here.


Graph navigation

In Software on Apr 10, 2009 at 00:01

JeeMon lets you zoom in (“drill down”) on the electricity / gas / water consumption data. Here’s how:

Picture 1.png

That’s 4 graphs: 1-day details with 5-minute resolution, 2 weekly summaries, and an entire year.

Picture 2.png

The weekly graphs only differ in the level of detail: on the left is the daily change each hour, whereas the graph on the right shows daily totals.

Clicking on a day in either of the weekly graphs causes the daily graph to display that day. And a click on any week in the year overview switches both weekly graphs to that week.

With two mouse clicks on this page you can display the details of any day in the past year. To easily go back to the initial display, a “Go to Today” button appears once a different graph has been selected (same as a refresh).

Note also that selecting a time range in the daily graph calculates the total consumption in that period, and hovering over a bar in the bar graphs displays the exact value as a tooltip.

No other zooming or panning is available, just these fixed choices. Simple and effective, IMO.


JeeMon demo (alpha)

In News, Software on Apr 9, 2009 at 00:01

Today I’m releasing a first version of the open source JeeMon home monitoring application, along with a demo dataset. Warning: this is an early alpha version – all features and bugs are still likely to change, a lot!

There are builds for Windows (x86), Mac OS X (x86 and ppc), and Linux (x86 + x86_64 + some embedded systems, such as the JeeHub’s MMnet1001 module).

It takes 3 steps to run this demo – there is no installation:

  • Get a runtime for your computer from this area
  • On Unix-like systems, run this: chmod +x JeeMon-*
  • Launch JeeMon

Here is the debug output this generates on my laptop:

Picture 1.png

And here’s a Windows screenshot with similar output:

Picture 1.png

Now point your browser to http://localhost:8888/ to see what JeeMon has to offer.

When started for the first time, JeeMon downloads a few extra files from the internet: “Jee-demodata”, “Jee-library”, and “Jee-library.update” (under 2 Mb total). These files are obtained from public servers at Dropbox, I do not track downloads or statistics.

On every subsequent startup, JeeMon looks for updates and refreshes the “Jee-library” file if there is a new version, so by simply starting JeeMon again a few days later you can track its development progress. To disable automatic updates, delete the “Jee-library.update” file.

I’m releasing this code to give you an early glimpse into the Jee Labs kitchen – and to gauge the interest and figure out how to improve JeeMon. If the demo works for you, great. If it doesn’t, you can either try to figure out what the problem is, or delete all the Jee-* files and try again at a later date. The new mailing list announced yesterday is open to anyone wishing to comment and make suggestions.

Let me reiterate that this JeeMon alpha release is for infinitely curious and technically interested people, not for those looking for a finished home energy monitoring solution. Running this JeeMon demo is bound to raise more questions than I can deal with – but it’s all open source, so feel free to explore this as much as you like.

Whatever. Enjoy …

Update – the demo dataset now contains data since 2009-01-01, i.e. a total of over 3 months of readings.


It gets a lot worse…

In Software on Apr 7, 2009 at 00:01

Yesterday’s post was about web server performance. Here’s the latest build of JeeMon with the same page:

Picture 1.png

That’s nearly 17 seconds to render the page now, versus 5 yesterday… whoops!

Here’s why I call this major progress anyway:

  • The “electricity.html” page no longer contains plot data values and is half its previous size.
  • All plot values are obtained via Ajax calls, running (mostly) in parallel.
  • This opens up the path to unobtrusive refreshes and automatic plot updates.
  • All plots now contain meaningful data, the previous version used some bogus values.

As you can see in the above graph, over 15 seconds is spent in waiting for the server to return all plot data.

And the reason this now takes so long is very simple: all values are calculated on-the-fly by the server from stored 5-minute values. That’s over 100,000 aggregated values to generate a simple 52-week bar graph!

All this should sort itself out once I start caching hourly and daily results in the database. For now, only the raw readings and the 5-minute aggregated counts get stored. The finecky bit is to make all aggregations / calculations consistent across arbitrary JeeMon restarts.

Here’s the latest version of the generated page:

Picture 2.png


Website debugging tool

In Software on Apr 6, 2009 at 00:01

Just to prove that JeeMon is far from ready for real-world use…

The Safari web browser has excellent tools to figure out where bottlenecks are. Here is an overview of where the loading time is going for the main electricity consumption page, as produced by JeeMon running on the MMnet1001 Linux module. That web page has several detailed graphs on it – here are the stats:

Picture 1.png

As you can see, all the time is consumed by the initial page generation on the server. That’s 4.64 seconds to generate the page – which is totally unacceptable. JeeMon serves that same page 50 times faster when running locally on my Mac laptop (!).

Here are the amounts of data transferred to serve that page:

Picture 2.png

The main page including the plot data is about 5 % of the total size – the other files are cached as static content, which is why they do not increase the transfer time.

There is a lot more information to be gleaned from these measurements, btw.

I have a hunch on where the bottlenecks in JeeMon are, but haven’t looked into it yet. Which is exactly how it should be: aim for solid / correct behavior first and use the proper instruments to measure performance and track the effects of optimizations later (I happen to use Safari, but similar tools exist for other browsers).

Stay tuned for updates, once I get to tweaking this…


JeeMon sneak preview

In Software on Apr 5, 2009 at 00:01

I’ve started work on a basic website design for JeeMon:

pastedGraphic-1.png

Here’s the gas consumption page:

pastedGraphic.png

The same JeeMon code is currently running on the “production” (ahem) JeeHub and on my development Mac.

These are real live webpages, with real (but not yet 100% correct) datasets. Still, this is just a tiny tip of a massive iceberg. Lots and lots of things left to do.

Well, at least it’s a pretty iceberg IMO ;)


Rethinking sensor nodes

In Software on Apr 4, 2009 at 00:01

The current “JeeNode Pulses” used around the house are getting a bit large in terms of code size – about 12 Kb of the available 14 Kb have been used up. A lot of this is due to debugging code, with each node generating a detailed report on its serial port. And then there’s the configuration code: displaying the current configuration on startup and interpreting commands entered on the serial port to change those settings.

Here’s serial output from a simple unit with 3 readings:

Picture 2.png

Most of this code becomes irrelevant once the nodes are installed and working properly, since the only thing that matters are the transmitted wireless packets. It seems a bit wasteful to have each node carry this logic around – and it’s inconvenient because settings can only be inspected and changed through a serial connection. Not so great for “remote” nodes…

A lot of this logic could be centralized, if only remote nodes were able to respond to wireless requests. But this requires some care: better not allow anyone anywhere to re-configure the JeeNodes in your house at will! IOW, we need a secure channel.

The simplest way is probably with a simple encryption protocol such as XXTEA, with a per-node “key” securely installed in EEPROM via the serial port. Once that is in place, we can securely send requests to any node to reconfigure and reboot it.

But why stop there? Why not generalize the entire transfer of state in both directions? What I’m thinking of is the following design:

  • Each node has a unique 128-bit key, known only the central server.
  • The central server can send encrypted configuration data, up to 60 bytes (1 packet).
  • The remote nodes can send readings data (perhaps up to 250 bytes).
  • Both sides keep track of the state of this data and only send differences.
  • The ack contains a CRC of the full state – if it is wrong, the full state will be re-sent.
  • Both the unique key and the configuration data are stored in EEPROM, and get copied to RAM on startup.
  • It might take a few packets to send all readings, but since only differences are sent, often a single packet will suffice.
  • The basic content of a packet would be: OFFSET, BITMAP, DIFFS, HASH.
  • The same mechanism could even be used to re-flash a remote node one day, given a suitable wireless bootstrap.

With such an approach, the code for JeeNode Pulses could become a lot simpler. No need to display text results on the serial port, since the central node has access to all relevant state (including all intermediate values which are not used as final readings). And no need to embed a command interpreter, since the server can produce the exact byte values needed for configuration – with a fancy GUI even, if needed.

The challenge now is to tie things together easily. How to define simple C code on the Pulse which is fully configurable on another system. This is sort of a poor-man’s SNMP… hmm, I’ve got some ideas to try out.


Real-time tracking

In Software on Apr 3, 2009 at 00:01

Here’s a real-time graph of data collected by JeeMon:

And a snapshot, in case the above real-time graph isn’t working right now:

Picture 2.png

It’s generated via a new site called pachube – I’ve extended JeeMon to feed that site through simple REST-style HTTP requests. Since the graph only has 15-minute resolution, that’s how often it gets updated.

Here’s another real-time feed, showing our gas consumption in l/hr:

Pachube is still in beta. I’ve only just started looking into it. You can define any number of feeds, each having any number of data values. Not sure how it deals with missing values, nor what to do about data which is collected at different rates. The graphs are somewhat configurable, as you can see - this will no doubt still evolve further.

Pachube can poll a server you designate (“automatic” mode), or you can feed it new data at your own pace (“manual” mode), as is being done here. The advantage of the latter is that it gets through firewalls and avoids the security issues of running a public server.

It’s an interesting approach, apparently pachube aims to become some sort of free exchange for producers and consumers of location-specific real-time data.

I’ll leave this setup running for now, but don’t be surprised if this post stops showing an up-to-date graph, since I might stop this feed one day.


This is getting out of hand

In AVR, Software on Apr 1, 2009 at 00:01

I’ve installed a preliminary JeeNode pulse upstairs, with a few 1-wire temperature sensors to track hot-water and central heating a bit. Here’s the real-time read-out, coming from the JeeHub:

Picture 4.png

These readings are coming from the following sources, mostly via wireless:

  • Electricity / gas meter, from a prototype JeeNode.
  • Weather data from 1 KS300 + 2 S300 commercial sensors.
  • Barometer and battery check, inside the JeeHub.
  • Room data and central heating temperatures, from a JeeNode Pulse
  • Some test values, from a second JeeNode Pulse on my desk.

But this display clearly won’t scale visually. Besides, it’s just a big pile of numbers. I really need to figure out a way to present this information nicely!

It’s fun to watch, seeing values update in real time…


Better delays

In AVR, Software on Mar 31, 2009 at 00:01

As described in this post on an Arduino forum, I’ve suggested some small changes to the delay() and delayMicroseconds() code in the Arduino/Wiring libraries:

Picture 3.png

(rest of delayMicroseconds unchanged)

Note that both changes only add new code, no lines have been removed or changed (the comments need to be adjusted, though).

The above changes only affect microsecond delays > 500 us and millisecond delays < 50 ms. In that range, a loop is used which should be accurate in the couple-of-microsecond scale and which does not lock out interrupts.

It should lead to a more accurate millis() value since interrupts can be serviced more often, and to a far more accurate delay for low-millisecond values. I hope something like this makes it into a future release of the Arduino IDE – timing capabilities are still a bit limited on the Arduino, IMO.


MuxShield front-end GUI

In Software on Mar 28, 2009 at 00:01

Here’s a little “muxview” front-end for the MuxShield:

Picture 2.png

(it’s not very exciting since all slaves are returning the same text)

This little application recovers the individual streams from the multiplexed data coming in over USB. Screens 1..5 show what is coming from each slave channel. Screen 6 displays what came in before the multiplexer was started.

Here’s a more interesting example with 5 RFM12B receivers (you won’t be able to read the text because I had to reduce the screenshot to fit this blog). The antennas are way too short, to force weak transmissions and bad reception. As you can probably glimpse, each of the slaves has trouble with the data, but in very different ways:

Picture 1.png

This application is written in Tcl/Tk, the script is available here. I’ll tweak it further over time to fit my needs.


The MuxShield is working!

In AVR, Software on Mar 27, 2009 at 00:01

It’s nice when things come together. The MuxShield is now able to re-program up to 5 attached slaves and then act as multiplexer for serial output from each of them.

Here’s how it works in practice:

  • Connect the MuxShield and plug a few slaves into it.
  • Make sure the switch is set to “PROG”.
  • Program a sample sketch for the slaves into it, e.g.:
    Picture 2.png
  • Connect to the MuxShield with a terminal:
    Picture 1.png
    (that’s 3x two lines of output, every 3 seconds)
  • Sample output with the switch set to “RUN”:
    Picture 3.png
  • In the “RUN” position, programming changes the MuxShield code of the ATmega328 itself.

I often forgot to put the switch in the right position while developing the MuxShield code, but once that is done it should no longer be an issue. Since the slave code is for an ATmega168, sending it to the MuxShield with the switch set to “RUN” will fail (the 328 bootstrap rejects the 168 re-programming request).

Note that the MuxShield also makes a fine stand-alone re-flash unit: just connect a slave and press reset with the switch set to “PROG”. Here’s a MuxShield with 5 slaves, all trying to say “Hi!” at the same time:

Picture 5.png

Source code of the MuxShield sketch is available here.

Onwards – I now have the perfect tool to develop and test multiple JeeNodes!


MuxShield custom bootstrap

In AVR, Software on Mar 26, 2009 at 00:01

I’ve made good progress on the MuxShield – only some minor changes to the bootstrap were needed, here is a diff:

Picture 1.png

These extensions still fit in the current 2K boot loader area, but only just!

To summarize: the bootstrap watches the switch connected to pin 7. If it is high (which is also the case if the MuxShield is not connected), then nothing changes. But if it’s low, the following happens:

  • Change the bootstrap baudrate back to 19200.
  • Respond as if this is an ATmega168 (different signature bytes).
  • Store all incoming flash data at an offset of 0×3800.

The end effect is really, really simple: if the switch is in the RUN position, nothing changes and the Arduino behaves just like an ordinary ATmega328 board. If the switch is in the PROG position, the board looks like an ATmega168 board while programming. After that, it resumes its ATmega328 program as before.

One more step is needed to make this truly plug-and-play for re-flashing up to 5 connected FTDI slave boards: on startup, the MuxShield code must check if the switch is in the PROG position. If so, it must program each of the slaves with the data loaded in flash memory starting at offset 0×3800. Then finish off by starting the serial multiplexing logic.

Anyway – 2 out of 3 steps are done, this is going a lot smoother than expected!

,

Multiplexing code for Arduino

In AVR, Software on Mar 25, 2009 at 00:01

Here is an extract of the code to multiplex 5 serial ports in software on an Arduino Duemilanove:

Picture 3.png

It’s a big bag of tricks, really. There is a timer running at 3x the baudrate, which detects start bits and then picks out data bits every three ticks. The crucial issue for the receivers is that the work is done in parallel for 5 input streams, but that these events are not happening at the same time. There are in fact 5 little state machine, each with their own independent state.

For the output, it’s slightly simpler: a buffer with all the 5 transmit bit states is scanned and sent out. The trick here is to fill that buffer with the proper bit patterns. When nothing is sent on a channel, its corresponding bit remains high in all values in the buffer. The transmit buffer has 30 entries, 3 per bit plus the start and stop bits. A simplification here is that all bytes are sent at the same time, i.e. start and stop bits occur at the same time on all active transmit channels.

The idea of multiplexing is that all received data is collected and sent out to the main (hardware) serial port, running at 57600 baud. Extra character codes 0×01 .. 0×05 are inserted into the multiplexed data stream to identify from which channel the data is coming. On the transmit / de-multiplexing side, the character codes 0×01 .. 0×05 are filtered out and used to indicate to which channel to send out the next characters.

If you connect two of these multiplexing shields back-to-back, you can send 5 independent bi-directional serial streams over a single 3-wire cable. Come to think of it, the following setup would make a great test for these multiplexing shields:

Picture 2.png

The full source code for the multiplexing sketch can be found here. It has been tested at 9600 baud (all receivers must run at the same speed). It probably works at higher speeds even, but this hasn’t been tested. At some point, the Atmega chip is going to get swamped while handling the barrage of timer 1 interrupts. I haven’t really tested the transmit part yet, i.e. the de-multiplexing side of things.


The logic of data logging

In Software on Mar 22, 2009 at 00:01

The point of several projects here is to monitor energy consumption, and probably also some weather data because it’s so easy to add. Where “monitoring” means logging as well as keeping measurement data in a database for quick access. But how do you make sure that all “readings” get logged, yet still retain the ability to adjust and extend the underlying database and software? It would be awful if each non-trivial change required conversion.

Here are a few basic design decisions I took for JeeMon:

  • Readings come in as text from the attached JeeNode, one line per reading.
  • All readings are time-stamped and appended to a logfile, in text format.
  • To keep things manageable, logfiles roll over to a new one on 0:00 GMT each day.
  • The first entry in a new logfile is the name and size of the previous one.
  • The database is treated as a cache: it can be reconstructed from the logs at any time.

And here’s a requirement I want to meet:

  • It must be possible to restart JeeMon at any time without losing readings.

The idea then is to “catch up” with the logs on each restart where possible, and to clear the cache db and reload it from scratch when the software changes are more substantial.

So what does it take to not lose any readings? Keeping in mind that JeeMon will run either on a little embedded Linux module right next to the central JeeNode, or on a personal Mac, Windows, or Linux computer. Well, first of all, this is why the central JeeNode has a backup battery and dataflash memory: it stays on at all times, and continues to receive and collect packets from all the other JeeNodes even when there is no JeeMon running. The duration of this autonomous operation will be fairly modest: a few hours or perhaps one day. Enough to handle brief power loss, to mess around with configurations, and to re-connect things occasionally.

This means there are now three places where data gets added and must be kept in sync with the rest:

Picture 1.png

(Four places if you also count the final destination: dynamically updating browser windows)

  • The dataflash memory gets a copy of each reading collected by the central JeeNode.
  • All readings must end up in the log files when JeeMon is running.
  • The cache database stores decoded values in an efficient internal format.
  • And lastly, the browser(s) present more or less detailed results, summaries, and derived info.

JeeMon will start up in logfile “catch up” mode: the current content of the dataflash memory will be checked against the last logfile entry. The first task is then to request old data from the JeeNode to bring the log files up to date. During this time, the JeeNode continues to add new incoming data to the dataflash.

Once all missed data has been transferred, the JeeNode switches to “real-time” mode, saving new readings to dataflash and sending them out to JeeMon at the same time. In real-time mode, the log files will track all readings as soon as they come in.

That’s just part of the story, though. Now JeeMon enters “db sync” mode. With information from the cache db, new entries and new log files are scanned and processed, with new readings added to the database. Once that is done, JeeMon switches to normal real-time operation.

All log files are kept online. Rebuilding a database from scratch is as simple as deleting the current one and restarting JeeMon. Since a full rebuild might take quite some time, the internal JeeMon webserver always starts up in a “please-wait” mode and switches to the real server after real-time operation has been enabled.

Part of the above logic has now been implemented. JeeMon resumes its logs, syncs / rebuilds its database as needed, and processes new readings while running. The firmware in the JeeNode to save and replay readings to and from dataflash is still work-in-progress, however.

So while the JeeHub / JeeNode needs to stay powered up, I can now restart JeeMon at will. Which is great, because that streamlines development.

, ,

Let's pick a few more names

In Hardware, Software on Mar 20, 2009 at 00:01

Not to worry: I’m not going to go crazy on the Jee<blah> naming used so far…

But I do need to give the different pieces some name. To be able to refer to them in these posts, but more importantly as names for the software for all this. I also need to introduce a basic structure (and some limits).

Picture 1.png

Here goes:

  • A JeeNode is this ATmega-with-4-ports-and-RFM12B thing. There may be lots of JeeNodes for various purposes. They can communicate with each other via wireless.
  • I use the Ports and RF12 libraries with JeeNodes, although this is not a hard requirement – both the JeeNodes and the libraries can be used in numerous other ways.
  • Each JeeNode has a letter ‘A’ to ‘Z’ assigned to it as node ID. Nodes are usually given a unique ID to avoid mixups, but this is not strictly required.
  • There can be up to 250 separate groups of JeeNodes. Nodes can only communicate with other nodes in the same group. Gateways between groups could be implemented later, if needed.
  • Most nodes will be pulse JeeNodes, i.e. running a specific piece of software called – you guessed it – “pulse”, which continuously monitors some attached sensors and reports the readings via wireless.
  • The JeeHub is either a JeeNode by itself connected to a Mac, Windows, or Linux PC via USB, or a JeeNode connected to Ethernet via a small dedicated Linux module.
  • This JeeNode-as-part-of-a-JeeHub is called the central node from now on. It runs a specific software configuration, also called “central”. There should always be exactly one central node.
  • The software running on the Mac, Windows, or Linux machine(s) is called the server from now on (how original, eh?). It consists of a system-dependent executable runtime called JeeMon plus the code and data for the application itself.

It probably doesn’t hurt to reiterate that “JeeNode” and “JeeHub” are hardware, whereas “JeeMon” and the Ports / RF12 libraries are software.

The above names are also used in the source tree repository now.

There are no doubt still infinitely many ways to lead you astray, but I hope that these definitions will help me place everything into an increasingly coherent context.


Energy monitoring

In Hardware, Software on Mar 19, 2009 at 00:01

This setup has been running here for a few months:

Picture 1.png

A light reflectance sensor detects the revolutions of the electricity meter. It is hooked up to a JeeNode, which counts the revolutions and sends its counter value to the central JeeHub node every 10 seconds.

For basic tracking of total electricity consumption, that’s all you need. Occasional packet loss has no influence on aggregated results since the counts are tracked near the sensor. The same works for gas and water consumption.

However, I also want to get a good estimate of the consumption rates in real time. This can be done by timing the rotation period, at millisecond resolution preferably. So the first improvement a while back was to let the sensor JeeNode send packets the moment a revolution is detected, and not just every 10 seconds. That way the central node can do millisecond-resolution timing, which is easy with a stable clock and Linux.

This works reasonably well, but only if reception is nearly perfect. Any lost packet messes up the preceding and the following timing estimate. Only with two packets sent right after a counter change can the period be measured. I kept the periodic 10-second sends enabled as fail-safe, but these late packets are useless for real-time rate calculations.

The next improvement was to let the sensor node do the millisecond timing. It wasn’t obvious how, though, as sensor nodes do not have a very accurate clock: a ceramic resonator has around 0.5% accuracy and probably also some temperature sensitivity. Turns out that it doesn’t really matter: as long as these real-time estimates are not summed together, errors won’t propagate any further.

So now the sensor node sends both the counter value and the last rotation period in milliseconds. The counter value is perfect for long term energy consumption monitoring – after all, it’s exactly the same as what the power company will use to charge me later. The millisecond period is a good estimate of current electricity use, and probably fairly stable from one measurement to the next. Absolute accuracy and drift are not so important here, because the main use of the real-time estimate is to see small changes as they occur.

Here’s an example of the entries logged on the JeeHub:

Picture 2.png

All log entries have the format “T [time] [millis] [type] …”. Where “time” is the time on the Linux module of the JeeHub in YYYYMMDD.HHMMSS format and “millis” is the current millisecond counter on the JeeNode inside the JeeHub. This might seem redundant, but keep in mind that the Linux module is not always on – it can pull data off the JeeNode when started up again. Additional timing data will come from the on-board DCF77 receiver once that is fully operational. So there are three sources of timing information inside the JeeHub: Linux, JeeNode, and the DCF77 atomic-clock reception. Each have different levels of accuracy and drift, but because all of them are logged, this can be fully determined later.

The raw text log files grow by around 0.5 Mbyte/day right now. Gzip compresses them to roughly 20% of that, so they can easily be kept around: the MMnet1001’s flash memory will hold several years of these full-detail logs.

Back to the actual log entries now.

The “HM2″ packets are from the electricity & gas metering node. Packets 56, 57, 58, 60, 61, 62, 63, 0, and 1 were lost (more on that later). The electricity revolutions went from 132 to 136 in this time frame, and the gas counter stayed at 139. The rotation time for the electricity meter in packet 54 was 74*256+209 = 19,153 ms. There is a third slot of four 0’s in each packet for measuring water consumption, currently waiting for a new sensor-enabled flow meter to be installed.

Looking at some calculated results, I think this setup will be able to track real-time power consumption with a resolution of under 1 Watt. The resolution actually improves with lower energy consumption rates – precisely what you’d want.

The VOLT and BARO entries are from sensors within the JeeHub. The ALT entry is weather data from a remote sensor and comes from another system for now.

A further improvement with the energy data was to switch to the latest RF12 library code and to use acknowledgements. Every time a packet has been properly acknowledged, the sensor node stops sending data until one of the counters changes. When no ack is received, data is resent every 3 seconds. With good reception most of the time, the number of packets sent (and logged) willl be quite small. And with packet loss, the data will usually arrive 3 or 6 seconds later anyway. Note how in the above log file excerpt each counter value from 132 through 136 was recorded, but retransmits were needed to get counts 134 and 135 to the JeeHub.

Final point: the wireless communication speed was raised from 38400 to 57600 baud – and reception appears to be at least as good as before.

Packet loss – I’ve been seeing long stretches of fairly bad reception at times. Until I realized that these coincided with our wireless headphone set being turned on. I don’t know what that thing is sending out into the ether, but it’s clearly interfering with the 868 MHz communication between JeeNodes. Oh well – it’s a good test for dealing with missed packets…

, ,

Collecting data with JeeMon

In Software on Mar 17, 2009 at 00:01

This is the information I’m currently tracking real-time:

Picture 1.png

(it’s in Dutch, but you can probably guess most of these)

And this is the setup I’m using for it:

Picture 2.png

The NAS in there is used as gateway to pass 868 MHz OOK data from a CUL receiver, which decodes signals from my KS300 weather station plus a couple of S300 temperature/humidity sensors. This will one day be replaced by on-board reception on the JeeHub, so that only the JeeHub needs to stay powered up at all times. All other sensors are hooked up to a couple of JeeNodes which transmit the readings wirelessly.

The Mac, Windows, Linux, and the JeeHub all run identical copies of the software, which is called “JeeMon”. It has a built-in web server, an embedded database, and a flexible set of network functions.

Each JeeMon instance will automatically self-update to the latest version on startup. During development on the Mac the JeeHub acts as transparent proxy, as if the different sensors were connected directly to the Mac (through a little Tcl-based system called “Tequila”). Once ready, the latest JeeMon release is wrapped into one file and placed on the internet. Finally, a restart of the JeeHub completes the upgrade.

That first screen dump above is a small test app on my Mac which bypasses JeeMon and connects directly to the JeeHub as Tequila client. I keep it open to check that data is coming in and gets saved on the JeeHub.

So this is the big picture for collecting energy/gas/water and environmental data in the house. The software can run on practically anything, can be accessed with a browser anywhere, and with proper security in place the various pieces can be connected and used across any network topology.

All of the above is working this very moment. The major task ahead is the full-scale processing, presentation, and interaction of it all. But that can now conveniently be done on my development machine, with HTML, CSS, JavaScript, etc.

Anyway, IMO this is a very flexible foundation for a 1-watt home monitoring server.


JeeHub software

In AVR, Software on Mar 13, 2009 at 00:01

Well, I’m quite pleased with how the software on the AVR side of the JeeHub is working out. When you look at the collapsed version of the current 270 lines of C++ code, you get this:

Picture 1.png

You can see the full source code here.

Everything becomes a DataSource, i.e. an object which is polled periodically and which can then submit readings for further storage or processing. It’s trivial to add more data sources without affecting any existing code.

On startup each data source is created and registers itself automatically. For example, the “power” object is set up to get polled/measured every 10 seconds. When polled, it takes a few ADC voltage readings, puts these into a struct, and hands it to the DataSource::newReading() method.

Currently, newReading() does nothing more than turn around and call the show() method of that same data source, which then sends a plain text version of the data out over the serial line. For unattended operation, readings will need to be saved in dataflash memory for retrieval at some later moment.

The above code compiles to 8104 bytes on an ATmega168 so far. Lots of room remaining!

,

JeeHub prototype

In AVR, Software on Mar 12, 2009 at 00:01

Voilá – the first JeeHub prototype exists!

JeeHub prototype

(click on the image to see the annotated version)

It’s a pretty simple setup so far, as can be seen on the backside:

DSC_0117

All the pieces work, at least in isolation. But the Linux board and the JeeNode haven’t been connected together yet (I need to figure out which serial port to use on the MMnet1001) and there is some strange interaction between the different interrupt-driven pieces on the JeeNode. Looks like both hardware and software troubles at this stage.

Here’s some output:

Picture 1.png

So it’s picking up RFM12B packets, reading external power and battery voltage levels, reading out temperature and barometric pressure, and decoding both the KlikAanKlikUit remote controls and the DCF77 clock signal. But there’s some stuff here which is definitely not kosher – could be interrupt handling issues.

Update – looks like gcc 4.3.0 has trouble with interrupt code. When I changed to 4.3.2, everything started working as expected.

There are also a few hardware glitches in this early prototype, such as the switching regulator interfering with the DCF77 receiver (will change to a linear one) and the 433 MHz radio interfering with the RFM12B (could add some decoupling or shielding).

This first version of the code has been added to the subversion repository.


Decoding 433 MHz KAKU signals

In AVR, Software on Mar 10, 2009 at 00:01

Here is a simple OOK receiver, hooked up to AIO port 3:

Decoding 433 MHz KAKU signals

The following interrupt-driven code will pick up signals from a “KlikAanKlikUit” remote control:

Picture 1.png

It should be relatively easy to adapt this to other units. Signals in different protocols can be decoded at the same time, since all decoding state is maintained in a separate state machine.

Sample output:

Picture 2.png

The KAKU units use a very crude signalling protocol, with no error checking other than sending a distinct bit pattern for 0’s and 1’s. The sample output has errors, in fact: the B7 after the B6 should have been a B6. It is probably best to only accept commands when they come in more than once in quick succession.

There can be quite a bit of noise from the receiver, in the form of random pulses. These will be ignored, but it may use up some CPU power to service all the corresponding interrupts.


Don't mess with the boot flash

In Software on Mar 9, 2009 at 00:01

Ok, so I tried to get the on-board 4 Mb dataflash of the MMnet1001 working, just out of curiosity…

That was a bad idea. Nothing went wrong for a while, but then I rebooted:

Picture 2.png

And then nothing. Whoops, I messed with the boot partition instead of the dataflash chip! Thankfully, tech support at Propox helped me out quickly. It’s actually quite easy to re-flash the MMnet1001: connect a USB cable to the USB device (not host) port pins, and then simply jump through a few basic software hoops. Neat – well done.

Ah, that looks better:

Picture 1.png

And so on.

So the 4 Mb dataflash remains a mystery for now. No problem, the 1 Gb NAND flash is the main deal, and it works just fine. I’ve added about 7 Mb of my own so far, but that still leaves plenty of free space:

Picture 4.png

The hardware clock works fine, but it needs to be set once before reading it out to get rid of an error message. IOW, set the time zone in “/etc/config/system” (I used “CET-1″), then set the proper date and time with “date”, then finish with “busybox hwclock -w -u”. Note that the “/sbin/hwclock” version doesn’t work for setting the clock (it seems to be ok for readout).

On the software side, I now have a Tclkit build cross-compiled via the OpenWrt SDK and working on the MMnet1001, so the rest of the work can be done on my development system and then copied to this amazing little Linux, eh… box? plug? chip?

Anyway, all is well.

Thank you, Andrzey, for helping out an AT91 newbie ;)


DCF77 clock reception

In AVR, Software on Mar 6, 2009 at 00:01

The DCF radio in Frankfurt sends out a constant beacon signal at 77 KHz which can be received by a cheap radio module all over Western Europe:

DCF77 clock reception

Update – the above connections are wrong. The diagram was showing the bottom view, all wires need to be flipped over to the other end of the DCF77 screw terminal block. No harm done, it works now.

The code has been added as “dcf77demo” in the Ports library.

Sample output:

Picture 1.png


OOK signaling with an RFM12B

In AVR, Software on Mar 3, 2009 at 00:01

Although the RFM12B was designed for FSK (frequency-shift keying), it can also be used for OOK (on-off keying) transmissions. The trick is simply to turn its transmitter on and off via the SPI interface.

This can be used to control simple RF-controlled devices such as the FS20 power control units by Conrad and ELV in Germany. Here’s a sketch which turns a remote device (lamp, etc) on and off:

OOK signaling with an RFM12B

It turns out that the 868 MHz version of the RFM12B can even transmit 433 MHz signals, at least for simple OOK. The following example turns a device on and off via the low-cost KlikAanKlikUit units sold in the Netherlands, using the same 868 MHz radio module as above:

OOK signaling with an RFM12B

Both demo’s have been added to the RF12 source code library. Other slow-rate bit-stream protocols similar to the above could easily be added.

No attempt has been made to receive OOK signals right now, though one could imagine reading out the RSSI bit to determine the presence of a carrier…

,

Ports on a standard Arduino

In AVR, Software on Mar 1, 2009 at 00:01

The Ports library described in recent posts works fine with any type of Arduino, Freeduino, etc – not just JeeNodes. It merely uses some conventions for the “DIO” and “AIO” pins of each of the 4 ports.

Here is an example using a HM55B compass module from Parallax, using an Arduino Duemilanove with a Proto Shield from AdaFruit:

Ports on a standard Arduino

There are actually two sensors in the above setup – the bigger sensor is a Parallax H48C 3-axis accelerometer (more on that below). Both need +5V to operate, so they won’t work with simple 4-pin ports on a JeeNode.

Ports are mapped to the Arduino pins as follows:

  • Port 1 DIO = Arduino digital pin 4 (PD4)
  • Port 1 AIO = Arduino analog pin 0 (PC0)
  • Port 2 DIO = Arduino digital pin 5 (PD5)
  • Port 2 AIO = Arduino analog pin 1 (PC1)
  • Port 3 DIO = Arduino digital pin 6 (PD6)
  • Port 3 AIO = Arduino analog pin 2 (PC2)
  • Port 4 DIO = Arduino digital pin 7 (PD7)
  • Port 4 AIO = Arduino analog pin 3 (PC3)

The ATmega register bits are listed in parentheses.

Here is the full code of the HM55B driver plus demo:

Picture 2.png

Sample output:

Ports on a standard Arduino

The H48C 3-axis accelerometer demo is very similar, source code can be found here. Sample output:

Ports on a standard Arduino

These two hookups both use some new utility code in the Ports library to shift a specified number of bits in and out of the DIO line while clocking the AIO line. Note also that these sensors needs two ports each, since they both use 3 IO lines for their SPI-like interface.

,

All together now

In AVR, Software on Feb 22, 2009 at 00:01

It’s time to combine everything:

Combi demo

This is a setup with all the sensor interfaces documented in recent posts:

  1. a SHT11 to measure relative humidity (and temperature)
  2. a BMP085 sensor to measure barometric pressure (and temperature)
  3. PIR + LDR sensors to demonstrate reading digital and analog signals
  4. a Nunchuk with 2-axis joystick, 3-axis accelerometer, and 2 buttons

A new “combi_demo” has been added to the Ports library. The code is essentially the concatenation of the individual demo source files (you can browse the real thing here):

Combi demo

This example illustrates how the Ports library lets you mix and match drivers and ports at will. Note that two I2C interfaces are running in parallel at different speeds (sure, the BMP085 and the Nunchuk could also have been tied to a single port, running at 100 KHz).

Sample output;

Combi demo

This example compiles to 9114 bytes of code with the Arduino 13 IDE, so there is still plenty of room left to add, say, a wireless driver.

,

Hooking up a BMP085 sensor

In AVR, Software on Feb 19, 2009 at 00:01

The BMP085 sensor measures barometric pressure (and temperature). I’ve hooked it up to port 2 via I2C:

Hooking up a BMP085 sensor

As SMD, it’s tiny …

Hooking up a BMP085 sensor

The 4 wire-wrap wires are connected to port pins 2..5, respectively: SDA (DIO), ground, SCK (AIO), and 3.3V power.

There’s now a bit-banging I2C master driver in the Ports library which works with any of the 4 ports, plus a new driver for the BMP085. Here’s the new demo code, now included with the library:

Hooking up a BMP085 sensor

It looks like the elaborate algorithm from the BMP085 data sheet to calculate pressure in Pascals (millibar) contains an error. After some guess-work and with information from the older SMD500 sensor, I think I’ve come up with the proper computations. The results are now as expected:

Hooking up a BMP085 sensor

This is a very sensitive MEMS sensor. It’s accurate enough in its lowest resolution mode to calculate altitude to 50 cm accuracy given the exact pressure at sea level – just moving the sensor up and down shows a matching variation! Higher-resolution modes have been not been implemented.

Update – There’s a problem with the calculations: with temperatures 25.0 °C and up, the pressure calculations fail and a result of 2.35 mBar is reported. Will need to look into this.

Update 2 – Thanks to a private email suggestion, I’ve finally been able to resolve the > 25 °C bug. The current source code no longer has the problem and shows consistent values also at higher temperatures.

,

Hooking up an SHT11 sensor

In AVR, Software on Feb 17, 2009 at 00:01

The SHT11 sensor measures relative humidity and temperature. It connects via 2 I/O lines using an I2C’ish protocol with a funny (optional) CRC check. I’ve hooked it up to port 1 on a JeeNode:

Hooking up an SHT11 sensor

There are just four wires to connect to this minute module. Here’s a close-up, with data and ground crossed over to port pins 2 and 3, respectively:

Hooking up an SHT11 sensor

The sensor is mounted free-floating on this little DIY breakout board so it picks up ambient air properties.

A driver for the SHT11 has been added to the Ports library, along with a small demo:

Hooking up an SHT11 sensor

A “sensor” object is defined at the top, tied to a specific “port” – the SHT11 class is derived from the Port class.

Sample output:

Hooking up an SHT11 sensor

The driver uses PROGMEM for the optional crc table. This causes a warning in Arduino-0013 when switching boards (which recompiles PortsSHT11.cpp), probably due to this issue, i.e. gcc bug #34734.

With thanks to Guido Socher for his excellent page about the SHT11 (his version does all calculations using fixed point integers, if you prefer to avoid floats).

Update – Changes checked-in and ZIP file updated. Thanks to Stephen Eaton for pointing out a mistake in the calculation constants for 3.3V.


Remote ports

In AVR, Software on Feb 15, 2009 at 00:07

Yesterday was about local “ports” on the Arduino’ish JeeNode. Today, this is extended to remote ports. Here’s a sketch which controls ports wirelessly:

Remote ports

It’s almost like the local one, except for some headers and declarations at the top and a new “bob.poll()” in the main loop. Where “bob” is defined as the object representing a remote node with id ‘B’ (id’s ‘A’ through ‘Z’ are available for general use). Welcome to the world of messaging and transport independence.

This is a test setup with two JeeNodes:

Remote ports

The receiver (top) is powered by a 3x AAA battery pack (it turns out that 3x NiMh ≈ 4V). The transmitter (bottom) is tied to the USB port. This test sends 25 packets per second through the air (data + ack at 80 msec intervals). Probably OK for a quick test, but not for prolonged use!

This test setup makes it easy to check range by walking around with the receiver. Turns out that it’s just fine for my purposes: this works across 3 stone walls / concrete floors. Data rate is ≈ 57600 baud.

The changes to the Ports library have been included in the new Ports.zip release and in subversion, see also the previous post. Two new examples have been added: blink_xmit and blink_recv. The receiver code decodes and interprets each packet and performs all port I/O requested by the transmitter. Input values are then sent back in the ack packet.

The Arduino-0013 IDE compiles “blink_xmit” and “blink_recv” to 3128 and 2644 bytes, respectively. So there’s plenty of room for more functionality.


Ports library for Arduino

In AVR, Software on Feb 14, 2009 at 00:03

Here is a “Hello world” sketch for JeeNodes, luxury edition:

Ports library for Arduino

It drives 4 LEDs, one connected to each the DIO + GND pins of each “port”. The LEDs on ports 1 and 4 blink at different rates, while LEDs 2 and 3 use the PWM output capabilities to continuously light up and then dim again (like Macs do in sleep mode). Only ports 2 and 3 support PWM.

Here’s a setup using four 4-pin headers on a JeeNode with a mini breadboard (port pins 1 and 6 are rarely used). Note how ports 1 & 2 are rotated 180° w.r.t. ports 3 & 4:

Ports library for Arduino

Each of the LEDs is connected to port pins 2 and 3 with a series resistor. The red LED is port 1, then counter clockwise ports 2 (green), 3 (green), and 4 (yellow). The wireless isn’t being used in this example.

This uses a new “Ports” library which is available as Ports.zip or via subversion. This code needs to be placed in the Arduino IDE’s “libraries/” folder. It includes this “blink_ports” example.

My next post will explain why I went through the trouble of creating this Ports library. For now it’s just a thin wrapper around the standard pinMode(), digitalRead(), digitalWrite(), analogRead(), and analogWrite() calls.

Update – sources now in subversion.

,

RFM12B library for Arduino

In AVR, Software on Feb 10, 2009 at 00:03

Here’s a driver for the RFM12B radio with Arduino’s and similar AVR boards. The radio is connected as follows:

RFM12B

The code is available as RF12.zip or via subversion. It’s packaged as an Arduino library. To try out the included example, you’ll obviously need two Arduino’s + radio modules, connected as above. Then:

  • unpack the above zip as new directory in your Arduino IDE’s “libraries/” folder
  • build and upload the included example sketch to both systems
  • check proper operation by opening the serial port at 57600 baud
  • you should see a “[RF12DEMO]” greeting on both
  • on one unit, type “0i 0i 1i” to set its node ID to 1
  • on the other, type “0i 0i 2i” to set its node ID to 2
  • send a test packet of 7x <N> bytes by typing “<N>s” (<N> = 0..9)
  • you should see the test packet on the other node
  • use “<N>a” to send a test packet and request an ack

The node ID is stored in EEPROM (the demo uses byte 0), so you only need to initialize ID’s once in each node.

This code uses interrupts to do most of the work in the background for transmission as well as reception. The packet buffer is limited to 66 bytes of data. All packets are verified with a 16-bit CRC.

Packets can be sent to a specific node (ID 1..30) or broadcast to all (ID 0). The demo code sends its test packets as broadcasts, but replies with acknowledgements to just the originator. Note that requesting an acknowledgement for broadcast packets only makes sense with two nodes.

Update – instructions for the demo are now slightly different (see the README).


Ignoring imprecise data

In Software on Feb 9, 2009 at 12:05

For some time, I’ve had a second set of calculations running which produces much more accurate and regular data than some of what’s on these past charts:

Picture 1.png

The difference with the graphs is that only readings which come in exactly at the time the counter changes are considered here. All retransmission (i.e. readings tagged with a late time stamp) are simply ignored. Only one retransmission happens to be in this example (the one marked “6+2″), but it still produces consistent results.

It looks like the gas measurements could be improved by tagging all readings in the sensor node instead of at the receiving end of the packets. That way retransmissions would still be tagged properly, even when readings arrive a bit late.

Hmm, this requires running a clock with milli-second accuracy on the sensor board for best results… not entirely trivial over extended periods of time. The sensor board will need to be synchronized to some other clock – either an on-board DCF77 radio or some data received from another node. And I’ll need to use a crystal instead of a resonator (0.005% vs 0.5% drift).


Still not right

In Software on Feb 8, 2009 at 23:55

The gas measurements are still giving me trouble:

Picture 3.png

I’ve changed scales and plotted all values to better see what’s going on over a small time frame. The electricity consumption looks ok now (these are night-time values). But the gas measurements keep jumping up and down by a factor of around 5 all the time!

Maybe there’s a problem with the sensor readout – there are about 4 readings coming in per minute. Or perhaps the automatic idle transmission of packets every 10 seconds is interfering with this cycle?

I don’t know what’s going on… needs more work!


Measurement anomalies

In Software on Feb 7, 2009 at 02:38

Here’s a more detailed view of the measurement quirks related to gas consumption:

Picture 1.png

The yellowish line is electricity usage, which hovers around 200 watt, while the blue line is the calculated gas consumption. What happens is that the heater cycles on and off approximately once every 15 minutes. When off, no gas is consumed so the meter just stops providing pulses. Then, after roughly 10 minutes it starts again at our heater’s standard rate of around 1.5 m3/hr. If you look very closely, you can see that the “zero” values are actually just above zero. That’s because whatever gas flowed until the rotation stops and whatever gas flows right after it starts all gets averaged-out over the entire period.

What needs to be done, I think, is to re-interpret the long averaged-out period as a period of continued flow at the original full rate, followed by zero flow:

Picture 3.png

I.e. the incoming data was treated as the single-hatched “B”, whereas it should be treated as the cross-hatched bar “A”. Both areas are the same.

Actually, this is not 100% accurate since we don’t know whether the meter stopped near the beginning or near the end of its revolution. But there is simply not enough information in the gas measurement data to disambiguate these cases.

However… as you can see, the electricity consumption also has some surprisingly regular small changes. Those measurements suffer from the same problem, but they have a much smaller impact due the the higher measurement rate. My hunch is that these small wattage changes are actually the electricity consumption of the heater, as it turns on and off!

But hey, who cares for now – these are just instant-by-instant consumption rates. The integral of these rates is the total consumption, and those are by definition “accurate” since I’m measuring the very same metering revolutions which the electricity and gas company use to charge me later on.

Update – Here are the results with the above adjustments:

Picture 1.png


Energy plots

In Software on Feb 6, 2009 at 00:03

And here’s the electricity and gas consumption for (part of) last Wednesday:

Picture 1.png

There are some quirks due to the way things are measured. The meter cannot accurately detect when gas consumption drops to zero, since that simply produces no readings. This can be resolved by adding fake readings when a very low consumption is reported, estimating the times when the gas flow stopped and then started again.

The electricity peaks at 19:45 are from the induction stove, the one around 20:15 is probably the close-in boiler.

This Tcl code generates the necessary HTML/JavaScript snippet to produce the plot:

Picture 3.png

All measurement data selection is handled by the “dataset loop” command, which takes a measurement parameter name, the selection range, and the names of the time and value variables to set during the loop.


Temperature graphs

In Software on Feb 5, 2009 at 13:30

Some weather & energy data has been coming in over here for over a month now, logged to text files, and patiently awaiting further processing. Here are the results of a few days of hacking around on the software side of things:

Picture 1.png

This is a test page served by a custom web server, with all data now stored in a database. I’ve used several bits of software which are dear and near to me:

  • the database used is my own trusty Metakit
  • the custom software is written in Tcl
  • the web server is based on my Rig modules
  • the plots are generated in the browser by Flot
  • it’s all in two files using using Tclkit / Starkits

The code is portable and works on Mac OS X, Windows, and Linux. The system requirements are minimal, so it runs just fine on my NSLU2 and Bubba NAS boxes.

Things are starting to come together nicely…

16 x 8 LEDs

In AVR, Software on Jan 30, 2009 at 21:51

Here’s the umptieth LED board:

DSC_0039.jpg

It’s been hooked up to SPI and FTDI connectors, and the two 8×8 LED matrices have been tied to an ATmega168 and an A6276 chip, described as a “16-Bit Serial Input, Constant-Current Latched LED Driver”. The voltage regulator and the I2C/power pins on the right haven’t been hooked up yet.

This shows the backside, which is surprisingly robust once done, but a huge pain to build:

DSC_0040.jpg

I use wire-wrap wire because its insulation can stand the soldering heat without problems and because the wire bonds easily with solder. For anything but one-offs, pcb’s are obviously a much better way to go.

Here’s the unit hooked up via a USB/FTDI board by ModernDevice:

DSC_0042.jpg

Here’s a better view with one LED actually turned on:

DSC_0044.jpg

The brightness and contrast are actually far better than in the above picture.

The design is made for scanning across columns while providing 8 bits of on/off data for the current column via 8 output pins on the ATmega. There’s a 16 MHz resonator on board, so this thing is fully compatible with the Arduino IDE.

Here’s a sketch which makes a single led turn on and meander across all positions:

Picture 1.png

This was used to verify that each dot is accessible and that there are no short circuits. The analogWrite() call is used to vary the display brightness, by controlling all LED driver outputs via PWM. Another useful property of this driver chip is that it needs only one resistor to limit the current through each of the leds.

Update: Silly me, with constant current on the columns, I have to cycle the rows of course. Will need to add an ULN2981 driver to allow the row pins from the ATmega to drive up to 16 leds at once. With this change the LEDs will cycle at 125 i.s.o. 62.5 Hz and can be slightly brighter.

Reliable comms with RFM12B

In AVR, Software on Jan 23, 2009 at 16:34

The driver code for RFM12B radio modules is starting to shape up nicely. It’s all interrupt-driven, so it works mostly in the background. Here’s a complete sketch for Arduino, allowing bi-directional sending of test packets with or without acknowledgement:

Picture 1.png

Most of this logic is the test code itself. The main interface calls are:

  • rf12_initialize() – call this one in setup()
  • rf12_recvDone() – returns 1 when a complete packet has been received
  • rf12_canSend() – call this to start sending when possible
  • rf12_sendStart(hdr,buf,len) – this starts actual transmission

All the interrupt stuff happens behind the scenes, the only requirements being that rf12_recvDone() should be called frequently and that rf12_sendStart() may only be called after rf12_recvDone() or rf12_canSend() return true, i.e. inside the above if statements.

The driver itself is under 200 lines of C code, it’ll be released as open source once the code settles a bit.

A simple RF pulse scope

In AVR, Software on Jan 10, 2009 at 23:45

Here’s a simple way to analyze RF pulse trains, as emitted by simple 433/868 MHz remote controls. A receiver for the proper frequency band is required, as well as some Arduino board connected to USB.

The code is a sketch which picks up all 0/1 transitions (i.e. presence/absence of the radio carrier) and reports the elapsed time between each transition:

Picture 1.png

The radio signal is tied to the analog comparator (pin 7), and the timer is set to count in 8 µsec steps. This rate was chosen to allow for pulses up to 2 msec.

The data is sent as a stream of bytes. Values 2..255 correspond to pulse durations of 16..2040 µsec. A “0″ is sent for longer breaks, a “1″ is sent if the data was not transmitted fast enough over USB, i.e. an overrun. Note that “0″ values are only sent once, i.e. when there is no signal no bytes are sent to the USB interface.

On the computer side, all sorts of software could be used/written to analyze these pulse trains. I used Tcl for a one-off analysis of KlikAanKlikUit remotes operating at 433 MHz, though it would be really neat to have some sort of logic analyzer GUI or web interface added to this.

Transmit via interrupts

In AVR, Software on Jan 8, 2009 at 16:37

Making the transmit logic of an ATMega MPU with RFM12B radio module interrupt-driven is not very hard, as it turns out. The idea is to use states to decide what to do when a new transmit interrupt comes in. These are the states I’m using:

Picture 1.png

And this is the interrupt handler:

Picture 2.png

Lastly, this code starts a new transmission (assuming txstate was TXIDLE):

Picture 3.png

The main trick used in the above is that the state usually increments on each interrupt and that negative states are used to send all the payload data bytes. So the choice of state values matters, i.e. TXCRC1 has to be zero.


More readings

In Software on Dec 19, 2008 at 15:55

Here is the latest display showing on my screen:

7474AFF5-C9D3-4DA7-B5E3-492EE65C15DB.jpg

Updated in real time, with new readings coming in every few seconds for electricity and gas and every few minutes from the weather station (still sitting here in the office for now, so the values are bogus).

Good RF with RFM12B

In AVR, Software on Dec 11, 2008 at 15:38

Yippie. Got a simple blinking LED demo running on RFM12B’s after lots of head-scratching. Had to understand ATmega’s SPI mechanism to get there, but these two boards now talk to each other over the air, uni-directionally:

Good RF

That’s an Arduino Mini Pro (which runs at 3.3V internally, as does the RFM12B). Power comes from the FTDI cable, but with a special hack to drive the raw power input from it, not the default VCC connection. This way, the on-board 3.3V regulator is used to lower the voltage for both the MCU and the wireless module.

The other board is a bit of a mess, has the same Mini Pro, but some other stuff which is partly a left-over from earlier experiments:

Good RF

Here a USB interface is used to provide 3.3V power. It’s functionally identical to the first board: with RFM12B’s the same setup can be used as transmitter or as receiver.

The top board is the receiver – its picture shows the green LED on, i.e. correct packet reception!

The hookup is very simple, just 5 wires between RF module and MCU, and 2 power pins / 1 pull-up / 1 antenna wire on the RF module is all it takes. The 5 wires are standard SPI plus an IRQ line:

  • MOSI (MCU) –> SDI (RFM)
  • MISO (MCU) <– SDO (RFM)
  • SCK (MCU) –> SCK (RFM)
  • SS (MCU) –> nSEL (RFM)
  • PD2 (MCU) <– nIRQ (RFM)

It took some work to get the software working. The trouble with this stuff is that until you get both transmitter and receiver working properly, it all remains a bit hard to debug. And there are quite a few registers to set up on the RFM12B to make it do its magic.

But after some tweaking, the rfm12xmit and rfm12recv sketches for the Arduino IDE started working. First with bit-banged SPI coding, but then also with the ATmega168’s built-in SPI hardware. Which is nice, because all exchanges now take place at 1..2 MHz (i.e. 8..16 µsec per 16-bit command). All I/O with the RF module uses simple busy polling for now.

The RF range and quality of reception with the RFM12B is considerably better than my earlier 868 MHz AM setup. Packets now easily get across 3 layers of stone/concrete and all of them appear to be arriving properly.

Update – here’s the C source code for the xmit and recv sides.

Update 2 – the connections for MISO and MISO as listed above were reversed – many thanks to José Xavier for pointing this out. The text above has been fixed.

Timer Interrupts

In AVR, Software on Dec 6, 2008 at 15:33

Here is some new interrupt-driven receiver code. This receiver is considerably better at identifying correct packets, regardless of what other bit patterns are coming in. The new code is based on a state machine which takes pulse times and gaps into account, detecting all valid sequences of pulse widths and rejecting all outliers. The main code is:

AA4D1C74-E3E3-41B7-BD67-90E432711496.jpg

Then, in the main loop, this code picks up the results:

EFC8BB55-ECC5-4553-9C8D-17CB1F9A92CC.jpg

Also, with an interrupt driven design the receiver no longer depends on timing loops and is free to spend its time in other logic. It is now possible to run multiple state machines in parallel, decoding a variety of different signal patterns.

Success @ 868

In AVR, Software on Dec 3, 2008 at 15:29

As expected, a lot of the bit-level reception problems with the 433 & 868 MHz wireless issues can be resolved with better packet framing and reception logic. Here’s part of the latest reception log, alll the way across those elusive 3 stone/concrete walls:

93E74FE4-4BB6-4209-A2EE-7700D52833DF.jpg

There are still times when proper reception stutters, but as long as reception recovers within a minute or so that’ll be fine for the purpose of low-rate data logging.

The send format now includes an old-fashioned SYN-bytes-plus-SOH header:

176528F4-75FD-4BB2-B7CA-5CF5EA7105A9.jpg

Reception takes a bit more code, here’s the basic frame reception loop:

86C83486-9950-4FB9-B4BE-8D5FB482ED49.jpg

And here’s the bit-level decoder:

23F292A5-C471-4C4E-A654-96D7A5E10263.jpg

The receiver + serial debugging code takes a mere 3208 bytes so far.