» Software Jee Labs

Fractional bits?

In AVR, Hardware, Software on Aug 28, 2010 at 00:01

To continue this little weblog series on bits, I’m going to go into bit fractions.

Yeah, right… there is no such thing, of course – unless you’re into probabilities or fuzzy logic, perhaps.

What I actually want to do, is describe “analog output” on the ATmega, using the Arduino library’s analogWrite() function. And throw in some bit manipulations along the way, to stay somewhat on topic.

The little secret with analogWrite() is that it doesn’t do what its name suggests. The ATmega has no way of generating an analog signal, i.e. a voltage level between 0 and VCC.

Instead, a pulse is generated, with a varying duty cycle. I.e. the “on” and the “off” times of the pulse will be different, the ratio of these times being proprtional to the 0..255 value passed as argument to analogWrite(). With “0″, the signal will be 100% off, with “255″ the signal will be 100% on. With “128″ the pulse will be on the same amount of time as off. With “1″, it will be on very briefly, and then off most of the time, and so on…

The neat thing is that you connect it to an incandescent lamp, or a motor, then the effect will be that these will light/turn at less than their full power, due to the time it takes for these devices to try and follow the pulse. So the effect is similar to a fractional adjustment: you can dim / slow down these devices by using analogWrite().

It even works with LEDs, although these turn on and off very fast. In this case, the reason is that our eyes can’t follow such fast changes, and so we perceive the result as dimmed as well. A whole industry was once created around this “persistence of vision” property of our eyes – it’s called TV…

Here’s a sketch which uses this pulsed output to control the brightness of a LED connected to DIO3 (i.e. D6):

Screen Shot 2010 08 27 at 16.44.38

Note that I didn’t have to define pin 6 as an output, analogWrite() does that.

What the above does, is ramp up gradually from 0 to 255, and then repeat:

Screen Shot 2010 08 27 at 16.49.06

Suppose we want it to fade in and out instead:

Screen Shot 2010 08 27 at 16.49.17

Try implementing this yourself.

Note that you’re going to need at least 9 bits of information to do this: 8 for the brightness level and 1 to keep track of whether you’re currently in the up ramp or in the down ramp:

Here’s one way to do it, using some bit trickery:

Screen Shot 2010 08 27 at 17.05.15

A few notes:

I’ve changed the “level” variable from an 8-bit byte to a 16-bit word
bit 8 toggles from 0 to 1 and back every 256 level counts
it’ll be 1 when level is 256..511, 768..1023, etc
when it’s 1, we flip the bits, i.e. 0 becomes 255, 1 becomes 254, etc
the analogWrite() function ignores all upper bits

If you think that was an obscure call to analogWrite(), try this one:

    analogWrite(6, level ^ -((level >> 8) & 1));

Maybe you can decypher it when written slightly differently?

    analogWrite(6, level ^ -bitRead(level, 8));

(hint: bitRead() always returns either 0 or 1)

It’s all pretty geeky stuff, and let’s hope you’ll never have to deal with code such as this again, but the point of this story is that there’s no magic. You just have to know what each operator does, and how to translate an integer from decimal to binary notation and back.

I’ll summarize my intuitive interpretation of bit operators below:

“X | Y” = take X and copy all the 1′s of Y into it
“X & Y” = take X and copy all the 0′s of Y into it
“X ^ Y” = take X and flip all the bits where Y has 1′s
“~ X” = flip all the bits of X
“- X” = arithmetic minus (same as “(~X) + 1″ !)
“! X” = 1 if X is zero, 0 otherwise
“X << N” = multiply X by 2, N times
“X >> N” = divide X by 2, N times

Some tricks based on this:

“~ 0” = all bits set to 1 (same as “-1″ !)
“~ 0 << N” = all bits 1, but N lowest bits set to 0
“bit(N) – 1” = a constant with N lowest bits set to 1
“X & (bit(N) – 1)” = the N lowest bits of X, the rest is 0
“X & ~ (bit(N) – 1)” = X, but with the N lowest bits set to 0
“!! X” = 0 if X is zero, 1 otherwise

An useful rule when writing logical expressions is: when in doubt, parenthesize! – see C operator precedence.

Sooo… use bit(), bitRead(), bitWrite(), bitSet(), and bitClear() wherever you can, since it usually makes the code easier to read. But there’s no need to get lost if you see ^&|~!’s in your expression – just slow down and decode such expressions step by step!

Flippin’ bits

In AVR, Hardware, Software on Aug 27, 2010 at 00:01

After yesterday’s post about setting and clearing bits, let’s explore reversing bits, i.e. changing them from 0 to 1 and back. And let’s do it by blinking an LED attached to DIO of port 1 – i.e. Arduino digital pin 4:

Screen Shot 2010 08 26 at 10.31.26

The “if (onOff = 0)” etc is the logic that toggles onOff between 0 and 1 on each pass through the loop:

    if (onOff == 0) onOff = 1; else onOff = 0;

But there are lots of ways to do the same thing, coded differently:

    if (onOff == 0) onOff = 1; else onOff = 0;
    if (onOff == 0) onOff = bit(0); else onOff = 0;
    if (onOff == 0) bitSet(onOff, 0); else bitClear(onOff, 0);
    onOff = onOff ? 0 : 1;
    onOff = (~ onOff) & 1;
    onOff = (onOff + 1) & 1;
    onOff = ! onOff;
    onOff = 1 - onOff;
    onOff = onOff ^ 1;
    onOff ^= 1;

See if you can figure out all of these.

Take your pick. Those last two use C’s XOR operator. I tend to prefer shorter source code, so I’d use that last notation (note that the resulting compiled code is not necessarily shorter than the other examples).

Now suppose you have a byte value “X” and you want to flip the 4th bit in it, while not changing anything else. That’s a bit more work. It could be done like this, for example:

    if (bitRead(X, 4) == 0) bitSet(X, 4); else bitClear(X, 4);

Or like either of these:

    X = X ^ bit(4);
    X ^= bit(4);

This shows clearly that the “^” XOR operator does exactly what we need: flip bits.

Back to blinking an actual LED, as done with the above sketch. Here’s a little mind bender – another sketch, doing the same using raw ports and the XOR operator:

Screen Shot 2010 08 26 at 10.58.10

The first example was doing things “the Arduino way”, using pinMode() and digitalWrite(). It compiles to 890 bytes of code. This second example goes straight to the hardware and uses 554 bytes of code:

Arduino digital pin 4 is bit 4 on the “D port” of an ATmega
“DDRD” is the “Data Direction Register”, where we set up pin 4 as an output
“PORTD” is the out “Port Register”, which controls the actual output signal

You can see the XOR in action in that last example. It takes all the output bits of port D (Arduino pins 0 .. 7), and flips just a single bit, i.e. bit 4.

Just for kicks, I’ll show you one more way to blink the LED:

Screen Shot 2010 08 26 at 11.03.44

This uses a relatively little-known feature of the hardware, which actually has “bit flipping” built-in. The “PIND” register is normally used for input, i.e. for reading the state of a pin as an input signal. But you can also write to that register. When you do, it will be used to flip output pins, but only for the bits which were set to 1. It’s essentially a built-in XOR.

That last example uses 550 bytes of code, most of which is overhead from the Arduino run-time library (setting up the milliseconds timer, etc). So what’s in a measly 4 bytes, right? Wrong. There is a minute, but sometimes important difference: the other approaches all had to read the register value first, flip the bit, and then write the value back. This last version only writes a (constant) value to a register. With interrupts, that can be very important: this last version can’t ever go wrong, it will always flip the requested bit. The other version could have an interrupt occur between the read and the write. It’s a known issue for the Arduino Mega. It can lead to code which runs for a week, and then fails mysteriously. Bugs like these are fiendishly hard to properly diagnose.

Bit-flipping can be quite useful for physical computing. Not only does it let you easily toggle specific bits, and change the state of some output pins, it can also be a way to clear a bit. Let’s say you need to generate a (very) quick pulse. Here are four ways to accomplish the same thing:

    bitSet(PORTD, 4); bitClear(PORTD, 4);
    PORTD |= bit(4); PORTD ^= bit(4);
    PORTD |= bit(4); PIND = bit(4);
    PIND = bit(4); PIND = bit(4);

That second one based on XOR works, because bit 4 is known to be one, so setting it to zero is always the same as flipping it. That’s also why the third PORTD/PIND example works, with PIND doing the XOR in hardware. Lastly, the fourth approach will only work if bit 4 was initially zero. It’s the fastest one, and does not suffer from the interrupt race condition mentioned above.

Ok, that’s enough flippin’ for one day!

Tomorrow, I’m going to go into, ehm… “fractional bits” (haha!) ;)

Update – see comment below on why “bitSet(PORTD, 4); bitClear(PORTD, 4);” are also interrupt-safe (mostly – but not on every pin of an Arduino Mega!).

Bit manipulation

In AVR, Software on Aug 26, 2010 at 00:01

Today I’d like to go into bit manipulation, ehm, a bit…

You need bit manipulation when you’re dealing with the individual bits in a byte, such as on the I/O ports of an ATmega, for example.

First the easy approach – use these predefined macros from the Arduino library:

bit(N) returns an integer with the N’th bit set to 1
bitRead(X,N) – returns the N-th bit of X as 0 or 1
bitWrite(X,N,B) – sets Nth bit of X to B (0 or 1)
bitSet(X,N) – sets the Nth bit of X to 1
bitClear(X,N) – sets the Nth bit of X to 0

This is why you might see code such as the following:

    bitSet(WDTCSR, WDIE);

This means: “set the Watchdog Interrupt Enable to 1 in the Watchdog Timer Control Register”. The WDTCSR and WDIE terms are predefined constants. WDIE is 6, for example.

Note that some of these routines can be written in terms of the others, i.e.

bitSet(X,N) is the same as bitWrite(X,N,1)
bitClear(X,N) is the same as bitWrite(X,N,0)

But what does it all mean?

Well, let’s dive in. First make sure that you are comfortable with “bit shifting”. The expression “bit(3)” is the same as “1 << 3″, which in turn is the same as doubling the value 1 three times, i.e. the value eight. So “bit(0)” is 1 doubled zero times (i.e. 1) and “bit(7)” is 1 doubled 7 times, i.e. 128. Bytes have 8 bits numbered 0 to 7, so all you need for (byte-sized) hardware registers is to remember that bits 0..7 map to (specific!) integers with values 1 to 128.

Setting a bit, is like OR-ing the bit with the rest of the value. The following statements are all identical:

    WDTCSR = WDTCSR | bit(WDIE);
    WDTCSR = WDTCSR | (1 << WDIE);
    WDTCSR = WDTCSR | (1 << 6);
    WDTCSR = WDTCSR | 0b1000000;
    WDTCSR = WDTCSR | 0x40;
    WDTCSR = WDTCSR | 64;

This, in turn, can be abbreviated in C as:

    WDTCSR |= bit(WDIE);
    WDTCSR |= (1 << WDIE);
    WDTCSR |= (1 << 6);
    etc...

Or you could write:

    bitSet(WDTCSR, WDIE);

It’s all the same. So OR-ing is about setting bits (to 1).

Likewise, AND-ing is about not clearing bits (to 0). Whoa, that’s confusing. This expression returns a value which is what X was, but only for bit N:

    X & bit(N);

So this will change X to a value with all bits except bit N set to zero:

    X = X & bit(N);

To put it differently: X will lose its original bits, except bit N, which will be left alone. All the bits are set to zero, except bit N.

Usually, you want the opposite, setting only bit N to zero. That too is accomplished with AND-ing, but you have to flip all the 0′s to 1 and all the 1′s to 0 first. Hang in there, it’s a slightly longer story. This sets bit N to zero:

    X = X & ~ bit(N);

Let’s examine what’s going on here. First, “bit(N)” is a value with only the Nth bit set. Now, “~ bit(N)” is a value with all the bits flipped around (“~” is called the complement operator in C), so that’s a value with all but the Nth bit set. Everything is 1, but bit N is 0.

Now we can tackle the expression “X & ~ bit(N)”. Since AND-ing is about “not clearing bits”, that means that the result of this expression is all the bits of X unchanged where “~ bit(N)” was one, which is almost everywhere. The only bit that differs is bit N – it is zero in “~ bit(N)”, therefore that particular bit will “not not clear …” (a double negation!): it will be cleared (to 0) in the result.

Finally, we replace X by that result. So X will change in precisely one bit: bit N. That bit will be cleared to zero, the rest is not affected. In short: we’ve cleared bit N.

Confused?

Well, that’s why the bit/bitSet/etc macro definitions were introduced. These expressions are all identical:

    X = X & ~ bit(N);
    X = X & ~ (1 << N);
    if (X & bit(N)) X = X - bit(N);
    bitClear(X, N);

That last one is clearest by far, because it conveys the actual operation with a well-chosen name: clear bit N, leave the rest alone.

So why would anyone ever choose to use anything but the bit/bitRead/etc routines?

Many reasons. Habit, perhaps. Coming from another environment which doesn’t have these macros. Being so used to this bit-manipulation that the use of words doesn’t really look any clearer. Whatever…

But another more important reason is that you can’t do everything with these bit/bitSet/bitClear routines. Sometimes you just have to go to the raw code. Such as when you need to set multiple bits at once, or flip bits. That’s why the ATmega datasheet has examples like these:

    WDTCSR |= (1<<WDCE) | (1<<WDE);

By now, you should be able to decode such a statement. It’s the same as:

    WDTCSR |= bit(WDCE) | bit(WDE);

In other words:

    WDTCSR = WDTCSR | bit(WDCE) | bit(WDE);

Which in turn is almost the same as these two statements together:

    WDTCSR |= bit(WDCE);
    WDTCSR |= bit(WDE);

That in turn, can be written as these two lines:

    bitSet(WDTCSR, WDCE);
    bitSet(WDTCSR, WDE);

Much clearer, right? Except… it’s not 100% identical!

The problem is a hardware issue: timing. The above two statements will set both bits, but not at the same time! For hardware register settings, that difference can be important. There is a fraction of a microsecond between the WDCE bit being set and the WDE bit being set. Unfortunately, in some cases that causes real problems – your code won’t work as expected.

Tomorrow, I’ll continue on this topic, but it’ll be a bit more fun, because there will be LEDs involved!

Dsc 1385 2

(Please ignore the cable on the left, I snatched the above picture from this post)

Update – see this excellent Wikipedia article for more details about bitwise operations.

JeePlug, Techniques

Simplified button interface

In AVR, Software on Aug 24, 2010 at 00:01

The Blink Plug has two pushbuttons and two LEDs. The buttons are simple miniature switches, but nothing is ever simple in microcontroller-land: reading out the state of a pushbutton reliably can be deceptively hard, due to mechanical bounce issues.

The Ports library has had a BlinkPlug class for some time now, including a “pushed()” function to do all the debouncing. Unfortunately, that function turned out to be a bit harder to use than I originally intended.

Time to add some more code to the BlinkPlug class!

I’ve added a new “buttonCheck()” member, which returns events instead of state. That makes it a lot easier to detect when people press a button, which is usually all you’re after anyway.

Here’s a new button_demo.pde example sketch, which illustrates the new functionality:

Screen Shot 2010 08 23 at 17.44.21

Sample output:

Screen Shot 2010 08 23 at 17.44.39

As you can see, it’s now a lot simpler to detect when people press or release one of the two buttons on a Blink Plug. Each time you call buttonCheck(), you’ll get one of the following events:

ALL_OFF, ON1, OFF1, ON2, OFF2, SOME_ON.

You have to keep calling “buttonCheck()” reasonably often, at least 10 times per second, if you don’t want to miss any events. Calling it all the time in the main loop is fine. Keep in mind that ON1, etc. will be returned only once for each actual button press.

You can still call “state()” whenever you want, to check the position of either button. But when you use buttonCheck(), you should not call the old – now deprecated – “pushed()” function, as these two will interfere with each other.

This code is now part of the Ports library (subversion and ZIP). Gory details are in Ports.cpp, near line 230.

JeeMon, Network

Pulling data from an EtherNode

In Software on Aug 17, 2010 at 00:01

Last month’s EtherNode sketch was an example of a simple web server which allows viewing incoming packets received by the RFM12B. Here’s a sample web page again:

Screen Shot 2010 07 13 at 231929

If JeeMon could access and pick up that data without requiring an extra JeeLink or JeeNode, then you could place the EtherNode wherever reception is best while running JeeMon on your desktop machine, or anywhere else.

In response to a request on the forum for just that, I started writing a little demo “application.tcl” for JeeMon to do this sort of web-scraping. Here’s what I came up with (code):

Screen Shot 2010 08 16 at 10.35.49

Sample console output:

Screen Shot 2010 08 16 at 10.42.48

The point here, is that it needs to periodically poll the EtherNode, get a web page from it, and skip the readings it has already seen before. That’s what most of the code in “EtherNodePull” does. Each packet that remains will be sent to the “GotPacket” proc, which just logs it on the console.

But that’s just one half of the required solution…

The bigger challenge is to also make JeeMon decode these packets, as if they came in through a serial USB link. There is quite a bit of logic in sketches/central/host.tcl to do that for a JeeNode or JeeLink running the “central” sketch (which is almost identical to RF12demo).

The reason this is more complicated, is that I want to be able to decode each packet in different ways, depending on the sketch running on the remote (sending) node. My network has more than just room nodes, and will be extended with many more node types in the future.

One workaround would be to collect all nodes of the same type in their own group, i.e. net group 1 for room nodes, net group 2 for the ookRelay, etc. And yes, that would work – but it’s not very convenient, and I’d need separate etherNodes to pick up the packets from each net group. Messy.

The approach I have used so far, is to maintain a config section for JeeMon, with information about the type of each node, organized by frequency band, net group, and node id:

Screen Shot 2010 08 16 at 10.52.23

It’s not automatic, but this way I just need to adjust one list whenever a new wireless node is brought online.

The current code in sketches/central/host.tcl is all about picking up packets, and mapping them thtough this configuration section to know what is what. It does this by setting up a pseudo “connection” whenever packets come in for the first time and includes logic to tear down this connection again when no new packets are received within a certain amount of time.

To use this approach with an EtherNode as data collection node, I need to re-factor the exisiting code and make the core mechanism independent of the Serial implementation. I also need to bring more of the code from central/host.tcl into the JeeMon code, so it can be re-used for EtherNodes.

Re-factoring is my middle name – I’ll update this post when the code changes are complete.

JeeNode, Network

Improved EtherNode

In Software on Jul 14, 2010 at 00:01

The EtherNode.pde sketch described in this post not so long ago has been extended a bit with some of the functionality of the RF12demo.pde sketch.

That way, an EtherNode can now also be used as sort of a replacement for RF12demo – through Ethernet i.s.o. USB: i.e. to collect data (by polling the server and scraping all new data from the home page) and to send out packets (via a GET request with a properly-formatted HTTP query string).

First of all, the configuration settings are now saved to EEPROM, so that the webserver will come back with the same settings after a restart:

Screen Shot 2010 07 13 at 23.13.11

As you can see, the “collect” mode has also been added. When enabled, the EtherNode will not reply with an ACK when receiving a packet. The issue here is that at most one node should respond with an ACK to broadcast packets. IOW, for each net group you use, you should normally have a single node running RF12demo or EtherNode with collect mode turned off. The others will be “lurking” (with collect mode enabled), i.e. they will be able to see all packets but they won’t reply.

Note: there is one other way to deal with collect mode and ACKs: if you have software such as JeeMon (or your own code) set up to explicitly generate an ACK, then again you should set collect mode on, so that the ATmega itself doesn’t generate the ACK as well. This is why this feature was called “collect mode” btw: the ATmega collects packets and passes it hrough, but it does not act as a full receiver which sends out ACKs.

Also new is that for debugging, you can now connect a serial console to see the full incoming requests:

Screen Shot 2010 07 13 at 23.12.22

The main screen hasn’t changed very much:

Screen Shot 2010 07 13 at 23.19.29

The only changes are a new link to a “send packet” page, and the fact that packets are now properly numbered from 0000 to 9999 (and then wrapping back to 0000).

The send packet page is new: it lets you send out a packet with up to 66 bytes of arbitrary data, to either a specific node, or as broadcast:

Screen Shot 2010 07 13 at 23.43.52

There is currently no way to request an ACK or send out ACKs via this mechanism. It’s just for sending out a data packet via a web request.

This is just the beginning of what’s possible with a JeeNode hooked up to Ethernet, of course. The sky is the limit, since everything else is a matter of uploading new software. Long live open source :)

Serial communication vs packets

In Hardware, Software on Jul 12, 2010 at 00:01

When you hook two devices up via wires, you’ve got essentially two options: parallel, i.e. one wire for each bit you want to transmit & receive (example: memory cards inside a PC). Or serial, where information gets sent across bit by bit over only a few wires (examples: ethernet, USB, I2C). Parallel can achieve very high speeds with little circuitry, but serial is more convenient and cheaper for large distances.

Serial communication is very common. The model even carries through to the way we think about the “command line” – a stream of characters typed in, followed by a stream of output characters. Not surprising, since terminals used to be connected via RS232 serial links.

Wireless connections are also essentially serial: you rapidly turn a transmitter on and off (OOK), or you change its frequency of operation (FSK), to get the individual bits across.

But there’s a lot more to it than that.

With two devices connected together, you get a peer-to-peer setup with a link which is dedicated for them. This means they can send whenever they please and things will work. The same can be done with wireless: as long as only two devices are involved, one device can send whenever it likes and the other will receive the signal just fine (within a certain range, evidently).

With such a peer-to-peer setup, the serial nature of the communication channel is obvious: A sends some characters, and B will receive them, in the same order and (almost) at the same time.

But what if you’ve got more than two devices? Ah, now it gets interesting…

With wires, you could do this:

Screen Shot 2010 07 11 at 11.20.41

It’s easy to set up, but it’s pretty expensive: lots of wires all over the place (N x (N-1) / 2 for N devices) plus lots of interfaces on each device (N-1). With 10 devices, that would be 45 wires and 90 interfaces!

Worse still, this is very hard to use with wireless, where each “wire” would need to be a dedicated frequency band.

The solution is to share a single wire – called multi-drop:

Screen Shot 2010 07 11 at 11.24.58

Now there’s one wire, a couple of “taps”, and one interface per device. Much cheaper!

Trouble is, you’ve now created a “channel” which is no longer dedicated to each device (or “node” as it is usually called in such a context). They can’t just talk whenever they like anymore!

Whole new slew of issue now. How do you find out when the channel is available? What do you do when you can’t send something right away – save it up? How long? How much can you save up? What if someone else hijacked the channel and never stops transmitting? What if all nodes want to send more than the channel can handle? How do you get your information out to a specific node? Can all nodes listen to everything?

Welcome to the world of networking.

All of a sudden, simple one-on-one exchanges become quite complex. You’ll need more software to play nice on the channel. All nodes need the same software revision. And you’ve got to deal with being told “not now”.

Note that these issues apply to wired solution sharing the same channel (RS485, Canbus, USB, Ethernet) as well as all wireless networks.

Simple OOK transmitters used in weather station sensors just ignore the issue. They send whenever they want to, in an après moi le déluge fashion… (“what the heck, I don’t care whether my message arrives”). This usually works fairly well when transmissions are short, and when lost transmissions are no big deal – they’ll send out a new reading a few minutes later anyway.

Another aspect of this shotgun approach is that it’s a broadcast mechanism. The sending node transmits its messages into the air without interest as to who receives them, or whether there’s anyone listening even. All it needs to do is include a unique code, so that the receiver(s) will be able to tell who sent the message.

For weather sensors, the above is ok. For security / alarm purposes, it’s a bit unfortunate – missing an intrusion alert is not so great. So what the simplest systems do is to yell a bit louder: repeat the alert message many times, in the hope that at least one will arrive intact. No guarantees, yet some very common security systems seem to be happy with that level of reliability.

For more robust setups, you really need bi-directional communication, even if the payload only flows in one direction. Then each receiver can let the transmitter know when it got a packet intact.

There’s a lot more (software) complexity involved to use a channel effectively, to get data across reliably with “ACK” packets, to detect new and lost nodes, to deal with “congestion” and external causes of bad reception, etc.

With JeeNodes and wireless comms via the RFM12B module, the basic RF12 driver is somewhere in the middle between unchecked uni-directional transmission and fully checked self-adapting configurations.

So what does this all mean for the “end user” ?

Well, first of all: wireless communication can fail. A node can be out of range, or a badly-behaved machine can be sending out RF interference to such an extent that nothing gets across no matter what nodes do. Wireless communication can fail, it’s as simple as that! But with bi-directional communication, at least all nodes can find out whether things work or not.

The second key property of communication via a shared channel, is that you can’t just send whenever you like. You have to be able to either save things up until later, or discard messages to let future ones through.

This means that treating a wireless channel as a serial link is really a very bad idea. Keep in mind that the baudrate can drop to zero – this means that you must be prepared to save up infinitely much data for re-transmission. And the more you intend to re-transmit later, the longer you’re going to have to need that channel when it becomes available. That will frustrate all the other nodes trying to do the same thing.

One way around this, is to use a RF link with very high data rates. That way there will be a lot of slack when nodes want to catch up. But then you still need to be able to buffer all that data in the first place. Not a great idea for limited devices such as an ATmega…

The better way is to design the system to work well with occasional loss of packets. Take an energy meter, for example: don’t sent the pulse or rotation trigger, but keep a count and send the current count value. That way, lost packets will not affect the accuracy of the results, they will merely be updated less frequently when the RF link is down.

The RF12 driver used in JeeNodes was designed for the context of a little data, sent on a periodic basis. The difference with a serial link, is that you don’t get garbled text on the other side, but packets (i.e. chunks of data). All you need to keep in mind is that occasionally an entire packet won’t make it.

This design also deals with multiple nodes. Each incoming packet can have a “node ID” so receivers can tell everything apart. Packets never get mixed up or combined or split in any way. Each packet is a verified and consistent amount of data.

Couldn’t we implement a virtual serial link?

Well, yes – there are well-known techniques to implement a virtual circuit on top of a packet-based communication channel.

But doing so would be a bad idea, for reasons which have hopefully become clear from the above. A virtual circuit would either have to act as perfect channel (not feasible with finite data storage) or drop characters in unpredictable places. It is far more practical to impose a packet / chunk structure on the sender, and then be allowed to drop chunks with clearly-defined boundaries when the RF link is out of service or overloaded.

The moral of the story: think in packets when using JeeNode wireless comms – you’ll get a lot more done!

Update – see some good comments by John M below, about IP, UDP, TCP, and the OSI model which describes all the levels of abstraction involved with networking, and all the standard terminology.

Musings

Software hell

In Software on Jul 6, 2010 at 00:01

I wish this were an exceptional tale. But it isn’t, it’s the norm…

I run a webserver for Jee Labs and a couple of other web sites and services.

Screen Shot 2010 07 02 at 23.55.19

I wanted to add support for logging the last hours of a new “#jeelabs” IRC channel. IRC might not be a good fit for me personally, but since several people have asked for it, I do want to support it. With say the last 3 days of that discussion logged as some public web-pages, I could drop by once in a while and try and add to the discussion going on at that time.

The “eggbot” system was suggested to me as a way to accomplish this:

Screen Shot 2010 07 02 at 23.55.50

Ok. Server is Debian, so “aptitude install eggdrop” should work, right? Debian is good at package management.

Wrong.

Turns out my server is still running Debian Etch, and “Etch is end-of-lifed”, as I just found out. The server ISP’s repository is gone. No updates, no installs, no security fixes, nothing.

Ok. Better upgrade Debian Etch to Debian Lenny first.

I’ve been there before, I know how to do it. Takes a few hours, but then it’s done. Server is running Lenny now, and all the main servers and services are still running fine (impressive!). This is a server in Germany, btw.

That was yesterday (well, ehm… night).

Today, quick check. Server ok. Weblog ok. Forum ok. Cool.

Oops. Ikiwiki isn’t working anymore. That’s the software I use to convert the Markdown pages into the Cafe web site. Fully scripted, based on Perl.

Ikiwiki wasn’t installed as Debian package, because the one in Debian was waaaay too old. Even on Lenny, it lags (by over a year – Debian Lenny still has 2.x, Ikiwiki has been at 3.x for some time now).

Ok, so let’s try to re-install it from source. By now, usually all my warning systems go into in high alert mode, because source installs are a different ball game (even with a “scripted” language such as Perl, how ironic!).

The Ikiwiki installation page is reassuringly short. Three steps. First two look pretty easy, right?

PERL5LIB=`pwd` PERL_MM_USE_DEFAULT=1 perl -MCPAN \
    -e 'CPAN::Shell->install("Bundle::IkiWiki")'
PERL5LIB=`pwd` PERL_MM_USE_DEFAULT=1 perl -MCPAN \
    -e 'CPAN::Shell->install("Bundle::IkiWiki::Extras")'

Wrong.

I’m now waiting for step 2 to complete.

More than 10,000 lines of text have scrolled over my terminal window so far. This sort of drivel:

Screen Shot 2010 07 03 at 00.05.40

More than two hours have passed.

More than 180 Mb of disk space has been consumed.

I’m forced to watch closely, because disk space is running low on the root partition. And this is running as superuser, so it’ll bring down the server if it fills up the disk.

Pinch me, am I really in the year 2010?

Screen Shot 2010 07 03 at 00.08.42

My complaint is not that some things are complex and require a lot of work and understanding.

My complaint is that some things are NOT complex and yet you get treated to an incredible amount of nonsense. Not to mention the fact that the problem isn’t solved until you invest in figuring out every detail of lots of different kinds of packages.

The IRC logging? Not solved yet. I can probably write a 200 line script with not a single dependency on other code which does what I need: track the last 100 hours of an IRC channel disscussion as a web page.

The Ikiwiki setup for the Jee Labs Cafe? Broken. One week of work to write a Markdown parser and generate the static pages myself, would be my estimate. Probably well under 1000 lines of code, all in a single script. Three orders of magnitude less disk space, and easier to adjust to my needs.

I’m still waiting for the second step of the Perl install to complete. It’s running a “BigInt” package without proper library support, apparently. Falling back to “Math::BigInt::FastCalc”, and running tests which take ages:

Screen Shot 2010 07 03 at 00.26.24

Could someone help me understand where a static wiki page generator needs to do cryptographically secure factorization of large primes or something?

I’m stumped. Three hours waiting now. I give up.

I’ve aborted the (non-) install. With 220 Mb disk space gone, and no clue how to get it back. Oh yes, wait, it’s all in “~/.cpan” of course – how obvious. Thank you, Ikiwiki & Perl, for breaking down. Thank you, Eggdrop, for making simple things hard. And thank you Debian Linux, for not giving me the option to go back to my previous setup.

And you know what ticks me off? All of the above can be avoided. It has been solved.

But we’re living in medieval times, clearly. Most people haven’t even heard of Starkits. What a cruel joke.

Ah, wait. Maybe I’m the only one who runs into issues like these. Maybe everyone else has servers which just work, and to which you can add functionality without getting stuck. Doing everything you want. And a breeze to upgrade, of course.

That must be it. It’s just me. I have totally unrealistic expectations.

Apologies for the rant. Normal transmissions will resume tomorrow…

Update – thanks to a backport tip in the comments, Ikiwiki is now working again.

▶ View 9 Comments

Arduino, ISP, JeeNode

Uploading? ISP? FTDI? Huh?

In AVR, Hardware, Software on Jul 4, 2010 at 00:01

There seems to be a fair bit of confusion in- and outside the Arduino world, and it’s spilling over to JeeNodes …

I’d like to go through some terms and buzzwords to try and clarify how to get your Arduino or JeeNode to do the thing you want it to do. I’m going to assume that you are familiar with the process of writing software (“code”), compiling it, and running it – at least on a Windows, Mac, or Linux computer (let’s call them all PC’s for now). With software for the ATmega, there are two approaches, depending on whether you use the Arduino IDE or not:

Screen Shot 2010 06 28 at 23.50.22

Both lead to the same result: a “hex” file with code that needs to be transferred from the PC to the ATmega.

The step which can be puzzling when starting out with physical computing and embedded hardware is how to get things across from your PC to that little Arduino or Arduino-like system you’re holding in your hand. And vice versa, since we often want to get results back or see some confirmation that things are working properly.

The confusion comes from the different conceptual levels involved, and things like:

do you mean how to connnect? – plug in a USB cable
do you mean how does the ATmega change its own code? – through a boot loader
do you mean via ISP (In System Programming)? – no, that’s normally not needed
do you mean via FTDI? – yes, that’s the name of the chip which is hooked up to USB
isn’t FTDI a connector? – yeah, that too, sort of… i.e. a 6-pin convention
isn’t a power + serial cable enough? – no, resets also play a key role
do I have to use the Arduino IDE? – no, you can also use a program called “avrdude”
what’s avrdude? – a program which can upload to a boot loader or an ISP programmer
do I need an ISP progranmmer? – nope, the boot loader does essentially the same thing
so why not just get rid of ISP? – because you need ISP to install the boot loader

Confused? Welcome to the club…

In case you’re wondering… the process is called “uploading” because the PC initiates this as outbound transfer:

Screen Shot 2010 06 29 at 01.51.50

To get your code (“sketch” in Arduino-speak) into an ATmega, you need three things, working together to make uploading happen under all possible conditions:

An electrical connection – to power the ATmega, to communicate with it, and to restart the ATmega when it is stuck or busy doing something else.
A common language / protocol – the communication must be understood by both sides, i.e. PC and ATmega.
Software on both sides of the connection – Sending something when the other side isn’t listening, or listening while no-one is sending will not have the desired outcome.

Let’s go through each of these separately.

1. An electrical connection

There are several ways to make the connection. With an Arduino, or any similar board which has a USB connector, you can simply plug in the USB cable:

Screen Shot 2010 06 28 at 23.52.41

Some boards use a separate USB interface (“FTDI adapter”), allowing reuse of that interface for multiple boards:

Screen Shot 2010 06 28 at 23.54.08

The end effect is the same: a connection which powers the ATmega and allows communicating with it using a simple serial protocol. There’s also a trick in this hookup to let the PC reset the ATmega whenever it wants.

2. A common language / protocol

Ah, now it gets interesting. First thing to note is that there is no single common language / protocol!

That’s right. It all depends on what you want to do. Here we want to upload code from the PC to the ATmega. That requires exchanging “ISP commands” over the connection. But once uploading is done, we really want to reuse the connection as a regular two-way serial link.

The way it works is that the PC will reset the ATmega just before uploading new code. This activates a “boot loader” on the ATmega. Now both sides will be in sync (briefly) so they can exchange the necessary information to make an upload happen. IOW, right after reset the protocol is “ISP commands”. Once the upload is done, the connection can be reused for any protocol you like – as determined by the code that was uploaded.

3. Software on both sides of the connection

Back to the software now. We need to send ISP commands over the connection.

As you may have guessed, that’s exactly what the Arduino boot loader on the ATmega understands. When reset, the boot loader in the ATmega gets control. It waits and listens for incoming STK500 ISP commands. If none come in within a second or so, it relinquishes control to whatever was previously uploaded to the ATmega.

On the PC side, we need software which resets the ATmega and then immediately sends all the ISP commands needed to transfer and program the contents of a hex file.

This is what “avrdude” does. You can either use it implicitly from the Arduino IDE by starting an “Upload” from the menu, or you can launch it manually from the command line – avrdude needs a few options to tell it where the USB port is, what baudrate to use, the type of ATmega, the protocol to use (i.e. STK500), etc.

There’s more…

The above describes the different pieces and concepts involved in getting code into an ATmega. The beauty of it is that once it works, it really works well. Supplying power, uploading, communication, control, debugging … all with one simple USB cable. You only need to go back a few years to appreciate just how much this approach simplifies embedded development.

But there’s one important detail: the ATmega has to have a functioning boot loader. Placing a boot loader into an ATmega is a bit more complicated (and involves other things such as “fuses”). It’s a chicken-and-egg problem.

This is where the ISP programmer comes in. An ISP programmer is a fairly simple piece of hardware. In fact, you can make your own, as I’ve described in several posts on this weblog. See this and this post for some quick solutions which require nothing more than a working Arduino or JeeNode.

The good news, is that you usually don’t have to worry about installing the boot loader – it’s all done for you. Once. For the mechanism described above, you’ll never need an ISP programmer.

Some people actually prefer to use the ISP technique for uploading their sketches. In fact, sometimes you have no choice, i.e. when you need the serial port at all times, or when you can’t spare the 1..4 Kb required by the boot loader code, or when working with ATtiny chips which don’t support bootloaders.

In thoses cases, you’ll need a setup with an ISP programmer. But for most people doodling around with ATmega’s and the Arduino IDE, the above boot loader mechanism is usually very convenient and the easiest to get going.

Either way, it helps to understand the process. I hope the above was helpful in that direction.

Arduino, ISP, JeeNode

Fixing a faulty ATmega (Arduino)

In AVR, Hardware, Software on Jul 2, 2010 at 00:01

After a recent post on how to repair an ATmega with a faulty (or even missing) bootloader and helping someone out with it, it occurred to me that this mechanism will work for any Arduino – not just JeeNodes.

Any 3.3V or 5V Arduino’ish system which lets you upload the isp_repair.pde sketch can be used to program any board with an ISP connector on it. The code is for ATmega328′s, because that’s all I use around here these days.

The trick is to hook up a few power and I/O lines in a specific way:

Screen Shot 2010 06 28 at 16.59.11

I’m using an Arduino Pro as example, but that’s just one of many you could use. Now connect these six wires:

ISP pin 1 = BLUE = Digital pin 4
ISP pin 2 = RED = VCC (+5V)
ISP pin 3 = GREEN = Analog pin 0
ISP pin 4 = PURPLE = Analog pin 3
ISP pin 5 = ORANGE = Digital pin 7
ISP pin 6 = BLACK = GND

Note: if your working board operates at 3.3V, then you should connect the RED wire to +3.3V, not +5V (that way signal levels and power supply will match). Also: the target board should not be connected to anything, since it will be powered through the ISP connector.

The only thing left to do is to upload the isp_repair.pde sketch to your working board, and open up a console window. You should see something like this:

That’s it – disconnect all the wires. The ATmega on the target board now has a boot loader and the standard pin 13 blink sketch. Ready again to accept whatever sketch you upload to it!

ISP, Techniques

Uploading without avrdude

In AVR, Software on Jul 1, 2010 at 00:01

In the future, I’d like to upload new firmware to JeeNodes, Arduino’s, and other AVR boards through channels other than a serial port or USB. Uploading to a “slave plug” via I2C would be neat, for example.

That means the standard avrdude won’t do. Besides, after having coded various types of ISP sketches recently, I realized that the upload mechanism is really quite simple. If all you need is STK500 compatibility (as used by several ISP programmers and by the Arduino boot loader itself), then avrdude is overkill.

So here’s a demo “rig” for JeeMon which does the same thing as avrdude, i.e. upload a sketch over a serial port:

Screen Shot 2010 06 27 at 23.19.45

That removes the need to compile and install avrdude. Better still, this should work as is on every platform supported by JeeMon.

(Note: the above code is now part of JeeMon, but the source code can also be found here on the web)

Onwards!

Update – the above code has been integrated into JeeMon as new Upload rig – with dudeLoader renamed to “stk500″ and readIntelHex now called “readHexFile”. Here’s a new demo “application.tcl” using this:

Screen Shot 2010 06 28 at 01.46.48

Now works with the Arduino boot loader as well as with the Flash Board ISP programmer (add “19200″ arg).

(Reminder: the Jee Labs shop will be closed from July 14th through August 14th)

Going for gold with the BMP085

In Software on Jun 30, 2010 at 00:01

The recent post about adding some battery savings logic got a lot of mileage out of a very simple change – more than 10x lower average power consumption.

Warning: getting power consumption down can be an addictive puzzle!

Jörg Binkele was very helpful, and sent me a scope image, measuring the voltage drop over a 10 Ω resistor in the power line (before the regulator). Here are the first 5 seconds after powering up:

Jeenode Start tb0

As expected, the node settles into a very low power mode most of the time, with an occasional blip once a second. FYI: one vertical division is 10 mA. The little horizontal bar at both ends is probably the trigger level.

One small surprise was the startup behavior. Well, that first 18 mA bump is really a very simple bug: in the first second when the timer is running and being polled, the node is not in low-power mode. Aha, of course – my power-down logic is at the end of loop(). Ok, trivial to change – just move the end of loop() to the beginning:

Screen Shot 2010 06 24 at 11.11.51

Yup, that seems to get rid of the 1 second hump @ 18 mA. Great. I don’t have an explanation yet for the initial 1.5 seconds, but I suspect that the RF12 driver is waiting in rf12_initialize() – there is still some oddness with RFM12B initialization after power-up. Oh well – that’ll be for another day.

But now it gets interesting – I told you it’s addictive! – the image above shows that each blip is ≈ 75 msec @ 18 mA. That’s when both the ATmega and the RFM12B are turned on.

Wait a minute. Why so long? Sure, the BMP085 needs to measure temperature and pressure, and that takes several tens of milliseconds. But why keep everything else running full throttle? There’s no need.

So I rearranged the core loop a bit, in a way where all major delays would be done with as much of the node’s hardware turned off as possible (bmp085demo.pde):

Screen Shot 2010 06 24 at 19.06.55

Here is the result, again courtesy of Jörg – and then annotated:

Detail Power use

There’s lots of info here. Please note that the time scale is 25 times more detailed than the first scope image. The fun part is that you can essentially tie each power level to a line of code.

For example, the first hump is when the timer hasn’t yet reached 1000 milliseconds (since the watchdog can only take steps of at least 16 ms), so the node waits. But since it uses idle mode i.s.o. normal mode, power levels are about half of what an ATmega usually consumes. With almost no effect on the code. All we’re doing is switch off to wait for the next timer interrupt – that’s 50% of easily obtained savings.

Then there are two medium peaks when the ATmega starts the BMP085 measurements, and in between it drops back to power-down levels. Then we waste some power sending results out on the serial port (this could be removed). Lastly, when it’s time to transmit the readings, we switch on the radio, make sure it gets its job done, and then loop, again in power-down mode with the watchdog to keep us going.

If you look very closely, you can even see how long the BMP085 is busy with measuring temperature (about 4ms) and pressure (roughly 20 ms). Exactly according to specs.

The blip on the first screen is about 4 divisions on the second screen, and as you can see, the node is now asleep most of that time. That’s probably another 10-fold improvement. I wouldn’t be surprised if this node will now run a year or so on one set of AA batteries. And it’s still reporting once a second.

The moral is: match your reasoning to measured facts, and you can get a lot of power savings. Each case will be different, but it’s not rocket science.

Thanks Jörg, but please don’t send any more scope shots … I need to kick this addiction again! ;)

(Reminder: last day of the June special in the shop!)

JeeBot, JeeNode

TwitLEDs finale

In AVR, Hardware, Software on Jun 29, 2010 at 00:01

The TwitLEDs project by Myra and Jean-Claude Wippler (daughter and father) has come to a conclusion. It was great. I’ll just summarize by pointing to the six posts on the Jee Labs weblog describing the technical details, and some pictures and videos, showing the results.

2010-06-22 – Something different…
2010-06-23 – TwitLEDs robot, part 2
2010-06-24 – TwitLEDs robot, part 3
2010-06-25 – TwitLEDs robot – action shots
2010-06-28 – TwitLEDS robot, part 4
2010-06-29 – TwitLEDS finale (this post)

Here you can see the finished TwitLEDs robot:

Dsc 1741

This is the, ehm… print head?

Dsc 1749

And last but not least, some videos. First a trial run for the print mechanism:

Next, a trial run for the vehicle, trying to stay out of trouble:

And here the result!

And one more:

Ok, that’s it. Myra and I both had oodles of fun building this and trying things out – hope you did too :)

Onwards!

(If you can’t view the videos on vimeo, see 1, 2, 3, and 4)

RFM12B as spectrum analyzer

In Software on Jun 27, 2010 at 00:01

Intrigued by a very interesting post by “loomi” on the discussion forum, I wanted to find out more…

What he did was use the RFM12B as a crude spectrum analyzer: sweep the frequency across its entire range, and use the RSSI threshold and status bit to find out whether there is any signal on that frequency.

First of all, I added an rf12_control() function to the RF12 library, to allow access to the low-level registers of the RFMN12B. This allows changing the frequency and RSSI settings, and reading out the RSSI status bit.

On the software side, a sketch is needed for the JeeNode which does the sweeping and measuring:

Screen Shot 2010 06 21 at 00.26.21

This reports one line of 476 digits 0..6 for each sweep. Each digit in this line of text represents the measured signal strength at that particular frequency.

The RSSI readout is very crude. All you can do is set a threshold and read out a single bit in the status register, telling you whether the signal is above or below the threshold. Tedious, but doable.

The fun part starts once you get to plotting this as a graph. I used JeeMon and wrote a Tcl/Tk script for it:

Screen Shot 2010 06 21 at 00.30.15

And here’s some sample output:

Screen Shot 2010 06 21 at 00.21.30

The green line is the default frequency setting of the RF12 driver.

Each sweep will add a set of black dots to the graph, but since the values are being accumulated in a “counts” array, the dots will creep higher and higher, drawing a bar as more sweeps come in. Using the current delays, one sweep takes a few seconds, so to get the above graph I had to keep JeeMon running for a few minutes.

When left running for some 15 minutes, the large bars will move out of range, but the smaller accumulated counts will now become clearer:

Screen Shot 2010 06 21 at 00.30.53

I haven’t figured out whether these values are valid, nor what could be causing all this RF activity. The peaks are fairly sharp though, so it would seem like a reasonable set of measurements.

TwitLEDs robot – action shots

In AVR, Hardware, Software on Jun 25, 2010 at 00:01

As promised in the previous post, some pictures and movies of the TwitLEDs robot in action.

First another view of the robot, with the different pieces:

Dsc 1747

We don’t have the proper “arena” yet, i.e. a big fenced-off floor area covered with glow-in-the-dark paint. Instead, these first trials used two pieces of foam board, taped together. It’s not perfect because the robot keeps running off track, and because the hump between the two pieces is quite high. Foam board isn’t really suited for this: it curls up too much from the moisture in the paint. We probably should have waited a bit longer for everything to dry completely…

The inital test code just printed out its name (povGlow) and the compilation date. In the first tests, one of the LEDs wasn’t working (a software bug, not hardware), so the text isn’t quite right:

Dsc 2039

You can clearly see the fading of the letters over time. This happens very quickly, but those faded letters then remain visible for quite a while. That’s why you can still see several trial runs “printed out” on the foam board.

So the basic idea of printing with light works, as you can see!

Here is a video (sorry, not inline), showing how the robot veers to the right as I put my hand in front of it to prevent it from running off the track. Note that I’m not touching the robot, I just briefly trigger the distance sensor. The on-board LED lights up in red when the correction takes place. It’s not very smooth, but it works.

Another picture, showing the decay of the printed text brightness:

Dsc 2067

Maybe 7 pulsed blue lasers (from DVD writers, perhaps?) could be used to create even more intense blue dots. As it is, with these blue LEDs the writing is very clear – but only in a relatively dark room. With the lights on, the text becomes virtually invisible. Even though the LEDs are bright enough to be painful to look at:

Dsc 1749

The quality of this “printer” is actually pretty good, considering how simple its technology is:

Dsc 2056

Finally, a run where the robot happened to stay a bit longer on track, allowing it to display its brief message a few times. Again as video – you can see the wobbling foam board, and this thing driving like a drunken duck, leaving its trail of fading messages behind.

All that remains, is to try and get some tweets into it in real time. Stay tuned…

Several days of testing have now drained the 4 alkaline AA to the point where the robot advances noticeably slower. Looks like we’re getting no more than an hour or two out of these batteries. Which is not really surprising: the DC motors must be eating quite a bit of current, and the LEDs probably draw up to 200 milliamps or so. Self-powered autonomous motion is really hard!

▶ View 8 Comments

TwitLEDs robot, part 3

In AVR, Hardware, Software on Jun 24, 2010 at 00:01

To continue on yesterday’s post, here is how we got a little autonomous robot going.

The main part was solved by picking the Asuro robot kit. It’s really low-end, but it has just enough functionality for this project, and at €50, it’s very affordable. In fact, Myra built a spare one because the first unit broke down. In the end, I was still able to fix it: a burnt out transistor (both H-bridges are done with discrete components!). So now we have two Asuro’s:

Dsc 1750

The nice bit about the Asuro is that it has an odometer on each wheel. IOW, there are IR LEDs + sensors to count the number of steps (8 per rev) made by the wheel, and the C library code includes logic to adjust the speed of the motor. It’s a bit crude, but because of this the Asuro can drive fairly straight. As we found out later, it has a bit more trouble doing so while driving slowly, so it’s still a bit wiggly.

But it works. Two small DC motors, some simple gears, motor axles soldered to the PCB (what a great low-cost solution), and room for an extension board. To give you an idea of how crude this thing really is: there is no on-board voltage regulator. When used with 4 alkaline AA batteries, you have to remove a jumper so the extra voltage drop over a diode gets the supply voltage down to under 5.5V …

The Asuro is full of such nifty cost-cutting tricks. It even includes a bidirectional IR link, over which new code can be uploaded. The IR link is very short-range, so it would have been insufficient for our purposes – but for quick code tweaks, the IR link works fine.

Speaking of code, here is the main avoider.c logic running on the ATmega8:

Screen Shot 2010 06 18 at 23.57.08

This is not an Arduino sketch, but I made it look a bit like one by using the same setup() and loop() functions.

This code has to be compiled with avr-gcc. The Asuro comes with several examples with Makefiles. I simply copied one and started extending it. A little optional USB-IR adapter is used to connect to the Asuro (an RS232 one is included with the Asuro kit, but I don’t have any RS232 ports). The whole setup works pretty well from the command line, especially considering that it’s a completely different setup than the Arduino IDE.

Anyway, back to the design of the TwitLEDs robot.

The challenge was to find a simple way to make this robot drive around without bumping into things. The Asuro has a row of switches at the front, but these are not very reliable, and besides: bumping and stopping and turning would mess up the LED strip being “printed” on the glow-in-the-dark paint.

So instead, we mounted a Sharp 10..80 cm distance sensor on top. It’s very easy to read out, since it produces an analog voltage, inversely related to the distance of objects in front of it.

The logic for collision avoidance is crude, but sufficiently effective: turn more and more right as you get nearer to an obstacle. The on-board LED turns from off to green to yellow to red as the distance decreases, so it’s easy to see what the robot is doing. I’ve tweaked it so that when it drives straight towards an obstacle, then turning will be quick enough to never bump into that obstacle.

Here’s the completed unit, with the Sharp proximity sensor on a little expansion board at the front, and the LED blinker glued to the side of the battery pack – the LEDs hover 1 .. 2 mm above the ground:

Dsc 1741

Myra’s idea was to have this thing drive inside a more or less circular “fence” made of corrugated cardboard. The whole area inside the fence will be made of some panels, covered with 3 layers of glow-in-the-dark paint.

There is a flaw in the current design, in that the robot can only evade an obstacle by turning to the right. If it approaches a wall obliquely from the right, it will do the wrong thing and drive straight into the wall. But we’re hoping that as long as it doesn’t over-correct, and by sending it off in a tangent, it will keep adjusting slightly to the right as it drives around and around in “sort of” circles.

More tests and tweaks will no doubt be needed.

But as it is, this thing really seems to stay out of trouble. We can let it loose in the room and it’ll veer to the right whenever it comes near an obstacle. Sometimes it veers to far, though. Again, I hope a more controlled “arena” will be sufficiently simple to keep it going.

When the robot does bump into its front switches, it turns both motors off, blinks for 10 seconds, and then starts off again. Time enough to pick it up and aim it in a different direction.

So that’s it. An autonomous unit with two independent computers, IR distance sensing, a short range IR link, and the JeeNode with a longer-range wireless RF link. Even including a JeeLink on the laptop side, the cost of all this was only slightly over €100.

Tomorrow, some pictures + movies. Gives me time to finish the Twitter link, and then on to the grand finale!

(Reminder: one week left for the June special in the Jee Labs shop!)

TwitLEDs robot, part 2

In AVR, Hardware, Software on Jun 23, 2010 at 00:01

Yesterday’s post introduced the robot Myra and I have been working on. Here’s the first part we built:

Dsc 1725

It’s basically a backplane for the LED blinker component. Or to put it differently: a simple persistence-of-vision (POV) unit, using a JeeNode, and Output Plug to drive a few LEDs. Only the output plug was soldered-in permanently. The removable JeeNode allows it to be easily programmed and re-used, and the removable LEDs allow trying out different units. This turned out to be important, because I only had a few green LEDs when starting this, and had no idea then as to what sort of LEDs would give the best POV results later on.

Myra did all the soldering. Here are the two LED mounts we ended up with:

Dsc 1745

The one on the left is the super-duper LED concoction we built as final version. The one on the right was great for initial testing.

Everything is held together with rubber bands, zip-lock ties, tape, and ample amounts of hot glue (once verified to work!) – hacking at its best, clearly:

Dsc 1742

Here’s the LED blinker with the final LED strip, side view:

Dsc 1743

Side view close-up – with the foam board cover:

Dsc 1746

Bottom view:

Dsc 1744

Seven blue LEDs, ready to shine very brightly and controlled by the JeeNode.

The software started out very simple, of course. Things like this, just to make sure it all works:

Screen Shot 2010 06 18 at 23.01.21

This is the main part of what is more or less the final twitLEDs.pde sketch:

Screen Shot 2010 06 18 at 23.03.06

I found a suitable font table by googling around a bit. This is needed to go from ASCII characters to dots-in-a-readable-pattern. No room for Unicode (don’t laugh: some tweets are in Japanese and Chinese, and they won’t show properly).

The amazing bit is that everything worked essentially on first go. It blinked! But does it blink in the proper pattern? Our first test consisted of Myra taking a long-exposure shot, as I waved this thing around in the air – with the lights off. Liesbeth tracked progress through all the shrieks and laughs… but from a safe distance :)

Dsc 1793

Yippie. It really works!

Tomorrow: driving around without bumping into things.

Something different…

In AVR, Hardware, Software on Jun 22, 2010 at 00:01

Ok, so I’ve got tons of projects on my plate to do and to finish. And tons more that are still very experimental, or haven’t even been started. Plenty to keep me busy, with the summer vacation period nearing fast.

Yet here’s something different. The timeline for this project was imposed by external factors: my daughter Myra doing a project for the last quarter of her second year at the Design Academy in Eindhoven.

She wanted to do something which triggers human interaction, and she wanted to try something with Physical Computing. “Terrific, I’ll help!” – I shouted, completely ignoring all the pending work on my own plate…

So here’s the start of a few articles about the “TwitLEDs” project we’ve been working on recently. All the basic ingredients work as I write this, but we have yet to finish the final setup and go through a last rehearsal.

What is it?

It’s a mix between a matrix printer and a persistence-of-vision (POV) display.
It’s called the TwitLEDs robot. And it’s hooked up to the internet.

Get it? No? Ok, then let me try again. I think this picture tells it best:

Dsc 2069

The idea is to have a little autonomous robot driving around, leaving messages behind on a floor covered with glow-in-the-dark paint. The messages are collected off Twitter using a configurable search term. This is done from a laptop and then sent to the robot by wireless.

There were several fairly non-trivial problems to solve here, with some experimentation needed to find a workable mix – as well as some time contraints. A few days of work would be the most I could set aside for this. Luckily, I didn’t get lost in too many dead alleys, so it worked out nicely.

Here are the pieces we used:

A low-cost robot kit called Asuro – based on an ATmega8, so I had all the software ready for it. In fact, I played around a bit with it a while back – as reported here.
A JeeNode for wireless connectivity.
An Output Plug to drive some LEDs.
Seven blue LEDs. I picked a bright one with a very focused beam (C503B-BAN-CY0C0461).
Glow-in-the-dark paint. Green stuff. Three coatings.
Some panels to create a floor. Covered with the green stuff.
Some cardboard to create an “arena” on the floor to contain the robot.
JeeMon running on a MacBook, with a JeeLink to send out the messages.
A fairly dark room. This just won’t work with the lights on, unfortunately.

As with every project, the first part is the hardest and the most critical success-factor: figuring out what to do, what not to do, and finding solutions within the many constraints we had to operate under. I’ll spare you the ideas that didn’t make it, and the (really neat) ideas we simply didn’t have time for.

Being the sole programmer on the team, I got to deal with all the software. Yummie! :)

The most important insight for me was that we could implement this project with three completely independent subsystems:

The LED blinker, driving 7 LEDs in the proper pattern, basically a POV unit (plus receiver).
The robot, moving around while continuously trying to stay out of trouble.
The server process running on the laptop, connecting to Twitter and sending messages into the air.

We started off with the LED blinker because it was a major component with few unknowns, i.e. we picked the low-hanging fruit first. Here’s a picture of it, still under construction:

Dsc 1726

More on the LED blinker tomorrow…

JeeNode, LowPower

A small improvement to rf12_sendStart()

In Software on Jun 21, 2010 at 00:01

The RF12 driver has a function called rf12_sendStart() which – you guessed it – is used to start sending out a new wireless packet.

Yesterday’s post was about battery savings, which I implemented by going into a power-down state right after sending out a new data packet.

That code included this little detail:

Screen Shot 2010 06 20 at 03.50.41

The reason for this is that the RF12 driver works mostly on an interrupt basis – i.e. in the background. Without the delay, the node would power down before the packet had actually been sent out. Then, when waking up, the transmission would fail due to the transmitter underrun caused by this logic.

By simply waiting a bit, the interrupts can continue to take place, and all the bytes will get sent to the RFM12B and from there into the airwaves.

Totally by chance, an EtherCard software problem was reported on the Jee Labs discussion forum, which turns out to be related to this very same issue. Sure enough, adding the “delay(5);” fixed that problem as well.

This felt a bit unsatisfactory as solution, so I’ve extended the rf12_sendStart() function to support a “synchronous mode”. The change is backwards compatible, i.e. the default behavior of rf12_sendStart() hasn’t changed.

Now, if you pass 1 as fourth argument, rf12_sendStart() will wait for the packet transmission to actually complete before returning to its caller. In general, calling rf12_sendStart() in sync mode is probably a good idea. Sync mode can be used for sending data packets as well as acknowledgement packets.

So the above code can be replaced by this single line:

Screen Shot 2010 06 20 at 04.02.11

This simplifies the bmp085demo.pde sketch slightly, and it also solves the problem that the EtherCard library may occasionally lock out interrupts on the SPI bus a bit too long to keep the RF12 driver running.

The code and documentation have been updated.

Onwards!

Battery savings for the Pressure Plug

In AVR, Hardware, Software on Jun 20, 2010 at 00:01

Reducing power consumption is fairly tricky, as you can see in the many previous posts about this topic. And to be honest, I haven’t quite gotten to the point where I want to be with the Room Board. My first “major” (ahem) setup with about a dozen nodes around the house didn’t quite go as I had hoped. Most batteries were empty within a month. A few nodes are still going, but those are hooked up to power adapters…

I’d like to revisit this issue and try to improve things a bit. To make the rooms sketch perform better, and also to make the code structure a bit clearer. The current rooms code is quite complex and hard to follow.

But before messing with the rooms.pde sketch, let’s tackle something simpler: the wireless sensor node based on the Pressure Plug, as described here, and then simplified here. I’ll use that last version as starting point.

The first point to note is that to get a substantial first power reduction, you have to focus on the portion of the code where it’s spending most of the time. Which, in the case of “bmp085demo”, is here:

Screen Shot 2010 06 17 at 01.55.06

That’s right: it’s waiting for the next second to “happen”. And even with a slow’ish sensor such as the BMP085 at maximum resolution, it’s spending more than 90% of its time there… waiting!

I’ll use an approach which might be a bit surprising: let’s not change any of the current logic. The idea is that once we’ve done our thing for the current second, we can go into low power mode, as long as we make sure to get back to normal operating conditions in time for the next second.

So what we’re going to do is add some code to the end of the loop() function. It’s functionally equivalent to adding it at the start, since loop(), eh, loops – but I think it makes a better point.

The end of loop() used to look as follows:

Screen Shot 2010 06 17 at 02.01.01

I’m changing it to this:

Screen Shot 2010 06 17 at 01.43.00

IOW, at this stage all the hard work has been done. We wait a bit for all interrupt-driven I/O to complete (serial and RF12). And then we need to figure out how much time remains until the next second. The power saving happens by spending that time in power off mode.

But this requires some preparation. When inducing a comatose state like this, you have to make absolutely sure that something is still able to get you out of coma and back up and running again. This is what the ATmega’s “watchdog” is for: we set it up to wake us up in 16 milliseconds, just before entering sleep mode. And then we keep doing that until it’s almost time to take another reading. The actual watchdog interrupt handler does nothing, btw – all we want is to get back out of power down.

Note that the radio also needs to be turned off and back on again. It’s the biggest power consumer when enabled. Turning it off and going into power down mode is what lets us go from a tens-of-milliamps current drain to a tens-of-microamps current drain.

All the logic for this is located in the loseSomeTime() function, which was adapted from a slightly different version in the rooms.pde sketch:

Screen Shot 2010 06 17 at 01.42.33

And that’s about it. The average power consumption of this sensor node will go down by an order of magnitude. It’ll still be 1..2 mA, but it’s a major improvement: this node should now run 2..3 months on AA batteries. The source code for bmp085demo.pde has been updated.

I’d like to stress that such gains require very little effort for many types of sketches. All you have to do is figure out where the sketch is spending most of its time, and deal with just that part of the code. Getting into yet lower power consumption levels would require more work.

Input Plug, oh my

In Hardware, Software on Jun 19, 2010 at 00:01

More fallout from the bad ISP programmer. As reported on the forum, there is a problem with the Input Plug. Fortunately, it’s fixable in software.

The problem is that the ISP programming of the ATTiny45 chip on the Input Plug had the same problem as the recent, ehm, Murphy debácle: the fuses weren’t always set properly.

So the ATTiny starts up running at 1 MHz i.s.o. 8. And hence the timing is off when driving pulses to it to control the analog multiplexer.

The solution was to add a “fix” parameter to the InputPlug class in the Ports library. It defaults to 0 (no fix needed), but when set to non-zero, the timing is adjusted to slow down by a factor 8. That way, channel selection will take a bit longer – but at least it’ll work:

InputPlug myPlug (3, 1); // plug on port 3, slow down for wrong ATTiny fuse

This code has been added to the Ports library. There is a downside: the fix means that channel slection now takes roughly 900 µs, instead of the intended 100 µs.

Next question is: how to make sure this won’t happen again? – Answer: build a test jig, so all plugs can be tested fully before shipping. Trouble is, all Input Plugs ship as follows (this is an early hand-soldered unit):

Dsc 1732

Without headers!

My mistake was of course that I would wing it, and avoid the testing. Bzzzt. Time to build a little test contraption:

Dsc 1733

The other side is filled with 17 tiny 1 kΩ SMD resistors, creating a voltage divider with 16 different voltage levels. Soldering this was quite a challenge, btw:

Dsc 1737

And now I can take advantage of the fact that everything is gold-plated. So the following will make contact, just with the pins being pushed in and pressing lightly against the board:

Dsc 1735

I’ve been using the same trick for some time now, to test other plugs. Here’s how to connect the whole thing to one of the ports:

Dsc 1736

Below is a little readout test – a better test setup would be to simply perform the check and display a go / no go signal, but for now I’m just reporting the 16 values:

Screen Shot 2010 06 16 at 19.00.21

Sample output:

Screen Shot 2010 06 16 at 20.24.30

As you can see, the selection is now working reliably (once connected).

So that proves the bug and gives me a new tool to test Input Plugs before shipping. And of course I’ll re-flash the ATtiny45 chips on all remaining Input Plugs in stock at Jee Labs. For those already out there: you can either use the software fix, or send your Input Plug(s) back to me and I’ll fix the fuse settings to make selection snappier, as originally intended.

Problem solved. Phew.

Does it feel good to have to confess to another goof-up on my end? Of course not. But worse than that would be to keep quiet on this – and I won’t. I’m human, I make humiliating mistakes (LOTS!), I occasionally cut corners, but I do my best to learn and deal with it all. There is no doubt whatsoever in my mind that every person and company goes through these sorts of mishaps. It’s just usually not out in the open for everyone to see. Well, at Jee Labs, it doesn’t work that way – this is open source: software, hardware, goofs … e v e r y t h i n g !

Home automation, JeeMon

A little GUI for the RGB LED strip

In Software on Jun 16, 2010 at 00:01

After yesterday’s post, this RGB strip control project is essentially done. Keep in mind that I’m doing this only to adjust what comes out as “white”. I’ll leave light levels and color variations up to nature, since all I need to do is look out the window…

But since wireless control is present anyway, I might as well create a little demo using the JeeMon software.

So here goes. The setup I’m using is as follows:

Screen Shot 2010 06 14 at 16.31.37

The JeeNode is configured as node 30 and the JeeLink as node 1 (in the same group). The manual way to send out a color command via serial USB is to type:

<red>,<green>,<blue>,<white>,30s

Where each of the four values are in the range 0..255 (0 = off, 255 = full on).

Now the GUI version (Mac OS X):

Screen Shot 2010 06 14 at 18.21.51

The code for this is as follows:

Screen Shot 2010 06 14 at 18.16.26

Like everything in JeeMon, it’s cross platform since it uses Tcl/Tk.

Here are the screen shots for Linux (Ubuntu) and Windows (W7, XP, 2K), respectively:

Screen Shot 2010 06 14 at 18.01.42 Screen Shot 2010 06 14 at 18.35.25

Screen Shot 2010 06 14 at 18.14.21 Screen Shot 2010 06 14 at 18.09.26

As you can see, the light gray background color was adjusted for the Mac – on some of the other platforms it needs to be darkened a bit to match the default label background.

Anyway. I’m not a GUI wizard, so this’ll have to do for now…

Home automation, JeeNode, Techniques

Remote RGB strip control

In Software on Jun 15, 2010 at 00:01

To continue this series on driving RGB strips with a JeeNode, here is a sketch which allows setting the brightness levels using PWM. It’s a bit long for a weblog post, but I thought it’d be useful, since there is quite a bit of trickery in here. Notes follow below:

Screen Shot 2010 06 11 at 23.17.29

This code does some hefty bit manipulation. The idea is to keep an array of 256 time “slots”, with bits to indicate when an I/O pin should be toggled on or off. The loop then continuously scans these slots at the rate of ≈ 32 microseconds per slot (this corresponds to roughly 120 Hz). The 8 bits in each slot map to the JeeNode’s I/O pins: bits 0..3 = AIO1 .. AIO4 and 4..7 = DIO1 .. DIO4.

A refinement is that only the first 255 of the 256 slots are scanned. This way, 0 is 100% off and 255 is 100% on.

The static “masks” array defines which setting gets mapped to which I/O pin. It depends on the way the output driver is connected to the JeeNode.

The main PWM timing loop is done fully in software. It will run slightly irregularly due to timer interrupts and RF12 driver interrupts, but the effects aren’t noticeable.

The RGBW values are stored in EEPROM, so that the LEDs come back on with the same settings after power cycling. The settings can be adjusted by sending a packet with the new values to node 30, group 5 @ 868 MHz.

This sketch could be extended to support “animations”, i.e. ramping up/down to specific levels, mood lights, etc – I’m not interested in that, I just want to be able to trim the color of my room lighting to a pleasant level of white:

Dsc 1723

All I did to adjust the strip on the right was send the following “RGBW” command out via a JeeLink:

255,140,40,120,30s

The above sketch does PWM, but this whole thing is still being turned on and off with an old-fashioned mechanical switch … if you still remember those :)

FWIW, I’m considering making a dual-channel “MOSFET Plug” – two of those could then be used to replicate this same setup for the LED strip on the left.

Tomorrow: a little GUI front end for all this.

Home automation, JeeNode

RGB driver board

In Hardware, Software on Jun 14, 2010 at 00:01

To follow up on yesterday’s post, I made a more permanent 4-channel board for driving an RGB LED strip plus the original white LED strip already in use.

A JeeNode is used to drive these strips, using a small custom JeePlug to hook up ports 1 and 4, as well as to use the LED voltage as power source for the JeeNode:

Dsc 1717

Here is the plug in more detail:

Dsc 1718

It includes an LP2950 linear regulator to put 5V on the PWR pin, that way this thing can be used with any input voltage from 12 to 24V. Here is the bottom side:

Dsc 1719

The driver itself uses four IRLZ34N MOSFETs, three of which will be used for the RGB strip, and the other for the white strip so that it too can be controlled and even dimmed remotely:

Dsc 1715

The back side is hand-wired using sturdy copper wire, although in this case only 300 .. 600 mA is being switched per color. The 10 kΩ pull-down resistors prevent the lights from turning on in the absence of control signals.

Dsc 1716

And here’s the completed setup:

Dsc 1714

The two wires sticking out are for attaching my 12V lab power supply during testing. The other green 2-pin screw terminal is for attaching the white LED strip.

The RGB ledstrips can now be installed next to the LED strips already in use here at Jee Labs:

Dsc 1722

Here’s a trivial test sketch to turn each of the 4 colors on and off:

Screen Shot 2010 06 11 at 13.17.35

The results can be seen in this brief video.

What remains, is to write the software so this thing can be controlled via wireless. I might add an LDR to sense ambient light levels, or even a Room Board. Perhaps some basic ramp-up / ramp-down PWM logic. Or maybe I’ll just duplicate this setup and drive the two sets of LED strips I have, independently and from a single JeeNode.

Hardware is easy, it’s always the software… the code will be presented tomorrow!

Reminders: 1) June discount and 2) Jee Labs will be closed from July 14th to August 14th.

Home automation, JeeNode

Driving an RGB strip with MOSFETs

In Hardware, Software on Jun 13, 2010 at 00:01

Another day, another project started. I have a nice 5-meter LED strip in the office, which provides indirect lighting at Jee Labs. Works fine, but despite being called “warm white”, it’s slightly greenish. Not so great…

Someone suggested adding an RGB strip next to it, to compensate for the color.

So to start off this multi-part series, I wanted to simply control RGB. This is very easy to do with three MOSFETs. I used IRLZ34N’s N-channel, since the LED strips are common anode. The circuit is trivial: source to ground, drain to LED cathode, and gate to an I/O pin. Rinse and repeat three times. I didn’t even add a pull-down resistor to the gate, though I probably should.

Here’s the circuit, fully connected to a test strip with 9 RGB LEDs using a separate 12V power supply:

Dsc 1664

I wrote a little test sketch which ramps the light levels up and down using PWM, and which goes through all the RGB combinations – using some bit fiddling:

Screen Shot 2010 06 07 at 13.17.10

The LEDs are pulsed in software, with a cycle time of 8160 µsec (32 * 255), i.e. roughly 120 Hz. No discernible flickering. A more advanced version ought to use interrupts to keep the PWM going in the background.

Here’s a video of the result – with apologies for the very low quality.

Since the JeeNode has RF on board, it will only be a small step to add wireless control. Stay tuned…

Update – Source and drain were listed the wrong way around, I’ve updated the text above.

▶ View 10 Comments

IR trigger for Nikon camera

In AVR, Hardware, Software on Jun 12, 2010 at 00:01

Here’s a fun project: sending out infrared remote codes to takes snapshots :)

This is the code:

Screen Shot 2010 06 10 at 17.19.44

This sketch is doing everything in software, and it’s sort of pushing the limits by pulsing a TSKS5400S IR emitter LED at 38.4 KHz using software delay loops.

The timing diagram on this website was a great help to get this working in no time. Here’s another site.

With the 33 Ω resistor in series, total current through the IR LED should be somewhere between 20 and 50 mA. Since the latter is beyond the ATmega’s current sourcing capapbility, I suspect that the pin voltage will actually drop quite a bit below VCC. A transistor or MOSFET could be added for more power. As it is, this seems to trigger reliably up to about a meter away.

Here’s the setup, triggering my trusty D40 Nikon:

Dsc 1692

… a self-portrait!

Lot’s of people have already done this ages ago, btw. Here’s a sketch which uses assembler to get the timing more accurate. But the above does work – the main point is to avoid digitalWrite(), which is relatively slow.

Musings

Long term decisions

In Software on Jun 11, 2010 at 00:01

This post is a question, but maybe I’m already answering it by posing the question in the first place …

The question is: what to do with the Jee Labs source code?

As you may remember, I went through a few mad gyrations recently w.r.t. setting up a community site using TikiWiki, and then quickly abandoning it again.

In the end, I’m glad I didn’t buy into TikiWiki. Too many compromises, and for my particular needs, far fewer benefits than I had originally hoped to see.

The current Jee Labs Café that has replaced it is very crude, visually, because I haven’t yet done the work to bring over the layout and style sheets from this weblog. But far more importantly, IMO, is that the information is finally organized in a sensible way – until someone tells me otherwise, anyway!

Screen Shot 2010 05 24 at 21.03.33

The pages are maintained by subversion (svn), a version control system which is well-known and widely used by software developers (as are the older “cvs” and the newer “git”), although I do find that a surprising number of people are not used to the whole idea of “source code control systems” and “code repositories”. Frankly, I couldn’t live without something like svn anymore, these days. It’s the only way to develop software and not lose your mind w.r.t. change management and long term maintenance – IMO.

Maybe I should go into the features and benefits of such an approach in another post.

My point here is that the entire Café website is managed by svn. This means that all older versions of pages, sources, documents, pictures, etc. are available. At least as importantly in this context, is that the pages can be edited by anyone with write access to the repository. It’s pretty simple: get a copy (“check out”) of the website, make changes, and write it back to the repository (“check in”). Writing is only allowed to those with a user name and password in the repository, so this is wiki-like, but not publicly writeable.

The interesting tidbit about the Café website is that all changes are automatically published on the web. I’m using ikiwiki as tool to take care of all that. Thats why all the pages for the Café can be maintained as simple Markdown text pages, and how the result ends up as HTML, with links and embedded images. I’m only using a very small subset of ikiwiki’s features, btw.

Another great side-effect of using ikiwiki is that the entire Café website consists of static web pages. No server load, inifinite scalability for free.

In practice, it’s all working out really well. I can easily update, extend, and maintain the Café website this way. And if anyone wants to participate and contribute more content for the wiki (one can always hope, eh?) then that would be very easy to fit in, since subversion can be used over the internet.

As far as I’m concerned, the daily weblog and the Café / wiki are in good shape for a long time to come.

There are two other areas. One is the Talk discussion forum, which uses bbPress. I’m not terribly excited by its functionality, nor by its (lack of) progress, but it’s holding out fine, sooo… if it ain’t broken, don’t fix it.

The fourth area of public Jee Labs activity is the source code being developed for use with JeeNodes and JeeMon. It’s been in my subversion repository for a while, then I moved it to Google Code, en then I moved it back to my subversion repository again, which is where it has been for many months now.

As fas as the repository goes, this way of managing source code works well, at least for me. But there are some sharp edges. For one, the web-browsing interface is based on ViewVC – which is a bit quirky, and which turns out to be CPU resource hog on my server. Probably just web-crawlers, constantly triggering CGI execution of the ViewVC Python scripts.

Also, I currently have no issue tracking system in place. Would be nice to get something going one day.

I’ve been looking into GitHub, which seems to attract a lot of attention (and open source projects), these days. It has git-based code repositories, with an optional svn interface, and it offers a wiki, issue tracker, and download area. Reminds me a bit of SourceForge, which seems to have fallen out of grace, these days.

My main concern with GitHub is that it’s yet another big central site, offering a mix of features and functions to attract as many developers as it can, with no guarantees that it won’t be gone a few years from now, or start doing “targeted” ads, or selling email addresses, or other nasty stuff.

There is, however, one tempting option out there: GitHub + Jekyll – Jekyll is similar to ikiwiki, in that it takes text files (including Markdown, yeay!) and generates a website from it. Jekyll even offers a migration path for WordPress sites. The interesting bit is that GitHub supports Jekyll out of the box, and that it can be set up to work with custom DNS domains. IOW, I could merge this daily weblog, the Café, and the code repository into a single consistent system. The result could be one or two static websites (hm, I guess that rules out blog comments), fully under version control, and using the Markdown format I’ve already standardized on. Needs more thought!

As it currently stands, I think I’m going to stick with WordPress + subversion, which is rock solid and has some really nice software tools for it (like Versions on the Mac). Maybe just integrate the source code areas with the Café, and use ikiwiki to allow browsing the latest version on the web, right next to the supporting doocumentation.

It feels a bit like “going it alone” by placing everything in my own subversion repository and maintaining it all on a privately managed server, but all I can say is that a lot of what I’ve done in the past has been around a lot longer – and still is! – than some of those fads…

The web is somewhat too ephemeral for my tastes…

▶ View 8 Comments

ISP, JeeNode, Techniques

A Happy Ending!

In AVR, Hardware, Software on Jun 10, 2010 at 00:01

The multi-ISP programmer I built and started using some two weeks ago, turned out to be quite a nightmare. Not only were incorrectly programmed ATmegas sent out to about two dozen people, I also had to go through about 70 kits, prepared as new stock just days after I started using this programmer. Twice!

Yes, twice. Because the first “fix” turned out to be insufficient. Doh.

This was a clear case of one bug hiding another, and another, and another. Yep, four bugs: a bug in the MemoryStream class in the Ports library, a timing problem exposed by fixing that bug, and two incorrect assumptions about how the “avrdude” utility works. I’ve now got an explanation for everything that went wrong.

There’s no doubt some interesting psychology at work here … I was so proud of my idea op programming multiple ATmega’s! The main idea was to create an AVRISP-compatible unit which stores everything sent through it and then just replays the saved stream as often as needed. Trouble is, I jumped to conclusions the minute a first “run” worked. Roll the presses! Print it! Tell it to the world!

Anyway. There is a happy ending to all this!

The latest version of the isp_capture.pde sketch in the Ports library has been working properly for over a week now, programming well over a hundred ATmega’s (and it now does auto shut-down a few minutes after use):

Isp Capture Output

The last bug was a very puzzling one: everything worked, but sometimes the fuses wouldn’t get programmed. It turns out that avrdude first reads the fuses, and only sends out commands to program them if the fuses don’t match the new value. Since the programmer needs to work with brand-new as well as previously-programmed chips, the replay mechanism would have depended on the prior state of the chips: not good.

The solution is very simple: I now always program each fuse twice, with two different (valid) values. The second one will remain in force, evidently. Since the replay code was already ignoring fuse mismatch checks, this now means that even if the first setting is skipped by avrdude, the second will always be emitted.

Here is the shell script to prepare the Flash Boards:

Screen Shot 2010 05 30 at 01.35.40

So this has now become a very useful tool at Jee Labs:

Dsc 1432

I love the on-board LiPo battery: I can grab it, use it where I need it, and put it back – no wires = freedom!

For pre-loading the fuses, boot loader, and RF12demo, it already saved me a huge amount of time. Its “burn rate” is up to 500 chips/hour. And Mr. Murphy taught me some valuable lessons along the way…

And now it’s time to move on!

ISP, JeeNode

Repairing a faulty ATmega

In AVR, Hardware, Software on Jun 9, 2010 at 00:01

This post is being written after a pretty nasty foul-up on my part w.r.t. sending out badly-flashed ATmega chips. It’s probably too late to be of use to anyone involved in that little débâcle, but I thought I’d describe a DIY fix here in full detail anyway. I’ve simplified the code to make it even easier to use than what was described here.

So what is this about, eh?

The ATmega328 chip used in JeeNodes (and Arduino’s, and various other similar board) comes with a serial boot loader pre-loaded into the high end of its flash memory. This is arguably one the most important features that made the Arduino popular: it lets you upload a new “sketch” into the ATmega over the serial or USB port.

Lots of little details, but the point here is that the boot loader itself can’t easily be damaged, even if you load a non-functional sketch into the ATmega. Just resetting it will let the boot loader overwrite the flash memory with a new version – hopefully a better one. Basically, no matter what goes wrong, there’s always the boot loader in the ATmega to upload a new sketch to it.

There’s a chicken-and-egg problem, though: how do you get that boot loader into the ATmega in the first place? Well … you don’t. It’s normally placed there by the supplier of your ATmegas.

Since Jee Labs is one of the suppliers of JeeNodes (for Europe, and Modern Device for the US), I do need to do a little more. I use an “ISP” programmer to place that boot loader there, along with the RF12demo sketch.

But there’s really not much to all this, and this whole boot loader / ISP stuff can easily be performed by anyone. Keep in mind that’s it’s just about getting that first step right: proper fuses and the boot loader stored in flash memory. Nothing more.

All you need is a working unit and a second target unit with the ATmega that needs to be repaired (or initialized for the first time – same thing). For Arduino’s, it’s all explained at http://arduino.cc/en/Tutorial/ArduinoISP.

As it so happens, there’s now even simpler software to do this, so I’m going to describe the process in terms of using one (working) JeeNode to repair another JeeNode (note that this still should work for Arduino’s as well).

The basic idea is still the same: we need to connect 6 wires. There are 2 wires for +3V and GND and 4 wires connected to ports 1 and 4. On the target JeeNode, the wires will be connected to the ISP connector:

Screen Shot 2010 05 29 at 01.37.23

The target JeeNode will receive its power from the working JeeNode, so don’t plug it into anything. Note that this can also be done with JeeNode USBs. All we need, is to make a few connections for power and I/O.

The working JeeNode should be plugged into a USB-BUB, or something similar. Do that now, because the following wiring step can be a bit sensitive to movement.

The following six connections need to be made:

Screen Shot 2010 05 29 at 02.57.58

On the working unit, the wires or wire jumpers can be inserted into the 6-pin port headers.

On the target side, you’ll probably just have gold-plated holes for the ISP connector. The nice thing about gold plating is that it easily makes contact. So one way to hook up the wires is to insert all six as shown above, and position the whole thing in such a way that there is a slight tension on the wires – just enough to make contact.

Ok, you should now have the working unit powered up, and the target unit receiving power and signals through these 6 wires. Good, leave them alone now.

To do the actual reflashing, I’ve added an isp_repair.pde sketch to the Ports library, which does everything. Upload that sketch to the working JeeNode, and open up the console. Here’s what you should see:

Screen Shot 2010 05 29 at 02.24.05

That’s it. The target ATmega has been set up with the proper fuse settings and a boot loader. You can remove all the wires, and hook up the JeeNode to a USB-BUB. It’ll now accept sketches like any other JeeNode. Fixed!

▶ View 10 Comments

Arduino, Network

EtheRBBBreadboard

In AVR, Hardware, Software on Jun 7, 2010 at 00:01

Please excuse the crazy title. This picture probably tells you more:

Dsc 1509

You’re looking at an RBBB from Modern Device, hooked up to the Ether Card from Jee Labs. The hookup is trivial, it need 6 wires: 2 for power and 4 for SPI:

Screen Shot 2010 05 29 at 04.18.26

And that’s all you need to create a webserver!

Here’s a sample screen (yep, it’s been running for almost 12 hours):

Screen Shot 2010 05 29 at 00.03.48

And here’s the code for it:

Screen Shot 2010 05 28 at 12.05.03

It presents a self-refreshing page with the “uptime”, i.e. how long the server has been running since power-up.

The code is available as “rbbb_server.pde” example sketch in the EtherCard library and is a simplified adaptation of the etherNode sketch, omitting the RF12 driver and calls, of course. It compiles to 6 Kb of code.

As with the etherNode sketch, the MAC address and IP address have to be set to suitable values in the source code before uploading it to the RBBB.

BTW… did you notice anything?

The RBBB is running at 5V. And it works. Because the Ether Card is compatible with 3.3V and 5V signals!

Which also means that the Ether Card can be used with any Arduino board. It’s not specific to JeeNodes and it’s not limited to being used with the Carrier Board, either.

The generic software for the Ether Card is contained in the “EtherCard” library, listed on the software page in the Café. It doesn’t depend on Ports or RF12 libraries, so this thing really is a completely independent product.

It just “happens” to fit gorgeously into a box alongside a JeeNode :)

So there you have it – the Ether Card can be used with just about any microcontroller setup. All it needs is a power supply of 3.6V or more, 4 SPI I/O pins, and the proper software to make it sing of course…

Onwards!

Why use acknowledgements

In Software on Jun 6, 2010 at 00:01

Yesterday’s post showed how to receive packets sent out by the “bmp085demo.pde” sketch, and report them back to the serial port. By reporting in the same format as the sender, this makes it possible to easily switch between a local direct-connect setup and a two-node wireless setup.

But there’s a bit of unnecessary complexity in there.

Note that I’m using the Easy Transmission mechanism, which is indeed easy to use – only four lines of C code is all it takes to turn a local node into a wireless node.

If you try out this setup, you’ll see that the receiver doesn’t always report packets, i.e. not every second. That’s not because it’s missing packets, but because the Easy Transmission mechanism includes logic to avoid sending anything if the data hasn’t changed.

There is a way to force sending packets with rf12_easySend.

But there’s another issue with this setup. The Easy Transmission mechanism uses acknowledgements and timers to make sure that all data arrives at its destination. If no ACK is received within one second, the data will be resent.

Trouble is, this is a bit silly: we’re sending out fresh readings every second anyway, in this setup! Why use timeouts and a retransmission mechanism, just to resend potentially outdated information in the first place?

This setup doesn’t need a retransmission mechanism, it adds no value.

In fact, we don’t even need ACKs: what’s the point of an ACK if the proper arrival at the receiver end doesn’t really matter to the sender? In this setup, the sender is going to send updated readings once a second anyway.

This is a clear case of overkill, so let’s simplify. Here is the improved sender:

Screen Shot 2010 05 26 at 22.11.57

It’s not shorter than before, because the rf12_easy* functions really are indeed quite easy and compact. But the code size is reduced a fair bit:

Screen Shot 2010 05 26 at 21.33.00

vs.

Screen Shot 2010 05 26 at 22.13.12

(Much of the remaining code size is due to the code needed for the BMP085 pressure calculations)

And here’s the receiver, which no longer sends ACKs:

Screen Shot 2010 05 26 at 22.08.15

The difference in size is due to no longer loading any transmit code:

Screen Shot 2010 05 26 at 21.38.27

vs.

Screen Shot 2010 05 26 at 22.07.28

More importantly perhaps, the receiver now reports readings once a second, including unchanged ones:

Screen Shot 2010 05 26 at 22.22.32

… and of course, that – without the ACKs – there will be about half as many packets flying through the air.

The moral of this story is: don’t include functionality if it doesn’t serve a need!

JeeNode, Sensors, Techniques

Switching from direct serial to wireless

In Software on Jun 5, 2010 at 00:01

The recent weblog post about the BMP085 sensor used on the Pressure Plug sends its readings out to the serial port once a second. It also included a few extra lines of code to send the results wirelessly via the RF12 driver. I added that code to illustrate how easy it is to go from a wired hookup to a wireless hookup with JeeNodes:

Screen Shot 2010 05 26 at 21.36.01

Sending it is only half the story – now I need pluck it out of the air again. I could have used the RF12demo sketch with – say – a JeeLink to pick up these transmissions, but then I’d get something like this back:

Screen Shot 2010 05 26 at 21.33.22

I.e. 6 bytes of raw data. One way to deal with this is to write some code on the other end of the serial port, i.e. on the receiving workstation, to decode the reported temperature and pressure values. That’s what I’ve been doing with JeeMon on several occasions. Then again, not everyone wants to use JeeMon, probably.

Another way is to simply create a special-purpose sketch to report the proper values. Such as this one:

Screen Shot 2010 06 24 at 18.42.53

Sample output:

Screen Shot 2010 05 26 at 21.27.22

I used the same format for the output as the “bmp085demo.pde” sketch, but since the two raw data values are not included in the packets, I just report 0′s for them. As you can see, the results is more or less the same as having the Pressure Plug attached directly.

The ACK logic in this sketch looks a bit complicated due to the bit masking that’s going on. Basically, the goal is to only send an ACK back if the sending node requests one. And since we assume that the sending node used a broadcast, we can then extract the sending node ID from the received packet, and send an ACK just to that node.

Tomorrow, I’ll describe how all this can be streamlined and simplified.

JeeNode, Network

JeeNode as web server

In AVR, Software on May 31, 2010 at 00:01

The new Ether Card add-on makes a lot of new fun stuff possible. Even with a measly 8-bit ATmega chip, it’s possible to create some pretty neat things. Like a little webserver for displaying the last received RF12 packets.

I’ve added an EtherCard library to the subversion repository, next to the Ports and RF12 libraries, etc. It’s now listed on the software page in the Café.

This is mostly the Ethernet code by Guido Socher and Pascal Stang, but I’ve moved some stuff around a bit, and kept the changes added by Andras Tucsni to make this work together with the RF12 driver.

A new “BufferFiller” class has been added, to help create web pages and such:

Screen Shot 2010 05 22 at 00.04.55

What it does is expand a PSTR given to it, by replacing $S, $F, and $E with strings copied from RAM, flash, or EEPROM, respectively. A bit like “printf” in standard C. It’s eaasiest to just show an example of use:

Screen Shot 2010 05 22 at 01.42.00

The $D code is expanded as integer, for convenience. And since BufferFiller derives from the Arduino’s “Print” class, all the usual “print()” and “println()” members are also available.

The above code will generate the following web page (and yes, it’s been up and running more than 3 days):

Screen Shot 2010 05 25 at 09.55.06

It’s part of the etherNode.pde sample sketch in the EtherCard library – a little web server which shows the last few incoming RF12 packets and continuously self-refreshes to present the latest info.

There’s also a configuration page:

Screen Shot 2010 05 21 at 18.46.32

The whole etherNode demo sketch is under 10 Kbyte, including both EtherCard and RF12 drivers, leaving plenty of room to implement more features.

BMP085 in high-resolution mode

In Software on May 27, 2010 at 00:01

The BMP085 is a pressure sensor, as used on the Pressure Plug:

Dsc 0735

It’s quite popular. Some people get more than one – I can only assume it’s for some sort of altitude application.

By popular demand, I’ve updated the BMP085 code to support all 4 resolution modes. There is now an optional second argument to set the oversampling:

Screen Shot 2010 05 22 at 11.45.55

Allowed values are 0..3 – the default is the same as before, i.e. 0 (no oversampling).

Sample output from the bmp085demo.pde sketch in the Ports library, adjusted to use maximum oversampling:

Screen Shot 2010 05 22 at 11.42.09

Note that the return values of some intermediate routines have been changed from 16-bit to 32-bit. The calculated results are the same as before: temperature in units of 0.1°C and pressure in Pascal (i.e. x 0.01 hPa).

JeeNode, Network

Meet the Ether Card

In AVR, Hardware, Software on May 24, 2010 at 00:01

Yesterday’s post was a sneaky way to show you a glimpse of an exciting new addition to the always-evolving Jee Labs product range – meet the Ether Card !

Dsc 1454

It’s a low-end Ethernet extension card, with a form-factor specifically made for the Carrier Board + Box:

Dsc 1467

It is based on the good ol’ trusty ENC28J60 chip, with all the components needed to hook it up to a JeeNode (or any other unit running the SPI bus at 3.3V).

I’ve started working a bit on the software side. The Ether Card is pretty standard in every respect, with the GPL2 code by Pascal Stang and Guido Socher working just fine with it. All it needs is a different chip select pin (PB0, Arduino pin 8) and proper interrupt guards to prevent the RF12 driver from interfering.

The card has been running smoothly for days on end here. It gets slightly warm, I’d estimate some 20°C above ambient. The regulator stays cool when powered from 5V. Total current draw is ≈ 150 mA, incl. JeeNode.

The Ether Card only uses through-hole parts, no SMDs. It has been added to the shop as kit and as PCB-only version. Here’s the PCB – in glorious blue-and-gold:

Dsc 1433

Here’s a sample web server requiring under 10 Kb of flash and showing a page with the last 25 RF12 packets:

Screen Shot 2010 05 19 at 11.33.18

So there you have it. JeeNodes can now handle wireless and wired networking.

It’s going to be oodles of fun to develop software for / on / with this Ether Card!

Credits: I would like to thank Andras Tucsni, who started the ball rolling by prototyping a complete working system and implementing the chip-select and interrupt changes needed for inter-operation with the RF12 driver on SPI. Andras also wrote the demo sketch which generated the above output. Knowing that it can be done and having working sample code makes a huge difference!

A subtle RF12 detail

In Software on May 20, 2010 at 00:01

Someone recently emailed about a baffling problem when trying to get two JeeNodes to alternately transmit and receive. It turns out that there is a subtle aspect to calling the rf12_canSend() routine in the RF12 driver, which really needs to be clarified.

The basic idea is to have two nodes running the same sketch. Each of them broadcasts a small packet every 3 seconds, and displays incoming data the rest of the time.

I used my favorite LED debugging technique to create a simple setup with two JeeNodes:

Dsc 1456

The idea is to blink the red LEDs on each send, and the green ones on each receive. Here’s a first attempt which doesn’t do what you’d expect:

Screen Shot 2010 05 17 at 17.40.42

There’s one minor and one major problem with the above code.

The minor problem is that since sending takes place in the background, you’ll probably not see the send LED blink at all: it’s only lit for a few microseconds. The same holds for the receiving end, although there it’ll probably be visible because serial I/O takes some time. The solution for this little problem is to insert delays so the LEDs stay on longer.

But the big problem with the above code is that it doesn’t work. Whoops!

The reason for this is that rf12_canSend gets called constantly, i.e. each time through the loop. Even when the timer hasn’t gone off, and we won’t be sending anything out.

The thing about calling rf12_canSend is that it is also being used to signal your intention to send out a packet. So when sending is possible, and rf12_canSend returns 1, the RF12 driver will stop reception and get ready to send your packet out. Which is what the subsequent rf12_sendStart will do.

The gotcha is that if rf12_canSend returns 1, then you have to also call rf12_sendStart.

How do we solve this? We can’t just exchange the calls to rf12_canSend and the timer poll, because we’d lose timer events (i.e. when the timer fires and just at that very moment rf12_canSend happens to return 0 – which it does, occasionally).

The problem can be solved by using a slight variation of the code:

Screen Shot 2010 05 17 at 17.51.10

I’ve added the complete code of this “pingPong.pde” sketch as example to the RF12 library.

Note: both nodes are set to ID 1. This isn’t a problem in this case, because the packets are sent as broadcasts. A node will never receive its own packet, since it is busy sending. With RF12, packets always go out to “everyone else but me”. When you’re only using broadcasts (and no ack’s), node IDs are irrelevant.

▶ View 10 Comments

Reflow profiles

In Software on May 19, 2010 at 00:01

This is part 7 of my reflow controller series.

Reflow soldering is a pretty simple process. Take a PCB, add solder paste, place components, and then let the whole thing go through a controlled reflow termperature profile. As I’ve described before.

In my (limited) experience, these temperature profiles are not nearly as critical as one might expect. Just preheat the thing while staying under the melting point of solder, then ramp up and keep it all well over the melting point for a while. Not too high and not too long, so nothing gets damaged. Then let it cool down. That’s basically it.

I’m going to use the following profile to start with:

do nothing for 10 seconds
heat up to 140°C
stay there for at least 30 seconds
heat up to 170°C
stay there for another 20 seconds
heat up to 250°C
stay there for 15 seconds
turn off and open up the grill for fast cool down
the finished board can be removed when under 150°C or so

In code:

Screen Shot 2010 05 12 at 23.00.15

I’m staying a bit longer at the high temperature because my grill is a bit uneven in its temperature distribution. I want the cooler spots to work properly as well. Hopefully that won’t damage anything.

So how do you go from a thermostat to a reflow controller? Simple: implement the profile. I added a RunProfile proc, which keeps calling itself over and over again, so it behaves like a background process. Its task is to adjust the target variable over time to match the requested reflow profile. When a step has been completed, it will be removed from the front of the profile list:

Screen Shot 2010 05 12 at 23.02.44

RunProfile is called in start, right after calling InitPlot.

Ok, let’s try it:

Screen Shot 2010 05 13 at 00.10.50

Hey, this is starting to look like something!

I changed the colors a bit and am now also plotting the target temperature for reference.

Latest source code available here.

But all is not well. While trying this out, I noticed that the current setup hangs once in a while. No more incoming data, so the plot and the control stops. I suspect that there’s an occasional conflict between sending out OOK commands and handling incoming packets – perhaps a bug in RF12demo. I worked around this by omitting all the redundant OOK commands.

There was also another case where the last temperature target was set, but the heater wouldn’t get turned on, i.e. HeaterControl returning 0.

So… progress, but not finished yet!

Temperature control

In Software on May 16, 2010 at 00:01

This is part 6 of my reflow controller series. Let’s see if we can get the reflow grill to a stable 150°C.

It’s not trivial: this grill has a slow startup time and a very noticeable lag in its response curve. Turning on the grill and turning it off at 150° is not the way to do it, since the stored heat will lead to a huge overshoot.

The proper way to do this is to use a PID control algorithm. PID stands for Proportional Integral Derivative. It’ll be interesting to try that, but first I want to try something simpler (actually, it is still PID, but just P and D).

The idea is to try and predict where the temperature will end up when we turn the heater off. I’ve got two relevant pieces of information for my particular grill, obtained from the last few trials:

The grill will heat up at about 2.5°C/sec once it gets up to speed.
When turned off at that rate, it appears to overshoot by about 40°.

Let’s try something easy first. Let’s find out what the grill does when turned off at 110°. I’ve added this line to the GotData proc to do so automatically:

if {$value >= 110} { set heat 0 }

Result:

Screen Shot 2010 05 12 at 17.29.53

Not bad – still some overshoot, so I’m going to assume an overshoot of 50° from now on. BTW, this is not the same as keeping the oven at a preset temperature, but it’s a start. In fact… I’m going to keep this line as a permanent safety valve:

if {$value >= 265} { set heat 0 }

It’s not essential, since the grill has a mechanical temperature cut-off as well, but that way I can be sure that this code will never try to push its heater beyond 265°C. It would have avoided yesterday’s runaway failure.

To improve on this, let’s assume that the overshoot is proportional to the heat-up rate. So if we turn off the heater while it’s heating up at 1.25°C/sec, it will overshoot by 25° – it seems plausible, since that probably means the heater hasn’t been on that long yet, so there is less “stored excess heat” in the system.

Next thing to do is to track the rate of change and base heater decisions on that. I’ve added a new HeaterControl proc which decides what to do for a given target temperature:

Screen Shot 2010 05 12 at 18.35.46

Note that heater control is simply a matter of setting the heat variable. It controls both the remote switch and the GUI checkbox, courtesy of Tcl’s built-in variable tracing facilities. This does the actual control, in GotData:

set heat [HeaterControl $value $lastv $x $lastx]

And in the start proc, this extra code will get the ball rolling:

variable target 0
after 10000 set [namespace which -var target] 150

IOW, after 10 seconds, JeeMon will attempt to maintain the grill temperature at 150°C. Let’s try it:

Screen Shot 2010 05 12 at 18.38.52

Woohoo, it works! A thermostat!

The “application.tcl” source code is available here. Next step is to add a reflow temperature profile.

Controlling the oven

In Software on May 15, 2010 at 00:01

This is part 5 of my reflow controller series.

Today, I’d like to be able to remotely turn the grill on and off. To avoid having to deal directly with high voltages (220V is scary!), I’m going to use an RF controlled switch – i.e. this FS20 module:

Dsc 1428

It’s perfect here, because it operates @ 868 MHz and can be controlled directly from a JeeLink or JeeNode, and because it has an on/off button right on the unit itself (unlike these). Which is great as emergency stop – we’re going to play with serious levels of electricity, current, and heat after all.

As it so happens, the RF12demo sketch I’ve been using to receive packets from the thermocouple node also supports sending FS20 commands out of the box.

So all that’s needed is to extend the GUI a bit with a control element, and hooking that up to send the proper FS20 command out.

This requires a few extra lines in the initPlot proc:

variable heat
pack [checkbutton .h -text Heater -variable [namespace which -var heat]]
trace add variable heat write [namespace which HeatChange]

And a proc called HeatChange, for which I’ll use a bit of test code for now:

proc HeatChange {args} {
  variable heat
  puts "heat = $heat"
}

The result is a window with an extra checkbox at the bottom:

Screen Shot 2010 05 12 at 024300

Clicking that button simply generates some test output:

heat = 1
heat = 0
heat = 1
heat = 0

Great. The GUI side is working. Here’s an updated version of HeatChange which actually sends out the proper FS20 commands:

proc HeatChange {} {
  variables heat conn
  if {$heat} {
    $conn send 54,32,1,17f
  } else {
    $conn send 54,32,1,0f
  }
}

The first 3 values are the house code and address bytes. They can be anything, because FS20 modules are configured by putting them in a special listening mode (press the button until the LED starts blinking). The next RF command sent to them will then be remembered, so that it will respond to that specific code from then on. Code 17 means ON, code 0 is OFF – that’s part of the standard FS20 protocol (see this German info page). The trailing “f” tells RF12demo to send everything out as an FS20 command.

IOW, to respond to these RF signals, put the FS20 unit in that special mode and then send one of the above commands by clicking on the checkbox in the GUI. You should now be able to manually control the remote switch.

Note: make sure you have the latest RF12demo. A nasty OOK bug was fixed a few days ago. If your JeeLink hangs: unplug, reconnect, then upload the latest code.

One more thing I’d like to do is include the heater status in the plot. That requires a few more changes. Here’s the latest “application.tcl” (I’ve collapsed the start and HeatChange code, since they are the same as before):

Screen Shot 2010 05 12 at 03.57.04

Let’s try this new setup, i.e. measuring and controlling 100% by wireless.

What I wanted to do is hook it up to my Ersa I-Con Nano temperature-controlled soldering station (with the soldering tip removed), because that would have been a great demo of how real temperature control works:

Dsc 1418

Unfortunately, that didn’t work – and drove home that there’s a real risk of fire involved in these experiments. Here’s what happened:

Screen Shot 2010 05 12 at 15.48.26

The temperature shot up to 450°C in seconds! – I think there’s a sensor in the very tip of the iron, and it wasn’t touching anything, so this heater went full blast – charring the thermocouple insulation on its way up. I switched the iron off manually, and then everything coooled off.

Second try, this time replicating yesterday’s setup:

Screen Shot 2010 05 12 at 16.02.57

Perfect. A step pulse and the response curve (grill was opened @ 175°C, like yesterday).

Warning: if you try these experiments, make sure you unplug your oven / grill / whatever when you’re done. Starting a fire while you’re tinkering with something else, or out of the house, or asleep is not a good idea…

Tomorrow, I’m going to create a feedback-control loop.

Oven temperature plot

In Software on May 14, 2010 at 00:01

This is part 4 of my reflow controller series.

Data is coming in over the serial USB connection. Quick, let’s visualize it!

There are two ways to do this: with some external tool such as Excel, or with the GUI facilities built into JeeMon.

Let’s do Excel first, because it requires less coding. Change yesterday’s “application.tcl” example to this:

proc start {} {
  set conn [Serial connect usb-A900adav 57600]
  oo::objdefine $conn forward onReceive [namespace which GotData]

  variable fd [open logfile.csv w]
  chan configure $fd -buffering line
  puts $fd "time,temperature"

  vwait forever
}

proc GotData {msg} {
  variable fd
  if {[Serial cmdParser $msg OK -node id -int1 temp]} {
    puts $fd "[Log now],$temp"
  }
}

The cmdParser function in Serial helps with decoding the “OK …” lines. It takes type arguments and variable names. In this case the node id header and an integer, to be stored in a variable called temp. The “-int1″ notation means: treat the int as having one decimal, i.e. convert 123 to 12.3, etc.

Run JeeMon and it will create a “logfile.csv” file with readings (in “Comma Separated Values” format). You may have to stop JeeMon before you can open the logfile with another application.

Using “Numbers”, a Mac OS X application similar to Excel, this is what I get:

Screen Shot 2010 05 10 at 20.11.19

You can see the sensor at room temperature, heating up as I touched the thermocouple, and then cooling off again gradually.

The other approach is to create the plot with Tk, the GUI toolkit which is built into JeeMon, as I did with the OOK Scope. This is more work, but you get a plot which updates in real time.

So now let’s make a graph. I turned the grill on until it reached about 175°C, then let it overshoot and cool back down to 175°C again, and then I opened the lid. This is the result over a period of some 8 minutes:

Screen Shot 2010 05 10 at 23.16.59

Looks like this little grill will overshoot by some 40°C, and that it can heat up about 2.5°C per second. It’s only 700 Watt, which probably explains it. Should be fine for reflow, though.

This is the code I used, i.e. “application.tcl” (source here):

Screen Shot 2010 05 12 at 01.04.40

All this is standard Tcl/Tk code, as documented here if you want to explore how it works. With some elbow grease, I hope to add such basic plotting facilities to JeeMon as utility code, hiding most of the distracting details.

Tomorrow, I’m going to add remote switching to control the oven.

Setting up JeeMon

In Software on May 13, 2010 at 00:01

This is part 3 of my reflow controller series. I’ve got a remote node sending out temperature readings once a second, and now I want to do something with that data.

First step is simply to get it into JeeMon and display values as they come in.

Warning: JeeMon is not Emacs, Eclipse, or Processing. For several reasons:

JeeMon is portable across a far wider range of platforms. The core also works on tiny embedded Linux boards such as this one, for example.
I want a system which can be wrapped up, shipped, and used elsewhere without installation hassles – even cross-platform. JeeMon can do that.
I prefer to use the same editing environment for everything I do because I work with lots of different environments, so I use “TextMate” on Mac and fall back to “vi” on Linux.
JeeMon can be grown into a “big” system, but it doesn’t need to. It can also be used as bridge for numerous other tools.
There’s a lot to like about big environments which take care of everything, but I prefer lower-level tools which let me get under the hood and tinker.

Does that make JeeMon primitive? I don’t think so. Infancy: yes. Limited scope: yes (so far). One-man activity: yes (so far). Perfect: nope (nothing ever is). Fit for its intended purpose: you bet!

All good things come in three, so to work with JeeMon, you need three things:

The JeeMon runtime, a single executable file.
A programmer’s editor. Pick your favorite. Get a good one. It’ll change your life, as developer.
Willingness to figure out how to glue things together using the Tcl programming language.

Let’s get started.

1. Set up the tools

There are JeeMon binaries for Windows, Mac OS X, and Linux as ZIP files, 2..3 Mb each:

Windows: 32-bit, 64-bit
Mac OS X: 32-bit (Universal)
Linux: 32-bit, 64-bit

Download the one you need, unpack, and you should end up with an executable called “JeeMon”. Feel free to rename it to “jeemon” (lowercase) or even “jm”. These files are 100% open source, but they’ve been wrapped up into single-file executables to get going fast. See the JeeMon page for more background info on the technology used inside JeeMon.

As for which programmer’s editor to use, you’ve probably already got a preference. It doesn’t matter what it is – stick with it and learn it well, is all I can say. A good editor lets you find references and definitions, colorize your source code, compare file versions, lookup documentation, create boilerplate templates, interface with a version control system, and much much more.

2. Get organized

This is going to be a moving target. I’m still exploring the best way to manage code and data, so that it is easy to use in the editor, in the Arduino IDE, and with JeeMon.

The Arduino IDE already sort of imposes a structure to use for its libraries and sketches. Fine.

We just need a good place for JeeMon. I suggest creating a new folder called “JeeMon”, right next to your “Arduino” sketches folder. On my Mac, that happens to be the Documents folder. This is where the above JeeMon executable should be placed.

Here’s a mock-up of the folder structure I have, when following the above guidelines:

Screen Shot 2010 05 09 at 17.42.47

It should be fairly similar on Windows and Linux, I expect.

3. Tie everything together

This is where the real work starts. We need to tell JeeMon how to hook up to the JeeLink, and what to do with incoming packets once connected.

Everything in JeeMon is always driven from a file called “application.tcl”. For this first trial, we can just create that file next to the JeeMon executable itself. Create a file called “application.tcl” with the following contents:

proc start {} {
    array set ports [SysDep listSerialPorts]
    parray ports
    vwait forever
}

In prose: call the listSerialPorts function in the SysDep module, and store the results in an array called ports. Then print that array. Then just stop and wait (but don’t exit, because then the output would be gone too).

In JeeMon, modules are called “rigs” btw – SysDep is a built-in rig. The above “application.tcl” file is also a rig. Once initialized, the system executes “application start” as its very last step. Which is how the “start” procedure in the above “application.tcl” file gets control. There’s no magic and very little syntax. Just some conventions.

Launch JeeMon. Sample output here, with three JeeNodes / JeeLinks hooked up:

ports(usb-A8007UsI) = /dev/tty.usbserial-A8007UsI
ports(usb-A900ad5m) = /dev/tty.usbserial-A900ad5m
ports(usb-A900adav) = /dev/tty.usbserial-A900adav

Your output will be different. The info you need to extract from the output is the connection name of your JeeLink. In my case it is “usb-A900adav”, so that’s what I’ll use in the following examples.

Stop JeeMon.

Replace the contents of “application.tcl” with the following code, but use the name of your interface in that second line, of course:

proc start {} {
    Serial connect usb-A900adav 57600
    vwait forever
}

In prose: call connect in the Serial rig, then wait forever. Communication takes place in the background.

When you start JeeMon again, you should see some output similar to this:

22:04:30.328        . (adav) [RF12demo] _ i31 g6 @ 868 MHz 
22:04:30.342        . (adav) DF I 42 4
22:04:30.343        . (adav) Available commands:
[...]

That’s output from the JeeLink. When the thermocouple node is turned on, you should see its packets being reported. Sample output:

22:04:34.348        . (adav) OK 33 209 0
22:04:34.350        . (adav)  -> ack
22:04:35.346        . (adav) OK 33 212 0
22:04:35.348        . (adav)  -> ack
22:04:36.344        . (adav) OK 33 206 0
22:04:36.346        . (adav)  -> ack
22:04:37.341        . (adav) OK 33 209 0
22:04:37.343        . (adav)  -> ack

Not much to write home about. But now you’ve got all the pieces in place to start doing more interesting stuff. GUI, networking, webserver, database, it’s all there in JeeMon, waiting to be activated and combined as needed.

There will be some minor differences between Windows, Mac OS X, and Linux, but not much really. If you’re following along and things don’t work as expected, please let me know. I’ll be happy to adjust these notes to cover as many possible details as needed to get going.

Let’s get back to the reflow side of things. We need to figure out how our thermocouple and our oven behave, and after that we need to find a way to control the oven temperature. No worries – one step at a time.

Tomorrow, I’ll set up a temperature graph. Two, in fact.

Musings

There IS a pattern

In Hardware, Software on May 11, 2010 at 00:01

If you’ve been following this daily weblog for a while, you may have noticed that it’s all over the place – as if I’m working on everything and nothing. At the same time!

But there’s a pattern, dear reader.

Not fully worked out, not fully planned, not static, but still… there is.

The pattern is that I’m currently trying to automate some of the stuff I need to do here to keep Jee Labs running. The shop has been growing steadily, which is great because it means I can keep doing this – which is exactly what I want to do. Indefinitely, preferably.

But the shop needs to run smoothly, so that I don’t end up becoming its slave. That means the daily production work needs to be automated as much as possible. Only then will it be possible for me to work on all sorts of fun projects, keep up this daily weblog, and fill it with – hopefully – interesting topics, day in day out.

This “self-automation” is why I created the Flash Board, for example. And why I’m redoing the reflow controller to work more reliably. I’ve also been automating like crazy recently to try and stay on top of this huge pile of parts called (haha!) inventory.

I have not lost track of the main focus of Jee Labs – the direction where it all started: energy use tracking and environmental monitoring around the house. It’s still my main focus. And now that the basic hardware works, with lots of configurations and sensors to play with, the next frontier is the software.

But software is a very finicky beast. With hardware, you hook up a few parts and it starts working – after some soldering and/or tinkering, evidently. Software is both primitive and complex in comparison: primitive, in the sense that you have to create these big house-of-cards constructions to get anywhere. Complex, because each of the ingredients is usually a massive chunk of code.

So I’m taking a lot of time to think through numerous aspects of the software. JeeMon is the house I’m building, but its core structure needs to support ideas which haven’t even been invented yet. In terms of software, that means it has to be very modular. I’ve currently got a few components in place, and the binding structure and modularity trade-offs are starting to become clear.

What I don’t have right now, is a clear enough view on the data storage side of things. So all the little JeeMon experiments so far have been side-stepping the issue of persistence (the IT word for “storage”). I’m showing things on screen. Great, but of limited utility.

What’s the big deal with persistence? Well, the moment software includes a storage mechanism, you get into the issue of how to make changes. Suppose you have a working system, and you want to change it in some fundamental way because of a new insight. How do you deal with the data it has already stored? It sounds trivial, but I think it’s everything but trivial – on a very fundamental level.

Storage is a big deal. It is crucial. And it comes up even with something as simple as displaying a moving average. How do you deal with a system restart when there are moving averages in graphs you want to show, for example?

Software development and persistence create opposing forces. Development means just that: to progress from one insight to the next as you go along and extend your understanding of the deeper issues in the problem space at hand. And then, ideally, to implement solutions in better – sometimes completely new – ways. As a developer, I constantly tear my software apart, to put it back together in improved ways (I probably do it 10x more often than most developers). This is a learning process, and the result – IMO – leads to simplicity, elegance, and almost as a side effect: robustness.

Persistence is the elephant in the room. It opposes change. Data saved on disk (or flash memory) has a structure, and changing that structure can be anything from awkward to nearly impossible. That’s why starting from scratch is so easy. That’s also why version 2 of anything in software can be so elusive. It’s not just data, btw – code is data too, in this context. Try folding a new idea into an existing bit of code …

Maybe I’m overstressing this a bit – m a y b e. But this is the main reason why I work on completely unrelated issues at times, such as streamlining the shop activities.

In the meantime, as background process, I keep exploring scenarios for the software and collecting insights from what others do or even just ask for.

So there you have it: ISP programmers, solder reflow controllers, even tangential activities such as 3D printing and CNC milling, they all get addressed here at Jee Labs. Meanwhile, I try to figure out the best way forward for many far-reaching design choices w.r.t. JeeMon.

The good news is that I think this detour is coming to an end. I think a simple, small, modular, and fun way of tying all sorts of hardware together via software is starting to shape up – in a vague hand-waving style right now, but that’s just a matter of pushing the ideas into code. And doing the grunt work.

Neither the ideas nor the code are the hard part. Ideas are cheap and plentiful. Code is easy and can be created gradually. No, the really hard part is to come up with a “pervasively modular” architecture. I.e. how to set up a system (and hence make choices) which can evolve, even in ways not yet imagined.

What I’m after is not a solution, but a tool. And that takes a bit more time…

As the well-known Chinese proverb goes:

Fish

I’m not a teacher, but the fish doesn’t interest me nearly as much as improving the whole process of fishing.

Arduino, Reflow

Setting up the thermocouple node

In AVR, Hardware, Software on May 10, 2010 at 00:01

This is part 2 of my reflow controller series. Unlike what was announced yesterday, I’m going to first describe how to set up the temperature sensing wireless node. JeeMon hookup to follow soon.

Our very first step could be to connect the thermocouple via a JeeNode and USB to the PC, but I’m going to do something more interesting and go straight for a wireless hookup. One reason for this is due to a problem with direct connections, but since this is going to be used in un-tethered mode anyway, it’s a good excuse to use a wireless configuration right from the start.

Here is the “thermoSend.pde” sketch I’m going to use (code here):

Screen Shot 2010 05 09 at 14.29.35

This contains all the ingredients needed for a simple basic sensor node: all we do is set up wireless, and then read out the thermocouple value and send it off once a second.

The easy transmissions code works with broadcast packets, so we don’t need to define a destination for the wireless packets, just this sensor node’s ID (1), the choice of frequency band (868 MHz), and a group ID (6). That’s done with the rf12_initialize() call.

The readings are converted to an integer in the range 0 .. 3300, representing a temperature range of 0 .. 330.0°C. Then we send that 2-byte integer in binary mode over wireless. That’s what the code inside loop() does. The map function is part of the Arduino library code.

The easy transmit system will take care of re-transmission if a packet is lost. All the transmission details are handled by rf12_easyPoll(), which needs to be called often.

One detail of the easy transmit system to keep in mind, is that only changed values are sent. No packets will go out if the reading is the same as the last one. In this scenario, that’s fine – there’s always some jitter in the readings, so we should see new packets at least a few times a minute, even when the sensor temperature is constant.

Ok, let’s get going. If you haven’t installed the Arduino IDE yet, go do it now.

Create the above sketch named “thermoSend” in the Arduino IDE. On my Mac, it ends up as a folder named “Documents/Arduino/thermoSend/” in my home directory. Plug in the JeeNode via an FTDI adapter such as the USB-BUB (or use a JeeNode USB) and check your Arduino IDE / Tools / Serial Port settings to make sure you’re hooked up to it. With multiple nodes on USB, it’s easy to mess up the wrong one – happens to me all the time…

Now the node is ready. Unplug, hook it up to a battery, and power it up.

You’re sending packets into the ether now. Quick, let’s try and collect those packets before they falll on the floor!

To do that, hook up a JeeLink or a second JeeNode. It needs to be running the RF12demo sketch, which is what it does out of the box if you got it from Jee Labs or Modern Device.

We need to connect to the JeeLink. Easiest way is to use the Arduino IDE. Make sure you are connected to the proper USB port. So again, check the Arduino IDE / Tools / Serial Port menu and select the JeeLink serial port.

Now open the Arduino IDE’s serial console. You should see something like this:

[RF12demo] A i1 g212 @ 433 MHz
Available commands:
[etc]

You are now in direct communication with the JeeLink. We need to set it up to listen to the proper transmissions. Sort of like tuning to the proper radio station. So type the following line and press “Send”:

2i 8b 6g

This sets node ID = 2, frequency band = 868 MHz, and net group= 6.

If all is well, and the wireless node is powered up, you should start to see packets come in, something like this:

OK 33 212 0
 -> ack
OK 33 209 0
 -> ack
OK 33 212 0
 -> ack
OK 33 209 0
 -> ack
OK 33 212 0
 -> ack

That’s a header byte (i.e. 33, associated with node ID 1) and 2 data bytes.

There won’t be packets every second, because only changed readings are sent. But there will be packets, and if you touch the thermocouple end, you should immediately see the rise in temperature:

OK 33 212 0
 -> ack
OK 33 226 0
 -> ack
OK 33 24 1
 -> ack
OK 33 44 1
 -> ack

No, I’m not suffering from sudden hypthermia: that’s not 4.4°C :) – what you’re seeing is binary overflow of the first data byte. A byte can only hold values 0..255. Anything higher and it will wrap around. That’s why two bytes are sent. The second byte contains the reading divided by 256, i.e. the number of wraparounds. You’re looking at the binary representation of an “int” on Atmel AVR chips.

So the above readings are: 21.2°C, 22.6°C, 28.0°C, and 30.0°C.

IOW, the actual temperature is: (byte1 + 256 * byte2) * 0.1°C

The next episode will be about hooking up JeeMon to the Jeelink and using it to read out the data and do something more meaningful with it. Stay tuned.

ISP, JeeNode

A capturing ISP programmer

In AVR, Hardware, Software on May 5, 2010 at 00:01

Meet the new, improved, autonomous, self-guiding, hassle-free, portable ISP programmer!

Dsc 1381

It works as follows:

hook up a JeeNode or JeeNode USB to your computer
upload the isp_capture.pde sketch to it
insert the Flash Board and hook it up to the target
program the target, using this as a standard STK500/AVRISP programmer @ 19200 baud
wait for the LED to blink twice
done

And this is where the fun starts:

connect the JeeNode to a battery or any other power source
insert the above Flash Board again and hook it up to the target
press the button on the Flash Board
wait for the LED to first turn off and then start flashing
done

You’ve just programmed another target MPU … look ma, no hands!

The first hookup went through a normal programming cycle, but it also stored everything in the EEPROM on the Flash Board: code, data, fuses, verification bytes, everything. That’s why I’m calling this a capturing programmer.

When pressing the button, it essentially repeats all the same steps.

This works for anything you can program with an AVRISP programmer: ATmega, ATtiny, whatever. And it will even capture multiple programming cycles, as long as they are started before the LED blinks twice, i.e. within a few seconds. So you could set up a script to call avrdude with a few different things to do – e.g. set fuses, program the flash, program the AVR EEPROM, set the lock bits.

There are some tricks under the hood to make this work. First of all, the baudrate as ISP programmer is set to 19200, so that the bootloader in the JeeNode doesn’t accidentally take over after reset (it listens at 57600 baud).

Another trick, and the main reason this all works transparently, is that the entire serial communication session is stored in EEPROM as is. When pressing the button, it simply replays the input to the programmer code as if it was coming from the serial port (and matches the results, also against EEPROM). There is some logic involved to be able to store both input and output streams, and keep them properly apart.

The EEPROM is connected via I2C to port 3. I used the MemoryPlug and MemoryStream classes from the Ports library to access it.

Lastly, there is some debugging built in. While used as AVRISP programmer, the serial port is in use for programming @ 19200 baud. But when in replay mode, the serial port is set to 57600 baud and used to report what the programmer is doing. Here is a transcript:

[isp_capture]
ISP bytes: 39680
Code size: -32768
Page size: 128
Data size: 1024
Signature: 86 00 00 01 01 01 01 03 FF 95 0F
Programming...
Done in 13.9 seconds.

That’s it. There is visual feedback when programming succeeds in the form of a flashing LED, so that this setup can be used without serial link. I’d like to add auto power-down one day, for serious battery use.

I’m going to use a bunch of Flash Boards here, pre-loaded with the different contents of ATmega’s and ATtiny’s I’m constantly preparing here at Jee Labs. Will probably also dedicate a bunch of JeeNodes to this, but that’s optional – any available JeeNode can be temporarily turned into an ISP programmer by simply uploading the “isp_capture.pde” sketch to it and inserting a Flash Board.

Now I can easily reprogram all those Room nodes in the house!!

PS. There’s nothing JeeNode-specific about this setup. The on-board wireless isn’t used (yet?).

PPS. For your convenience, I’ve tagged all related posts on this weblog with ISP.

Communication 101 – Networks

In Hardware, Software on May 4, 2010 at 00:01

After yesterday’s introduction, let’s tackle the more advanced version of communication: networking.

For example a set of machines connected via Ethernet:

Screen Shot 2010 04 26 at 13.48.00

(often, the hub will also be connected to a router and/or internet)

While similar to “point-to-point” communication on the surface, and sometimes even at the hardware level, this is completely different in terms of software.

The difference between dedicated links and networks, is that the network is shared for use by multiple nodes. It can’t deal with everything at once, so again, “time” is a crucial aspect of networking.

First of all, you’ve got to take turns talking, How do you take turns? Not so easy. In face-to-face conversations, we use an intricate system of pauses, expressions, gestures, and behaviors to effortlessly take turns in a conversation. On the phone, it’s a little harder, on a phone with a noticeable time lag, it can be frustratingly hard.

With computers, one thing you have to do is break each transmission into chunks – i.e. “packets”. Then it’s a matter of getting all the packets across, one by one, in the correct order, until the task is done. Or abort between packets, and allow changing roles so that other nodes can start to transmit.

Ethernet is a common way to connect computers, and TCP/IP is the standard used to do make communication possible. The whole internet is built on it. On a network, everyone has to play by mutually agreed rules. These can be fairly complex, which explains why microcontrollers need some extra hardware and code to be able to act as Ethernet node – and it can easily reach the limits of what they can do.

The nice bit about networking is that once you’re “on” a network, any node can communicate with any other node. But again, there is quite a bit of machinery and logic needed to make that possible.

The two key features of the serial communication described yesterday, is that they are reliable and throttled. Well, not so on a network. Packets get lost or damaged in transit. Someone could be messing with the cable while two units are exchanging packets, and disrupt the whole process. Even if two nodes are working 100%, they can still fail to exchange a packet!

So with networking, you have to deal with things like timeouts, acknowledgement packets (which can get damaged as well), and re-transimssions. You also have to deal with flow control, to avoid sending more data than a receiver can handle. Imagine sending a steady stream of packets: if one of them gets lost, we have to detect the failure (takes time), re-send the packet (takes more time), and catch up with the new incoming data!

Before you know it, yesterday’s “Serial.println(value);” example turns out to require a lot of logic and error-handling.

It gets worse: what if the receiving node isn’t even connected right now?

The transmitter should to be able to detect this so it can decide what to do.

Sometimes, there is no alternative but to ignore it. Sometimes, that’s not even such a big deal – for example with a sensor which is periodically sending out new readings. It’ll fail, but once the receiver is back online, it’ll all start to work again.

If you’ve ever ever looked into a “networking stack” (i.e. its code), or even implemented one yourself, you know that writing this sort of code is a complex task. It’s not easy to communicate reliably when the communication channel is unreliable.

This daily weblog is a nice analogy, btw. I send out daily “posts” (packets), but this one for example continues where yesterday’s discussion left off. In a way, by assuming that you, dear reader, will follow along for more than one post, I’m creating a “story” (virtual circuit), based on daily chunks. It’s a fairly robust communication stream operating at a constant rate. Based on nothing but web pages. So now you can think about packets, next time you watch a YouTube video :)

What about wireless networks?

Screen Shot 2010 04 26 at 13.48.08

The good news is: wireless networks have to deal with most of the same issues as wired networks, and these issues are well understood and solvable by now.

The bad news is: wireless networks have to deal with a lot more unreliability than wired networks. All it takes to disrupt a wireless network is some RF interference from an old arcing motor, or even just walking out of range!

It’s possible to maintain the illusion of a serial connection over a network – it’s called a virtual circuit. TCP/IP does that: when you talk to a web server, you often exchange a lot more than will fit in one data packet. So TCP/IP sets up a mechanism which creates the illusion of a serial link – a virtual pipe between sender and receiver. Put stuff in, and it’ll come out, eventually.

Except that this illusion can break down. There’s no way to maintain the illusion of a permanent “connection” when you unplug the receiver, for example. Or walk out of range in a wireless network.

There’s no way even to give guarantees about communication speed. You might be at the very edge of a wireless network, with the system frantically trying to get your packets across, and succeeding perhaps once an hour. Oof – made it – next – yikes, failed again – etc.

In a wireless sensor network (WSN), which keeps sending new readings all the time, the illusion of a virtual circuit – i.e. virtual serial connections – can break down. And what’s worse: when it does, it’ll do exactly the wrong thing: it will try to get the oldest packet across, then the next, and so on. Which – if the network is really slow – is just going to lead to a permanent backlog.

What you want in a WSN, is to occasionally drop the oldest readings, which are more and more obsolete anyway, if that helps recover and obtain the most recent readings again.

A bad WSN’s can give you lots of data you don’t want, whereas a good WSN will give you the latest data it can. The trick is to stop trying to send an old value as soon as you’ve got a new and more up-to-date reading.

This is the reason why the RF12 driver used with JeeNodes uses a packet delivery model, instead of a virtual circuit model. In networking terms: UDP instead of TCP. Also: best effort instead of (semi-) guaranteed in-order delivery.

Nice bonus: packet delivery is far simpler to implement than virtual circuits.

What has worked so well for teletypes and serial consoles for several decades, is not necessarily a good idea today. Not in the context of networks, particularly WSN’s. For another example, you need look no further than the web: part of HTTP’s success is based on the fact that it is “state-less”, i.e. not fully connection-based.

So all I have to do now, is to convince the whole world that thinking of communication as serial connections can be awful in some scenarios!

Heh, piece of cake: just make the whole world read this weblog, eh?

Communication 101

In Hardware, Software on May 3, 2010 at 00:01

Triggered by a recent post on the discussion forum, it occurred to me that I may be taking way too many concepts for granted. My problem is that I’ve been around computers too long. Not so much in being socially inept (just a bit introvert :) – no, the issue is that I seem to have become pretty familiar with how they work, from silicon to database to web-server.

This is a huge hurdle when trying to share and explain what I’m doing, and it probably makes this weblog harder to dive into than I intended, as a friend recently pointed out – an insight for which I’m very grateful.

So after this little bit of personal communication, let me get to the point: what I’d like to do from time to time on this weblog is to go into some key topic, relevant to the projects here at Jee Labs, but doing it in such a way that will hopefully bring more insight across to people who share the enthusiasm for all this Physical Computing stuff, but not necessarily all that techie background.

Not to worry – this is not the start of a textbook :) – nor a treatise. Just trying to clarify some stuff. Succinctly, as much as I can. If you know all this, or if it bores you – bear with me for one or two posts, I will go back to other topics again in the next posts. When I make mistakes, or say nonsense, please do correct me. I live to learn.

Today, I’ll kick this off with Communication (Wikipedia) – in the context of information exchanges between computers and peripherals, and within the hardware of various types of systems.

First off: why communicate? Because that’s what computers do, essentially. Number crunching (i.e. the literal sense of “to compute”) is secondary by now.

Examples, in the context of physical computing:

sending a measurement value to our PC
sending information to a display
sending data to an attached chip or device
sending a control signal to turn things on or off
sending packets by wireless to another node

How can information be sent? In short: as bits, from a software perspective, or as electric / magnetic / optical / mechanical signals from a hardware perspective. You could say that Physical Computing is about bridging those software and hardware perspectives. Sensing “stuff” and making “stuff” happen. With the fascinating part being that there is computation and change awareness (state) and decision-taking involved via the microcontroller which sits in the middle of it all.

This is a big topic. Let’s narrow it down. Let’s focus on communication in the form of bits, because ultimately just about everything goes through this stage.

Screen Shot 2010 04 26 at 13.38.51

Let’s take that first example: sending a measurement value to our PC. How do you do that? Simple, right? Just put something like this in your sketch:

Serial.println(value);

Whoa… not so fast. There’s a lot going on here:

we select an interface (Serial)
we fetch the measurement value from a variable (value)
we convert that data to a text string (println)
we transmit the text, character by character, over a serial link
somehow that serial link uses electrical signals to travel over a wire (hint: FTDI)
somehow this finds its way into a USB port (hint: device driver)
somehow this is picked up by an application (hint: COM or /dev/tty port)
somehow this appears on the screen (hint: serial console)

And what’s probably the most complex aspect of this entire process: it takes time. What appears to happen in less than the blink of an eye to us, is in fact a huge time consumer as far as the microcontroller is concerned.

If we ignore the details, you have to admit that this works pretty well, and that we can indeed easily get results from a microcontroller board to our PC.

That’s due to two key features of this comms channel:

the connection is reliable: what is sent, will arrive, eventually
the connection is throttled: sending is slowed down to match the reception speed

It’s easy to take this for granted, but not everything works that way. When you send data to an attached LCD display for example, the software has to take care not to send things too fast. LCD displays need time, and there are limits to how fast information can be presented to them. The Arduino “LiquidCrystal” library is full of little software delays, to avoid “overrun”, i.e. sending stuff faster than the LCD can properly handle.

The trouble with delays is that they hold everything up!

Well, that’s the whole point of delays, of course, but in a microcontroller, it means you don’t get to do anything else while that delay is in progress. Which has its own problems, as I’ve described earlier.

If you think about it for a moment, you might see how delays in fact make communication virtually impossible: because if you’re supposed to wait for all the little steps to complete, how can you possible handle incoming data, which has no idea that you’re currently waiting in delays and not listening to anything?

I won’t go into the (hardware-based) solutions which work around this issue, but it has been solved, to a certain extent. This is why data coming in from the USB link will usually arrive just as expected, yet at the same time sending out data usually slows down the microcontroller in clearly noticeable ways. Try making a blinking LED, and then sending a string with Serial.println in between each blink. Sure enough, either the blinking or the serial output will become slower or even irregular.

Communication of of data takes time. We don’t have infinitely fast connections. Even something as simple as “Serial.println(value);” is full of nasty side-effects and details, especially on a microcontroller such as the ATmega.

It’s good to keep this in mind. This is one reason why something as seemingly simple as collecting data from multiple serial streams requires a fairly advanced library such as NewSoftSerial, and why even that clever code has its limitations.

Tomorrow, I’ll talk about packets, networking, and wireless communication.

Decoding a pulse train

In AVR, Hardware, Software on Apr 27, 2010 at 00:01

The (planned) Input Plug uses an ATtiny chip to decode a 16-channel selection code using a single I/O pin.

Here is the relevant part of the schematic:

Screen Shot 2010 04 19 at 19.50.37

The DIO pin is connected to PB4 of the ATtiny. The ATtiny PB0..PB3 pins in turn are hooked up to the A..D select inputs of the analog multiplexer. So all the ATtiny needs to do is decode an incoming “pulse train” on its PB4 pin, and send out the decoded value on its PB0..PB3 output pins.

My first attempt used a simple serial protocol: a start bit and 4 data bits, clocked at a fixed rate:

Screen Shot 2010 04 19 at 19.52.59

This worked, but it was a bit flakey because the internal clock of the ATtiny is not accurate enough to ensure everything stays in sync for the entire duration of the transmission, i.e. across 5 bit periods. This is clearly explained in the following image (from here):

async1

It’s better to use a self-clocking format, for example 5 pulses of varying width, because then only the length of each individual pulse matters. Above a certain threshold = 1, below = 0. And we can reset when there are no pulses.

Here’s the a transmit test sketch, which sends a 0..15 counter every 100 ms:

Screen Shot 2010 04 19 at 19.57.36

As you can see, the pulses are 4 or 8 µs wide, with one pulse every 12 µs. Roughly.

Note that interrupts have to be disabled during each “transmission”, since these pulses are very short and need to be fairly strictly controlled.

The decoder on the ATtiny (ATtiny85 in this test) is also quite simple. It waits for a pulse start and then counts in a loop until the signal drops to zero again. Counts above a certain value are treated as “1″. Missing pulses and pulses which are way too long cause the decoder to be reset and start from scratch:

Screen Shot 2010 04 19 at 20.01.21

This can be compiled with avr-gcc, and it’s of course just a few bytes (even an ATtiny is overkill here):

Screen Shot 2010 04 19 at 20.02.44

I’m using yesterday’s Flash Board for ISP programming. Makes a good test that it also works with ATtiny MPUs.

Here’s the test setup (with the ISP programmer disconnected):

Dsc 1362

And sure enough, the LEDs display a little running 4-bit counter, driven by data sent over a single DIO pin.

Should be good to go for the Input Plug!

ISP, JeeNode

Preparing ATmega’s with ISP

In AVR, Software on Apr 25, 2010 at 00:01

Here’s a second use of yesterday’s ISP plug: pre-loading ATmega’s with a fixed sketch and bootloader.

This uses a very nice trick from the Arduino Boot-Cloner: store the code to be sent to the target MPU as PROGMEM data inside the ISP sketch itself!

I’ve adapted that Boot Cloner so that it takes its data from a separate C include file, added a couple of other features, and speeded the whole thing up a bit.

The result is called “isp_prepare” (code here).

Here’s what you’ll see on the USB serial port when starting it up:

Screen Shot 2010 04 18 at 23.14.22

The list reports the files which have been integrated into this sketch at build time, using this C code at the top of isp_prepare.pde:

Screen Shot 2010 04 18 at 23.23.36

Here is the contents of that include file, with most of the data bytes omitted:

Screen Shot 2010 04 18 at 23.24.51

The data consists of “sections” of code, to be programmed into the target ATmega using ISP. In this case there are two sections, a RF12demo sketch starting at address zero, and a bootloader in high memory.

Here’s is isp_prepare in action:

Screen Shot 2010 04 18 at 23.14.47

So the steps to load RF12demo onto a JeeNode with a fresh ATmega are as follows:

upload this isp_prepare.pde sketch to a “master” JeeNode
insert the ISP plug described yesterday,
connect the target JeeNode to this master JeeNode via ISP
enter G to start programming and wait for it to complete
done: disconnect, now the target JeeNode is ready to run with RF12demo on it

I’ve also included a “data_blink.h” header file, if you want to preload the ATmega with the standard Arduino blinking LED demo. Just change the include in isp_prepare and upload the sketch again.

I can now use this setup for initializing all the ATmega’s here at Jee Labs, since it means I no longer have to start up the Arduino IDE or use avrdude.

Convenience!

ISP, JeeNode, JeePlug

ISP plug

In AVR, Hardware, Software on Apr 24, 2010 at 00:01

For one of my projects, I needed a quick way to reflash an AVR via ISP. Didn’t want to have to use any of the several ISP programmers around here, so I made my own “ISP plug” for use on a JeeNode:

Dsc 1353

The pins are connected as follows:

ISP pin 1 = MISO = DIO1
ISP pin 2 = VCC (+3.3V)
ISP pin 3 = SCK = AIO1
ISP pin 4 = MOSI = AIO4
ISP pin 5 = RESET = DIO4
ISP pin 6 = GND

Here the top view, made from a little JeePlug board:

Dsc 1354

Bottom side:

Dsc 1355

Note that this ISP setup draws 3.3V from the JeeNode and uses it to power the target, so the target board should not have its own power and it should not draw more than say 100 mA of current @ 3.3V.

The code for this is called “isp_flash.pde” and is derived from the “ArduinoISP” sketch. I added software-based (bit-banged) SPI communication for use with the above I/O pins, and did a fair bit of cleanup of the source code.

The nice thing about this ISP programmer is that it works out of the box with Arduino IDE 0018:

upload the ISP_flash.pde sketch to the JeeNode
insert the ISP plug and hook it up to the test circuit (or one of these …)
burn the boot loader using the Arduino IDE:

Screen Shot 2010 04 18 at 18.04.06

The result will be a properly initialized ATmega, with protected bootloader and the default blink demo pre-loaded.

It’s not terribly fast because it uses a 19200 baud connection, but it’s simple and can be a life-saver if you ever damage the bootloader on an ATmega, or want to prepare blank ATmega’s for use with a JeeNode.

This plug can in fact be used as ISP programmer for any type of ATmega or ATtiny with “avrdude”:

Screen Shot 2010 04 18 at 18.14.42

You’ll need to adjust the serial / USB port, fuse settings, and sketch as needed, of course.

Home automation

(Not so) Home Easy

In Software on Apr 23, 2010 at 00:01

Here’s the Elro / Home Easy AB440R remote control:

Dsc 1351

On the surface, it looks like a very clean 433 MHz protocol:

Screen Shot 2010 04 17 at 12.16.22

Here are some bit patterns, as I see them coming in:

Screen Shot 2010 04 17 at 12.15.26

All the pulse widths are in the expected range (first two are timestamp and pulse count):

Screen Shot 2010 04 17 at 12.38.38

This is completely in line with the protocol description I found. Given that it uses a special 0-0 pattern for extended group codes, I decided to pass all the transitions back, leaving the interpretation to higher-level code:

Screen Shot 2010 04 17 at 12.31.28

Here’s what comes out:

Screen Shot 2010 04 17 at 12.30.50

Several very odd problems with this:

the incoming bit patterns are less than 50 bits
the KAKU decoder also matches these patterns
I’m getting spurious empty lines reported

The KAKU decoder match is very surprising, since it only fires when receiving exactly 12 bits! As it turns out, the other way leads to false positives as well: the KAKU transmitter currently triggers the HEZ decoder – but that’s more logical, since the HEZ decoder isn’t checking exact packet sizes yet.

Something very fishy is going on, but I can’t see where :( – despite this, I’ve updated the code.

Oh well, can’t win ‘em all…

JeeMon, JeeNode

A mini scope

In Hardware, Software on Apr 22, 2010 at 00:01

Triggered by a discussion on the forums, here’s a little demo of how to build your own “Poor Man’s Scope”:

Screen Shot 2010 04 21 at 20.36.10

I connected an LDR between AIO1 and ground, and played around with light levels during this screen shot. The last samples were all over the map because I inadvertently touched the LDR leads.

All you need to reproduce this experiment is: an LDR, a JeeNode, and the JeeMon runtime. Oh, and the Arduino IDE to compile and upload a sketch to the JeeNode.

The sketch is adapted from the arduino-arduinoscope.pde demo. This is not nearly as fancy (or even flashy) as that Processing-based version, but it’s very easy to set up and get started with, IMO.

Anyway, here is the sketch running on the JeeNode:

Screen Shot 2010 04 21 at 20.44.32

It does a lot more than I’ll be using here, since I’m only picking up and displaying the first analog value, i.e. AIO1 on the JeeNode.

To get started with JeeMon, you could go read this page to get some background, but all you need really is the executable, so you can just download the appropriate one: for Windows, Mac, or Linux.

Then:

unpack the ZIP file, so you end up with a “JeeMon” executable
create a directory called “JeeMon-rigs”
in that directory, create a file called “application.tcl” (contents shown below)
adjust the serial port of the JeeNode to your setup (look at the top of application.tcl)
launch JeeMon, and you should get the real-time scope display shown above

Here is the application.tcl file, written in Tcl and using the built-in Tk GUI subsystem for visual display:

Screen Shot 2010 04 21 at 20.53.36

I’ve sprinkled some comments in the code. There are a fair number of pesky little details (as in all programming languages), but even without knowledge of Tcl/Tk you should be able to more or less see what’s going on.

There’s a lot one could improve or add to this thing. There always is. This is just a starting point.

But there you go: a Poor Man’s Scope in less than 100 lines of JeeNode sketch + JeeMon rig code. Fun!

Home automation, X10

X10 RF reception

In Software on Apr 20, 2010 at 00:01

One more decoder, a home automation protocol this time – a X10 RF transmitter:

Dsc 1342

That’s a Marmitek SS13, sending out over 433 Mhz (European model).

This is a fairly simple protocol (documented here), although synchronization and packet detection works a bittle differently from the other decoders so far:

Screen Shot 2010 04 16 at 00.06.22

Sample output:

Screen Shot 2010 04 16 at 00.05.35

Easy!

Home automation, JeeNode, Sensors

More OOK decoders

In Software on Apr 19, 2010 at 00:01

O(o)k … so there are now several decoders for OOK transmissions: 3 for 433 MHz and 4 for 868 MHz:

Screen Shot 2010 04 15 at 10.36.43

Sample output:

Screen Shot 2010 04 14 at 23.20.21

The code is here. It has been set up as an Arduino sketch, but the decoder logic is not Arduino-specific, nor even ATmega-specific in fact. The sketch compiles to under 5 Kb, leaving plenty of room for future extensions.

All that is needed is to call the decoders with pulse widths in microseconds. They all run in parallel, but normally at most one will succeed for any given pattern of pulses. The actual decoding completion happens during the final long pulse (2000 µs or more, in most decoders).

Here is the rest of the sketch, which drives the decoder from pulses measured in an interrupt routine:

Screen Shot 2010 04 15 at 10.36.58

Note that there is something odd in this code: all the decoders are fed the same pulses! In real use, separate receivers will need to be connected for 433 Mhz and 868 Mhz, and then pulses fed to the appropriate decoders only. The reason I did it this way was for testing: I can simply replace the OOK receiver with whatever I’m currently trying out to see if it works – no reset needed.

So that’s it for now: 7 decoders, capable of decoding packets from several dozen different types of sensors and transmitters.

The data is reported as a set of hex nibbles, which need to be separated into the specific bit patterns for each type of packet. Also, checksums still need to be verified, to weed out any remaining false positives.

But that’s another topic, for another day.

Note – the RFM12B receiver cannot receiver OOK signals. I hooked up extra hardware to make the above work.

Home automation, Sensors

Visonic sensors

In Software on Apr 18, 2010 at 00:01

Another set of sensors is made by Visonic. Here’s the door sensor:

Dsc 1341

It has three sources of triggers: a reed switch, normally kept closed with a magnet on the door or window, an extra set of ordinary contacts, and an internal switch which is released when the case is opened.

There are other sensors using the same protocol, such as a PIR motion detector and a “universal transmitter”.

The sensors send OOK data on the 868 MHz band, but it took me a while to figure it all out. The main reason being that I couldn’t find a preamble for these radio packets – apparently there is none! – instead, the tail of the packet is a good reference point, i.e. the 72 pulses preceding the long pause at the end can be reliably decoded.

Each bit is either a short-long pulse pair, or a long-short pulse pair. This can in fact be used to resync even before the end of the packet is reached, because short-short and long-long combinations indicate that we’re off sync.

The total packet has 72 pulse transitions, i.e. 36 bits of data. I add 4 extra zero bits to create a 5-byte result. Here is the main decoding logic:

Screen Shot 2010 04 14 at 16.28.07

Here is some sample output:

Screen Shot 2010 04 14 at 16.35.43

I’ll postpone decoding of the actual bit pattern to another day but note that you don’t really need to interpret all the bits: just look which patterns your sensor is sending out under various conditions, and you can match the incoming packets accordingly.

Sensors

Improved OOK Scope

In Software on Apr 17, 2010 at 00:01

In yesterday’s post, I used the OOK Scope to find the distinctive pulse pattern emitted by a specific sensor to be able to write a decoder for it.

Keep in mind that this is a bit like looking for a needle in a haystack: there are often over a million pulses coming in within just minutes. That’s why I had to compare the “sensor-off” baseline with the “sensor-on” pattern. And even then, you never know for sure where pulses come from and especially whether they came together in quick succession, as expected for genuine RF packet data.

So I added some filtering: instead of showing all pulses, I only look for pulse sequences with pulse widths and pulse counts in a certain range, followed by a brief period of silence (i.e. a relatively long pulse).

I also added some configurability, i.e. whether to display using a logarithmic horizontal scale, whether to “decay” old data so new pulses are weighted more heavily in the counts.

One last very convenient feature added was a “peak detector”, i.e. trying to identify the most prominent peaks in the graph and reporting their microsecond values in a status line at the bottom (the first value is a pulse count).

I didn’t implement a GUI to adjust these values – I simply set them as needed in the code and restart JeeMon:

Screen Shot 2010 04 13 at 20.58.19

Here’s the result of running the OOK decoder with these settings and waiting for a packet from the Cresta sensor:

Screen Shot 2010 04 13 at 19.52.25

Bingo: very clean peaks, as well as the actual durations of the main pulses, i.e. 392 µs, 576 µs, 880 µs, and 1056 µs. As I’ve been able to determine from previous experiments, these are just two groups of pulses: 392/576 = 0 and 880/1056 = 1. The reason is probably skew – these receivers appear to have a completely different response to carrier-start and carrier-end transitions.

Here’s a plot when leaving the OOK Scope on and accumulating for one hour:

Screen Shot 2010 04 13 at 21.05.53

There’s a bit more garbage now, with additional noise and a few more peaks being “detected”. I’ve added code so that pressing ESC clears the counts and starts from scratch – this can help isolate a specific transmission.

This version of the OOK Scope was used to establish more accurate pulse widths for the Oregon and Cresta sensors, and that’s was led to the values used in those decoders. The latest source code of the OOK Scope has been checked in, replacing the previous version. It’s still very small: under 200 lines of Tcl/Tk code.

Note how much you can do with just an OOK receiver and software: it’s like a special-purpose scope. For these experiments it’s in fact better than a storage scope or a logic analyzer, because of the custom pulse filtering!

Sensors

Cresta sensor

In Software on Apr 16, 2010 at 00:01

Another day, another sensor:

Dsc 1339

This is a Cresta unit (no serial number on it), with a simple outside temperature sensor (which has been outside for a some time). Here’s the OOK scope fingerprint:

Screen Shot 2010 04 12 at 13.50.25

Here’s the baseline, i.e. the background signal with the sensor switched off:

Screen Shot 2010 04 12 at 14.06.37

Some clear peaks, at values 100, 135, 175, and 195, roughly. These correspond to pulse intervals centered around 400 µs, 570 µs, 890 µs, 1080 µs, respectively.

My hunch is that the 400/570 and 890/1080 pairs are really just the same pulse, skewed by the size of the preceding pulse.

In this case, it turns out that the sensor sends 3 copies of the packet, with a 1..3 counter value inside. Seeing that counter in the raw bits indicates that this sensor is not using Manchester encoding, simply 1 = 1 and 2x 0 = 0:

Screen Shot 2010 04 13 at 20.11.30

I get better results with the sensor at least 1 meter away from the receiver – a very strong signal probably adds extra spikes or shifts pulses too far apart.

Time to write a decoder for this thing:

Screen Shot 2010 04 13 at 20.15.06

With the base unit turned on to see what the values being reported are, I was again able to determine the basic details of the packet layout. Each 8-bit byte is followed by a 9-th parity bit. So this hex data isn’t as easy to read:

Screen Shot 2010 04 13 at 20.18.56

But the expected temperature values are in there, and the 1..3 channel selection is as well. Again, I’ll save the checksum verification and the post-processing in JeeMon for another day.

Onwards!

Sensors

Oregon sensors

In Software on Apr 15, 2010 at 00:01

Let’s try to decode the OOK data coming from an Oregon Scientific THGR810 sensor:

Dsc 1340

It sends at 433 MHz (the above picture was accidentally set to Fahrenheit). I used the OOK scope to figure out which pulse widths to use in the decoder. Ended up splitting pulses shorter and longer than 700 µs.

Anyway, now I can write a bit-decoder for it, using the same finite state machine approach I’ve used before for KAKU and FS20. But there’s one difference: these bits are Manchester encoded, i.e. 0 has two short pulses, while 1 is one long pulse. The Manchester encoding can be deduced from the fact that when you replace two short pulses my one marker, you get more or less consistent packet lengths. Since successive 1′s must flip the interpretation of the signal, there’s one extra bit of state to carry around in the decoder.

Most of these things can be determined by trial and error. Same for the synchronization pattern, the exact bit offset to start the data, and the bit / nibble / byte order of the data.

It helps tremendously that the sensor has an LCD display, showing what values it is transmitting, of course.

My basic approach is to collect lots and lots of pulses, and save them to file as Tcl scripts. Then I weed out impossible runs of bits, keeping only what looks like potential 0/1 sequences of a decent length. Then just keep collecting, until some “packets” end up being received more than once.

Once I have a dozen or more packets which keep showing up, it’s time to look at the preamble, to try and figure out what the common prefix could be. It’s usually pretty obvious by then. The only uncertainty being at which bit the preamble stops and the actual data starts.

Anyway, for the THGR810, this is some of the data I ended up receiving:

Screen Shot 2010 04 12 at 17.43.19

If you look closely, you can already see the reported values in reverse hex in the data.

That’s it!

Here’s the main part of the decoding logic for this sensor:

Screen Shot 2010 04 13 at 16.11.39

And here’s the received data:

Screen Shot 2010 04 12 at 12.18.44

The last value is the seconds timer, showing that this sensor is reporting values roughly every 70..90 seconds. I did some resets and fiddled with the channel switches, to be able to later determine which bits they correlate to.

Another run, showing the temperature dropping from 21.1° ro 20.1°:

Screen Shot 2010 04 13 at 00.37.49

Will do the checksums and separation into readings in JeeMon … another time!

An OOK Scope

In Software on Apr 13, 2010 at 00:01

Using yesterday’s OOK plugs, I wanted to get a feel for what sort of RF pulses are flying around here at Jee Labs. It’s a first step to picking up that data and trying to decode some of it.

Meet the OOK Scope, a sketch which reports every pulse it receives over the serial port, and a matching Tcl/Tk script for JeeMon to display the results graphically, and in real time.

Here’s the “ookScope.pde” sketch:

Screen Shot 2010 04 11 at 14.17.41

The analog comparator is used to generate interrupts on each transition of the input signal across the 1.1V bandgap reference voltage (the digital pin-change interrupt would also have worked – I’ve use both in the past).

One byte of data is sent over the serial port for each transition, containing the elapsed time since the previous one.

Note that the serial port can easily become a bottleneck for all these pulses: at 57600 baud, it can “only” report some 5700 pulses per second, so rapid pulse trains under 170 µs or so will hit the serial port bandwidth limit.

Reporting pulse lengths with 2 bytes of data would halve this bandwidth, so a trick is used to be able to report pulse lengths from 20 µs to over 3 ms using a single byte of data. It works by encoding large values more coarsely, sort of like a poor man’s logarithm:

pulses are measured at 4 µs resolution, using the Arduino micros() function
pulses of 20 µs or less are ignored, they are simply too short to deal with
24..508 µs pulses are reported as byte values 6 .. 127 (4 µs granularity)
512..1020 µs pulses are reported as values 128 .. 191 (8 µs granularity)
1024..1532 µs pulses are reported as values 191 .. 223 (16 µs granularity)
1536..2044 µs pulses are reported as values 224 .. 239 (32 µs granularity)
2048..2556 µs pulses are reported as values 240 .. 247 (64 µs granularity)
2560..3068 µs pulses are reported as values 248 .. 251 (128 µs granularity)
3072..4604 µs pulses are reported as values 252 .. 255 (256 µs granularity)
all longer pulses are also reported as value 255

In other words: short pulses get reported fairly precisily, longer ones less so.

Due to the way AGC (Automatic Gain Control) works, most receivers will end up constantly generating pulses, because the gain is adjusted continuously until something is detected, whether actual RF signals or noise.

So this sketch tends to generate a huge amount of data over the serial port. In my case, I’m usually seeing over 2000 bytes of data per second coming in.

That’s a lot of bytes. Trying to make sense of it is not trivial – needles and haystacks come to mind …

So let’s just start by plotting the frequency with which pulses of different durations come in. This is an excellent task for JeeMon, with its built in Tk graphical user interface.

Here’s a first plot, based on some 1,000,000 samples:

Screen Shot 2010 04 11 at 15.34.35

The shortest pulses are represented by the lines at the top (i.e. starting at 24 µs), the longest ones are lines furher down. The width of the lines is the number of times such pulse lengths were received. The whole graph is scaled to fit horizontally, with some statistics showing in the window title.

Every 10th pulse width is marked in blue. Every 100th is marked in red.

Note that these are condensed values, as listed above. The “widths” are shown as values 5..255 on the vertical axis.

What the graph shows, is essentially noise. There’s a gradual decrease in pulse widths, i.e. short noise pulses are more frequent than longer noise pulses.

There is a very surprising peak of pulse lengths around 880 µs – I haven’t figured out where they are coming from, but that’s clearly not noise.

Let’s change the graph by switching to a logarithmic scale on the horizontal axis from now on (note that the vertical scale is sort of logarithmic as well, due to the above encoding). Here’s the logarithmic version, from roughly 1,000,000 samples:

Screen Shot 2010 04 11 at 15.26.09

Interesting: there’s another source of pulses around 1100 µs – again, no idea where those come from.

Here’s a graph where I held down a button on an FS20 remote control:

Screen Shot 2010 04 11 at 15.44.50

Sure enough – peaks around 380 µs, 412 µs, and 584 µs. The FS20 protocol works with 400 and 600 µs pulses – the variations are probably due to skew in the receiver, with different skews depending on the length of the preceding pulse.

Here’s the Visonic “fingerprint”, also on 868 MHz:

Screen Shot 2010 04 11 at 16.05.53

Let’s try the 433 MHz band now – a nice clean histogram using the ELV 433 MHz OOK receiver:

Screen Shot 2010 04 11 at 15.14.58

And here’s one using the Conrad 433 MHz OOK receiver:

Screen Shot 2010 04 11 at 15.12.35

Odd, there’s a blind spot in there. Also, the Conrad receiver reports only 1/10th the number of pulses reported by the ELV receiver. Either it’s filtering out noise much better, or it’s simply a lot less sensitive!

The ookScope code (sketch and Tcl/Tk script) can be found here.

Turning the protocol around

In Software on Apr 8, 2010 at 00:01

Yesterday‘s post made it possible to send a message to a remote LCD screen. The remote node is set to constantly receive packets, and once a valid packet comes in, it displays the text.

There is no acknowledgement, so the sender does not know whether the remote node actually did receive the message, nor even whether it is turned on.

This can be solved by “turning the protocol around”: instead of sending packets to a remote receiver, the remote node periodically polls a central node to see if there is new data. The communication becomes bi-directional.

Let’s first change the code in the remote node to do this periodic polling (code here):

Screen Shot 2010 04 07 at 19.26.48

I’m using the “easy transmission” calls in the RF12 driver. They are used in a somewhat unusual way: by sending out empty packets, and then waiting for a valid acknowledgement packet to come in. This is done by checking the return value of rf12_easyPoll() – it will be 1 when an ack has been received.

But now there’s a problem. The JeeLink will immediately send an ACK back when it gets the data:

Screen Shot 2010 04 07 at 19.28.25

That’s not what we want, because the acknowledgement will contain no message text.

Instead, the RF12demo needs to be configured to use “collect” mode instead of “respond” mode. That will stop it from sending out the ACK:

1c

Now something else happens. Lots of packets will start coming in, in quick succession, because the remote node’s easy transmission setup is not getting any ACKs. So it keeps retrying:

Screen Shot 2010 04 07 at 19.31.40

We’ll need to change the JeeMon “application.tcl” script to use ACKs instead of direct packets (code here):

Screen Shot 2010 04 07 at 19.52.21

This change is fairly subtle. The SendToLCD code hasn’t changed. But it’s used in a completely different way now: instead of just sending out one packet, we set up a connection and keep it going – and therefore listening…

The trick is now to check what’s coming in, and when an empty packet from node 9 is coming in (“OK 41″, i.e. 32+9), then we send an ACK (128+9) with the message text. And since we’re done, we also quit the application.

There’s one small GOTCHA… it doesn’t work!

The problem is a bug in the RF12demo code, which prevents it from sending out packets that look like an ACK. I’ve fixed it now, so you’ll need to update your JeeLink or JeeNode – whichever you use as central node.

Once you’ve done that, it should work: power up both nodes and then launch JeeMon in the same way as before. If all is well, a message will again appear on the LCD screen:

Dsc 1327

Note that I’ve extended the remoteLCD sketch to support 2 lines of output: everything after a newline ends up on the second line of the display.

Sooo – similar result, but using a completely different mechanism.

This approach has as benefit that the remote node will stubbornly keep trying to obtain a message to display, even when packets get lost. But in its current form, that’s also a drawback: when JeeMon is not running, the LCD node will keep on retrying forever. With the easy transmission mechanism, it normally retries 8 times, once a second, but since we restart the packet every 5 seconds, it ends up constantly retrying, each second.

There are some other advantages with this remotely-initiated protocol. One of them is less important in this particular case: since the remote node is in control, it doesn’t have to keep the receiver on all the time, and can go into low-power sleep mode any time to save on battery consumption.

The second advantage is that the remote node doesn’t need to know where to get its data from. The easy transmission mechanism uses broadcasts, so whichever node decides to respond to it, gets its message displayed. As long as exactly one node always does, to prevent constant retries.

A third advantage is that the central node can detect when a remote node is up and when not, and act accordingly. This is a topic for another weblog post…

P.S. The Dimmer, Gravity, and Lux plugs have been added to the shop.

Remote LCD from JeeMon

In Software on Apr 7, 2010 at 00:01

Yesterday‘s post got a message to an LCD display, using a JeeLink with RF12demo to send the packet over the air. It works, but the packet had to be manually encoded as decimal bytes. Not very convenient.

Today, I’ll use JeeMon to streamline this somewhat. There will be several iterations of this example, so that it can also be used as an introduction to JeeMon and the Tcl programming language it uses.

To that end: here is a list of Tcl links which may be useful if you want to learn more about Tcl. Having said that – the code below should be fairly clear, even if you don’t want to dive in further.

This first example is mostly to get started with JeeMon.

Read the Getting Started section to obtain a suitable version of JeeMon (Windows, Mac, several Linux’es).

Got that? Ok, make sure it launches and no errors appear. Most likely result of doing so should be a message with “Can’t start (yet), …“. Good. It works.

Let’s set up JeeMon to send a message to the remote LCD.

Create a directory called “JeeMon-lcd1″.
Inside, create a file “application.tcl” with a text editor, and place the following Tcl program code in that text file (you can download it here):

Change the “usb-A600dVNz” in the above program to match the USB device you’re using to send the message with (can be either a JeeLink or a JeeNode). You can use the Arduino IDE to find out the proper name (probably COMn on Windows). Do not use the port of the remote LCD node, since that JeeNode is listening to incoming packets, not its serial port.
Launch JeeMon from the command line, by typing this command: “JeeMon lcd1” - JeeMon will now load the code contained in the JeeMon-lcd1 directory.

If all went well, you should see two things happen:

Some debug output will be printed:

Sending: 72,101,108,108,111,44,32,87,111,114,108,100,33,9s

the text “Hello, World!” appears on the LCD!

Dsc 1307

To be continued…

Update – only moments ago, a bug in the RF12 driver was fixed. Make sure to get the latest one and re-build / upload the remoteLCD.pde sketch again if needed.

JeeLink, JeeNode

Demo remote LCD

In Hardware, Software on Apr 6, 2010 at 00:01

It’s hard for me to realize just how confusing all this Jee stuff can be, and it’s certainly good to be reminded of that once in a while. Not everything makes sense for people who aren’t immersed into all this every day.

I want to improve that. In fact, it’s the main reason I’ve been chasing around in search of a good forum + wiki setup. As it stands, I’m hesitant to start writing down lots of things and, eh… yes, documenting all this Jee stuff. But this will change. Promise.

For starters, some good news on the forum + wiki front: I’ve decided to standardize on Markdown as formatting language. It’s ancient (i.e. proven), it’s clean, and with just one or two extras it will be good enough for most of the things I need. The reason I’m bringing it up here, is that I’ve just finished converting all the wiki pages to Markdown, as well as 90% of the current Jee Labs documentation pages. The forum remains bbForum, but it has now been adjusted to use Markdown formatting as well. So has WordPress. In fact, this is the first weblog post written in Markdown (in Textmate, my favorite text editor).

Yeay for one set of conventions, and yeay again for simplicity!

Ok, back to the main topic of this post.

Here’s a demonstration on how to send a text string to a remote LCD via wireless JeeNodes. This is the hardware I’m going to use for that remote LCD node:

a JeeNode USB
an LCD Plug with 2×16 LCD display
one extension cable
plus a hookup to USB to power this whole thing

Dsc 1300

Ok let’s set up a simple demo sketch for the remote receiver end (code here):

Screen Shot 2010 04 05 at 20.09.58

Very simple, really: wait for an incoming packet and display the bytes as characters on the LCD. Plus some initialization code.

On the sender end, I’m using a JeeLink running the pre-loaded “RF12demo” sketch:

Dsc 1302

First step is to put the JeeLink in the right mode:

8b 4g 1i

That’s the 868 MHz band, group 4, node ID 1.

Now we can send a message to node 9, which is running the remoteLCD sketch, by sending a packet with the individual character codes to the JeeLink (the “9″ at the end is the destination node ID):

84,101,115,116,32,49,50,51,9s

The result:

Dsc 1305

Yippie. No wires. Magic! :)

Not quite a compass, yet

In Software on Mar 25, 2010 at 00:01

This is an attempt to read out the magnetometer / compass of the Heading Board introduced yesterday.

The demo sketch has been extended, and is still deceptively trivial:

Screen shot 2010-03-23 at 17.30.23.png

The meat of the code is in the HeadingPort class implementation in the Ports.cpp source file.

Trouble is – it doesn’t work :(

Sample run #1:

Screen shot 2010-03-23 at 16.28.33.png

Sample run #2, using another sensor:

Screen shot 2010-03-23 at 16.29.04.png

The barometer readings are more or less consistent, but probably both wrong. The local weather station and my Pressure Plug in the ookRelay both report that the current air pressure is currently around 1017 hPa.

As for the temperature readings, well … the temperature sensor in the Pressure Plug reports 20.9°C, and it’s about one meter away, also on my desk.

What’s worse is that the magnetometer readings are missing, and both come out as zero.

Trying again with a different setup, I get this third sample output:

Screen shot 2010-03-23 at 17.25.11.png

The values don’t change when I rotate the sensor. I can’t make anything of this right now, best guess is that the I2C bus transactions are perhaps flakey. Not impossible – this is a pretty simplistic software-based bit-banging I2C implementation after all, which doesn’t support things like clock stretching. I could try it out on a real hardware I2C bus via the Plug Shield. Another day…

The code has been added to the Ports library, as usual.

JeeNode, Sensors

Heading Board

In Hardware, Software on Mar 24, 2010 at 00:01

Another new plug: the Heading Board.

This one is a pretty odd combination: a 2-axis compass, a barometric pressure sensor, and a thermometer:

It’s based on the HDPM01 module by HopeRF, and the board underneath is just to re-route the pins to the proper headers. As you can see, it barely fits.

Note that this board requires two port headers and sits on top of a JeeNode in the same way as a Room Board. This is necessary, even though the sensor uses I2C, because it also needs two more signal lines: a XCLR control signal, and a 32 KHz clock, i.e. 4 I/I pins in all.

The clock signal is tied to the IRQ line, because this is also the OC2B timer 2 output. So this board ties up timer 2 to generate a 32800 Hz clock, and it interferes with using the IRQ pin (which is shared by all port headers). Furthermore, one of the AIO pins is unused. So it’s a slightly odd fit for the JeeNode, really. Oh, well… soit.

Now let’s try it out.

There’s a new HeadingClass in the Ports library, which handles all the basic logic to access this board.

Here’s the heading_demo.pde sketch to read out the temperature and pressure sensor:

Screen shot 2010-03-22 at 23.58.36.png

Very simple stuff, given that most of the code is in the HeadingPort class.

Sample output:

Screen shot 2010-03-22 at 23.58.24.png

I’m not convinced of the accuracy of these sensors. Both the temperature and the pressure values vary a bit, and differ about 0.5% from the readings I see on the Pressure Plug.

Tomorrow, you’ll see why this is called a Heading Board!

Gravity Plug

In Hardware, Software on Mar 22, 2010 at 00:01

The plug story continues…

The Gravity Plug contains a 3-axis accelerometer with 2..8g settable range:

As before, a new GravityPlug class has been added to the Ports library, along with a “gravity_demo” sketch to illustrate its use:

Screen shot 2010-03-19 at 13.08.33.png

Sample output:

Screen shot 2010-03-19 at 13.26.31.png

Values are X, Y, and Z, in that order.

I’m not sure about the high bit readout with the current code, maybe this needs some tweaking to get the ranges right, but you can see the effect of moving the plug around a bit.

The Y axis wraparound may be due to the chip not being completely flat on the board. This is a minute chip, which can’t be soldered by hand because the pads are too small and on the bottom side of the chip. I applied solder paste manually and then used the reflow grill – but even that is tricky, it’s very difficult to get an even-yet-small amount of paste on there! A solder paste stencil will no doubt solve this.

Lux Plug

In Hardware, Software on Mar 21, 2010 at 00:01

Here’s a second simple plug which works:

(the silkscreen markings are incorrect, this plug responds to I2C addresses 0×29, 0×39, or 0×49)

The Lux Plug measures incident light intensity which can be converted to a 16-bit Lux value in the range 1 .. 65535. A 16x multiplier can be used to increase the dynamic range to 20 bits.

A class named – surprise! – “LuxPlug” has been added to the Ports library, as well as a “lux_demo” sketch:

Screen shot 2010-03-19 at 01.31.30.png

Sample output:

Screen shot 2010-03-19 at 01.29.01.png

The two first values are the raw readings from two internal sensors. The TAOS datasheet explains how to derive the Lux value from them. This calculation is included as the “calcLux” member in the class (to be called after getData() has obtained a reading).

Onwards!

Dimmer Plug

In Hardware, Software on Mar 20, 2010 at 00:01

Looks like the Dimmer Plug is working as intended – yippie!

I connected a LED with series resistor between the L1 output and PWR (i.e. 5V in this case).

Added a new “DimmerPlug” support class in the Ports library, and a “dimmer_demo” example sketch:

Screen shot 2010-03-18 at 22.42.06.png

The result is a LED which blinks briefly 4 times per second, with increasing intensity. When the maximum intensity has been reached, it restarts from off. Note that the blinking and brightness control is done by the plug – the sketch just keeps feeding it new settings.

There are 16 independent channels, the outputs can be either open-collector or totem-pole, and by adding a MOSFET, transistor, or darlington stage, much larger currents and voltages can be controlled by this plug.

Another fun application would be to connect up to 5 RGB LEDs, for full 24-bit color control of each one of them.

The above code has been added to the Ports library.

JeeLink

JeeNodes on Bifferboard

In Hardware, Software on Mar 17, 2010 at 00:01

The Bifferboard described yesterday is a bundle of fun (well, for geeks like me anyway…).

But this weblog is about physical computing, not just embedded hardware or software.

So let’s hook up a JeeLink!

The first hurdle is a silly one: my Bifferboard only has a single USB slot, and it needs the USB stick to run off. So for this experiment I added an unpowered USB hub:

Did I mention that this is a standard Debian setup? Here’s the kernel log with the JeeLink plugged in:

Screen shot 2010-03-15 at 12.45.30.png

Note how it recognizes the FTDI serial link without having to install or configure anything. Perfect!

The other factoid which can be gleaned from this info is that it took about 36 seconds from the start of the kernel boot to this stage. From this, I’d estimate a Bifferboard with JeeMon to boot up in well under 60 seconds.

To verify that the USB recognition is really complete, I used this little test in “try/application.tcl”:

And sure enough, it recognizes and identifies the USB device:

Let’s try one more thing – by changing try/application.tcl to this code:

Screen shot 2010-03-15 at 13.01.48.png

Sample output:

Screen shot 2010-03-15 at 13.01.23.png

That’s a Bifferboard receiving JeeNode packets via a JeeLink – yippie!

I can now go back to developing JeeMon further on my desktop machine, in the knowledge that software updates are one little restart away (due to JeeMon’s built-in self-updating over internet) and that apps will probably work as is on a Bifferboard. Long live platform independence!

JeeMon on Bifferboard

In Hardware, Software on Mar 16, 2010 at 00:01

The Bifferboard is an interesting computer which was mentioned on the forum a while back:

It has an ethernet jack, and all it needs is 5V power and a USB stick with Linux on it. You can get an impression of its diminutive size with the JeeLink next to it.

So let’s try it out!

The Bifferboard will work with several different Linux distributions. I decided to use Debian because it’s stable, easy to manage, and has lots of packages ready to install. And because I’m very familiar with it.

Setting up Linux is fairly easy, using these instructions. The hardest part has been done and is fully automated with two scripts: formatting and creating a USB stick to contain a complete Debian 5 (lenny) system. This needs to be done from Linux on your desktop machine, I used Ubuntu.

The only tricky part is that a matching kernel needs to be flashed onto the Bifferboard itself. My older unit has 1 Mb flash, newer units have 8 Mb flash. There are two ways to flash the unit: via a special serial console cable and via a direct ethernet cross-over connection. I tried the ethernet thing, but couldn’t get it to work, and since I had already hooked up a serial line for debugging, I ended up using that method.

The serial port is available as 3.3V I/O signals, but it requires soldering an extra header to the Bifferboard:

(only 3 lines are needed: RX / TX / GND, but I’m swimming in 6-pin connectors around here…)

Those pins are then tied to a USB-BUB, which in turn is hooked up to my Linux setup. The connection is 115200 baud, and shows the kernel boot log on startup.

Once the reflashing is done and the USB stick has been inserted, you end up with a standard Debian setup, which can be accessed over the network via SSH:

Screen shot 2010-03-15 at 12.19.06.png

If there is interest, I can set up a wiki page with more details about all this.

Now the fun part: the Bifferboard is x86 compatible, so it will run the standard JeeMon build for 32-bit Linux!

Screen shot 2010-03-15 at 12.16.03.png

There’s about 25 Mb of usable RAM (no swap space), which should be plenty to run just about anything in JeeMon with some other processes alongside it.

The one thing to keep in mind, is that this is a low-power board. It’s equivalent to a 486SX, i.e no hardware FPU, and it looks like it runs about 100 times slower than my (modern) Mac. But hey, you do get something in return: an very small unit and a power consumption of only 1.5 .. 2.5 watt!

Software- and hardware-I2C

In Software on Mar 15, 2010 at 00:01

Until now, the Ports library supported software I2C on the 4 “ports”, whereas hardware I2C required the Arduino’s Wire library. The unfortunate bit is that the API for these two libraries is slightly different (the Ports library uses a slightly more OO design).

See the difference in resulting code with and without Plug shield, for example.

Triggered by an idea on the forum, I decided to extend the Ports library, so that “port 0″ gets redirected to Analog 4 (PC4) and Analog 5 (PC5), i.e. the hardware SDA/SCL pins on an ATmega.

So now you can use the Plug Shield with the same code based on the PortI2C class as with JeeNodes. Simply specify port zero when using this with an Arduino and a Plug Shield!

Here’s the Arduino example again, which no longer requires the Wire library:

Screen shot 2010-03-13 at 19.27.28.png

Note the use of port 0 to connect to hardware I2C pins.

The benefit is that all plugs for which code exists based on the Ports library (Pressure, UART, LCD, etc) can now be used on an Arduino with a Plug Shield.

A nice extra is that this will also work on an Arduino Mega, without requiring two extra patch cables to hook up to the hardware I2C pins.

Long live simplicity!

Home automation, JeeMon

FritzBox call log

In Software on Mar 12, 2010 at 00:01

I’ve been using the FRITZ!Box 7170 as ADSL modem and telephone interface for some time now:

Screen shot 2010-03-08 at 12.50.41.png

It has a nifty feature: when you enable the built-in call monitor by typing in “#965“, you can connect to TCP/IP port 1012 to get real-time messages about all phone calls.

This is easy to hook up to JeeMon. I added a new “FritzBox.tcl” file with the following code:

Screen shot 2010-03-08 at 13.15.39.png

(Some experimental code was omitted from the above listing)

To enable this, the following line was added to application.tcl:

Here’s an example of an incoming call, as shown in the JeeMon log:

Screen shot 2010-03-09 at 13.51.12.png

And here’s an outgoing call:

Screen shot 2010-03-08 at 12.55.09.png

Neat! I’ll hook this up as events once the JeeMon event system is ready.

These experiments are very useful to see how the current design is working out. The above code depends on the “MessageStream” class in the “Serial” rig. It doesn’t really add much value in this case, but some class hierarchies seem to be emerging. This socket connection is an example of a similar-but-not-quite-the same case as serial connections, requiring its own class. A socket connection is very similar to a serial connection once initialized – but this doesn’t show as clearly in the code as I would like and there’s already a bit of code duplication.

Will probably need some re-shuffling later, to better match concepts / design and implementation.

JeeMon, X10

Outbound X10

In Software on Mar 11, 2010 at 00:01

I’ve added support for controlling X10 devices via the CTX15 module described in the past two days:

Screen shot 2010-03-09 at 20.31.20.png

First thing was to extend the ookRemote GUI, by adding these definitions to ookRemote.tcl:

Screen shot 2010-03-09 at 20.34.10.png

With this change, the configuration section in “config.txt” supports CTX15 buttons:

Screen shot 2010-03-09 at 20.34.56.png

These new buttons will send X10 commands to the CTX15 module. But this needs a bit of wrapping to turn them into frames, including the proper checksums. So I added two methods to “sketches/ctx15try/host.tcl”:

Screen shot 2010-03-09 at 20.37.16.png

The “send” method overrides the existing one and sets up a frame before calling the original method with it.

That’s it. I can now make the Marmitek AM12 switch “clunk” with my mouse – audible feedback! :)

JeeMon, X10

JeeMon interface for X10

In Software on Mar 10, 2010 at 00:01

Now that the CTX15 has been connected via a JeeNode, it’s fairly straightforward to tie it into JeeMon.

First I created a new folder called “sketches/ctx15try/” with a file “host.tcl” in it, using this code as quick test:

Screen shot 2010-03-08 at 14.37.36.png

The reason there is nothing more, is that I changed the ctx15try.pde sketch a little to return lines starting with “CTX15″ – since these are the easiest to tie into JeeMon. The change affects these lines in yesterday’s demo:

Screen shot 2010-03-08 at 14.33.08.png

I’ve also wrapped the received frame in {}’s, because that simplifies processing in JeeMon (Tcl interprets $’s and []‘s unless escaped).

To activate everything, I added an extra line to the config.txt configuration file, so that the output from the JeeNode USB (serial# A900adwo) automatically gets picked up as sketch:

Screen shot 2010-03-08 at 15.01.22.png

Here is an example of the above code running:

Screen shot 2010-03-08 at 14.36.50.png

Good. CTX15 events are coming in. They are easily recognizable as (unit#, command) pairs.

But as you can see, these pairs don’t always arrive in the same frame. So we have to add a bit of logic to get things right. Here’s an updated version of host.tcl:

Screen shot 2010-03-08 at 14.51.50.png

The solution used above is as follows: if it looks like a unit code (A..P + 1..16), then save that as last one seen. If it’s anything else, use the last saved location. I’m not sure this covers all possible scenarios, but for this very simple test it seems to work. Here’s a screen shot of the real-time status window:

Screen shot 2010-03-08 at 14.50.48.png

As you can see, the CTX15 event was turned into an “AOFF” command for the “ctx_A01″ device, which was created on-the-fly by this code.

Hmm, looking further, I think the “A” prefix needs to be stripped from “AOFF”. Oh well, it’s all scripted, I’ll adjust that when I get to using this stuff for real.

So that’s it. A few dozen lines of C code in the JeeNode to act as pass-through and do the polling, and a few dozen lines of Tcl code to tell JeeMon how to interpret the incoming data. Now we get notifications whenever any X10 activity is detected on the mains power line.

Onwards!

Home automation, JeeNode, X10

Xanura CTX15

In Hardware, Software on Mar 9, 2010 at 00:01

Another popular home automation module is the Xanura CTX15:

That’s a live 220V power-line connection in the top right corner!

I’ve connected the CTX15 module through a UART Plug, with a 4.7 kΩ resistor in series with the RX signal (yellow from CTX15). This is because the signal swings up to 5V, whereas the UART plug only accepts 3.3V voltage levels. The module is powered from the PWR pin, which on a JeeNode USB carries 5V.

The CTX15 is a bi-directional interface, it can send as well as receive A10-type power-line commands (a superset of X10). The trouble is that it needs to be polled to read out what has been received and buffered so far.

Here’s a sketch which takes care of that:

Screen shot 2010-03-08 at 12.08.53.png

And here’s some sample output:

Screen shot 2010-03-08 at 12.08.40.png

As a test, I powered up yesterday’s XM10E test setup as well, to send out on and off commands to unit A.1 every 3 seconds. As you can see with the CTX being read out every 5 seconds, multiple received packets will sometimes be combined and returned as one reply.

JeeMon goes embedded

In Hardware, Software on Mar 5, 2010 at 00:01

Testing out a new setup … I started by launching the JeeMon runtime executable as follows:

Then I went back to my Mac and opened a browser window:

Screen shot 2010-03-02 at 15.56.12.png

It works! You’re looking at a little web server running on a tiny Technologic TS-7500 board:

ARM based, 64 Mb RAM, µSD card socket on board, running Debian (lenny). It’s not very high powered, but in return it only uses 400 mA @ 5V, i.e. it can run off USB power.

I haven’t yet figured out how to get the FTDI USB serial driver on there, but once that’s solved this little unit will be able to act as host for USB-connected JeeNodes or JeeLinks.

The RAM and storage is more than sufficient to run even a very elaborate JeeMon setup, I expect.

This might be an interesting low-end always-on home sensor & automation system!

Pushing data as web client

In Software on Mar 3, 2010 at 00:01

Another way to get data to another place is via push HTTP requests, i.e. acting as a client for another web server somewhere on internet.

The Pachube site is one place to send measurement results. I’ve set up a test feed to use with JeeMon, with two data streams. One of the nice features is that you can easily produce and embed graphs from that system:

Screen shot 2010-02-26 at 02.45.13.png

Here’s how I implemented it in JeeMon. First, I created a configuration section:

Screen shot 2010-02-25 at 22.37.12.png

(I’ve omitted most of my private API key, but anyone can sign up and get their own …)

Next, the code, in the form of a “pachube.tcl” file / module / rig:

Screen shot 2010-03-02 at 03.11.12.png

Note how the configuration file in fact contains “settings” which are simply evaluated as a script: the two lines starting with “Fetch …” are handled by the Fetch proc in pachube.tcl, using the JeeMon notification mechanism to get called back whenever either of the “meter1″ or “meter2″ readings changes. The Update proc then updates the “latest” variable accordingly. Lastly, SendToWebSite periodically does an HTTP PUT request in the background, supplying all the info needed to successfully submit to the Pachube website.

Btw, this simple example is flawed, in that it does not calculate averages – it just sends the last reading. But things like averaging require a sense of history, and persistence. Haven’t added that to JeeMon yet…

The missing link, as usual, is a line in the application file to start the ball rolling:

There’s a lot of flexibility at the protocol and network levels. Such as creating an XML request via templates, if the remote server needs XML. This isn’t limited to HTTP requests, or to using port 80 – send emails, FTP files, etc.

There are quite a few details in the above code which I won’t go into. Again, I’m doing this mostly to show how little code it takes to initiate periodic HTTP client requests to an outside website.

Apart from sites such as Pachube, this could also be used from a tiny embedded Linux running JeeMon, to submit incoming data to a different setup elsewhere on the LAN, for example. IOW, JeeMon can be a front-end for other systems. It’s not about lock-in. It’s a switchboard. It can glue systems together. It bridges gaps.

JeeMon as web server

In Software on Mar 2, 2010 at 00:01

JeeMon comes with a nifty built-in HTTP/1.1 web server (using coroutines to handle requests in parallel).

So let’s create another real-time status display, but as web application this time:

Screen shot 2010-02-25 at 17.56.25.png

As before, I’m leaving out the lipstick – just showing the core functionality.

Here’s what I did to generate this page. First of all, I’m re-using some functionality which is already in the GUI version, to dynamically construct a matrix with the proper columns and rows, so the “statusWindow” code has to be running for the above to work. For real use, that code should probably be reorganized into its own rig.

The main task was to create a HTML template in a file called “statusPage.tmpl”:

Screen shot 2010-02-26 at 00.41.23.png

This uses a simple templating mechanism based on Tcl’s “subst” command. The meta tag sets up the page to self-refresh every 5 seconds. The last line tells the Wibble web server to deliver the page as HTML.

To launch the server, this line was added to the main application file (port 8080, current dir is doc root):

That’s it – there’s very little to activating a web server in JeeMon, as you can see. Which is important, because on tiny embedded Linux systems, an HTTP server will probably be the only option to present information.

Creating a full-blown site with CSS, JavaScript, and Ajax is a matter of adding more files – the usual stuff…

Did I mention that it’s all 100% open source, so you can browse / extend / change all of this? – I did? Oh, ok ;)

Node discovery

In Software on Mar 1, 2010 at 00:01

This post isn’t about GUIs, but about the dynamic behavior behind them.

And about wireless. Let’s take the voltmeter setup I described recently, and make it send its readings over the air – it’s a JeeNode after all, so it has wireless connectivity built in.

I extended the original sketch to send readings out roughly once per second:

Screen shot 2010-02-25 at 15.57.03.png

Extending the host side to also work from received packets is trivial:

Screen shot 2010-02-25 at 16.49.26.png

The problem now, is that this node will start sending out packets, but JeeMon has no idea what to do with them. One day, automatic node type registration would be great to add, but for now I’ll just use the configuration file to associate nodes with sketches. Here’s an extract of my current “config.txt” file:

Screen shot 2010-02-25 at 16.00.30.png

For this example, I added the “3/ { type analogPlug }” line to the “868:5/” entry (i.e. 868 MHz, netgroup 5). And sure enough, JeeMon will now launch the GUI for this node, displaying results coming in over the air in real-time:

Screen shot 2010-02-25 at 16.03.00.png

Here’s the relevant part of the log output:

The fun part is that a timeout mechanism has also been implemented. If no packets come in from the analog node for 5 seconds, the connection is torn down and the window is closed again. The timeout value is sketch-specific and is included in the above configuration info.

In other words: node is ON => window appears, node is OFF => window goes away. Look ma, no hands!

Note: it’s not good UI practice to open and close windows like this. In a more refined setup, these windows would be panels in a larger window, or they could just be grayed-out to signal their disconnected state, for example.

The analog node is surprisingly convenient this way. I can use it anywhere, turn it on, look on the screen, and turn it off when I’m done. It wouldn’t be hard to add another wireless node to show the results on an LCD screen, btw. If this is extended with a way to assign node IDs ad-hoc, and a way for a sketch to identify itself over the air, then a whole family of wireless “instruments” could be created – using nothing but a JeeNode and some plugs.

This illustrates a guiding principle in JeeMon: don’t ask, just do it! – JeeMon tries to be as dynamic as possible, adapting to changes in status and configuration by adjusting everything it does accordingly. In the case of nodes, new ones are discovered and stale ones are weeded out, in real-time. With objects getting created and destroyed automatically. What these objects do, as side-effect, is up to each one of them,

Right now, this dynamism in JeeMon is still fairly messy. There’s too much state in too many places which needs to be managed carefully. But with a bit of luck, this can eventually be simplified further to create a very consistent and predictable behavior.

A real-time status GUI

In Software on Feb 28, 2010 at 00:01

One more post about setting up a GUI with JeeMon…

I wanted to have a basic real-time display of all the readings coming in, sorted by source. Like this:

Screen shot 2010-02-25 at 15.48.08.png

(the size of this screen shot was reduced to fit this post)

Everything in this window is dynamic: rows get added when new sources start reporting their data, and columns get added when new parameter types are reported. I’m not storing any information persistently yet, which is why this status window looks as follows right after starting up JeeMon:

Screen shot 2010-02-23 at 21.31.15.png

Not very exciting, but all you have to do is wait …

Let’s see what it took to create this GUI. First, I added a generic “Notifier” rig to JeeMon, and a “Reading” method for all sketches to use. The rooms sketch, for example, now reports its incoming results as follows:

Screen shot 2010-02-23 at 22.06.04.png

Similar changes were made to the ookRelay and RF12demo “host.tcl” sources.

The rest of the GUI code is in “statusWindow.tcl”:

Screen shot 2010-02-24 at 01.16.42.png

The tabular display is based on a very nice Tcl/Tk open source package called Tablelist by Csaba Nemethi, which has been added to the JeeMon core library.

The coupling with the rest of the system is accomplished by the “Notifier attach readings * …” call at the end of the setup code. This registers a callback which will be invoked when anything in the “readings” notification group changes. Notifiers are going to be used a lot in JeeMon because they provide such a manageable way to glue independent parts of an application together.

And lastly, adding the following line to the main application starts the ball rolling:

There’s quite a bit going on here which I won’t go into. I just wanted to illustrate how little code it takes to create a basic, fully dynamic, cross-platform, self-updating status display window. There – how’s that for adjectives!

JeeMon, Sensors

Voltmeter with GUI

In Software on Feb 27, 2010 at 00:01

Let’s make a 5-digit voltmeter with a JeeNode, an Analog plug, and JeeMon. But instead of a readout on an LCD screen, I’m going to show the results on the desktop. To start with that last part, here’s my little display on a Mac:

Screen shot 2010-02-23 at 20.23.25.png

You’re looking at my feeble attempt to create a good-looking / souped-up GUI, btw :)

Anyway. Here’s the sketch I’m running on the JeeNode, with an Analog plug inserted into port 3:

Screen shot 2010-02-23 at 11.01.16.png

Trivial stuff. It sends out lines with “VOLT <value>” readings twice per second.

The important part is that this sketch starts up with “[analogPlug]” as greeting. To hook up to this sketch, all we need to do is create a directory called “analogPlug”, with a file called “host.tcl” in it.

This new “analogPlug” directory is a sub-directory of “JeeMon-sketches”. This directory, in turn, has been registered as being used for sketches – with this command in my main application code:

Back to the “JeeMon-sketches/analogPlug/host.tcl” file. Here’s its contents:

Screen shot 2010-02-23 at 11.40.34.png

That’s the whole kaboodle.

Now, whenever I plug in a JeeNode running the above “analogPlug.pde” sketch, a window pops up showing readings in real-time. When I unplug the JeeNode, the window is closed again. It’s all automatic, as long as JeeMon is kept running.

This mechanism is not limited to JeeNodes or Analog plugs, of course.

OOK remote control

In Software on Feb 26, 2010 at 00:01

The JeeMon story continues. Here’s a fun little panel for controlling some switches around the house:

Screen shot 2010-02-22 at 21.42.14.png Screen shot 2010-02-22 at 23.31.35.png

Screen shot 2010-02-23 at 20.19.28.png Screen shot 2010-02-22 at 21.51.23.png

These screen shots show Windows (7 and 2K), Mac OS X, and Linux Ubuntu, respectively – long live diversity!

This GUI was created with these configuration settings:

Screen shot 2010-02-26 at 02.09.33.png

Those settings were loaded by adding this line to the main application code:

Which in turn loaded the following “ookRemote.tcl” source file I wrote, as module / rig:

Screen shot 2010-02-26 at 02.08.48.png

That’s the entire app. Adding lipstick (i.e. a fancier visual design) is left as exercise for the reader.

Note how the design is split in a configuration section and a source code file – I’m probably going to use this approach often in JeeMon. The idea is that people who don’t care about the code behind all this stuff can just adjust the configuration file (using a visual interface, hopefully, one day). While those who like to tinker can view the full source code, and explore and extend it at will, and more importantly: ad infinitum.

It’s called open source for a reason!

PS. There’s another major reason for the split between configuration settings and source code files: turnkey deployment. You can put all the source code files in a ZIP archive called “JeeMon-rigs.zip”, so that all you need for a new installation with JeeMon, is: 1) your archive, 2) the proper JeeMon runtime, and 3) a config file to match the target setup. The config file is optional, btw – more on that another time…

Output and Expander plug fixes

In Software on Feb 25, 2010 at 00:01

While testing an Output Plug, I found a little mess-up: the documentation pinout is wrong, all even and odd I/O pins on the 2×6 output connector were swapped. Here is the correct pinout:

I’ve updated the documentation.

The “expander” sketch is also very confusing, it was still for the PCA8574A, whereas the current Expander Plug uses an MCP23008.

Here’s an improved sketch, with a little running-light demo:

Screen shot 2010-02-22 at 12.38.18.png

This code has been checked in to replace the “expander.pde” sketch in the Ports examples.

Secure transmissions

In Software on Feb 23, 2010 at 00:01

For some time, I’ve been thinking about adding optional encryption to the RF12 wireless driver. I’ll leave it to the cryptography experts to give hard guarantees about the resulting security of such a system…

The basic goal is to provide a mechanism which lets me get messages across with a high level of confidence that an outsider cannot successfully do the same.

The main weakness which most home automation systems such as FS20 and KAKU completely fail to address is message replay. The “house code” used by FS20 has 16 bits and the address has 8 bits, with the naive conclusion being that it takes millions of attempts to send out messages to which my devices respond. Unfortunately, that’s a huge fallacy: all you have to do is sit outside the house for a while, eaves-dropping on the radio packets, and once you’ve got them, you’ve figured out the house code(s) and adresses in use…

I don’t care too much about people picking up signals which turn the lights on or close the curtains. You don’t need technology to see those events from outside the house anyway. I do care about controlling more important things, such as a server or a door opener.

Here are my design choices for optional encryption in the RF12 driver:

The cipher used is David Wheeler’s XXTEA, which takes under 2 Kb of code.
The keys are 128 bits, they have to be stored in EEPROM on all nodes involved.
All nodes in the same net group will use either no encryption or a shared encryption key.
A sequence number of 6, 14, 22, or 30 bits is added to each packet.

To start with the latter: XXTEA requires padding of packets to a multiple of 4 bytes. What I’ve done is add the sequence number at the end, using as many bytes as needed to pad to the proper length, with 2 bits to indicate the sequence number size. Encrypted packets must be 4..62 bytes long. It’s up to the sender to decide what size packets to send out, and implicitly how many bits of the sequence number to include. Each new transmission bumps the sequence number.

To enable encryption, call the new rf12_encrypt() function with a pointer to the 16-byte key (in EEPROM):

Screen shot 2010-02-21 at 18.37.36.png

Encryption will then be transparently applied to both sending and receiving sides. This mechanism also works in combination with the easy transmission functions. To disable encryption, pass a null pointer instead.

The received sequence number is available as a new “rf12_seq” global variable. It is up to the receiver (or in the case of acks: the originator) to ascertain that the sequence number is in the proper range. Bogus transmissions will decrypt to an inappropriate sequence number. To make absolutely certain that the packet is from a trusted source, include some known / fixed bytes – these will only be correct if the proper encryption key was used.

This new functionality has been implemented in such a way that the extra code is only included in your sketch if you actually have a call to rf12_encrypt(). Without it, the RF12 driver still adds less than 3 Kb overhead.

I’ve added two sample sketches called “crypSend” and “crypRecv” to the RF12 library. The test code sends packets with 4..14 bytes of data, containing “ABC 0123456789″ (truncated to the proper length). The receiving end alternates between receiving in encrypted mode for 10 packets, then plaintext for another 10, etc:

Screen shot 2010-02-21 at 22.42.27.png

As expected, the encrypted packets look like gibberish and are always padded to multiples of 4 bytes. Note also that the received sequence number is only 6 bits on every 4th packet, when the packet size allows for only one byte padding. The strongest protection against replay attacks will be obtained by sending packets which are precisely a multiple of 4 bytes (with a 30-bit sequence number included in the 4 bytes added for padding).

So this should provide a fair amount of protection for scenarios that need it. Onwards!

Declarative configuration

In Software on Feb 22, 2010 at 00:01

Things are starting to come together quite nicely with JeeMon – all in less than 1000 lines of Tcl code so far. That’s not to say that it’s perfect yet – I’m still massively rearranging some of these code structures – but there is already some functionality to play with, which helps expose the strengths and weaknesses so far.

The basic approach is declarative in nature: specifying what should happen, but not necessarily in the same place as when or how all the work should be done.

Here’s my first tentative configuration file:

Screen shot 2010-02-21 at 14.31.44.png

For each serial interface, an action is specified when that interface comes on-line. For example I can now plug in my A8007Up6 device (a USB-BUB with a JeeNode) and JeeMon will open a terminal window showing the output stream. Unplugging closes the window again (and automatically cleans everything up).

Right now, I’m editing this config file by hand, but at some point that could be augmented with a set of GUI or web-based configuration panels (even remotely).

There are a lot of challenges ahead. One of them is decoupling things so that code and data end up organized in the most convenient, logical, and flexible way. I’m trying very hard to avoid any calls to the “Config” module from generic JeeMon code. This allows you to set up the configuration file structure and hierarchy in any way you like.

My application file currently looks as follows:

Screen shot 2010-02-21 at 15.42.58.png

As you can see, I’m setting up a periodic port scanner and supplying a “SerialEvent” callback to decide what to do when serial ports get added or removed. In those decisions, most of the logic is driven from the “config.txt” configuration file. I’m about to implement a similar callback for radio events, i.e. RF12 nodes coming online and dropping off again.

In a way, the above two files are the entire application.

The rest is utility code to make the above convenient and concise. That rest might well be 99% of the code, but it’s all structured as loosely-coupled “rigs” (modules), which can be used / overridden / ignored at will.

The way I see it, there can be three sources for such utility code: your own code, plug-ins from others, and a couple of rigs included in the JeeMon core itself. The above two files require just the core, using the built-in rigs (Config, Gui, JeeSketch, Log, Serial, and SysDep).

Serial USB devices

In Software on Feb 21, 2010 at 00:01

For JeeMon, I’d like to be able to auto-detect USB device insertions and to identify FTDI serial numbers (with names like “FTE54GKC” and “A9007CNE”).

Cross-platform… i.e. on Windows, Mac, and Linux!

It looks like it can be done.

On Windows, we can browse the good ol’ registry, as follows:

Screen shot 2010-02-20 at 15.24.16.png

Sample output:

This was tested on Win2000 and Windows 7, in the hope that everything in between works the same.

On Mac OS X, it’s easiest (for a change):

Screen shot 2010-02-20 at 15.26.02.png

Sample output:

On Linux, it’s a bit messy because there are so many different distributions:

Screen shot 2010-02-20 at 15.27.43.png

Sample output:

Tested on Debian 5 (Lenny), Ubuntu 9.10 (Karmic), and Gentoo.

The above code can be found in the subversion repository, i.e. here.

I’m testing this all at once on a single machine btw, courtesy of VMware Fusion. And using JeeMon’s built-in self-update mechanism to quickly get new versions across while debugging and tweaking things.

By calling “SysDep listSerialPorts” periodically, we can automatically detect a change and see which plug was inserted (FTDI only for now). Without depending on any external libs or executables.

Onwards!

RF transport independence

In Software on Feb 20, 2010 at 00:01

With the basic serial interface access and dispatch in place in JeeMon, it’s time to move on to JeeNode / JeeLink packet processing.

What I want is to be able to forget all about how readings got to JeeMon. It shouldn’t make a difference whether I’m using a directly connected Room Node and grabbing the readings off its serial USB connection, or whether they came in through the air via the RF12 wireless driver – or by any other means. Take a snapshot with your cell phone, send the picture to EverNote, have it OCR’d, and then grab the readings off the web … whatever!

The way I’ve tied this into JeeMon, is to let the interface to RF12DEMO act as de-multiplexer. This is purely a decision at the RF12DEMO listener level. Each incoming packet is examined to determine which node it came from. Then we need a way to map from nodes to listener class – i.e. find out what sketch is running on the remote node. This is hard-coded for now:

Screen shot 2010-02-18 at 22.57.41.png

What this does is actually a bit more elaborate: a RF12DEMO listener will set up and manage a bunch of extra “RF12_PacketStream” listeners, one for each node. When packets come in, they will simply be dispatched to the corresponding stream. Each packet stream can process its packets in a different way.

The fun part is that these packet streams can use the same listener classes as with direct serial interfaces. All they need is an extra “decode_RF12″ method:

Screen shot 2010-02-18 at 23.02.04.png

The task of decode_RF12 is to re-cast the incoming packet as messages of the same structure as what would come in over a serial connection.

Here’s the “rooms” listener as example:

Screen shot 2010-02-18 at 23.04.34.png

This one class encapsulates all the protocol details of room nodes, both serial and via RF12. When a 4-byte data packet comes in via RF12 (as $a..$d), the bits are decoded and an “onMessage” call is generated with a “ROOM” message id and the 5 decoded readings.

Here is a log of this code in action, one message per line:

Screen shot 2010-02-18 at 22.37.15.png

The way to read these messages is as key-value pairs, i.e. id = OK, type = RF12, name = usb-A900ad5m, etc.

The first two lines show an incoming OK message from node 21 (53=21+32), which is then turned into a ROOM message, tagged as coming from the “rf12-868.5.23″ packet listener.

The next 3 lines are more involved: first an EM10 message came in over USB, then an OK message came in which got dispatched again, as the same EM10 message. That’s because I’m running JeeMon with a direct connection to the ookRelay board, even though it transmits all its information over RF12. So everything from the ookRelay comes in twice (great for debugging).

The point is that the two EM10 messages have the same info. It no longer matters how the message got here (but it is traceable, using the remaining details). And all the code to accomplish this is in a single source file, right next to the sketch running on the ookRelay board.

This design makes it possible to develop an application using only the serial USB connection, and then later add logic to send the information via RF12 (or not). Infrared, XBee, Twitter, anything: transport independence!

Note that nothing is done with these decoded messages at this stage. This messaging framework is independent of higher-level application decisions, such as where to store / send / show msgs, or even when to process them.

Serial port encapsulation

In Software on Feb 19, 2010 at 00:01

This post continues to look a bit into the new JeeMon design.

Let’s focus on serial interfaces first, mostly USB. There’s a “Serial” module which does all the basics. On the Mac, if I want to open device /dev/tty.usbserial-A900ad5l, then the following call with do everything:

Serial connect usb-A900ad5l 57600

The name of the device would be different on Windows and Linux (COM5, or USB1), but that’s all.

By default, this creates a new Serial object, which logs all incoming text to stdout. To send a command out, we need to keep a handle to this object:

set conn [Serial connect usb-A900ad5l 57600]
$conn send "some text"

Nice, but not very exciting…

Let’s take it one step further. The “JeeSketch” module does the automation described in the previous post, i.e. detect the running sketch, associate it with a class, instantiate an object for it, and call the methods of that object whenever text lines come in over that serial port. Here’s a complete JeeMon custom “application.tcl” program:

Screen shot 2010-02-18 at 14.44.59.png

First, all the sketch drivers are made available with one or more “register” calls. This lets the appropriate classes take over for each new serial connection – depending on what sketch is running. That’s all it takes. Servicing such serial ports now becomes an event-driven background activity.

The listeners are defined in separate files, one for each type of sketch:

Screen shot 2010-02-18 at 15.05.58.png

The ookRelay/host.tcl file looks like this, for example:

Screen shot 2010-02-18 at 14.51.58.png

This structure makes it easy to manage stuff that belongs together. Projects can be exchanged (or archived, or revision-controlled), with all the pieces needed to use them in one place. And as far as I’m concerned, it won’t be limited to JeeNodes etc, or Tcl, or a specific platform. This has to remain general enough to hook up to any hardware and use with any language (via networking, files, and direct launching of executables/scripts). My goal for JeeMon is not to limit anyone’s options, but to create a simple switchboard between whatever is needed.

(I’m still mucking around with the organization and naming of code and files, as you can see)

Nice, but still not very practical…

The problem with the above is that it doesn’t deal with devices getting plugged in or unplugged. Well, unplugging is the easy bit – the above code will automatically clean up after itself on connection loss, so that part is covered.

Wouldn’t it be nice if we could just plug in new devices and get them to automatically start doing something?

I implemented such a mechanism in a recent revision of the code, but I’m hesitant to add it again – because it was Mac OS X specific, where USB devices connected via the FTDI driver include a unique code. On Windows and Linux, you just get COM<n> and USB<n> devices, where “n” seems to be related to the order and number of device insertions.

I haven’t looked into “libusb” yet. Should I? Will it help for Windows too? Do I need to start learning about USB enumeration? What OSS-compatible tools and libs are there?

Update – on Linux, it looks like /sys/bus/usb/devices/* has all the info needed to identify USB devices. So that only leaves Windows – good: at most one lib or dll to deal with.

▶ View 9 Comments

In the kitchen

In Software on Feb 18, 2010 at 00:01

Ok, so maybe it’s time to start describing some of the new stuff cooking in the kitchen lab.

I’ve been exploring software designs to use as basis for JeeMon, that new switchboard-type application for physical computing devices. The idea is to treat the system as one big message-passing machine – a bit like Message-Oriented-Middleware (MOM), which has been around for ages, but without getting sucked into any heavy-weight conventions.

Messages can be passed around, queued, stored, duplicated, filtered, transmitted, returned, ignored, sorted, etc. After all, living organisms do nothing but send chemical messages around, so why not do the same for an infra-structure focused on physical computing (sensors, displays, actuators) and home automation?

If you’ve been following this weblog for a while, you’ll have noticed that almost all the output I generate in sketches is of the form “identifier arg1 arg2 …”. So for example, packets received by RF12demo look like:

OK 61 9 2 8 79 243 87 13 0 15 0

A barometric reading from the BMP085 sensor on the pressure plug may get reported as:

BMP 233 10019

And so on. One or more upper case letters, optionally followed by digits. Then the payload (which may also include floating point and string values, not just ints).

Another convention I’ve been sticking with is to report the name of the sketch on the serial port during startup:

[ookRelay]

Or an identifier plus the current configuration settings:

[RF12DEMO] W i23 g5 @ 868 MHz

These two conventions can be used for an object-oriented design. With a bit of preparation, the name of such sketches can be automatically associated with a class, and each line treated as a method call.

Here’s the skeleton of the first level of code I use for decoding RF12demo.pde output:

Screen shot 2010-02-16 at 00.15.27.png

Another class definition, for the ookRelay.pde sketch:

Screen shot 2010-02-16 at 00.16.11.png

Or to put it differently: with JeeMon running, you can hook up a device (JeeLink, JeeNode, Arduino, etc) containing some sketch and the matching class will automatically be associated with that serial port, once the sketch has been identified. A new object is then created, and its methods will be called whenever the device sends new output (messages?) over the serial line.

Simple!

What I would like to do, is manage the sketch (C/C++) and the class (Tcl) together, perhaps as two files in a common development directory for that project. That way the interface between the two pieces of hardware essentially vanishes into the background. The point being that each class can then do all sorts of things, such as storing results in a database, sending it to another system over the network, updating web server pages, popping up a GUI window to show incoming data in real-time, etc.

This mechanism is very simple, even under the hood. This matters, because it has to work even when JeeMon is running on low-end embedded Linux hardware. But at the same time, such a MOM + OO design will allow considerable flexibility for abstract / high-end use later.

PS. If you’re familiar with Tcl, you might be surprised to see all this “oo” stuff in here. That’s because JeeMon uses Tcl 8.6 with built-in object system. Multiple inheritance, mixins, filters, dynamic class change support, delegation, prototypes / per-object methods, it’s all in there. I’m also using ensembles and there’s an interesting coroutine-based web server waiting in the wings (called wibble).

For the record: nothing ever gets added just to be buzz-word compliant. If a feature simplifies application-level concepts and leads to a cleaner implementation in JeeMon, it’ll be used, else I’ll pass. Life’s too short to jump on bandwagons.

CNC, JeeNode

Wireless works, sort of…

In Software on Feb 10, 2010 at 00:01

Ah, now we’re getting somewhere!

This is my current test setup:

Screen shot 2010-02-07 at 21.23.55.png

The JeeNode on the printer side implements a packet pass-through system, receiving command packets from the JeeLink and sending back response packets. Here is the sketch which does all the work:

Screen shot 2010-02-08 at 01.02.16.png

On the Mac/PC side, some Tcl code was added to go through the RF12demo text-mode protocol to send and receive arbitrary data, using the “a” command. Still work in progress, but the basic transport encapsulation works.

The tricky part is timing … it always is with this sort of real-time control stuff. Unfortunately, the current G3 software on the CupCake isn’t quite as responsive as defined in the specs. Some responses take way over 80 milliseconds to come back from the motherboard. This is the case when scanning the SD card, as well as when stopping the extruder motor.

So what this sketch does is wait up to 500 ms for a reply to come in. Even if there isn’t one, an acknowledgement packet will be sent back. The new code on the Mac in turn waits up to 1 second for that ack to come back.

If no ack came back, then there was an error in the wireless connection (this can happen either during the request or during the ack, there is no way to tell!). Probably best thing to do would be to resend the command.

If an empty ack came back, then the response packet did not arrive within 500 ms. In this case, we could send an empty command and wait for its ack. This hasn’t been implemented yet, but it will allow dealing with even the slowest responses, simply by polling a few more times with an “empty command”.

But hey – it works, and the output is the same as before:

Screen shot 2010-02-07 at 21.26.39.png

This is probably the first wirelessly controllable CupCake in the world :)

What I should mention though is that this doesn’t yet work reliably due to those very loose timing behaviors and the fact that packet errors are not yet dealt with. Test runs fail occasionally – mostly in the SD card access code, i.e. while grabbing all the filenames with NEXT_FILENAME.

CNC, JeeNode

Connecting to a CupCake

In Software on Feb 9, 2010 at 00:01

To follow up on yesterday’s post, I wrote some test code to request the extruder nozzle temperature from the JeeCake and send the results out over wireless. Here is the full “jeeStatus.pde” sketch:

Screen shot 2010-02-07 at 16.19.43.png

And sure enough, it works:

Screen shot 2010-02-07 at 16.08.55.png

You’re seeing the nozzle cool down, after heating it up via ReplicatorG. The connected JeeNode has been given node ID 19, and it’s transmitting in group 5 of the 868 MHz band, so I can simply track these incoming packets through the JeeLink which is already collecting all sorts of data anyway.

So much for the easy part – the real software will be more work!

New date / time / RTC library

In AVR, Software on Feb 5, 2010 at 00:01

Not so long ago, I had the opportunity to work a bit on something which has bugged me for a long time – the lack of date and time handling in connection with RTC chips. There are a few libraries out there, but I think I could do better – i.e. make it simpler, smaller, yet sufficiently powerful for real day-to-day use.

Seeing where this was going on the Arduino developer mailing list (and disagreeing with just about everything that happens over there), I decided to put my money time where my mouth is, and build my own library.

Here’s the header file of the new RTClib Arduino-compatible library:

Screen shot 2010-02-04 at 13.52.13.png

This lets you do date / time calculations, and it provides two different ways to implement a clock: via a hardware chip or using the built-in millis() timer.

RTClib has been checked into subversion, see the CODE page for details on how to get it.

It includes four example sketches:

datecalc illustrates how to do calculate with dates and times
ds1307 interfaces with a DS1307 RTC chip, connected via the Wire library
plugrtc interfaces with the RTC Plug, connected via the Ports library
softrtc demonstrates how to do the same with just software

One fun trick I added, inspired by a comment from Limor Fried, is to allow initializing a DateTime object from the DATE and TIME strings generated by the C compiler. That means you can run that “softrtc” sketch without hardware support, and it’ll automatically have its clock set to the compilation date of the sketch, i.e. fairly close to correct. Not good enough for general use, but great during quick debug cycles when you’re re-compiling your sketch all the time anyway.

Note that to use RTClib, you need to include the “Wire.h” library – even if you don’t use it!

The inability to properly deal with libraries, particularly in a resource-constrained embedded processor context, is one of the aspects of the current Arduino direction which irritates me – see an older post for more details.

Arduino, JeeNode

Fascinating concurrency

In AVR, Software on Feb 4, 2010 at 00:01

There is a new language for the Arduino / JeeNode / ATmega328, called Occam-π.

I found out about it yesterday, at http://concurrency.cc/ – it’s high level, and it supports parallel programming. The current development environment release is for Mac OS X, with Windows and Linux coming soon.

Here is a complete program with 4 blinking LED’s, one on each DIO pin of the JeeNode ports:

Screen shot 2010-02-03 at 01.13.19.png

That’s it. Compiles to roughly 2 Kb. Each extra blink adds just 20 bytes, btw.

And yes, it really makes four LEDs blink at an independent rate:

There is slightly more to it than that, but this is mind-blowing stuff. The “parallelism” is simulated, of course. Looks like the ATmega can do around 6000 context switches per second (i.e. parallel task switches).

There is a roughly 20 Kb interpreter part that needs to be uploaded once (which is why this requires at least an ATmega328). After that, the IDE will upload just the bytecode for your program, i.e. 2 Kb in the above case.

B R I L L I A N T .

Imagine hooking up the RF12 driver to this – there’s plenty of room for the extra 3 Kb or so. And for doing all sorts of things… in parallel! My earlier complaint post about how awful it is to do several things at once on an ATmega board might just have been wiped off the table.

Looks like I’ve got some very serious learning ahead of me to try and get to grips with all this.

▶ View 21 Comments

JeeMon, Sensors

Temperature plots

In Software on Jan 22, 2010 at 00:01

With over a dozen nodes now active, JeeMon is finally starting to collect a bit more useful data to work with:

Screen shot 2010-01-22 at 12.03.23.png

Looks like some temperatures have a granularity of 1°C – might be a few cheap DS1820 units I had…

The Mason-like templating code in “try-temps.html” to generate this plot is still far from obvious:

Screen shot 2010-01-20 at 01.52.42.png

Once I get to spend some real time on all this, it’ll look totally different. JeeMon as a whole is going to change completely in fact – until then, you can find some technical background information in an older post.

Let’s just say that it gets the job done right now…

JeeNode, LowPower

Sleep mode fix

In AVR, Software on Jan 20, 2010 at 00:01

The Rooms sketch (latest code here) had problems with the sleep mode added about a month ago:

Screen shot 2010-01-17 at 14.05.55.png

The first value is the current consumption in µA.

For some reason or other, the node would stop working and enter a permanent-on mode, drawing over 7 mA and draining the battery in a matter of days. Not good.

It seems to be related to the way the power down mode was implemented. To get absolutely rock bottom power draw, I was using the RF12′s watchdog timer. The ATmega watchdog time draws slightly more current and isn’t quite as configurable.

I’ve now reverted to using the ATmega watchdog anyway. Here is the modified logic:

Screen shot 2010-01-18 at 12.09.39.png

The watchdog is set to interrupt every 16 milliseconds, constantly. When the node is powered down, this will wake it up again. What the new loseSomeTime() code does, is simply to power down a couple of times, until the target delay time has been reached. There is some inaccuracy in these timings since the watchdog timer is free-running, but this should not matter too much when waiting for a second or so.

The new code has been running fine for over a day on six nodes. Here’s a sample from my power tracker:

Screen shot 2010-01-19 at 16.56.22.png

That’s 110 µA right now, 265 µA in the last minute, and 230 µA in the last hour.

At around 250 µA average, the power consumption is a bit higher than before, but my main concern is first to get the nodes running reliably. Even @ 250 µA, the AA batteries should last several months.

The average power down current draw is 110 µA/sec, 40 µA of which is due to the PIR sensor. The total current draw while transmitting is around 29 mA – during reception it’s about half, i.e. 14 mA. Still, due to the very brief on-times, this current consumption averages to only about 400 µA during 1 second for a normal send + ack sequence. Under optimal RF conditions, the long-term average consumption of a node will be under 150 µA. I’m confident that further optimizations could reduce this to well under 100 µA.

But there is one known flaw, which can be observed to happen once in a while: the nodes always wait for an ack in full-power mode, i.e. with the receiver on. This will normally be within milliseconds, but if the connection is flakey or if the central node is unresponsive, then nodes can spend a great deal of time in full-power mode. This needs to be fixed one day.

I’ve started re-flashing all the room nodes and replacing their dead batteries, here at Jee Labs – let’s see how it goes this time around.

More C++ template trials

In AVR, Software on Jan 18, 2010 at 00:01

Here are some more results in trying to use templates for embedded software on JeeNodes. This time, it’s about using Jee Plugs with bit-banged I2C on one of the 4 ports and with built-in TWI hardware on pins A4/A5.

Let’s start with an extract of the Latest JeeLib.h header:

Screen shot 2010-01-17 at 11.50.27.png

I’ve omitted all the actual code, for brevity in this example. The full code is here. The previous version of JeeLib has been moved to the avr-jeelib-first branch in subversion.

The Port<N> class is essentially the same as in an earlier post. It generates efficient code for raw pin access, given fixed pin assignments known at compile time.

The above code adds a templatized PortI2C<N,W> class, where N is again the port number (1..4) and W is a constant which determines the bus speed. As with Port<N> class, this leads to a class definition which requires no state and which can therefore consist entirely of inlined static members.

A HardwareI2C<W> is also defined, with the same API as PortI2C<N,W>, but using the hardware TWI logic in the ATmega. The point is that in use, PortI2C<N,W> and HardwareI2C<W> objects are interchangeable.

You can see how all this templating stuff complicates the naming of even simple classes such as these.

The final class implemented above is DeviceI2C<P,A> – it does the same as DeviceI2C in the original Ports library, but again using templates to “bring in” the underlying port classes and the device I2C address.

Here is an example sketch built with all this new code:

Screen shot 2010-01-17 at 12.48.05.png

It supports two bit-banged I2C devices on ports 1 and 2, respectively, as well as a third I2C device driven by the built-in TWI hardware.

This compiles to 980 bytes (using Arduino IDE 0017).

The good news is that this generates pretty efficient code. It’s not 100% inlined – but quite a bit is, especially at the lower levels, so the result looks like a pretty good implementation of a high-level I2C driver which can be used for both bit-banged and hardware-supported I2C, all by changing one declaration at the top of the sketch.

But there are quite a few inconveniences with this approach…

First of all, note that the declarations at the top are fairly obscure. I did my best to simplify, but all this template mumbo-jumbo means you have to understand pretty well how to declare a port, and how to declare an I2C device object for that port. The “typeof” keyword in there is a GCC extension, without which these declarations would have looked even more complex.

The major trade-off is that the above example essentially generates separate code for each of these three I2C devices. There is virtually no code sharing. This can lead to code bloat, despite the fact that each version generates pretty good code. In practice this might not be so important – it is not likely after all that you’ll need all three types of I2C buses in the same sketch. Just keep in mind that you’re trading off efficient hard-wired coding against the need to generate such code for each type of I2C access you might need.

So would this be a good foundation to build on? I don’t know yet…

C++ templates do seem to get a lot more of the logic “into” the compiler. Instead of passing say an I2C device address as constant to an object, we generate a highly customized class which is special-cased to implement just one device at just one address. With the result that the compiler can perform quite impressive optimizations. In the above example, there are lots of cbi and sbi instructions in the generated code, just as if you were to dig in and hand-craft an optimized implementation for exactly what you need. All from a small general-purpose library!

But it comes at a price. There is no (non-template) “DeviceI2C” class anymore. Writing a class on top to implement access to the UART Plug for example, means this class has to use templates as well. It’s a bit like “const” – once you start on the path of templates, they will start to permeate all your your code. Yikes!

The other “cost” is that all templates have to end up in header files. The size and complexity of the “JeeLib.h” header file is going to increase immensely. Not so great if you want to get to grips with it and just use the darn stuff. On the plus side, I was pleasantly surprised that error messages are fairly good now.

These drawbacks may be acceptable if all the template code can indeed remain inside the library – I sure wouldn’t want to impose the need for all library users to learn the intricacies of templates. So maybe it’s all doable – but this approach has major implications.

Is it all worth it? Hm, big decision. I do not like big decisions, not at this stage…

New library experiments

In AVR, Software on Jan 14, 2010 at 00:01

Encouraged by the previous post, I started writing a bit more code based on C++ features such as templates and namespaces, in the form of an Arduino library (compatible with it, not depending on it):

Screen shot 2010-01-09 at 02.08.27.png

Things to point out:

Less #defines (none so far), I turned bitRead() and bitWrite() into template functions (getBit and setBit), so that they can be used with 1..4 byte ints, just as the macros.

The Port class is inside a new “Jee” namespace, so there is no conflict with the existing Ports library.

Atomic access hasn’t been solved. Some of this won’t work if I/O pins get changed by interrupt code. I’m hoping to solve that at a higher level.

There are compatibility definitions for pinMode(), digitalRead(), and digitalWrite() using the normal Arduino pin numbering conventions (only Duemilanove so far). These are in their own namespace, but can also be used without any qualifiers by adding “using namespace Jee::Arduino;” to the sketch.

Totally incomplete stuff, untested, and not 100% compatible with Arduino in fact.

The other thing I want to explore, is to treat the choice of what a pin does as a pin initialization mode. Hence the enum with Mode_IN .. Mode_PWM definitions. The underlying code is PortBase::modeSetter(), but it hasn’t been written yet. It’s all steamy hot vapor for now.

Update – I’ve placed the code in subversion, but the API is going to be in flux for a long time.

Update #2 – Atomic access is actually better than I thought. With constant pin numbers, setBit() will use the cbi/sbi instructions, which are atomic.

C++ templates

In AVR, Software on Jan 12, 2010 at 00:01

A recent post described the performance loss in the Arduino’s digitalRead() and digitalWrite() functions, compared to raw pin access.

Can we do better – i.e. hide the details, yet still get the benefits of raw I/O? Sure.

If you’ve used JeeNodes and in particular the “Ports” library, you’ll have noticed that there is a C++ class which hides the details of each port (i.e. JeeNode port, not ATmega port). Let’s look at that first:

Screen shot 2010-01-06 at 12.40.09.png

I’ve omitted the implementation, but there are still lots of secondary details.

The main point is that this is now C++, and uses a “Port” object as central mechanism. Each object has one byte of data, containing the port number (1..4).

Due to heavy inlining, there is almost no additional overhead for using the Port class over using digitalRead() and digitalWrite(), on which they are based. I verified it by running similar tests as in the recent post about pin I/O:

Screen shot 2010-01-06 at 12.48.50.png

Using the definition “Port orig (1);” – and sure enough the results are nearly the same.

There are two issues which make this approach sub-optimal: using the slow digital read/write calls, and storing the port number in a memory location which needs to be accessed at run time. There is no way for the compiler to optimize such calls, even “orig.digiRead()” should be the same as writing “bitRead(PORTD, 4)” in this example.

That’s where C++ templates come in. Check out this definition of a new “XPort” class (named that way to avoid a name conflict) and an example of use for port 1:

Screen shot 2010-01-06 at 12.54.02.png

(As you can see, I’m switching to a different, and hopefully clearer, API along the way)

There’s some funky <…> stuff going on. We’re in fact not declaring one class, but a whole family of classes, parametrized by the integer included in the <…> notation on the last line.

The big difference, is that each class now has that integer value “built-in”, so to speak. So we can define member functions which directly pass that value on to the corresponding bitRead() and bitWrite() macros. And then all of a sudden, all the overhead vanishes: since the member needs no access to object state, it can be made static, and since all the info is known in the header, it can be made inline as well.

So the above template is C++’s modern way of doing far more at compile time, allowing the optimizer to generate much better code.

Note that templates come with some pitfalls: first of all, it’s very easy to inadvertently generate huge amounts of code, so very careful inlining and base class derivation is essential. The second problem is that templates tend to be “instantiated” as late as possible by the compiler, which can lead to confusing error messages when the templates are wrong or used wrongly.

I’m still just exploring this approach for embedded use. The potential performance gains are substantial enough to give it a serious try. My hope is that the hard work can be done in a library, so that everyone else can just use it and benefit from these gains without having to think much about templates, let alone implement new ones. The “one” object declared above acts like any other C++ object, so using it will be just as easy as non-template objects.

Does the above lead to fast code? You bet. Here’s a test sketch:

Screen shot 2010-01-06 at 13.05.29.png

And here’s some sample output:

Screen shot 2010-01-06 at 13.06.14.png

As you can see, values 5 and 6 are virtually the same as values 7 and 8. We’ve obtained the performance of direct pin access while using a high-level port-style notation to access those pins. This is why templates are so attractive for embedded use.

The timings are different from the previous post because the loops are coded differently. In this case, only the relative comparisons are relevant.

Arduino, JeeNode

Pin I/O performance

In AVR, Hardware, Software on Jan 6, 2010 at 00:01

There was a discussion on the Arduino developer’s mailing list about the impact of a small change to the digitalWrite() function, and for some time I’ve been hearing that digitalWrite() has a huge amount of overhead.

Time to find out.

Here is the sketch I used to measure how often a pin I/O command can be issued using various mechanisms:

Screen shot 2010-01-05 at 11.42.53.png

The logic is that I’m counting how often the same command can be called between timer overflows, i.e. every 1024 µs (one byte, incrementing @ 16 MHz / 64), before the timer tick count changes again.

And here’s the sample output:

Screen shot 2010-01-05 at 11.42.14.png

There’s a small amount of jitter, which tells me the loops are syncing up almost exactly on the timer ticks. Interrupts have not been disabled, so the timer interrupt is indeed being serviced – once for each loop.

What these values tell me, is that we can do about:

10 analog 10-bit readings per millisecond with analogRead()
128 pwm settings per millisecond with analogWrite()
220 pin reads per millisecond with digitalRead()
224 pin writes per millisecond with digitalWrite()
1056 pin reads per millisecond with direct port reads
1059 pin writes per millisecond with direct port writes

(I’ve corrected the counts by 1000/1024 to arrive at these millisecond values)

So the Arduino’s digital I/O in IDE version 0017 can do roughly 1/5th the speed of direct port access on a 16 MHz ATmega328.

But WAIT! – There’s a large systematic error in the above calculations, due to the loop overhead. It looks like the loop takes 1024000/1251 = 819 ns overhead, so the actual values are quite different: digitalRead() -> 3712 ns, direct port read -> 151 ns. Now the values are more like 1/25th!

So let’s redo this with more I/O in each loop iteration (all 4 ports):

Screen shot 2010-01-05 at 11.55.31.png

The sample output now becomes:

Screen shot 2010-01-05 at 11.56.59.png

With these results we get: one digitalRead() takes 4134 ns, one direct port read takes 83 ns (again correcting for 819 ns loop overhead). The conclusion being that digitalRead() is 50x as slow as direct port reads.

Which one is correct? I don’t know for sure. I retried the direct port read with 16 entries per loop, and got 67 ns, which seems to indicate that a direct port read takes one processor cycle (62.5 ns), as I would indeed expect.

Conclusion: if performance is the goal, then we may need to ditch the Arduino approach.

Update – Based on JimS’s timing code (see comments): digitalRead() = 58 cycles and direct pin read = 1 cycle.

Update #2 – The “1 cycle” mentioned above is indeed what I measured, but incorrect. The bit extraction was probably optimized away. So it looks like direct pin access can’t be more than 29x faster than digitalRead(). As pointed out by WestfW in the comments, digitalRead() and digitalWrite() have predictable performance across all use cases, including when the pin number is variable. In some cases that may matter more than raw speed.

Update #3 – Another caveat – Lies, damn lies, and statistics! – is that the register allocations for the above loops make it extremely difficult to draw exact conclusions. Let me just conclude with: there are order-of-magnitude performance implications, depending on how you do things. As long as you keep that in mind, you’ll be fine.

▶ View 15 Comments

JeeMon – one year later

In Software on Jan 4, 2010 at 00:01

JeeMon is a little web server / database / reporting tool I wrote a while back. It has been in use at Jee Labs for over a year now. Here is the electricity use for 2009:

Screen shot 2009-12-31 at 17.27.15.png

Couple of glitches, such as incorrect readouts when power failed and the sending node got reset to zero counts.

Here’s the gas consumption for 2009:

Screen shot 2009-12-31 at 17.28.06.png

Gas consumption (heating and hot water) is relatively high for this house, which is an open split-level design with lots of windows. Well isolated, but there’s simply no way to keep heat from rising up through the open stairways.

Here are the temperature and humidity readings for the past month:

Screen shot 2009-12-31 at 17.38.39.png

These are commercial S300 and KS300 sensors, and they seem to have frequent glitches. It looks like those could easily be filtered out, though.

Still, we managed to get €600 back on the 2009 energy bill, and with about 3000 kWh and 2000 m3 we’re 30% below the average consumption in this residential neighborhood, for both electricity and gas.

Being aware of what’s going on makes a difference, IMO. It has become a habit to turn off all the lights when I leave a room, and closing curtains right away when it gets dark. When we go out, we turn down the thermostat. Who needs high tech, when common sense is all you need? It’s so obvious, yet so effective …

The JeeMon database for all of 2009 is 26 Mb, i.e. tiny when considering that this includes every reading and some aggregated series as well. In fact, it contains a lot more data than what’s shown in the above graphs.

I’ve got several ideas and plans for JeeMon in 2010. I want to make it far more modular, so that nearly all its current functionality becomes available as easy-to-extend plug-ins. And it needs to be fullly configurable – as it is, JeeMon is still little more than a one-off implementation. But it has served its purpose very nicely already.

Better UART code and GPS

In Software on Jan 3, 2010 at 00:01

The UartPlug class which was recently implemented used per-byte access via I2C to retrieve each received character.

This isn’t terribly efficient, since it requires sending several bytes back and forth for each received character: first we address the UART to check its status, then we access it again to get one character from the FIFO. That’s six bytes of data for each received character.

It didn’t work well enough at 38400 baud, so I added a small buffer as optimization. The UartClass API hasn’t changed. The code which does this is:

Screen shot 2009-12-29 at 15.36.49.png

Nothing fancy, just a small buffer (10 bytes currently) to speed things up. The essence is that when lots of data is coming in, we get 10 bytes at a time, for a total I2C overhead of 15 bytes. That’s a 4-fold reduction in bandwidth.

The setup I’m experimenting with is this FV-M8 GPS receiver:

The hookup looks daunting, but that’s because the connector is a bit small so I made a little breakout board for it:

Screen shot 2009-12-29 at 15.44.56.png

Here’s some sample output:

Screen shot 2009-12-29 at 16.05.09.png

The “gpsdemo” sketch is simply a serial pass-through, converting 38400 baud to 57600 along the way:

Screen shot 2009-12-29 at 15.46.40.png

But it still loses character along the way, as seen in this snippet:

Screen shot 2009-12-29 at 15.17.26.png

The RMC line has a partial line at the end, after the “*7B” checksum.

My guess is that this isn’t the UARTs fault, nor the I2C bus. What seems to be happening is that the Serial class does not buffer its output, so all output causes the sketch to become unresponsive for input. In theory the 57600 baud link should be able to get data out fast enough, but I suspect that in practice there is an occasional hickup.

If this is indeed the cause, then it’s another example of how wasting time can lead to problems.

Back to the GPS. After a few minutes it finds its position in 3D. The following info comes in 5 x per second:

Some additional details are reported once a second:

Screen shot 2009-12-30 at 01.53.35.png

(ignore the last few garbled lines, I really need to lower that baud rate a bit)

Here’s an even more accurate fix I found in the data stream:

$GPGSA,A,3,12,22,09,28,18,26,17,27,,,,,1.32,1.02,0.85*02

So that’s up to 8 satellites – indoors, with stone/concrete walls and coated double-glazing windows. Impressive… older units I tried never even got a fix indoors!

To interpret this GPS data, check out Mikal Hart’s TinyGPS library, it can easily be hooked up to the UART plug – just use this code at the top of his test_with_gps example:

… and init nss to match the baud rate of your GPS unit.

Wasting time

In Software on Jan 2, 2010 at 00:01

No, not your time or mine… the topic I want to go into here is how to let a micro-controller deal with things that happen over time.

The way an Arduino sketch works is as follows:

Screen shot 2009-12-28 at 14.59.53.png

The little slashes represent real code, where “stuff happens” so to speak. You can see that there is a clear flow of control, from the start of the code to the end, with delays and calls to other bits of code.

The trouble with this is that it can’t deal with multiple events. You can’t count pulses on an input pin and keep a LED blinking at the same time, for example.

In the old days, interactive computer interfaces were written in this same way. You’d enter a couple of values and then you got some results back. If you made a mistake halfway in the sequence, you had to restart and enter everything all over again.

Then “command menus” were invented: elaborate decision trees of the form “do you want to do X, Y, or Z?” – now at least it was possible to go back just one step. Today, user interfaces are event driven, responding to whatever interaction the user initiates instead of presenting choices. The user is in control, not the computer.

The traditional event-driven logic uses a dispatch / switchboard approach:

Screen shot 2009-12-28 at 15.00.03.png

The more modern approach is to use callbacks and closures. Fancier still is to use coroutines or continuations.

Could we use something similar for micro-controllers? Sure.

Callbacks are not that convenient in C, since it does not support the closures concept which would make it convenient (nor coroutines or continuations). Also, callbacks would have to be represented by pointers to C functions (2 bytes), whereas dispatch codes can probably be represented as a single byte (as long as we don’t have more than 256 of them). Efficiency and memory space matters on small 8-bit chips such as the ATmega.

This requires a change in mindset when writing sketches. A good way to get into such an event-oriented style is to imagine that there is no delay() function. In event-oriented code, you’re not allowed to wait for time to pass.

So how can we blink a LED if we can’t wait to switch it on and off? Well, instead of delays, we have timers. We’re still not allowed to loop until the timer expires, but we can tell this new type of timer to generate an event when its time has come.

For a low-end implementation, an event can simply be a small integer dispatch code.

So instead of waiting until we can turn the LED on or off again, we tell the system to wake us up again at the right time. This approach is also a perfect match for low-power nodes, by the way: sleep and wake-up on events can be taken literally to enter power-down mode and start up again.

The code might look something like this:

Screen shot 2009-12-28 at 16.00.55.png

Sure, it takes some more work than this delay-based code:

Screen shot 2009-12-28 at 16.03.06.png

… but it has the huge benefit that it’s now fairly simple to deal with other activities at the same time. What you need to do is turn each of those activities into one ore more events as well. They can then be added right next to the blink event code, as additional case(s) in the above “switch” statement.

An event-based dispatch mechanism adds a lot of flexibility. Tasks such as counting pulses, blinking LEDs, serial communication, wireless packet reception and transmission – these can all be processed as they occur. With as nice bonus that low-power sleep modes can become fully automatic: when there are no events, go to sleep!

The trick is to ban all uses of “delay()” and “delayMicroseconds()” and to never use “busy loops” to wait for something to happen. This has far-reaching implications, because all libraries must also play by these rules.

I’m going to explore this approach further. Maybe one of the existing nanokernels could be used as foundation. To qualify, it must be truly minimal (and I’m picky!) – i.e. it has to fit into an ATmega328 without claiming too much flash or RAM memory space.

Update – here’s a web page by “inthebitz” which illustrates the problems described above. It’s an absolutely genuine attempt to help people get started, and it no doubt does a very fine job at that, but it also shows how everything gets serialized time-wise. For slow things, it’ll be just fine of course.

JeeNode, JeePlug

JeeNodes and complexity

In Hardware, Software on Dec 29, 2009 at 00:01

In yesterday’s post, I commented on a couple of aspects of the Arduino world which are giving me some trouble.

Now it’s time to do the same for all this “JeeNode” and “JeePlug” stuff I’ve been working on in 2009. Because there too, there are some ugly edges, uncomfortable trade-offs, and unsolved issues.

Let me start by pointing out the obvious: the Arduino world and Jee Labs are totally and utterly incomparable in scale. In terms of number of people involved, money involved, business interests, variety, maturity, promotion, reach, ambition, … they really differ in every aspect you can think of.

Except technology: Arduino’s and JeeNodes both live in the space of Physical Computing. And JeeNodes are compatible to Arduino’s in terms of software and PC interfacing. That’s not so surprising, since I designed JN’s to get going as fast as possible with as few key changes as possible.

These key changes were:

Voltage levels – run at 3.3V to support more sensors and wireless
Hardware ports – multiple identical interface connectors
Hardware extension – I2C buses which allow daisy-chaining
Software – connector-independence through the “Ports” library
Physical shape – a much smaller RBBB-like form factor

Two key additions driving some of this were:

Wireless – built-in bi-directional wireless link
Low power – selecting all hardware so it can run off batteries

Because one major use for all this is remote battery-powered operation, a minimal PC interface was chosen (3.3V signals for use with an FTDI adapter).

So is all this Jee stuff merely about creating wireless nodes? Not really. That just happens to be my area of interest. Staying focused is good, and I definitely intend to stick to this for quite some time.

So far so good. I thought I had it all figured out.

Then reality – and complexity – set in. Here are a number of issues:

Connector choices – ports use 6-pin 0.1″ headers as connectors: cheap and widely available. Not polarized, so you can plug things in the wrong way (and you will) – it took a while, but it has been standardized, and the POF approach solves polarization where it matters most: during experimentation.
Library dependencies – the “Ports” and “RF12″ (wireless) libraries have become inter-dependent, so you now have to include both of them all the time. As far as I can tell, this is not a software modularity issue but an Arduino IDE problem. This is starting to become a major inconvenience for small projects.
Class dependencies – classes such as “MilliTimer” are very small and self-contained but the more they get used, the more they lead to the above library dependency troubles. If adding modular and useful code leads to headaches, then something is definitely wrong.
Wireless functionality – a small driver takes care of packet reception and transmission in the background. It’s very low level but an easy transmission layer on top now adds robust communication for sensor networks. More convenience functions like these need to be added.
Multi-node development – it’s tedious to deal with multiple interfaces when more than one JN is plugged in, because USB interface names are meaningless. There is currently no way to auto-detect when a JN is plugged in or removed, let alone do something specific depending on what sketch that JN is running.
I2C ≠ I2C – ports support a software (bit-banged) I2C bus, but the ATmega also has pins which support hardware I2C. The underlying driver code is completely different. There is no generalized layer yet to hide these differences, so right now code written to use I2C plugs via the Ports library needs to be changed to work with hardware I2C and the Arduino’s “Wire” library. This must be fixed.
Ports vs. Plug Shield – due to the I2C differences described in the previous item, code written for ports on a JN needs to be adapted to work with the Plug Shield on an Arduino, and vice-versa. Tedious.
External power – the various power pins on a JN are all tied together, because the voltage drop of diodes would hamper ultra-low power use. That means you can’t hook up a JN to FTDI while a battery is tied to one of the other power pins (unless the battery has an on/off switch and you’re careful).

Some of the above issues have already been sorted out. Others can be addressed in future hardware designs. Some, such as “external power” are likely to remain as is for now. And some just need more or better software.

Adding or modifying software is easy. The hard part is avoiding restrictive decisions which have long-term consequences. Here’s the most challenging one: should I maintain 100% software compatibility with the Arduino world and benefit from all the shared knowledge already built up, or should I start off on a new journey and redo whatever is needed from scratch to reach a workable level again? I haven’t decided.

Screen shot 2009-12-28 at 00.33.52.png

When you hit a wall, do you look for a way around it or do you tear it down? Depends on the wall, I s’pose.

Arduino

Arduino and complexity

In Hardware, Software on Dec 28, 2009 at 00:01

This is – in a nutshell – the Arduino world:

Screen shot 2009-12-27 at 18.53.04.png

In prose: an ATmega board, some hardware peripherals, an environment for embedded software development, a front-end for visualization, and a website to bring all these facets together for a growing worldwide community.

For each of these, there are alternatives and variations. These add variety and increase the number of choices for everyone interested in this low-cost physical computing world. The “Arduino” name is conquering the world, and it sticks … even as “duino” suffix. Names are a great way to create a brand.

But what is Arduino, really? Is it an ATmega? Is it a board that fits in the palm of your hand? Is it a standard for connecting other boards? Is it a standard way of structuring software? Is it defined by a set of standard libraries? Is it the dual aspect of the IDE, i.e. Wiring vs. Processing? Is it the main web site or the discussion forum?

The answer depends no doubt on who you ask. And if this is a healthy ecosystem, then all of these will evolve and improve over time. In my opinion, the current state of an ecosystem is far less important than its ability to evolve.

That’s where complexity comes in.

It’s not hard to construct a great system, even an elaborate one. But what is extremely hard, is to come up with a system which supports evolution of all the pieces involved. Because with multiple pieces, you have to make decisions. You have to interconnect the pieces, and that requires making choices. And once you do, you reduce the number of future paths, including many you don’t even know about yet.

Making choices can be good. Hooking up an Arduino to a PC via a serial connection is a great low-cost solution for uploading, debugging, and interacting. The transition from an RS232 connection + a separate power supply to a USB connection with “built-in” 5V power is an excellent example of how evolution can lead to substantial progress.

Technological evolution can take years. And as in life, that’s where the really interesting stuff happens.

Unfortunately in the Arduino world, I’m hitting some nasty edges which tell me we need more generality or flexibility. I’ll describe a few, in terms of interfaces between the different pieces shown above:

Voltage levels – the most widely used Arduinos are based on 5V, meaning that all their I/O pins are also 5V-based. More and more sensors can only be used with a supply voltage up to 3.6V or so. Interfacing them to a standard Arduino requires the use of level converters. Often, resistors will do – but you can’t simply ignore the issue. This is an area where I2C can help.

Hardware modules – the Arduino “shield” concept doesn’t extend very far, because each shield determines which pins it needs. Pins are a scarce resource, and each pin location on the connector is fixed. It’s hard to design shields which will actually stack with others. Even more so now that there is a new Arduino Mega with slightly different pin allocations.

Software modules – libraries are a great concept: code which is there when you need it, but doesn’t get in the way (i.e. included) if you don’t. Right now, adding a library which depends on say the “Wire” library means that you have to include the Wire library in your sketch even if you don’t ever use it. This is almost a show-stopper.

C/C++ code – on the surface, programming for the Arduino board is like programming in C and C++, with a bunch of very common issues resolved and a wide variety of useful functions available, ready to use. Newbies can get going fast and old hands can find their way around in no time. Except… the IDE does some funky things with headers and declarations. And it breaks a couple of fairly basic C/C++ conventions.

Automation – the IDE works with a specific scenario in mind. As of the latest 0017 release, multiple sketches and consoles are supported. This solves some of the issues, but it’s hard to automate this further: after uploading, I still have to manually launch the serial console. If I have some other app talking to the hardware, there is no mechanism to upload a new sketch and then resume that app because the serial USB connection is dedicated.

Visualization – it’s interesting to see how the Arduino / Wiring / Processing approach is bridging a huge range of hardware in an integrated manner. But there are several choices involved which reduce flexibility towards entirely different approaches. Some visualizations (as well as other types of data processing) would benefit from using a more internet-/web-centric approach. Lower-end hardware cannot support the Java environment used by Processing, so that code can’t be leveraged. There are alternatives. Scripting language support would be useful.

Don’t get me wrong: all these issues are tough ones! It’s probably impossible to solve them all and to make everyone happy. These issues are definitely not things open source software is obliged to address from day one. But my point is that this is not about where we are today, but about having the mechanisms to evolve to where we want to be tomorrow.

If you’ve followed this weblog for a while then you can probably guess that I’ve got some ideas and suggestions on how to move forward on all this. I would really like to help take the existing Arduino platform further – but I’m not sure at this point that it can be done within the current constraints. I don’t even know whether anyone else considers the issues listed above to be important. All I know is that as I continue to add more software and design new hardware, the “push-back” from the current Arduino design choices is increasing… a lot.

Do I want to change the Arduino world (i.e. make a case, convince people, find compromises) – or do I want to sidestep some of its decisions by starting all the way from scratch? I haven’t decided.

Carrier detection

In AVR, Software on Dec 26, 2009 at 00:01

Wireless communication is a complex process. The ISM bands used by the RFM12B module get used in various ways – some quite simplistic. For “serious” use, what you want is to avoid transmissions interfering with each other – which means at most one transmitter should be active at any given time for a specific frequency band.

Easier said than done. This is what CSMA/CA is all about. It’s quite similar to ethernet: you listen to the “ether” and wait until there is no carrier before starting to send yourself. If everyone does so, then there will be fewer “collisions” – i.e. messed up packets.

CSMA/CA isn’t perfect. It doesn’t scale all that well if there are lots of nodes, all trying to find a free slot to send their packets out. There is always a non-zero probability of collision, when two nodes decide at nearly the same time that there is no-one else transmitting. And then, BOOM! … as they say.

One solution for that is another acronym: TDMA. Basic idea: have all the nodes agree to only send in specific time slots allocated to them. This requires a central coordinator, as well as pretty accurately keeping track of time (which is non-trivial with RC-based watchdog timers for sleep modes used with low-power nodes!).

In the ISM band, at least the 868 MHz one used in Europe, a simpler solution is used: keep the air-waves more or less free. The rule is that each transmitting node should send no more than 1% of the time, on average. With one or two dozen nodes and a bit of randomness in timing, the idea is that collisions will be relatively rare.

I recently added the 1% rule to the easy transmission mechanism in the RF12 driver (for the 868 MHz band only). So with the rf12_easy…() calls, adhering to the 1% rule is now automatic.

The 1% rule is a very simple system, yet it works surprisingly well. I’ve got over a dozen nodes around the house, sending out packets whenever they feel like it. The off-the-shelf commercial weather station nodes I use are very simplistic – they just send, ignore collisions, and send new data again a minute or two later. Those nodes probably don’t even have receive capability.

The RFM12B transceiver module in the JeeNode is slightly more sophisticated, in that communication can take place in both directions. So the receiver can send back an “ack” packet, and the originating node will have some idea of whether its last data packet ever made it to the destination (note that ack’s can get lost as well – but that’s another story).

Still – the RF12 driver used in JeeNodes is just as careless as the other nodes: it starts sending the moment it feels like it – except if a packet for its own net group is currently being received. Sending packets with the RF12 driver can still easily mess up whatever is currently going on in the air.

Well, as of today, things will improve a bit further. I’ve extended the RF12 driver code to look at the RSSI status bit before starting to transmit. If a carrier is detected, even one that isn’t being recognized by the RFM12B, then transmission will be delayed a bit. Here is the latest code in the RF12.cpp driver:

Screen shot 2009-12-21 at 02.16.28.png

This isn’t perfect – nothing ever is – because most nodes will be polling very frequently and then start to send right after the carrier drops. So the chance increases that nodes two and three will both try sending when node one finishes. But let’s assume that the RSSI signal doesn’t drop to 0 for all nodes at exactly the same time. If that turns out to be insufficient, a timer-based exponential back-off mechanism can be added later.

And there is a substantial benefit: nodes will no longer mess up packets which are currently “on the air”. As more nodes are being added around here, I expect this change to cause less degradation due to collisions.

In summary: the RF12 driver is a very simple system. No CSMA/CA, no TDMA. There are definitely limits as to what it can be used for. There are some severe limits on how much data it can send, given that (on 868 MHz) each node can only send up to 1% of the time, but this is also why such a simple approach is actually quite effective. And why the RF12 driver is still just a few Kb on an 8-bit ATmega, leaving lots of room for application specific code. I happen to think that the RF12 approach strikes a pretty decent balance between simplicity and effectiveness – and I’m always open for suggestions on how to take this further.

OOK reception with RFM12B ?

In AVR, Software on Dec 25, 2009 at 00:01

Yesterday’s post described a setup to see the RSSI and DQD status bit reported by the RF12 driver in real time.

One of the interesting results is that I can see the RSSI light come on when pressing a button on the FS20 remote transmitter – even though that’s an OOK signal, not FSK!

When adjusted to run at 433 MHz, the RSSI indicator also lights up with the KAKU remote.

In both cases, the DQD signal appears useless – it just shimmers all the time.

The RSSI signal is encouraging, though. It turns out that getting it to blink reliably did depend on setting the threshold right. At -103 and -97 dBm, it was on all the time – only the -91 dBm value produced a usable signal. I hope that’s the case with all units.

Could this be used to receive FS20 or KAKU?

Well, I just had to try. My idea was to continuously poll the RSSI status bit and then “mirror” its value to a DIO output pin. Then use a second JeeNode to treat this as a normal OOK pulse train.

Here’s the “rssiMirror” sketch I used:

Screen shot 2009-12-20 at 17.07.34.png

Does it work? Unfortunately … no :(

Time to hook up the Logic Analyzer to see what’s going on. I connected the above digital output to the first channel, and a real OOK receiver on the second channel:

Guess what… the RSSI signal is indeed detecting the presence of a transmitted signal, but it’s way too slow!

Here’s the same sample, zoomed in on the real OOK pulse train:

As you can see, there’s a pretty good sequence of transitions, 400 µs and 600 µs apart. Oh well, so much for the RSSI status bit – it’s nice to detect the presence of a carrier, but not more than that.

Next thing I tried was the DQD signal. After tweaking the DQD threshold to 3, this is what came out:

Yeah, sure, it seems to track the signal, but not reliably, and with a huge number of extra transitions. Note how the top timings are all multiples of 25 µs apart – that’s because it takes 25 µs to read out the DQD status bit. Coarse, but fine enough in principle to track 400 / 600 µs pulses from an FS20 remote.

So, again: nice, but no dice. Neither the RSSI nor the DQD status bits are fast and accurate enough to decode a slow OOK pulse train with.

Next attempt was to try and pick up the ARSSI signal, direct off the RFM12B module – as mentioned in this forum discussion. There’s a German forum which describes where to pick up that signal:

rfm01.JPG.jpeg

And sure enough, here’s a scope capture of an FS20 transmission:

Yeah, it’s there alright. But the signal is a bit weak. I’d rather not dedicate the analog comparator or ADC to it, and besides – that still leaves the need to compare against the average level – there’s a nasty 0.4V bias in that signal.

Here’s the same signal, AC coupled:

And here’s a zoomed-in area, showing what looks like pretty decent 400 µs and 600 µs pulses:

So yes, a small self-adjusting comparator can proabably turn this into a nice digital pulse train – but it’ll require some extra components, and I’m a bit out of my league on designing such a circuit.

Oh well – perhaps this information will help someone else further along. It’s been a good learning experience for me, even if the result is not quite what I had hoped…

Tomorrow, I’ll describe another – successful! – outcome from this RSSI / DQD exploration.

RF12 status lights

In Hardware, Software on Dec 24, 2009 at 00:01

Here is a little setup to see what’s going on in the ether, wirelessly speaking, that is:

Four LEDs, blinking according to the following status signals:

RECV – blinks briefly for each received packet
RSSI – shows the value of the RSSI bit in the RF12 status
DQD – shows the value of the DQD bit in the RF12 status
ALIVE – blinks at 1 Hz, just to show that this node is alive

I changed some thresholds to get better results: RSSI threshold -91 dBm and DQD set to 7 (normally -103 and 4, in the current RF12 driver).

Here is the sketch which drives the LEDs accordingly:

Screen shot 2009-12-20 at 16.05.18.png

In this example, I’ve set up the RF12 parameters to receive group 5 in the 868 MHz band.

The result is quite interesting: DQD is flashing constantly, in an irregular pattern. RSSI blinks only once in a while, and some of these cause RECV to light up – as expected: only packets with the proper FSK format are filtered out, and only some of those are for this specific net group.

The reason for doing this will become clear tomorrow…

JeeNode, JeePlug, LowPower

Power tracker – software

In AVR, Software on Dec 22, 2009 at 00:01

Yesterday’s post described a small circuit to track power consumption of JeeNodes to help optimize sketches for minimal power consumption.

Let’s put that circuit to use, with a bit of software to measure actual power consumption. The basic idea is to continuously measure current and then integrate these measurements to determine the sum of all power consumption intervals, regardless of levels.

The reason for this is that we’re not really interested in current draw but in the amount of charge consumed by the JeeNode. As far as the battery is concerned, drawing 1 mA for 1 hour is the same as drawing 100 µA for 10 hours (in the ideal case, anyway). The fancy way to say this is that we need to measure Coulombs, not Amps. Or rather micro-Coulombs, i.e. µC. That’s really easy once you realize that 1 µA is the same as 1 µC per second. Or to put it differently again: 1 mAh = 3600 mC. So a 1000 mAh battery is really nothing but a 3600 C charge.

Ok, back to the problem at hand: measuring average current draw per second.

Here is a simple sketch which does all the auto-ranging and integration:

Screen shot 2009-12-19 at 19.34.25.png

It reports averaged power drain in µA/sec (the second value is the number of samples per second). The 10% correction which I had to apply in my setup could be due to a number of factors – most likely it’s due to resistor tolerances (they are all 5%).

Here’s an interesting case with the latest rooms node:

Screen shot 2009-12-19 at 20.01.21.png

As you can see, the baseline power drain is a fantastically low 56 µA/sec in this case, but once or twice a minute it goes up to 14 mA/sec for several seconds. Not sure what’s going in here – need to investigate (now that I can!).

It would be nice to automatically detect the baseline, i.e. the average low-level sleep consumption, and things like the peak current and the percentage of the total consumption caused by such peaks. Extending the software to handle this is more work.

With slightly more elaborate software, it will be possible to place the power measurement plug between the measuring JeeNode and the JeeNode under test, and then leave it alone. A 1-day or 1-week average should give an excellent estimate of battery lifetimes.

UartPlug class

In AVR, Hardware, Software on Dec 20, 2009 at 00:01

Here is a utility class for the UART Plug:

Screen shot 2009-12-18 at 13.42.26.png

The interface is exactly the same as the Serial class, so it can be used interchangeably. Here is the updated “uart_demo” example in the Ports library:

Screen shot 2009-12-18 at 13.38.44.png

Here’s a test setup with a second JeeNode running RF12demo plugged in:

Sample output:

Screen shot 2009-12-18 at 13.38.29.png

Both input and output are supported by this UartPlug class – this demo is essentially a serial pass-through.

The UART supports accurate baud rates all the way up to 230400, which is in fact beyond the current I2C rates of the Ports library. Even at 57600 baud, I’ve seen several serious overruns with the above demo. One reason is that it’s only reading out one byte at the time, going through a multi-byte I2C bus sequence for each one (!). Note also that the Serial class does not buffer its output, so it can easily bog down everything else.

The UART hardware can support both hardware handshaking and XON/XOFF throttling, so this would be another way to avoid buffer overruns.

Up to say 9600 baud the UartPlug class should work fine, even with several UART plugs on the same I2C bus.

Battery life – refinement

In AVR, Hardware, Software on Dec 19, 2009 at 00:01

Yesterday’s post described how to estimate the battery life of a JeeNode running the “rooms” sketch with the SHT11, ELV PIR, and LDR sensors. To summarize:

the code started out using 370 µA average current, i.e. roughly 7 months on 3 AA cells
of these, 200 µA were caused by the 1-second periodic wakeup “blip”
another 120 µA were due to the actual measurements and packets sent every 30 seconds
and finally, the remaining 50 µA come from the PIR + JeeNode current draw in sleep mode

Yesterday’s post was also about reducing that 200 µA blip consumption to somewhere around 20 µA.

Today, let’s tackle the other power “hog”: the 300 ms @ 12 mA spike. Here is that pattern again:

The high peak at the end is the RF transmission of a packet, followed by a “knee” during which the node is waiting for the ack packet in RF receive mode.

Note that the main power drain is NOT caused by wireless communication!

This period of over 300 milliseconds is when the ATmega is polling the SHT11, waiting for a fresh reading. Twice in fact: once for the temperature and once for the humidity measurement.

So the explanation is very simple: we’re polling and waiting in full-power mode. Quelle horreur!

The fix requires a small modification to the SHT11 driver in the Ports library. Instead of a fixed delay, it needs to be extended to allow using an arbitrary function to waste some time. Here’s the modified code:

Screen shot 2009-12-18 at 01.00.53.png

A new second arg has been added, specifying an optional function to call instead of delay(). The code remains backward compatible, because this argument defaults to zero in Ports.h:

So now all we need to do is define a delay function which goes into real power down mode for a little while:

… and then adjust the two measurement calls to use this function:

Screen shot 2009-12-18 at 01.05.08.png

Does all this make a difference? You betcha:

That’s a substantial fraction of a second in which the ATmega will draw considerably less than 12 mA. How much less? Let’s expand the vertical scale:

Most of the time, the voltage is around 50 mV, i.e. 1 mA over 47 Ω. That’s the SHT11 current draw while active. There are two measurements – so everything behaves exactly as expected!

A couple of quick wake-ups remain, to check whether the SH11 measurement is ready. And so does the wireless TX/RX peak, of course. Here is an isolated snapshot of that RF activity (200 mV/div and 4 ms/div):

Approximate current draw: TX = 35 mA, RX = 20 mA. Total time is about 10 ms.

Looks like we’ve reduced the power consumption of this once-per-30-second spike by perhaps 90%. As a result, the node now consumes about 20 (blip) + 20 (spike) + 50 (sleep) = 90 µA on average. Even with much smaller 800 mAh AAA cells, the battery life of these low-power nodes should now be over a year.

There are several conclusions to take home from this story, IMO:

The biggest drain determines battery lifetimes.
Measuring actual current profiles always beats guessing.
A simple USB storage scope is plenty to perform such measurements.

If I had followed my hunches, I’d no doubt have spent all my time on getting the current draw of packet transmissions down – but as these experiments show, their effect on power drain is minimal.

There are more optimizations one could explore. There always are. But the gains will be limited, given that the ELV PIR sensor consumes 30..40 µA, and that it needs to be on at all times anyway, to be able to detect motion.

Sooo… end of story – for now :)

All source changes checked in. The entire rooms sketch still compiles to under 8 Kb of code.

JeeNode, LowPower

Battery life estimation

In AVR, Hardware, Software on Dec 18, 2009 at 00:01

To get an indication of battery power drain, I measured the voltage drop over a 47 Ω resistor in series with the 5V supply, using a JeeNode with Rooms Board and the latest version of the “rooms” sketch.

Here’s the blip I see, once a second (100 mV/div and 20 msec/div):

That’s a 40 msec pulse of about 5 mA, in other words: 5 mA during 4 % of the time, which averages out to around 200 µA current draw continuously. Not bad, but not stellar either: it’s 4 times the sleep mode current.

Every once in a while, a much longer and bigger spike shows up:

Which looks like roughly 350 msec @ 12 mA.

Let’s assume the big one is a real transmission. It sort of fits: the spike at the end is a brief transmission at ≈ 35 mA total, followed by a 20-ish mA period of reception (waiting for the ack to come in).

Very roughly speaking, the area of that extra spike at the end is about the same as the 5-to-10 mA step at the start of this period. So as an estimate, we’re consuming about 12 mA during 350 msec – let’s round down to 300 msec.

Let’s also assume these bigger current patterns happen every 30 seconds, when the node is reporting changed values (everything other than motion gets reported at that rate in the latest “rooms” sketch).

So 1% of the time (300 ms every 30s), power consumption is 12 mA. This averages out to 120 µA continuous current consumption.

In other words: a JeeNode running this latest rooms sketch with the SHT11 and ELV-PIR sensors, is consuming roughly 200 (blip) + 120 (spike) + 50 (sleep) = 370 µA.

Using a 2000 mAH 3-cell AA battery, this should lead to a 225-day lifetime – over 7 months.

Can we do better? Sure.

It’s basically a matter of figuring out what’s going on during those 40 and 350 msecs, respectively. Interestingly, more can be gained by improving non-transmitting “blips” than twice-a-minute high-power RF packet exchanges.

Do those 40 ms @ 5 mA every second look a bit suspicious? Yep – that’s the “idling” power level. What happens is that I was a bit too pessimistic in the time spent in sleep mode. This was the code:

Screen shot 2009-12-17 at 00.51.26.png

Looks like this is about 40 ms off, and so the code ends up waiting for the 1 sec timer to expire… in idle mode!

Let’s change this to end up closer to the desired time:

Screen shot 2009-12-17 at 02.59.16.png

Here’s the new blip (different scales):

We’re down from 40 to 10 msec blips – tada!

That translates to an average 50 µA current draw from the blips, bringing the total down to 220 µA. Which translates to a 375-day battery life: over a year!

Now we’re cookin’ … but could we do even better? Sure.

Note that only the 2 ms spike at the end of the 5 mA blip is the actual active period. The time up to then we’re just waiting in idle mode – and wasting power.

We could shorten the sleep timer to 994 ms, since we don’t care whether readings are taken exactly 1 second apart. Now the RFM12B-based watchdog timer will wake us up just 2 ms short of the target time. And sure enough, the 5 mA blip is down to around 3 ms – shown here with an even further expanded time scale:

But that’s silly. We’re tweaking a millisecond timer, and we’re not even interested in an “exact” 1000 ms cycle in the first place! It makes much more sense to just use the RFM12B wakeup timer to get us close to that 1 second cycle, and then immediately take a measurement. Here’s the corresponding code change in periodicSleep():

Screen shot 2009-12-17 at 02.35.36.png

Does this make a difference? Definitely:

One final remark: the above battery lifetime estimates do not take into account the increased power consumption when motion is detected and more packets are sent (up to once every 5 seconds). On the plus side, when no light / temperature / humidity changes happen, the packet frequency will drop further, to once-a-minute.

The above changes have been checked into the source code repository.

Update – I just found out that the DSO-2090 scope has a high-pass low-pass filter option:

Sure wish I’d found out about that feature sooner… it’s so much more informative: the initial ramp is probably the clock starting up, and the little peak could well be the LDR pull-up during ADC conversion!