Planet NoName e.V.

The Kinesis Advantage Contoured is an ergonomic keyboard which I have been using for four years. It has many features that make it a great keyboard, and it’s certainly the best I have ever used.

However, as you can also read on many places on the internet, many users have a tiny problem with the keyboard: the modifier keys sometimes get stuck. For example, you press Shift, followed by A, release Shift, press n. What appears on your screen is AN instead of An. You can only get out of this state when pressing Shift once again. It seems like the firmware doesn’t register the key release.

Last year, I decided to write Kinesis about it and they offered me a firmware upgrade. I was surprised to learn that a firmware upgrade actually means getting shipped a new microcontroller with a new firmware on it :-). After swapping the microcontroller, I noticed that the problem did not appear as often as it used to, but it still appeared. Of course, that is not satisfactory for perfectionists and heavy keyboard users :-).

The Humble Hacker Keyboard

The Humble Hacker Keyboard is a project by David Whetstone, who started out his project by modifying existing keyboards, amongst them the Kinesis Contoured. Thankfully, he documented his modifications in his blog.

After reading what he did, I decided to replicate his approach and swap the electronics of my Kinesis. It seemed reasonably straight-forward: Use a Teensy++ 2.0 USB development board, connect the different parts of your Kinesis to it and use David’s Open Source firmware. So I ordered the Teensy++ and had a look at how to modify my Kinesis so that I could test whether everything works before destructively modifying the keyboard.

The flex connectors

In comparison to David’s version of the Kinesis Contoured, my version was newer and used flex connectors for connecting the left/right keyboards to the controller board, as you can see on the picture above. Thankfully, Kinesis is very friendly to the enthusiast community around its products. Rick revealed that they are using the Cvilux CF01131V000 flex connectors. Unfortunately, it seems impossible to order just a few Cvilux parts to Europe.

From an earlier modification I still had a few spare Kinesis parts lying around, so I decided to unsolder the flex connectors from those for a quick test. Unfortunately, the grid of the flex connectors is different from the usual 2.54mm grid, so you cannot simply use such a connector on a breadboard. Volker came up with the idea of soldering a pin header directly to it, and that worked (see the picture). I then used a bunch of jumper wires to connect the flex connector to the Teensy++ and then the Kinesis keyboard parts to the flex connector. After flashing the firmware, I could press keys on the Kinesis and have them appear on my screen. Yay! :-)

Given that I could not order the Cvilux flex connectors directly, I spent quite a bit of research into finding a compatible part. In the end, I found and used the Molex 39-53-2135 which you can order from RS components. They even have free shipping for students :-).

The prototype board

Now that the proof of concept was done, it was time to design a prototype board on which I can mount the Teensy++ and connect all the Kinesis keyboard parts — essentially, one could call it a break-out board for the Teensy++. It really doesn’t contain any electronics.

I created the layout and board files with the free version of CadSoft EAGLE, based on David’s descriptions in his blog posts. I am not very experienced in EAGLE, so thankfully my friend Moritz helped out with creating an EAGLE library for the Molex flex connector and cleaning up my board.

We then milled the prototype board at Volker’s lab and soldered the flex connectors and pin headers on the prototype board. Here are two pictures of the board, one before soldering and the other one after putting it into the Kinesis:

Note that the thumb keyboards don’t use a flex connector — they are soldered directly onto the controller board :-(. We decided to solder them to a pin header so that we can connect and disconnect them at will.

After connecting the prototype board, most of the keyboard worked properly. There unfortunately was one little mistake in the layout (and subsequently in the board), which we were able to correct with a bit of wire. I then decided to leave the prototype board in to test it for a bit of time until the PCBs were manufactured and delivered.

Unfortunately, we only noticed that there have to be two holes in the center of the board in order to be able to actually close the keyboard case. Furthermore, the microcontroller doesn’t have a lot of space, so you need to be careful when designing a board. I decided to eat my own dogfood and leave the Kinesis half-open for a while.

The PCB

Now that it was clear that the idea basically works, Volker and I decided to take measurements of the original board and design a proper board — with holes in it and LEDs on it. It took Volker about two hours to route the board, and the result looks pretty good.

Afterwards, we submitted the board file to BILEX-LP, a cheap PCB manufacturer in Bulgaria. It took them a few working days to manufacture the board and a few more for shipping it, but it finally arrived about two weeks after we placed the order. You can see the PCB on the picture below:

Of course, not everything was perfect, both with our board file and with the manufacturing:

1. One connection was disconnected because of a cut on the board. Not detecting this in the quality control was the manufacturers fault. But you have to expect this when ordering at a cheap PCB manufacturer — the other 5 boards look fine. We were able to fix this with a bit of wire.
2. The bottom hole is too small for actually fitting the screw through it. That’s not a big deal though, since the upper hole is good and the PCB is held in place by the connection through the upper hole and the connected keyboards.
3. The drilling holes for the Teensy++ (in the middle of the board) are 0.8mil instead of 0.9mil. Since the Teensy++ EAGLE library was downloaded from their website, we thought it’d be fine, but we should have double-checked. Luckily, you can still mount a pin header on the board, you just need to push a bit harder :-).

So all in all, the PCB is usable, even though it’s not perfect. I then soldered the ports onto it and we placed it in my Kinesis. Note: the Teensy++ faces the board, so the USB connector is also facing the board. Therefore, we plugged in the USB cable and positioned it so that the connected cable fits, then soldered it onto the pin header.

For the USB connection, Volker suggested to just rip out the board on the other side of the Kinesis case, then cut the cable and directly solder a USB cable to it. That way, we re-use the original Kinesis strain-relief and their cable, which is a pretty good idea. You can see the result here:

The firmware

As expected, David’s Humble Hacker Keyboard firmware does not suffer from the stuck-modifier bug that motivated me to start this whole modification in the first place. However, I noticed that some key presses were registered twice, so I implemented debouncing and sent a pull request. The whole implementation was finished surprisingly quickly, the code really is pretty clean and easy to understand.

You can find my version of the firmware at http://code.stapelberg.de/git/kinesis-firmware/. You can also download the .hex firmware image that you can flash on your Teensy++ — but beware: I use neo-layout.org, so I remapped all the keys in the firmware :-).

The layout and board files

In case you also want to replace the controller board of your Kinesis, or as a foundation for similar projects, here are the EAGLE files:

• PCB with proper holes (2013-03-18): brd, sch, brd (PNG), sch (PNG),
• PCB as manufactured (2013-02-28): brd, sch
• git repository

Note that I also have 3 spare PCBs to sell. You can have one for 10 € plus shipping, but without any warranty or support. First come, first serve.

Conclusion

Of course it was not strictly necessary to build my own keyboard controller. I could have just lived with the bug. But it was fun and it’s a great feeling to have a (mostly) selfmade keyboard with an Open Source firmware to type on :-).

The total amount of time spent on this project was about a full work week.

I also want to say a big thank you to David Whetstone for publishing his experiments and firmware, to Moritz Augsburger for the help with the early EAGLE files and last but not least to Volker Wunsch who spent a lot of time with me and helped me a lot in order to get this project done.

OpenWrt is a nice FOSS Linux firmware (primarily) for wireless routers, which I use for many years. Even though I never experienced a problem with my routers, I’d like to be prepared for hardware failures, software failures and getting my router compromised. Here is a short description of each scenario so that it is clear what I mean:

• Hardware failure: The flash in my router dies and after rebooting, neither the read-only part nor my configuration can be read, so the device does not work anymore. The only solution is to buy a new router, or have a hot spare ready. This is the worst case.
• Software failure: After my network provider’s intern decides to fuzz the customer IP range instead of their testbed, he discovers an implementation flaw in the router’s PPPoE software and subsequently, the router deletes all the read/write data (i.e. my configuration). The solution is to reformat the read/write part of the flash and restore the latest backup. This story covers not only software failure, but also human failure when upgrading the router firmware.
• Compromised router: Some software has a security vulnerability and an attacker obtains access to the router. Since a backdoor might have been installed, I need to re-flash the firmware image and restore my configuration.

All of these points imply having a backup. But did you actually verify that your OpenWrt backup works? What’s your disaster recovery plan for each of the scenarios above?

Backing up

OpenWrt ships with a feature to provide a tar archive containing all your configuration files. You can find it in LUCI’s “System → Backup” tab. Obviously, you need to repeat this step after every configuration change you do.

If you have installed any additional packages, you also need to save the list of packages:

opkg list-installed | cut -f 1 -d ' '

If you have installed any services that ship an init script (e.g. OpenVPN), you also need to keep a note of which ones are enabled/disabled in LUCI’s “System → Startup” tab.

Restoring

The correct order in which to restore your router to a working state is this:

1. Flash your firmware image (save the original one whenever you flashing, or at least note which precise version you used).
2. Configure your router so that it can access the internet.
3. Re-install your packages:

   opkg update && for i in $(cat /tmp/pkgs); do opkg install $i; done
   

4. Restore your configuration by uploading the tar archive in LUCI’s “System → Backup” tab.
5. Re-enable any services you have installed (e.g. OpenVPN) in LUCI’s “System → Startup” tab, because that information is not contained in the tar archive.

Restoring to a different device

In case you have a different router, for example because a hardware failure occured or because you want to setup that hot spare I have been talking about, you need to watch out for one little subtlety in the process:

The MAC addresses of the radio interfaces need to be replaced before restoring the backup. Otherwise, OpenWrt will not apply your wireless configuration to the interfaces it finds.

In order to do that, simply edit the relevant file with a text editor and re-pack the tarball:

mkdir /tmp/fix && cd /tmp/fix
tar xf ~/Downloads/backup-OpenWrt-2013-03-13.tar.gz
vi etc/config/wireless
tar czf ~/Downloads/backup-OpenWrt-2013-03-13-fixed.tar.gz *

Building your own firmware image

In case you are dissatisfied with the dependency on the internet in step 3 of the restore procedure, you could build your own firmware image which contains the extra packages that you use. I don’t want to describe that process in depth, but it seems worth pointing out that this is one way to go.

Alternatively, you could also keep a copy of the extra packages, scp them to the router and install them with opkg. Depending on your number of extra packages, one of these will clearly seem easier :-).

About 4 years ago, I started tracking my configuration files with git. The advantages of storing configuration files in some repository are numerous:

1. You can destroy your configuration/computer and easily revert to a known good state.
2. You can easily distribute and update the same set of configfiles across multiple machines (especially virtual machines or other test setups).
3. You can refer other people to your configfiles if you store them in a public repository.

This article describes the way I use git to track my configfiles. My solution is simple and should be easy to understand even if you have not used git extensively before.

Please note that I am not trying to convince you to use my script(s), or my configfiles. Also, I am not going to support this — if you have any troubles with it, you have to debug it on your own. I am merely describing what works fine for me, in the hope that it will be useful to somebody else.

initialize.sh

Obviously, the first step to get the configfiles repository on a “fresh” computer is to clone the git repository: git clone git://code.stapelberg.de/configfiles

Afterwards, run the script initialize.sh: cd configfiles && ./initialize.sh. This script will create a corresponding dotfile for each file in your configfiles repository, e.g. when you have “configfiles/foorc”, it will create “~/.foorc” as a symbolic link:

$ ls -l .zshrc
lrwxrwxrwx 1 michael staff 32 2010-09-30 07:57 .zshrc -> /home/michael/configfiles/zshrc

Every already existing file, e.g. your current ~/.zshrc, will be preserved in ~/.configfiles.bak.

Files located in subdirectories

While this approach works well for a large number of configfiles, there are exceptions. MPlayer for example stores its configfile in ~/.mplayer/config. Since folders are reserved for a different purpose in my way of doing things, the solution for this problem is a file called MAPPING. It contains a simple lookup table of (filename → path) pairs, e.g. mplayer.config ~/.mplayer/config. initialize.sh takes care of creating the destination parent folders if necessary.

Furthermore, you can use that mechanism to group files together by giving them a filename with common prefix. As an example, all my emacs-specific configfilse start with “emacs-”, e.g. “emacs-init.el” and “emacs-zkj-notmuch.el”. I find this much clearer than just naming the file configfiles/init.el.

Host-specific files

If you have files which are specific to a certain host, you can store them in a folder named like your host (use “hostname” to see how the folder needs to be called precisely). To have a standard ~/.Xmodmap, but overwrite it on a single machine called midna (with a weird keyboard), use configfiles/Xmodmap and configfiles/midna/Xmodmap.

I have read about other people using branches to have different configfiles for different hosts. While that may be a more complete solution for some extreme cases, I find it way too hard to manage.

update.sh

In order to actually update the configfiles repository, one would normally run git pull. To make this happen automatically, I call update.sh from my zshrc:

cfgfiles=$(dirname $(readlink ~/.zshrc))
# If the configfiles are in a git repository, update if it’s older than one hour
find $cfgfiles -maxdepth 1 -name .git -mmin +60 -execdir ./update.sh \;

Whenever I log into a system which is automatically managed, it will update the configuration files. On the other hand, if I don’t use a system, it will not spend any bandwith/cpu time on updating.

The script update.sh itself is a wrapper around git pull, but has a few nice properties:

1. Only one instance of the script will run, so that you can open multiple shells/terminal emulators quickly and not have any race conditions.
2. The script uses git stash to preserve your local, uncomitted changes.
3. Instead of operating inplace (and breaking programs that run precisely when git replaces a file, e.g. your shell), the script works in a temporary directory.
4. It runs the update in the background so that you don’t have to wait for a terminal emulator window to open. This implies that the shell process which triggered the upgrade will not pick up changes to zshrc automatically.

Detecting and handling errors

Because update.sh runs its upgrade in the background, it will not post any messages about success or failures to stdout/stderr — that might interrupt what you are currently doing in foreground. Instead, it is entirely quiet in case everything is okay.

When something fails (e.g. the server which hosts your git repository is down), update.sh will create a file called ERROR in your configfiles directory. My zshrc is configured to pick this up in its prompt and display it prominently in red:

setup_prompt() {
    local _cfg_nag

    if [ -f "$(dirname $(readlink ~/.zshrc))/ERROR" ]; then
        _cfg_nag="%F{red}cfg-git-error%f "
    else
        _cfg_nag=""
    fi

    PROMPT="%K{cyan}%F{black}%m%k%f ${fg_green}%~${fg_no_colour} \$(get_git_prompt_info)$_cfg_nag$ "
}

setup_prompt

So whenever something is wrong, you will end up with a prompt like this:

In order to figure out what went wrong, view the file last-update.log in the configfiles folder. Afterwards, just delete the file ERROR.

Committing changes

There is nothing special to watch out for when committing your changes. That is, you could edit ~/.zshrc, then go to the configfiles directory and commit your changes with git commit -a. Don’t forget to git push your changes afterwards. Also, git add -p might come in handy to only add specific parts of a file.

Pitfalls

In order to help with trouble-shooting, here are a few mistakes I have done in the past:

• I thought it’d be a clever idea to check out the configfiles repository once in /etc/configfiles and run initialize.sh for each user. The obvious problem is that as a user, you don’t have permission to write to /etc, thus you cannot upgrade. So this idea only works out when you log in as root by default, which you should avoid.
• You should not use git stash on your own in the configfiles repository. update.sh tries to stash any changes and will then just unconditionally try to pop the latest stash entry after pulling.

When writing data to a file descriptor (file, socket, …) in C, it is recommended to use a loop to write the entire buffer and keep track of how many bytes write() could actually write to the file descriptor. This is how to write data to a file in C in a naive way:

#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <err.h>
#include <errno.h>

int main() {
    int fd = open("/tmp/pattern.example",
                  O_CREAT | O_TRUNC | O_RDWR,
		  S_IRUSR | S_IWUSR);
    if (fd == -1)
        err(EXIT_FAILURE, "Could not open() file");

    const char *data = "This data illustrates my point.";
    /* This is WRONG, don’t do that: */
    write(fd, data, strlen(data));

    close(fd);
    return 0;
}

…and here is how to write data with the aforementioned pattern:

const char *data = "This data illustrates my point.";
int written = 0;
int n;
while (written < strlen(data)) {
    if ((n = write(fd, data + written, strlen(data) - written)) < 0) {
        err(EXIT_FAILURE, "Could not write() data");
    }

    written += n;
}

In case it is not entirely obvious what happens here: write() returns the amount of bytes it wrote, and that might be less than you specified. Therefore, we keep track of how many bytes were written and try to write the rest, until finally all data was written successfully. Be careful, though: a return value of -1 signals an error, so you need to handle these carefully.

The reason I am writing about this pattern is to illustrate it with real-world examples. We recently received a bug report for i3 (ticket #896, direct link omitted due to spam bots) which stated that i3bar would crash in a certain setup when switching workspaces. This report was only reproducible on OpenBSD, which tends to use conservative buffer sizes for many things.

It turned out that the cause for the crash was an error in our write code, which would fail to properly call write() multiple times. This never came to our attention previously because the data we send upon workspace switches got larger only recently and the buffer sizes on Linux still fit all of the data in a single write() call.

Another interesting behavior of some system calls is that they might return an error which means that you should just repeat that call. Two such error codes come to mind: EAGAIN and EINTR. The former is only relevant for non-blocking file descriptors, and means that performing that write() would block the process. EINTR means the system call was interrupted by a signal.

The same piece of code which contained the bug I talked about earlier was also not prepared to handle EAGAIN: when you switched workspaces often enough, the scheduler might give i3 so much CPU time — and none to i3bar — that i3 filled up the socket buffer and write() returned -1 with errno set to EAGAIN.

In conclusion, the correct write pattern looks like this:

const char *data = "This data illustrates my point.";
int written = 0;
int n;
while (written < strlen(data)) {
    if ((n = write(0, data + written, strlen(data) - written)) < 0) {
        if (errno == EINTR || errno == EAGAIN)
            continue;
        err(EXIT_FAILURE, "Could not write() data");
    }

    written += n;
}