A couple of weeks ago, from 15th to 19th March, I participated in the virtual annual meeting of the German Association of Applied Mathematics and Mechanics (GAMM). For me this meeting was the first time I presented my work to an audience of applied mathematicians with a broad background and no inherent interest in quantum chemistry. With only 15 minutes for my talk in the "scientific computing" track preparing the material was quite a challenge. I hope I still managed to convey the main ideas of our recently published LDOS preconditioner in a broadly accessible way. My slides are attached below.

Link	Licence
A novel black-box preconditioning strategy for high-throughput density-functional theory (Slides)

I recently learnt about the Emacs package project.el, which is used to figure out which files and directories belong to the same project. This is used under the covers by Eglot, for example.

In practice, a project is recognized by looking for Git repositories, which is a decent first approximation that often just works.

But what if the detection fails? For example, maybe you want to anchor your project-based commands in a parent directory that contains multiple Git repositories.

Luckily, we can provide our own entry to the project-find-functions hook, and look for a .project.el file in the parent directories:

;; Returns the parent directory containing a .project.el file, if any,
;; to override the standard project.el detection logic when needed.
(defun zkj-project-override (dir)
  (let ((override (locate-dominating-file dir ".project.el")))
    (if override
      (cons 'vc override)
      nil)))

(use-package project
  ;; Cannot use :hook because 'project-find-functions does not end in -hook
  ;; Cannot use :init (must use :config) because otherwise
  ;; project-find-functions is not yet initialized.
  :config
  (add-hook 'project-find-functions #'zkj-project-override))

Now, we can use touch .project.el in any directory to make project.el recognize the directory as project root!

By the way, in case you are unfamiliar, the configuration above uses use-package, which is a great way to (lazily, i.e. quickly!) load and configure Emacs packages.

NXP’s Eclipse-based MCUXpresso IDE is the easiest way to make full use of the hardware debugging features of modern NXP micro controllers such as the i.MX RT1060 found on the NXP i.MX RT1060 Evaluation Kit (MIMXRT1060-EVK), which I use for Teensy 4 development.

For projects that are fully under your control, such as imported SDK examples, or anything you created within Eclipse, you wouldn’t necessarily need Compilation Database support.

When working with projects of type Makefile Project with Existing Code, however, Eclipse doesn’t know about preprocessor definition flags and include directories, unless you would manually duplicate them. In large and fast-changing projects, this is not an option.

The lack of compiler configuration knowledge (defines and include directories) breaks various C/C++ tooling features, such as Macro Expansion or the Open Declaration feature, both of which are an essential tool in my toolbelt, and particularly useful in large code bases such as micro controller projects with various SDKs etc.

In some configurations, Eclipse might be able to parse GCC build output, but when I was working with the QMK keyboard firmware, I couldn’t get the QMK makefiles to print commands that Eclipse would understand, not even with VERBOSE=true.

Luckily, there is a solution! Eclipse CDT 9.10 introduced Compilation Database support in 2019. MCUXpresso v11.3.0 ships with CDT 9.11.1.202006011430, meaning it does contain Compilation Database support.

In case you want to check which version your installed IDE has, open Help → About MCUXpresso IDE, click Installation Details, open the Features tab, then locate the Eclipse CDT, C/C++ Development Platform line.

For comparison, Eclipse IDE 2021-03 contains 10.2.0.202103011047, if you want to verify that the issues I reference below are indeed fixed.

Bug: command vs. arguments

Before we can enable Compilation Database support, we need to ensure we have a compatible compile_commands.json database file. Eclipse CDT’s Compilation Database support before version CDT 10 suffered from Bug 563006: it only understood the command JSON property, not the arguments property.

Depending on your build system, this isn’t a problem. For example, Meson/ninja’s compile_commands.json uses command and will work fine.

But, when using Make with Bear, you will end up with arguments by default.

Bear 3.0 allows generating a compile_commands.json Compilation Database with command, but requires multiple commands and config files, which is a bit inconvenient with Eclipse.

So, let’s put the extra commands into a commandbear.sh script:

#!/bin/sh

set -eux

intercept --output commands.json -- "$@"
citnames \
  --input commands.json \
  --output compile_commands.json \
  --config config.json

The "command_as_array": false option goes into config.json:

{
  "compilation": {
  },
  "output": {
    "content": {
      "include_only_existing_source": true
    },
    "format": {
      "command_as_array": false,
      "drop_output_field": false
    }
  }
}

Don’t forget to make the script executable:

chmod +x commandbear.sh

Then configure Eclipse to use the commandbear.sh script to build:

1. Open Project Properties by right-clicking your project in the Project Explorer panel.
2. Select C/C++ Build and open the Builder Settings tab
3. In the Builder group, set the Build command text field to: ${workspace_loc:/qmk_firmware}/commandbear.sh make -j16

Verify your build is working by selecting Project → Clean… and triggering a build.

Enabling Compilation Database support

1. Open Project Properties by right-clicking your project in the Project Explorer panel.
2. Expand C/C++ General, select Preprocessor Include Paths, Macros etc. and open the Providers tab.
3. Untick everything but:
   • MCU GCC Built-in Compiler Parser
   • MCU GCC Build Output Parser
   • Compilation Database Parser
4. Select Compilation Database Parser, click Apply to make the Compilation Database text field editable.
5. Put a full path to your compile_commands.json file into the text field, e.g. /home/michael/kinx/workspace/qmk_firmware/compile_commands.json. Note that variables will not be expanded! Support for using variables was added later in Bug 559186.
6. Select MCU GCC Build Output Parser as Build parser.
7. Tick the Exclude files not in the Compilation Database checkbox.
8. Click Apply and Close.

You will know Compilation Database support works when its progress view shows up:

If you have an incompatible or empty compile_commands.json, nothing visible will happen (no progress indicator or error messages).

After indexing completes, you should see:

1. Files that were not used as greyed out in the Project Explorer
2. Open Declaration in the context menu of a selected identifier (or F3) should jump to the correct file. For example, my test sequence for this feature in the QMK repository is:
   • in tmk_core/protocol/chibios/main.c, open init_usb_driver
   • open usbStart, should bring up lib/chibios git submodule
   • open usb_lld_start, should bring up MIMXRT1062 port
3. Macros expanded correctly, e.g. MIMXRT1062_USB_USE_USB1 in the following example

Slow file exclusion in projects with many files

Bug 565457 explains an optimization in the algorithm used to generate the list of excluded paths, which I would summarize as “use whole directories instead of individual files”.

This optimization was introduced later, so in MCUXpresso v11.3, we still have to endure watching the slow algorithm for a few seconds:

Conclusion

NXP, please release a new MCUXpresso IDE with a more recent CDT version!

The improvements in the newer version would make the setup so much simpler.

I bought the cheapest compatible BTicino intercom device (BT 344232 for 32 €) that I could find on eBay, then soldered in 4 wires and added microcontrollers to make it smart. It now connects to my Nuki Opener Smart Intercom IOT device, and to my local MQTT Pub/Sub bus (why not?).

Background

In my last post about the BTicino intercom from November, I described how to use a Teensy microcontroller to reliably interpret SCS bus signals and drive a Nuki Opener (Smart Intercom).

Originally, I had hoped the Nuki developers would be able to fix their device based on my SCS bus research, but they don’t seem to be interested. Instead, their support actually suggested I run my microcontroller workaround indefinitely!

Hence, I decided to work on the next revision to clean up my setup in terms of cable clutter. I also figured: if I already need to run my own microcontroller, I also want to connect it to my local MQTT Pub/Sub bus for maximum flexibility.

Unfortunately, the Teensy microcontroller lacks built-in WiFi, or any kind of networking.

I switched to an ESP32-based microcontroller, but powering those from the SCS bus seems like a bad idea: they draw a lot of power, and building small high-quality power supplies is hard.

This made me scrap my previous plans to make my own SCS send/receive hardware.

Instead, I wondered what the easiest yet most reliable approach might be to make this intercom unit smart. Instead of building my own SCS hardware, could I use the intercom unit itself to send the door unlock signal, and could I obtain the unit’s already-decoded SCS bus signal?

Finding the signals

Based on my previous research, I roughly knew what to expect: closest to the bus terminals, there will be some components that filter the bus signal and convert the 27V into a lower voltage. Connected to that power supply is a microcontroller which deals with all user interface.

To learn more about the components, I first identified all ICs (Integrated Circuits) based on their labeling. The following are relevant:

• TI LM393: Dual Comparators
• TI LP2951: Adjustable Micropower Voltage Regulators, 5V output
• Microchip PIC16F684: 8-bit microcontroller

I connected my development intercom unit to my SCS bus lab setup and used my oscilloscope to confirm expected signal levels based on the pinout from the IC datasheets.

I settled on the following 4 relatively easily accessible signals and soldered jumper wires to them:

• 5V and GND: 5V, 100mA. Our QT Py microcontroller uses 7mA.
• OPEN5V: activates the button which unlocks the door
• SCSRX5V: converted SCS signal

Converting the signals

Because the BTicino intercom units runs at 5V, but more modern microcontrollers run at 3.3V, we need to convert between the two voltages:

1. We need to convert a 3.3V signal to OPEN5V to trigger opening the door.

2. We need to convert SCSRX5V signal to 3.3V so that I can use an ESP32 microcontroller to read the signal and place it on MQTT.

Here’s the corresponding schematic:

Microcontroller selection

I eventually decided to task a dedicated microcontroller with the signal conversion, instead of having the WiFi-enabled microcontroller do everything, for multiple reasons:

• Reliability. It turns out that using a hardware analog comparator results in a much higher signal quality than continuously sampling an ADC yourself, even when using the ESP32’s ULP (Ultra Low Power) co-processor to do the sampling.

• Easy implementation. Converting an SCS signal to a serial signal is literally a single delayMicroseconds(20); call in the right place. Having a whole microcontroller for only this task eliminates any concurrency concerns. I have not had to debug or change the software even once in the last few months.

• Easy debugging/introspection. I can connect a standard USB-to-serial adapter and verify the signal is read correctly. This quickly narrows down issues on either side of the serial interface. Issues with the microcontroller side can be reproduced by sending serial data.

Here are the 2 microcontrollers I’m using in this project, plus the Teensy I used previously:

If ESP32 boards such as the TinyPICO had a hardware Analog Comparator, I would likely use just one microcontroller, but keep the serial interface on a GPIO for easy debugging.

Why the Adafruit QT Py?

The minimal function we need for our signal conversion device is to convert an SCS signal (5V) to a serial signal (3.3V). For this conversion, we need a hardware analog comparator and an output GPIO that we can drive independently, so that we can modify the signal.

Additionally, the device should use as little power as possible so that it can comfortably fit in the left-over energy budget of the intercom unit’s power supply.

The smallest microcontroller I know of that comes with a hardware analog comparator is the Adafruit QT Py. It’s a 32-bit Cortex M0+ (SAMD21) that can be programmed using the Arduino IDE, or MicroPython (hence the name).

There are other SAMD21 boards with the same form factor, such as the Seeeduino XIAO.

Why the TinyPICO ESP32 board?

When looking for a WiFi-enabled microcontroller, definitely go with something ESP32-based!

The community around the Espressif ESP32 (and its predecessor ESP8266) is definitely one of its biggest pluses: there are tons of Arduino sketches, troubleshooting tips, YouTube videos, reference documentation, forum posts, and so on.

The ESPs have been around since ≈2014, so many (largely-compatible) boards are available. In fact, I started this project on an M5Stack ESP32 Basic Core IoT Development Kit, deployed it on an Adafruit HUZZAH32 Breakout Board and ultimately ported it to the TinyPICO. Porting between the different microcontrollers was really smooth: the only adjustments were pin numbers and dropping in a TinyPICO helper library for its RGB LED, which I chose to use as a power LED.

I chose the TinyPICO ESP32 board specifically for its small form factor and convenience:

The TinyPICO is only 18mm × 32mm, slightly smaller than the Teensy 4.0’s 18mm × 35mm.

In comparison, the Adafruit HUZZAH32 breakout board is gigantic with its 25mm × 44mm. And that’s without the extra USB-to-serial adapter (FT232H in the picture above) you need for programming, serial console and powering the board!

The TinyPICO does not need an extra adapter. You can plug it in and program it immediately, just like the Teensy!

I’d like it if the next revision of the TinyPICO switched from Micro USB to USB C.

If the TinyPICO is not for you (or unavailable), search for other boards that contain the ESP32-PICO-D4 chip. For example, DFRobot’s ESP32-PICO-KIT or Espressif’s own ESP32-PICO-KIT.

Prototype

After testing everything on a breadboard, I soldered a horizontal pin header onto the QT Py, connected it to my Sparkfun level shifter board and soldered the remaining voltage divider components “flying”. The result barely fit into the case, but worked flawlessly for weeks:

Backpack PCB for the QT Py

After verifying this prototype works well in practice, I miniaturized it into a “backpack” PCB.

The backpack contains all the same parts as the prototype, but with fewer bulky wires and connectors, and using only SMD parts. The build you see below uses 0602 SMD parts, but if I made another revision I would probably chose the larger 0805 parts for easier soldering.

Assembly

To save some space in the intercom unit case, I decided to solder the jumper wires directly onto the TinyPICO instead of using a pin header. I could have gone one step further by cutting the wires at length and soldering them directly on both ends, without any connectors, but I wanted to be able to easily unplug and re-combine the parts of this project.

From top to bottom, I made the following connections:

The TinyPICO USB port is still usable for updating the software and serial console debugging.

Here’s the TinyPICO connected to the QT Py inside the intercom unit:

The QT Py is powered by the intercom unit’s supply, and the TinyPICO I’m powering with an external USB power supply and a cut-open USB cable. This allows me to route the jumper wires through the intercom unit’s hole in the back, through which a USB plug doesn’t fit:

Software / Artifacts

You can find the Arduino sketches and KiCad files for this project at https://github.com/stapelberg/intercom-backpack

For debugging, I found it useful to publish every single byte received from the SCS bus on the doorbell/debug/scsrx MQTT topic. Full SCS telegrams are published to doorbell/events/scs, so by observing both, you can verify that retransmission suppression and SCS decoding work correctly.

Similarly, signaling a doorbell ring to the Nuki Opener can be debugged by sending a message to MQTT topic doorbell/debug/cmd/ring.

Initially, it wasn’t clear to me whether the WiFi library would maintain the connection indefinitely. After observing my microcontroller eventually disappearing from my network, I added the taskreconnect FreeRTOS task, and things have been stable since.

Nuki Opener: verdict

I now have a Nuki Opener running next to my own microcontroller, so I can see how well it works.

setup

Setting up the Nuki is the worst part: their colorful cable is super flimsy and loose, often losing contact. They should definitely switch to a cable with a mechanical lock.

The software part of the setup is okay, but the compatibility with the SCS bus is poor: I couldn’t get the device to work at all (see my initial post), and had to resort to using my own microcontroller to drive the Nuki in analogue mode.

I’m disappointed that the Nuki developers aren’t interested in improving their device’s compatibility and reliability with the SCS bus. They seem to capture/replay the entire signal (including re-transmissions) instead of actually decoding the signal.

in my day-to-day

The push notifications I get on my iPhone from the Nuki are often delayed. Usually the delay is a few seconds, but sometimes notifications arrive hours later or just don’t arrive at all!

While the push notifications are sent from a Nuki server and hence need the internet to function, the Nuki Bridge (translating Bluetooth Low Energey from the Nuki Opener to WiFi) allows configuring notifications in the local network via web hooks.

The Nuki Bridge’s notifications are much more reliable in my experience.

People sometimes ask why I use the Nuki Opener at all, given that I have some infrastructure of my own, too. While opening the door and receiving notifications is something I can do without the Nuki, too, I don’t want to spend my spare time re-implementing the Nuki app (on multiple platforms) with its geo fencing, friend invitations, ring to open, etc. In addition, the Nuki Opener physical device has a nice ring sound and large push button to open the door, both of which are convenient.

Conclusion

My intercom is now much smarter! Doorbell notifications make their way to my various devices via MQTT, and I can conveniently open the door from any device, as opposed to rushing to the intercom unit in the hallway.

Compared to the previous proof-of-concepts and development installations, I feel more confident in the current solution because it re-uses the intercom unit for the nitty-gritty SCS bus communication details.

The overall strategy should be widely applicable regardless of the specific intercom vendor/unit you have. Be sure to buy your own unit (don’t solder into your landlord’s intercom unit!) and test in a separate lab setup first, of course!

Debian Code Search now offers an OpenAPI-based API!

Various developers have created ad-hoc client libraries based on how the web interface works.

The goal of offering an OpenAPI-based API is to provide developers with automatically generated client libraries for a large number of programming languages, that target a stable interface independent of the web interface’s implementation details.

Getting started

1. Visit https://codesearch.debian.net/apikeys/ to download your personal API key. Login via Debian’s GitLab instance salsa.debian.org; register there if you have no account yet.

2. Find the Debian Code Search client library for your programming language. If none exists yet, auto-generate a client library on editor.swagger.io: click “Generate Client”.

3. Search all code in Debian from your own analysis tool, migration tracking dashboard, etc.

curl example

curl \
  -H "x-dcs-apikey: $(cat dcs-apikey-stapelberg.txt)" \
  -X GET \
  "https://codesearch.debian.net/api/v1/search?query=i3Font&match_mode=regexp" 

Web browser example

You can try out the API in your web browser in the OpenAPI documentation.

Code example (Go)

Here’s an example program that demonstrates how to set up an auto-generated Go client for the Debian Code Search OpenAPI, run a query, and aggregate the results:

func burndown() error {
	cfg := openapiclient.NewConfiguration()
	cfg.AddDefaultHeader("x-dcs-apikey", apiKey)
	client := openapiclient.NewAPIClient(cfg)
	ctx := context.Background()

	// Search through the full Debian Code Search corpus, blocking until all
	// results are available:
	results, _, err := client.SearchApi.Search(ctx, "fmt.Sprint(err)", &openapiclient.SearchApiSearchOpts{
		// Literal searches are faster and do not require escaping special
		// characters, regular expression searches are more powerful.
		MatchMode: optional.NewString("literal"),
	})
	if err != nil {
		return err
	}

	// Print to stdout a CSV file with the path and number of occurrences:
	wr := csv.NewWriter(os.Stdout)
	header := []string{"path", "number of occurrences"}
	if err := wr.Write(header); err != nil {
		return err
	}
	occurrences := make(map[string]int)
	for _, result := range results {
		occurrences[result.Path]++
	}
	for _, result := range results {
		o, ok := occurrences[result.Path]
		if !ok {
			continue
		}
		// Print one CSV record per path:
		delete(occurrences, result.Path)
		record := []string{result.Path, strconv.Itoa(o)}
		if err := wr.Write(record); err != nil {
			return err
		}
	}
	wr.Flush()
	return wr.Error()
}

The full example can be found under burndown.go.

Feedback?

File a GitHub issue on github.com/Debian/dcs please!

Migration status

I’m aware of the following third-party projects using Debian Code Search:

If you find any others, please point them to this post in case they are not using Debian Code Search’s OpenAPI yet.

This week I started my new position as a postdocotoral researcher at the Applied and Computational Mathematics (ACoM) research lab at RWTH Aachen University. The lab consists of two interdisciplinary research groups, namely the group of Prof. Dr. Manuel Torrilhon, who works on the mathematical modelling and simulation of technical processes (e.g. plasma or gas flow processes) as well as the group of Prof. Dr. Benjamin Stamm, which I am now joining. Ben's research focus is the numerical analysis of PDEs and linear algebra problems, which arise e.g. in electrostatics or quantum chemistry. This includes principle questions related to eigenvalue problems, but also concrete applications such as improving the performance of the polarisable continuum model, a standard solvation model in electronic structure theory. I see a good fit between our respective research backgrounds and I am happy for this opportunity to extend my research horizon and contribute to the research in Ben's group and the ACoM over the next years.

During my time in Aachen Ben and I want to continue to work on the numerical analysis and the development of mathematically-motivated methods for density-functional theory (DFT). One aspect we have in mind, for example, is to port Ben's recent work for constructing good initial guesses for the self-consistent iterations in molecular DFT (and Gaussian basis functions) to plane-wave DFT. As part of this research we will make use and extend the density-functional toolkit (DFTK), the density-functional theory code I started in Paris. Beyond our work at the ACoM I expect DFTK and its suitability for multidisciplinary research also to be helpful for reaching out to other researchers in the mathematics, computer science and physics departments in Aachen. In particular I see a good fit of DFTK within the JARA-CSD, a joint research initiative between RWTH Aachen and the Jülich research centre.

Having known Aachen already a little from my previous visits in Ben's group I am very much looking forward to work here. Not only is the city very pretty and welcoming, but also the interdisciplinary orientation of RWTH Aachen resonates well with me. I'm looking forward to the many interesting discussions to come and to becoming part of strengthening the interdisciplinary links in Aachen, while at the same time continuing to work at the boundary of chemistry, physics and mathematics.

Data compression is an incredibly useful tool in many situations, be it for backups, archiving, or even filesystems. But of the many compressors that there are, which of them is the one to be preferred? To properly answer this question, we first have to answer a different, but related one: What exactly makes a compression algorithm good?

Time for a closer investigation to find the best of the best of the best!

Pareto optimality or what makes a compression algorithm good

Pareto optimality is a situation where no […] preference criterion can be better off without making at least one […] preference criterion worse off […].

The concept of Pareto Optimality is tremendously useful, far beyond the scope of this blogpost, by acknowledging competing interests.

In our particular case of optimal compression, one of the obvious candidates is compression size. The smaller the resulting file is, the better the compressor, right?

But what if we are just compressing so we can speed up upload to a remote location? At this point, it doesn’t matter if the compression is supremely superior to everything else if we have to wait twice as long for it to happen compared to simply upload the uncompressed file.

So for our algorithms, we have at least two criteria that come into play: actual achievable compression and compression cost, which we will measure as “how long does it take to compress our content?"¹. These two are the criteria this blogpost is focused on.

A practical example

Practically speaking, let’s assume two compression tools A and B, having both two compression levels 1 and 2, which we use on a sample file. The results might look like the following:

We find that while both algorithms increase their compression size with level, B.1 is unconditionally better than A.2, so there is no reason to ever use A.2 if B.1 is available. Besides that, A.1, B.1 and B.2 continuously increase in compression effectiveness as well as time taken for that. For easier digestion, we can visualize these results:

Here we clearly see what could already be taken from the table above, but significantly more intuitive. Remembering our definition of Pareto optimality from before, we see that A.2 is not Pareto optimal, as B.1 is both better in time taken as well as in compression effect. This shows more intuitively that in this scenario there is no reason to use A.2 if B.1 is available. For A.1, B.1 and B.2 the choice is not so clear-cut, as the resulting file size can only be reduced further by investing more time into the compression. Hence, all three of them are Pareto optimal and constitute the Pareto frontier or the Pareto set.

One wants to always strive for choosing a Pareto optimal solution whenever possible, as non-Pareto-optimal solutions are always to some degree wasteful.

With this insight into Pareto optimality, we can now put our knowledge into practice and begin our journey to the Pareto frontiers of compression algorithms.

Setup and test procedure for real-world measurements

Data was gathered on a Linux system with a given sample file for the resulting filesize compared to the original as well as the time the process took for compressing. First, the sample file was read from disk entirely, ensuring it would be present in Linux' filesystem cache, then compressed with each compression program at each level 15 times for a decent statistic, the compressed result is routed directly into /dev/null. All presented compression times presented are the median of these 15 runs, compression size was only measured once after verifying that it was deterministic².

Unless explicitly denoted otherwise, all compressors were run in their default configuration with a single compression thread. Furthermore, all applications on the machine used for benchmarking except the terminal holding the benchmarking process were closed to reduce interference by other processes as much as possible.

The tests for decompression were done analogously (with the obvious exception that the sample file is compressed according to the testcase in play beforehand), single-threaded decompression as well as routing the decompressed result to /dev/null.

All tests were run on several different CPUs: Intel i7-8550U, Intel i5-3320M³ (both mobile CPUs), Intel i7-7700K and AMD Ryzen 7 3800X (both workstation CPUs). While the absolute compression times changed, the Pareto frontiers did not, hence in this blogpost only the plots for the i7-8550U are shown exemplarily.

Case study: initramfs-compression

Just recently, the current maintainer of mkinitcpio, Giancarlo Razzolini, announced the transition from gzip to zstd as the default for initramfs-compression. The initramfs is a little mini-system, whose only objective is to prepare the hardware and assemble all disks so that the main system is readily prepared to take over operation. The initramfs needs to be loaded (and hence decompressed) at every boot as well as recreated occasionally, when components contained in it are updated.

mkinitcpio supports a variety of compression algorithms: none, gzip, bzip2, lzma, lzop, lz4 and most recently zstd.

Quantifying these algorithms and plotting the Pareto frontier for compression yields:

We find that the change of defaults from gzip to zstd was well justified, as gzip can no longer be considered Pareto optimal for this type of file. Choosing lzma as the default would make it even smaller, but this would be paid by a noticable higher resource usage for compression (which has to be invested on every update affecting the initramfs), so from the data zstd is certainly the wiser choice⁴.

This can also be seen when we take a look of Pareto optimality of decompression (after all, this decompression needs to happen on every single system boot):

We clearly see that zstd is blasting all other algorithms out of the water when it comes to decompression speed, making it even more of a good choice for this use case. Given these numbers, it is even more of a good choice for an initramfs, not only does it compress fast, it also decompresses impressively fast, six times faster than lzma which was previously known for its quick decompression speed despite high compression factors.

Given the data for zstd, it cannot be ruled out completely that zstd simply hit a non-CPU-bound on decompression, but even if it did, the conclusion for the choice of algorithm does not change.

dracut

If you happen to be on a Linux distribution that uses dracut for initramfs-generation, the conclusions that can be drawn for dracut-initramfs-compressions are the almost identical given the compression and decompression data for a dracut-initrd, the Pareto frontier remains mostly unchanged with just some more levels of zstd in it.

Real world usage recommendation

To use zstd in mkinitcpio, simply use version 30 and above. You can modify the compression level (default is -3) by adding a specific level to COMPRESSION_OPTIONS, but given the data, this doesn’t seem to provide much of a benefit.

For dracut, add compress="zstd" to /etc/dracut.conf to get zstd compression at the default level.

Case study: borgbackup

In the next scenario we will investigate the impact of compression when backing up data with borgbackup. Borg comes with an integrated option to compress the backupped data, with algorithms available being lz4, zstd, zlib/gzip and lzma. In addition to this, borg has an automated detection routine to see if the files backupped do actually compress enough to spend CPU cycles on compressing (see borg help compression for details).

For this scenario, we do not only have one definite sample, but will consider three different samples: A simple text file, being a dump of the Kernel log via the dmesg command, representing textual data. A binary, in this particular case the dockerd ELF binary, (rather arbitrarily) representing binary data. Finally, an mkinitcpio-image, intended as somewhat of a mixed-data sample. We do not consider media files, as these are typically already stored in compressed formats, hence unlikely to compress further and thus dealt with by borg’s compressibility heuristics.

The resulting Pareto frontiers for compression are, starting with least effective compression:

We see that effectively, except for some brief occurrance of lz4 at the top, the relevant choices are lzma and zstd. More details can be seen in the plots linked in the column headers. Hence, as backups should be run often (with a tool as borg there is little reason for anything else but daily), zstd with a level slightly below 10 seems to be a good compromise of speed and resulting data size.

Real world usage recommendation

Adding --compression=auto,zstd,7 to the borg command used to create a backup will use zstd on level 7 if borgs internal heuristics considers the file in question to compress well, otherwise no compression will be used.

This flag can be added on-the fly, without affecting existing repositories or borgs deduplication. Already backupped data is not recompressed, meaning that adding this flag for use with an existing repository does not require a reupload of everything. Consequentially, it also means that to recompress the entire repository with zstd one effectively has to start from scratch.

Case study: Archiving with tar

Compression can not only be used for backup-tools like borg, it can also be used to archive files with tar. Some compressors have explicit flags in tar, such as gzip (-z), lzma (-J) or bzip2 (-j), but any compression algorithm can be used via tar’s -I flag.

Working with tar poses two challenges with regard to the compressor:

• streaming: tar concatenates data and streams the result through the compressor. Hence, to extract files at a certain position of a tarball, the entirety of the data before that file needs to be decompressed as well.
• compress-only: as a further consequence of this, tar lacks a feature testing if something is well compressible, so uncompressible data will also be sent through the compressor

If we want to investigate a good compressor without consideration for the input data, we aim for picking a Pareto optimal compressor that takes two properties into consideration:

• fast decompression, to be able to easily and quickly extract data from the archive again if need be
• good performance on non-compressible data (this particularly means that incompressible data should especially not increase in size).

Compression capabilities

To investigate the decompression capabilities of certain compressors, we can reuse the dataset used on the borg case study and add some incompressible data in form of a flac music file to the mix. As tar has a larger variety of usable algorithms, we include lz4, gzip, lzma, zstd, lzop and brotli as well.

We can exemplarily see the effectiveness of these on the dockerd elf binary (other datasets can be found below) first:

In summary, the Pareto frontiers for the different types of data overall turn out to be (with c for compression and d for decompression):

As can be seen in the linked plots, we again find that while lzma still achieves the highest absolute compression, zstd dominates the sweet spot right before computational cost skyrockets. We also find brotli to be an interesting contender here, making it into the Pareto frontier as well. However, with only sometimes making it into the Pareto frontier, whereas lzma and zstd robustly defend their inclusion in it, it seems more advisable to resort to either lzma or zstd as this only provides a sample binary and actual data might vary. Furthermore, when it comes to decompression brotli is not Pareto optimal anymore at all, also indicating lzma and zstd as being the better choice.

Impact on incompressible files in detail

We will take another closer look at the incompressible case, represented by a flac file and strip away bzip2 and lzma, as we could tell from the linked plots that these two clearly increase the size of the result and are hence not Pareto optimal (as they are already beaten by the Pareto optimal case “no compression”).

The results have a clear indication:

The recommended choice of algorithm for compression is either brotli or zstd, but when it comes to decompression, zstd takes the lead again. This is of course a cornercase, the details of this might change with the particular choice of incompressible data. However, I do not expect the overall impression to significantly change.

Real world usage recommendation

Concluding this section, the real-world recommendation resulting from this seems to simply use zstd for any tarball compression if available. To do so, tar "-Izstd -10 -T0" can be a good choice, with -T0 telling zstd to parallelize the compression onto all available CPU cores, speeding things up even more beyond our measurements. Depending on your particular usecase it might be interesting to use an alias like

alias archive='tar "-Izstd -19 -T0" -cf'

which allows quickly taring data into a compressed archive via archive myarchive.tar.zst file1 file2 file3 ….

Case study: filesystems

Another usecase for compression is filesystem compression. Conceptually similar to what borg does, files are transparently compressing and decompressed when written to or read from disk.

Among the filesystems capable of such inline-compression are ZFS and btrfs. btrfs supports ZLIB (gzip), LZO (lzop) and zstd, whereas ZFS supports lz4, LZJB (which was not included in these benchmarks as no appropriate binary compressor was found), gzip and ZLE (zero-length-encoding, only compressing zeroes, hence also not tested). zstd support for OpenZFS has been merged, but apparently hasn’t made it into any stable version yet at time of writing according to the OpenZFS documentation.

This case is situated similar to the tar-case study, and as all compressors available for ZFS and btrfs have already been covered in the section above, there is no reason to reiterate these results here.

It shall, however, be noted, that at least for btrfs, the standard flag for filesystem compression adopts a similar heuristic as borg and hence the case of incompressible data might not be so relevant for a btrfs installation. That being said, the conclusion here is a recommendation of zstd, and as we have seen in the last section, the question of incompressible files doesn’t change the overall recommendation.

Real world usage recommendation

If you want to save diskspace by using compression, the mount option compress in combination with zstd is generally a good choice for btrfs. This also includes the compressibility-heuristics (compress-force would be the option that compresses without this heuristics). For ZFS, the general recommendation is consequentially also zstd once it makes its way into a release.

Conclusion

Concluding the experiments laid out in this blogpost, we can effectively state an almost unconditional and surprisingly clear recommendation to simply use zstd for everything. The exact level might depend on the usecase, but overall it has demonstrated to be the most versatile, yet effective compressor around when it comes to effectiveness and speed, both for compression and especially decompression. Furthermore, it has, in contrast to most other contenders, flags for built-in parallelization, which not used in this blogpost at all, and yet zstd still stomped almost the entire competition.

Only if resulting filesize should be pushed down as much as possible, without any regard for computational cost, lzma retains an edge for most kinds of data. In practice, however, the conclusion is to simply use zstd.

_{Thanks to Joru and corvus for proofreading and helpful comments.}

A bit over a year ago we published our adcc code. In this work the aim was to develop a toolkit for computational spectroscopy methods focused on rapid development and interactive hands-on usage (see the blog article for details). Our target back then was to simplify method development involving the algebraic-diagrammatic construction approach (ADC) to compute excited states energies and properties. ADC has been a research focus both of myself as well as the group of Andreas Dreuw and the ADC family of methods have proven in the past to be greatly suited for describing photochemistry and spectroscopic results.

Employing mainly thread-based parallelism and (apart from our recent inclusion of libxm) basically no options for swapping stored tensors to disk, adcc is naturally restricted to problems that fit into the main memory of a single cluster node. This is fine for developing and testing new ADC methods, but can be limiting for employing ADC methods in practice: The code can currently only treat small-sized to medium-sized molecules.

In parallel to adcc we therefore started working on the Gator project in collaboration with the groups of Patrick Norman and Zilvinas Rinkevicius (both KTH Stockholm). We now release in a first version. Apart from an interface to adcc, Gator features a response library capable of the complex polarisation propagator (CPP) approach for simulating properties such as excited-states polarisabilities or enabling a direct computation of spectra including broadening. Additionally it contains a newly developed ADC(2) module with MPI-based distributed computing capabilities. For this the integral driver of the Veloxchem code from KTH is used, which allows the ADC(2) computation to be performed in a direct fashion (i.e. without storing the two-electron-integral tensor). This makes ADC(2) simulations in Gator more memory efficient and allows them to be distributed over a few cluster nodes. In this publication we provide an overview of Gator's current capabilities. The full abstract reads

The Gator program has been developed for computational spectroscopy and calculations of molecular properties using real and complex propagators at the correlated level of wave function theory. At present, the focus lies on methods based on the algebraic diagrammatic construction (ADC) scheme up to third-order of perturbation theory. A Fock matrix-driven implementation of the second-order ADC method for excitation energies has been realized with an underlying hybrid MPI/OpenMP parallelization scheme suitable for execution in high-performance computing cluster environments. With a modular and object-oriented program structure written in a Python/C++ layered fashion, Gator enables, in addition, time-efficient prototyping of novel scientific approaches as well as interactive notebook-driven training of students in quantum chemistry.

Automatically resolving and installing dependencies is one of the core features of package managers (and one of the most convenient). However, this can lead to packages being installed that have been pulled as a dependency for another package, but are no longer needed¹. This can have slightly unfortunate side effects, such as crowding the upgrade dialog when a bunch of packages that are no longer needed receive updates in high frequency (looking at you, Haskell packages in Arch…).

To ease house cleaning, at least on distros using pacman, we can simply find these unneeded packages that used to be installed as a dependency via pacman -Qtd². This will list all unrequired packages that have been originally installed as a dependency instead of being specifically installed by the user.

We could either periodically run this by hand, or we can simply tell pacman to just do it for us regularly. To do so, we create a new file /etc/pacman.d/hooks/unneeded-packages.hook³ containing

[Trigger]
Operation = Install
Operation = Upgrade
Operation = Remove
Type = Package
Target = *

[Action]
Description = "Checking for unneeded packages"
When = PostTransaction
Exec = /usr/bin/pacman -Qtd

This will run pacman -Qtd on every pacman-operation that can change our package state for every possible target package after all package transactions have been completed.

While this works, the output is not the most neatly arranged, so we can just modify our Exec a little to make it more beautiful, either by calling a script that does that or simply by a semi-beautiful, but compact one-liner, changing our pacman-hook to

[Trigger]
Operation = Install
Operation = Upgrade
Operation = Remove
Type = Package
Target = *

[Action]
Description = "Checking for unneeded packages"
When = PostTransaction
Exec = /usr/bin/bash -c "set -o pipefail && /usr/bin/pacman -Qtd | sed 's/^/  - /'  || /usr/bin/echo '  :: No unneeded packages found.'"

This now neatly displays superfluous packages right after ever pacman -Syu, so you can clean up right away if you really don’t need those packages anymore.

Cleaning up superfluous packages

The quickest way of removing superfluous packages (and all its dependencies, that are only needed by these packages) is by running pacman -Rnsc $(pacman -Qdtq)², which can be easily aliased to something like pacclean, being readily available at your fingertips afterwards without much typing.

off-topic remark: dependency tracking

On the note of package dependencies: if you want to find out which package(s) pulled in a specific package, pactree -r specific_package_name will give you insight into that.

These past two days I have participated in the Tri-Lab Support Team (TST) meeting of the CESMIX, the newly founded Center for the Exascale Simulation of Material Interfaces in Extreme Environments at the Massachusetts Institute of Technology. Within the next few years the idea of the CESMIX is to develop a multi-level simulation stack all the way up from DFT over MD to flow simulations to be able to discover novel materials suitable for extremely high temperatures under atmospheric conditions. The prototypical application for such materials would be heat shields for example in space crafts returning to earth or supersonic planes.

One novel aspect of the project is to include progress from modern compiler techniques and programming language design when building the software stack. In particular the challenge is that multiple codes will be involved in the project that feature a large variety of programming languages (FORTRAN, C++, Julia, python, ...). On top of that one goal is to keep track of simulation errors using uncertainty quantification (UQ) and use that insight to construct a multi-fidelity workflow. In such an approach the data generation for the simulation does not only employ a single accurate model, but in fact features multiple simulation layers based on cheaper and cruder models as well as more costly and accurate ones. Using the deduced knowledge of the error one can dynamically switch between these models and reach a compromise between accuracy and computational cost, but in a way that the result still has the quality to be comparable to experiments. As the employed fidelity layers the CESMIX project targets classical molecular dynamics or ab initio molecular dynamics with various kinds of density-functional theory (DFT) methods ... and this is the angle of my involvement in the project.

In particular using our density-functional toolkit (DFTK) the idea is to be able to quickly prototype parts of the workflow. Then, building on DFTK's design as a multi-disciplinary platform (see my related blog articles), we want to start incorporating new techniques (GPU platforms, UQ, multi-fidelity) to see how they could fit. I am excited about the opportunity to contribute to a project, which shares a lot in philosophy to DFTK itself. In particular I am looking forward to seeing how DFTK will play out in a real-world research scenario for connecting the needs from the modelling side with the approaches of the computer science folks.

With respect to the TST meeting, I briefly gave an overview about DFTK to show where we are. Slides and a short demo are attached below.

Link	Licence
DFTK.jl: A multidisciplinary Julia code for density-functional theory development (Slides)
Benchmarks and DFTK demo (Tarball)

While I had heard the abbreviation MQTT many times, I never had a closer look at what MQTT is.

Here are a few quick notes about using MQTT as Pub/Sub bus in a home IOT network.

Motivation

Once you have a few IOT devices, an obvious question is how to network them.

If all your devices are from the same vendor, the vendor takes care of it.

In my home, I have many different vendors/devices, such as (incomplete list):

• Nuki Opener Smart Intercom
• Sonoff S26 Smart Plug (WiFi-controlled socket outlet)
• Aqara Door & Window Sensors
• IKEA Home Smart (formerly TRÅDFRI) Smart Lights

Here is how I combine these devices:

• When I’m close to my home (geo-fencing), the Nuki Opener enables Ring To Open (RTO): when I ring the door bell, it opens the door for me.
• When I open the apartment door, the Smart Lights in the hallway turn on.
• When I’m home, my stereo speakers should be powered on so I can play music.

A conceptually simple way to hook this up is to connect things directly: listen to the Aqara Door Sensor and instruct the Smart Lights to turn on, for example.

But, connecting everything to an MQTT bus has a couple of advantages:

1. Unification: everything is visible in one place, the same tools work for all devices.
2. Your custom logic is uncoupled from vendor details: you can receive and send MQTT.
3. Compatibility with existing software, such as Home Assistant or openHAB

Step 1. Set up an MQTT broker (server)

A broker is what relays messages between publishers and subscribers. As an optimization, the most recent value of a topic can be retained, so that e.g. a subscriber does not need to wait for the next change to obtain the current state.

The most popular choice for broker software seems to be Mosquitto, but since I like to run Go software on https://gokrazy.org/, I kept looking and found https://github.com/fhmq/hmq.

One downside of hmq might be that it does not seem to support persisting retained messages to disk. I’ll treat this as a feature for the time being, enforcing a fresh start on every daily reboot.

To restrict hmq to only listen in my local network, I’m using gokrazy’s flag file feature:

mkdir -p flags/github.com/fhmq/hmq
echo --host=10.0.0.217 > flags/github.com/fhmq/hmq/flags.txt

Note that you’ll need https://github.com/fhmq/hmq/pull/105 in case your network does not come up quickly.

MQTT broker setup: displaying/sending test messages

To display all messages going through your MQTT broker, subscribe using the Mosquitto tools:

% sudo pacman -S mosquitto
% mosquitto_sub --id "${HOST}_all" --host dr.lan --topic '#' --verbose

The # sign denotes an MQTT wildcard, meaning subscribe to all topics in this case.

Be sure to set a unique id for each mosquitto_sub command you run, so that you can see which subscribers are connected to your MQTT bus. Avoid id clashes, otherwise the subscribers will disconnect each other!

Now, when you send a test message, you should see it:

% mosquitto_pub --host dr.lan --topic 'cmnd/tasmota_68462F/Power' -m 'ON'

Tip: If you have binary data on your MQTT bus, you can display it in hex with timestamps:

% mosquitto_sub \
  --id "${HOST}_bell" \
  --host dr.lan \
  --topic 'doorbell/#' \
  -F '@Y-@m-@dT@H:@M:@S@z : %t : %x'

Step 2. Integrate with MQTT

Now that communication via the bus works, what messages do we publish on which topics?

MQTT only defines that topics are hierarchical; messages are arbitrary byte sequences.

There are a few popular conventions for what to put onto MQTT:

• The Homie convention

• Home Assistant has its own convention, but allows full configuration. Home Assistant does not support the homie convention yet.

• openHAB refers to Home Assistant and Homie.

If you design everything yourself, Homie seems like a good option. If you plan to use Home Assistant or similar, stick to the Home Assistant convention.

Best practices for your own structure

In case you want/need to define your own topics, keep these tips in mind:

• devices publish their state on a single, retained topic
  • the topic name could be e.g. stat/tasmota_68462F/POWER
  • retaining the topic allows consumers to catch up after (re-)connecting to the bus
• publish commands on a single, possibly-retained topic
  • e.g. publish ON to topic cmnd/tasmota_68462F/Power
  • publish the desired state: publish ON or OFF instead of TOGGLE
  • if you retain the topic and publish TOGGLE commands, your lights will mysteriously go off/on when they unexpectedly re-establish their MQTT connection

Integration: Shelly devices with MQTT built-in

Shelly has a number of smart devices that come with MQTT out of the box! This sounds like the easiest solution if you’re starting from scratch.

I haven’t used these devices personally, but I hear good things about them.

Integration: Zigbee2MQTT for Zigbee devices

Zigbee2MQTT supports well over 1000 Zigbee devices and exposes them on the MQTT bus.

For example, this is what you would use to connect your IKEA TRÅDFRI Smart Lights to MQTT.

Integration: ESPHome for micro controllers + sensors

The ESPHome system is a ready-made solution to connect a wide array of sensors and devices to your home network via MQTT.

If you want to use your own ESP-based micro controllers and sensors, this seems like the easiest way to get them programmed.

Integration: Mongoose OS for micro controllers

Mongoose OS is an IOT firmware development framework, taking care of device management, Over-The-Air updates, and more.

Mongoose comes with MQTT support, and with just a few lines you can build, flash and configure your device. Here’s an example for the NodeMCU (ESP8266-based):

% yay -S mos-bin
% mos clone https://github.com/mongoose-os-apps/demo-js app1
% cd app1
% mos --platform esp8266 build
% mos --platform esp8266 --port /dev/ttyUSB1 flash
% mos --port /dev/ttyUSB1 config-set mqtt.enable=true mqtt.server=dr.lan:1883

Pressing the button on the NodeMCU publishes a message to MQTT:

% mosquitto_sub --host dr.lan --topic devices/esp8266_F4B37C/events
{"ram_free":31260,"uptime":27.168680,"btnCount":2,"on":false}

Integration: Arduino for custom micro controller firmware

Arduino has an MQTT Client library. If your microcontroller is networked, e.g. an ESP32 with WiFi, you can publish MQTT messages from your Arduino sketch:

#include <WiFi.h>
#include <PubSubClient.h>

WiFiClient wificlient;
PubSubClient client(wificlient);

void callback(char* topic, byte* payload, unsigned int length) {
    Serial.print("Message arrived [");
    Serial.print(topic);
    Serial.print("] ");
    for (int i = 0; i < length; i++) {
      Serial.print((char)payload[i]);
    }
    Serial.println();
  
    if (strcmp(topic, "doorbell/cmd/unlock") == 0) {
  		// …
    }
}

void taskmqtt(void *pvParameters) {
	for (;;) {
		if (!client.connected()) {
			client.connect("doorbell" /* clientid */);
			client.subscribe("doorbell/cmd/unlock");
		}

		// Poll PubSubClient for new messages and invoke the callback.
		// Should be called as infrequent as one is willing to delay
		// reacting to MQTT messages.
		// Should not be called too frequently to avoid strain on
		// the network hardware:
		// https://github.com/knolleary/pubsubclient/issues/756#issuecomment-654335096
		client.loop();
		vTaskDelay(pdMS_TO_TICKS(100));
	}
}

void setup() {
	connectToWiFi(); // WiFi configuration omitted for brevity

	client.setServer("dr.lan", 1883);
	client.setCallback(callback);

	xTaskCreatePinnedToCore(taskmqtt, "MQTT", 2048, NULL, 1, NULL, PRO_CPU_NUM);
}

void processEvent(void *buf, int telegramLen) {
	client.publish("doorbell/events/scs", buf, telegramLen);
}

Integration: Webhook to MQTT

The Nuki Opener doesn’t support MQTT out of the box, but the Nuki Bridge can send Webhook requests. In a few lines of Go, you can forward what the Nuki Bridge sends to MQTT:

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"net/http"

	mqtt "github.com/eclipse/paho.mqtt.golang"
)

func nukiBridge() error {
	opts := mqtt.NewClientOptions().AddBroker("tcp://dr.lan:1883")
	opts.SetClientID("nuki2mqtt")
	opts.SetConnectRetry(true)
	mqttClient := mqtt.NewClient(opts)
	if token := mqttClient.Connect(); token.Wait() && token.Error() != nil {
		return fmt.Errorf("MQTT connection failed: %v", token.Error())
	}

	mux := http.NewServeMux()
	mux.HandleFunc("/nuki", func(w http.ResponseWriter, r *http.Request) {
		b, err := ioutil.ReadAll(r.Body)
		if err != nil {
			log.Print(err)
			http.Error(w, err.Error(), http.StatusInternalServerError)
			return
		}

		mqttClient.Publish(
			"zkj-nuki/webhook", // topic
			0, // qos
			true, // retained
			string(b)) // payload
	})

	return http.ListenAndServe(":8319", mux)
}

func main() {
	if err := nukiBridge(); err != nil {
		log.Fatal(err)
	}
}

See Nuki’s Bridge HTTP-API document for details on how to configure your bridge to send webhook callbacks.

Step 3. Express your logic

Home Assistant and Node-RED are both popular options, but also large software packages.

Personally, I find it more fun to express my logic directly in a full programming language (Go).

I call the resulting program regelwerk (“collection of rules”). The program consists of:

1. various control loops that progress independently from each other
2. an MQTT message dispatcher feeding these control loops
3. a debugging web interface to visualize state

This architecture is by no means a new approach: as moquette describes it, this is to MQTT what inetd is to IP. I find moquette’s one-process-per-message model to be too heavyweight and clumsy to deploy to https://gokrazy.org, so regelwerk is entirely in-process and a single, easy-to-deploy binary, both to computers for notifications, or to headless Raspberry Pis.

regelwerk: control loops definition

regelwerk defines a control loop as a stateful function that accepts an event (from MQTT) and returns messages to publish to MQTT, if any:

type controlLoop interface {
	sync.Locker

	StatusString() string // for human introspection

	ProcessEvent(MQTTEvent) []MQTTPublish
}

// Like mqtt.Message, but with timestamp
type MQTTEvent struct {
	Timestamp time.Time
	Topic     string
	Payload   interface{}
}

// Parameters for mqtt.Client.Publish()
type MQTTPublish struct {
	Topic    string
	Qos      byte
	Retained bool
	Payload  interface{}
}

regelwerk: MQTT dispatcher

Our MQTT message handler dispatches each incoming message to all control loops, in one goroutine per message and loop. With typical message volumes on a personal MQTT bus, this is a simple yet effective design that brings just enough isolation.

type mqttMessageHandler struct {
	dryRun bool
	loops  []controlLoop
}

func (h *mqttMessageHandler) handle(client mqtt.Client, m mqtt.Message) {
	log.Printf("received message %q on %q", m.Payload(), m.Topic())
	ev := MQTTEvent{
		Timestamp: time.Now(), // consistent for all loops
		Topic:     m.Topic(),
		Payload:   m.Payload(),
	}

	for _, l := range h.loops {
		l := l // copy
		go func() {
			// For reliability, we call each loop in its own goroutine
			// (yes, one per message), so that when one loop gets stuck,
			// the others still make progress.
			l.Lock()
			results := l.ProcessEvent(ev)
			l.Unlock()
			if len(results) == 0 {
				return
			}
			for _, r := range results {
				log.Printf("publishing: %+v", r)
				if !h.dryRun {
					client.Publish(r.Topic, r.Qos, r.Retained, r.Payload)
				}
			}
			// …input/output logging omitted for brevity…
		}()
	}
}

regelwerk: control loop example

Now that we have the definition and dispatching out of the way, let’s take a look at an actual example control loop.

This control loops looks at whether my PC is unlocked (in use) or whether my phone is home, and then turns off/on my stereo speakers accordingly.

The inputs come from runstatus and dhcp4d, the output goes to a Sonoff S26 Smart Power Plug running Tasmota.

type avrPowerLoop struct {
	statusLoop // for l.statusf() debugging

	midnaUnlocked          bool
	michaelPhoneExpiration time.Time
}

func (l *avrPowerLoop) ProcessEvent(ev MQTTEvent) []MQTTPublish {
	// Update loop state based on inputs:
	switch ev.Topic {
	case "runstatus/midna/i3lock":
		var status struct {
			Running bool `json:"running"`
		}
		if err := json.Unmarshal(ev.Payload.([]byte), &status); err != nil {
			l.statusf("unmarshaling runstatus: %v", err)
			return nil
		}
		l.midnaUnlocked = !status.Running

	case "router7/dhcp4d/lease/Michaels-iPhone":
		var lease struct {
			Expiration time.Time `json:"expiration"`
		}
		if err := json.Unmarshal(ev.Payload.([]byte), &lease); err != nil {
			l.statusf("unmarshaling router7 lease: %v", err)
			return nil
		}
		l.michaelPhoneExpiration = lease.Expiration

	default:
		return nil // event did not influence our state
	}

	// Publish desired state changes:
	now := ev.Timestamp
	phoneHome := l.michaelPhoneExpiration.After(now)
	anyoneHome := l.midnaUnlocked || (now.Hour() > 8 && phoneHome)
	l.statusf("midnaUnlocked=%v || (now.Hour=%v > 8 && phoneHome=%v)",
		l.midnaUnlocked, now.Hour(), phoneHome)

	payload := "OFF"
	if anyoneHome {
		payload = "ON"
	}
	return []MQTTPublish{
		{
			Topic:    "cmnd/tasmota_68462F/Power",
			Payload:  payload,
			Retained: true,
		},
	}
}

Conclusion

I like the Pub/Sub pattern for home automation, as it nicely uncouples all components.

It’s a shame that standards such as The Homie convention aren’t more widely supported, but it looks like software makes up for that via configuration options.

There are plenty of existing integrations that should cover most needs.

Ideally, more Smart Home and IOT vendors would add MQTT support out of the box, like Shelly.

Was in diesem Post als System Unit bezeichnet wird, ist eine systemd Unit deren Prozesse im system.slice laufen, was den meisten Service Units entspricht. Unter systemd haben Benutzer standardmäßig nicht die Rechte den Status solcher Units via systemd selbst zu beeinflussen. Es ist jedoch möglich Benutzern oder Gruppen für bestimmte Befehle dies mittels polkit zu erlauben.

systemd prüft bei der Ausführung der meisten Befehle mittels polkit, ob der den Befehl initierende Benutzer dies darf, bei Befehlen wie systemctl status oder systemctl show geschieht dies nicht. Für die Befehle zum Start, Stop oder Neustart einer Unit kann man diese Befugnis sehr spezifisch erteilen, bei Befehlen die über eine andere Policy gehandhabt werden ist eine spezifische Einschränkung nicht möglich, zum Beispiel das De-/Aktivieren von Units. Bei systemctl start|stop|restart wird org.freedesktop.systemd1.manage-units verwendet, für systemctl enable|disable ist es org.freedesktop.systemd1.manage-unit-files, auch wird im systemd Code letzterer beider Befehle nicht nach so vielen spezifischen Details gefragt wie bei den ersten drei.

Ein beispielhafter Anwendungsfall für solche Regeln wäre es, wenn eine Benutzergruppe auf einem Entwicklungsserver in keiner Form als root operieren darf. Den Entwicklern soll es jedoch möglich sein, die von ihnen entwickelte Software auf den Server zu deployen und exakt in der Form auszuführen, in der die Software später in einer produktiven Landschaft operiert. Hierzu bedürfe es also einer globalen Service Unit und einer polkit Regel, die es den Benutzern erlaubt den Service via systemctl start|stop|restart zu handhaben. Ob dies nun menschliche Benutzer oder der Benutzer einer continuous integration Lösung ist, der die Ergebnisse des Builds deployed sei dahingestellt, auf magische Art und Weise geschieht das Deployment natürlich auch immer absolut korrekt.

Die Service Unit foobar.service

[Unit]
Description=foobar is the most creative name for anything in the whole universe.

[Service]
Type=simple
ExecStart=/opt/foobar/executable
User=foobar
Group=foobar
# Imagine lots of other useful settings here

[Install]
WantedBy=multi-user.target

Die dazugehörige polkit Regel /etc/polkit-1/10-foobar.rules

polkit.addRule(function(action, subject) {
    if (action.id == "org.freedesktop.systemd1.manage-units" &&
        (subject.isInGroup("developer") && action.lookup("unit") == "foobar.service" &&
        (action.lookup("verb") == "start" ||
         action.lookup("verb") == "stop"  ||
         action.lookup("verb") == "restart"))) {
        return polkit.Result.YES;
    }
});

Der für granularere Restriktion ist neben der Gruppen auch der Benutzername als subject.user verfügbar.

Last Thursday I was invited to give a virtual talk at the Scientific Computing Seminar of working group of Prof. Nicolas Gauger at TU Kaiserslautern. Since the research in Prof. Gauger's group mostly concerns topics which are not directly related to electronic structure theory and density-functional theory (DFT), I chose to present my current research from a rather broad and introductory angle. Main focus of my talk was thus to hint at the interdisciplinary challenges arising in high-throughput methods in DFT simulations, followed by a summary of a few of my recent projects in the field.

I was glad for the opportunity to spread the word about the difficulties with high-throughput methods in DFT. Firstly because I think it is an absolutely fascinating topic, but secondly because it is one where input from fields beyond the standard "culprits" of chemistry, physics and materials science is beneficial to solve the upcoming problems. In fact exactly this hope to get other fields and people with non-standard backgrounds involved was one of the driving forces behind our density-functional toolkit DFT code, which I also briefly presented.

I hope that my talk got some of the audience more interested in DFT and I look forward to continue the discussions at a later point, hopefully meeting the Gauger group in person in Kaiserslautern.

Link	Licence
High-throughput density-functional theory calculations: An interdisciplinary challenge (Slides)

I recently bought a Nuki Opener, which “turns your existing intercom into a smart door opener”.

Unfortunately, I have had a lot of trouble getting it to work.

I finally got the device working by interjecting my own micro controller between the intercom bus and the Nuki Opener, then driving the Nuki Opener in its Analogue mode:

The rest of this article outlines how this setup works at a high level.

Prerequisites

For reliable interpretation and transmission of SCS bus data, we’ll need:

1. SCS receive/transmit circuits. These can be prototyped on a breadboard if you have the required diodes, transistors, resistors and capacitors.

2. A microcontroller with an Analog Comparator. If your microcontroller has one, you’ll find a corresponding section in the datasheet. This function is sometimes abbreviated to CMP or AC, or might be part of a larger Analog/Digital Converter (ADC).

3. A UART (serial) decoder. Most microcontrollers have at least one UART, but if you don’t have one available for whichever reason, you could use a software UART implementation, too.

SCS receive circuit

An R-C network, directly connected to the SCS bus, is used for incoming signal conditioning.

The resistor values have been chosen to divide the voltage of the input signal from 28V down to approx. 2V, i.e. well within the 0-3.3V range for modern microcontroller GPIO pins.

A zener diode limits the 28V level to 3.3V, which should be safe for most microcontrollers.

Simulation: https://tinyurl.com/yxhrkejn

SCS transmit circuit

We directly connect the gate of a mosfet transistor to a GPIO pin of our microcontroller, so that when the microcontroller drives the pin high, we use the 100Ω resistor to attach a load to the SCS bus.

For comparison, the KNX bus, which is similar to the SCS bus, uses a 68Ω resistor here.

Simulation: https://tinyurl.com/y6nv4yg7

SCS lab setup

Use a lab power supply to generate 28V DC. I’m using the Velleman LABPS3005SM because it was in stock at Galaxus, but any power supply rated for at least 30V DC will do.

As the DIY home automation blog entry “A minimal KNX setup” describes, you’ll need to place a 47Ω resistor between the power line and your components.

Afterwards, just connect your components to the bus. The supply/ground line of a breadboard will work nicely.

Micro Controller choice

In this blog post, I’m using a Teensy 4 development board that is widely available for ≈20 USD:

With its 600 MHz, the Teensy 4 has enough sheer clock frequency to allow for sloppier coding while still achieving high quality input/output.

The teensy tiny form factor (3.5 x 1.7 cm) works well for this project and will allow me to store the microcontroller in an existing intercom case.

The biggest downside is that NXP’s own MCUXpresso IDE cannot target the Teensy 4!

The only officially supported development environment for the Teensy 4 is Teensyduino, which is a board support package for the Arduino IDE. Having Arduino support is great, but let’s compare:

I also have NXP’s MIMXRT1060-EVK eval kit, which uses the same i.MX RT1060 micro controller family as the Teensy 4, but is much larger and comes with all the bells and whistles; notably:

1. The MCUXpresso IDE works with the eval kit’s built-in debugger out of the box! Being able to inspect a stack trace, set breakpoints and look at register contents are invaluable tools when doing micro controller development.
2. The MCUXpresso IDE comes with convenient graphical Pin and Clock config tools. Setting a pin’s alternate function becomes a few clicks instead of hours of fumbling around.
3. The NXP SDK contains a number of drivers and examples that are tested on the eval kit. That makes it really easy to get started!

Each of these points is very attractive on their own, but together they make the whole experience so different!

Being able to deploy to the Teensy from MCUXpresso would be a killer feature! So many NXP SDK examples would suddenly become available, filling the Teensy community’s gaps.

Signal Setup

On a high level, this is how we are going to connect the various signals:

Step 1. We start with the SCS intercom bus signal (28V high, 22V low):

Step 2. Our SCS receive circuit takes the bus signal and divides it down to 2V:

Step 3. We convert the voltage-divided analog signal into a digital SCSRXOUT signal:

Step 4. We modify our SCSRXOUT signal so that it can be sampled at 50%:

Step 5. We decode the signal using our micro controller’s UART:

Micro Controller firmware

Once I complete the next revision of the SCS interface PCB, I plan to release all design files, schematics, sources, etc. in full.

Until then, the following sections describe how the most important parts work, but skip over the implementation-specific glue code that wires everything together.

Analog Comparator

The Analog Comparator in our microcontroller lets us know whether a voltage is above or below a configured threshold voltage by raising an interrupt. A good threshold is 1.65V in my case.

In response to the voltage change, we set GPIO pin 15 to a digital high (3.3V) or low (0V) level:

volatile uint32_t cmpflags;

// ISR (Interrupt Service Routine), called by the Analog Comparator:
void acmp1_isr() {
  cmpflags = CMP1_SCR;

  { // clear interrupt status flags:
    uint8_t scr = (CMP1_SCR & ~(CMP_SCR_CFR_MASK | CMP_SCR_CFF_MASK));
    CMP1_SCR = scr | CMP_SCR_CFR_MASK | CMP_SCR_CFF_MASK;
  }

  if (cmpflags & CMP_SCR_CFR_MASK) {
    // See below! This line will be modified:
    digitalWrite(15, HIGH);
  }

  if (cmpflags & CMP_SCR_CFF_MASK) {
    digitalWrite(15, LOW);
  }
}

This signal can easily be verified by attaching an oscilloscope probe each to the SCSRX voltage-regulated bus signal input and to the SCSRXOUT GPIO pin output:

Analog Comparator Modification

There is one crucial difference between SCS and UART:

To transmit a 0 (or start bit):

• SCS is low 34μs, then high 70μs
• UART is low the entire 104μs

UART implementations typically sample at 50%, the middle of the bit period.

For SCS, we would need to sample at 20%, because the signal returns to high so quickly.

While setting a custom sample point is possible in e.g. sigrok’s UART decoder, neither software nor hardware serial implementations on micro controllers typically support it.

On a micro controller it is much easier to just modify the signal so that it can be sampled at 50%.

In practical terms, this means modifying the acmp1_isr function to return to high later than the Analog Comparator indicates:

volatile uint32_t cmpflags;

// ISR (Interrupt Service Routine), called by the Analog Comparator:
void acmp1_isr() {
  cmpflags = CMP1_SCR;

  { // clear interrupt status flags:
    uint8_t scr = (CMP1_SCR & ~(CMP_SCR_CFR_MASK | CMP_SCR_CFF_MASK));
    CMP1_SCR = scr | CMP_SCR_CFR_MASK | CMP_SCR_CFF_MASK;
  }

  if (cmpflags & CMP_SCR_CFR_MASK) {
    // Instead of setting our output pin high immediately,
    // we delay going up by approx. 40us,
    // turning the SCS signal into a UART signal:
    delayMicroseconds(40);
    digitalWrite(15, HIGH);
  }

  if (cmpflags & CMP_SCR_CFF_MASK) {
    digitalWrite(15, LOW);
  }
}

You can now read this signal using your laptop and a USB-to-serial adapter!

On a micro controller, we now feed this signal back into a UART decoder. For prototyping, this can literally mean a jumper wire connecting the output GPIO pin with a serial RX pin. Some micro controllers also support internal wiring of peripherals, allowing you to get rid of that cable.

SCS RX (receive)

With the SCS intercom bus signal bytes now available through the UART decoder, we can design a streaming SCS decoder. The decoder self-synchronizes and skips invalid SCS telegrams by checking their checksum. We start with a ring buffer and a convenience working copy:

constexpr int telegramLen = 7;

typedef struct {
  // circular buffer for incoming bytes, indexed using cur
  uint8_t buf[telegramLen];
  int cur;

  uint8_t tbuf[telegramLen];
} scsfilter;

Each byte we receive from the UART, we store in our ring buffer:

void sf_WriteByte(scsfilter *sf, uint8_t b) {
  sf->buf[sf->cur] = b;
  sf->cur = (sf->cur + 1) % telegramLen;
}

After every byte, we can check if the ring buffer decodes to a valid ring signal SCS bus telegram:

bool sf_completeAndValid(scsfilter *sf) {
  const uint8_t prev = sf->buf[(sf->cur+(telegramLen-1))%telegramLen];
  if (prev != 0xa3) {
    return false; // incomplete: previous byte not a telegram termination
  }

  // Copy the whole telegram into tbuf; makes working with it easier:
  for (int i = 0; i < telegramLen; i++) {
    sf->tbuf[i] = sf->buf[(sf->cur+i)%telegramLen];
  }

  const uint8_t stored = sf->tbuf[5];
  const uint8_t computed = sf->tbuf[1] ^
    sf->tbuf[2] ^
	sf->tbuf[3] ^
	sf->tbuf[4];
  if (stored != computed) {
    return false; // corrupt? checksum mismatch
  }

  return true;
}

int sf_ringForApartment(scsfilter *sf) {
  if (!sf_completeAndValid(sf)) {
    return -1;
  }

  if (sf->tbuf[3] != 0x60) {
    return -1; // not a ring command
  }

  if (sf->tbuf[1] != 0x91) {
    return -1; // not sent by the intercom house station
  }

  return (int)(sf->tbuf[2]); // apartment id
}

SCS TX (send)

Conceptually, writing serial data to a GPIO output from software is done with e.g. the Arduino SoftwareSerial library, but there are plenty of implementations for different micro controllers. This technique is also sometimes called “Bit banging”.

I started with the the Teensy SoftwareSerial::write implementation and modified it to:

1. Invert the output to drive the SCS transmit circuit’s Mosfet transistor gate, i.e. low on idle and high on transmitting a 0 bit.

2. Return to idle 70μs earlier than the signal would, i.e. after ≈34μs already.

The modified write function looks like this:

#define V27 LOW
#define V22 HIGH

#define scs0() do { \
  while (ARM_DWT_CYCCNT - begin_cycle < (target-43750/*70us*/)) ; \
  digitalWriteFast(11, V27); \
} while (0)

size_t SCSSerial::write(uint8_t b)
{
  elapsedMicros elapsed;
  uint32_t target;
  uint8_t mask;
  uint32_t begin_cycle;

  ARM_DEMCR |= ARM_DEMCR_TRCENA;
  ARM_DWT_CTRL |= ARM_DWT_CTRL_CYCCNTENA;
  ARM_DWT_CYCCNT = 0;

  // start bit
  target = cycles_per_bit;
  noInterrupts();
  begin_cycle = ARM_DWT_CYCCNT;
  digitalWriteFast(11, V22);
  scs0();
  wait_for_target(begin_cycle, target);

  // 8 data bits
  for (mask = 1; mask; mask <<= 1) {
    if (b&mask) {
      digitalWriteFast(11, V27);
    } else {
      digitalWriteFast(11, V22);
    }
    target += cycles_per_bit;
    scs0();
    wait_for_target(begin_cycle, target);
  }

  // stop bit
  digitalWriteFast(11, V27);
  interrupts();
  target += cycles_per_bit;
  scs0();
  while (ARM_DWT_CYCCNT - begin_cycle < target) ; // wait
  return 1;
}

It works!

With the approach described above, I now have a micro controller that recognizes doorbell rings for my apartment and ignores doorbell rings for my neighbors. The micro controller can unlock the door, too, and both features are available through the Nuki Opener.

How is the Nuki Opener?

It took over 2 months before I saw the Nuki Opener working correctly for the first time.

I really hope the Nuki developers can work with what I described above and improve their product’s reliability for all customers with an SCS intercom system!

The device itself seems useful and usable, but time will tell how reliable it turns out in practice. I think I noticed push notifications when the door rang coming in rather late (many seconds later).

I’ll keep an eye on this and explore the various Nuki APIs more.

Appendix: Project Journal

• 2020-09-26: I buy a Nuki Opener (Nuki Opener #1), but despite connecting it correctly, it never successfully opens the door. I start learning about the SCS home automation bus system that our intercom uses.
• 2020-09-28: I publish an SCS bus decoder for sigrok and contact the Nuki Support.
• 2020-10-15: I buy another Nuki Opener (Nuki Opener #2) to test their old firmware version, because downgrading firmware versions is impossible. Opener #2 actually opens the door, so I assume we are dealing with a firmware problem [turns out incorrect later].
• 2020-10-16: I publish a detailed analysis of the Nuki Opener not sending the correct signal for the Nuki developers to go through.
• 2020-11-03: I update my new Nuki Opener #2 to the latest firmware and realize that my old Nuki Opener #1 most likely just has some sort of hardware defect. However, Opener #2 has trouble detecting the ring signal: either it doesn’t detect any rings at all, or it detects all rings, including those for my neighbors!
• 2020-11-16: In their 13th (!) email reply, Nuki Support confirms that the Opener firmware is capturing and matching the incoming ring signal, if I understand their developers correctly.
• 2020-11-18: I suggest to Nuki developers (via Nuki Support) to decode the SCS signal with a UART decoder instead of comparing waveforms. This should be a lot more reliable!
• 2020-11-23: My self-designed SCS receiver/transmitter/power supply PCB arrives. The schematics are based on existing SCS DIY work, but I created my own KiCad files because I was only interested in the SCS bus interface, not the PIC microcontroller they used.
• 2020-11-25: Working on the intercom, I assume some wire touched an unlucky spot, and my BTicino intercom went up in smoke. We enabled the Nuki Opener’s ring sound and started using it as our main door bell. This meant we now started hearing the ring sound for (some) of our neighbors as well.
• 2020-11-26: My Teensy 4 microcontroller successfully decodes the SCS bus signal with its Analog Comparator and UART decoder.
• 2020-11-28: My Teensy 4 microcontroller is deployed to filter the SCS bus ring signal and drive the Nuki Opener in analogue mode.

Auch wenn ohne Mateflaschenumfallgeklirr, Einlassbändchenkratzrandertastung, Bratnudelstandfettdampfklamottengeruch und Messeglaswurstecho sicher einiges an Congressfeeling fehlen wird – (Aufkleber-)Goldschatzauffindungsimpressionen sollen trotzdem nicht zu kurz kommen.

Hierfür haben wir die Aktion „Schicke Sticker von den Sticker-Schickern“ ins Leben gerufen.

Wenn ihr Lust auf eine klebrige Überraschung habt, könnt ihr uns bis zum 4. Dezember 2020 einen adressierten und frankierten Rückumschlag (am besten C5) an unsere neue Adresse senden und wir schicken euch Aufcccleber und andere Goodies zurück.

Die Aktion wird – außer der Umschläge – kostenlos sein. Dennoch freuen wir uns natürlich über allgemeine Spenden auf unser Konto oder einen Beitrag via PayPal, um alle Kosten zu decken und den möglichen Rest in unseren neuen Space zu investieren.

Bei Fragen könnt ihr euch auf Twitter und Mastodon an uns wenden.

Einen Tag vor Ablauf unserer Umzugsfrist haben wir die allerletzten Reste aus den alten Räumlichkeiten rausgekehrt und die Schlüssel unserem ehemaligen Vermieter übergeben. Die Ära Boveristraße ist somit endgültig Geschichte.

Unser Elektroniklabor, die Holzwerkstatt, das FabLab und unsere Küche haben in der Weinheimer Straße ein neues Zuhause gefunden haben. Auch unser Olymp wird in anderer Form bei den Heidelberger Breidenbach Studios weiterleben.

Wir freuen uns auf die nächsten 10(0) Jahre RaumZeitLabor! Auf abgeschlossene und verworfene Ideen und Projekte, Vorträge, Feiern, Ess- und Mottopartys, Nerd am Herd, Bits & Bites und Stick-Nachmittage. Auf 3D-Druck- und Lasercut-Sessions, Programmier-Workshops, Agenda Aktion, GnoPN, Large Hackerspace Conventions, Film- und Analogspieleabende, Mario-Kart-Turniere, GameJams, Exkursionen und Museumsbesuche, Ausflüge zu Veranstaltungen und vor allem auf ganz, ganz viel Popcorn!

Last week Wednesday I was invited to give a talk at the group seminar of the AG Christoph Jacob at TU Braunschweig, Germany. Christoph was especially interested in our recent publication on a posteriori error estimation in Kohn-Sham problems and so I decided to use the opportunity to give a broad introduction into the topic. Since the main audience for my talk were chemists I motivated our work from the context of high-throughput density-functional theory calculations, which are becoming more and more of interest in practice. In the second part of my talk I tried to lay out the basic ideas and challenges of a posteriori error estimation and what are the key aspects to understand and overcome if one wants to provide a useful error bound for a particular problem (see also this article for more details). In the last part of the talk I turned the attention specifically to our recent contribution, discussing error estimates in the context of simple Kohn-Sham problems. Many of the delicate details of our work I did not touch upon, but nevertheless this last section of my talk gives a very good overview of our approach, our main underlying assumptions and our results. While we have focused solely on plane-wave basis sets in our work, I briefly hinted whether extensions to other basis sets with similar methods can be achieved.

Unfortunately the whole "visit" to Braunschweig was completely virtual, which really was a shame as I would have loved to return to the city and spend a day with the group. Despite these circumstances though the discussions with Christoph and his group did not not really fall short. During the day I had many video call sessions where I had ample opportunity to discuss with PhD students about their projects and in this way get a good idea of the interesting research going on in Braunschweig. As that's typically my favourite part about visiting a group I'm very happy this worked out so well (thanks to everyone who I managed to talk to!) and I already look forward to properly meeting everyone in person when the usual conference schedules are back in place.

The slides from my talk are attached below.

Link	Licence
Challenges and prospects of a posteriori error estimation in density-functional theory (Slides)

Last week (24th and 25th September) I attended the 4th annual meeting of the Moansi (Modelling, analysis and simulation of Molecular Systems) work group of the GAMM. I already attended the Moansi meeting last year, where I very much enjoyed both the broad range of talks at the interdisciplinary border of maths, chemistry and physics as well as the familial atmosphere with lively and stimulating discussions. In the same spirit I was looking forward to this year's addition, where in light of the global pandemic, however, the workshop had to be virtual. Nevertheless I was able to take away many stimulating impressions from the two days of the meeting and I already look forward to next year, where we hopefully meet again in person.

During the meeting I myself presented our recently published work on an LDOS-based SCF preconditioner, which is especially suitable for DFT calculations in inhomogeneous systems (metallic slabs or clusters). See this blog article and my slides for details.

Link	Licence
Black-box inhomogeneous preconditioning for density-functional theory (Slides)

I have long been looking for a way to make my intercom a little more pleasant.

Recently, a friend made me aware of the Nuki Opener, which promises to make existing intercom systems smart, and claims to be compatible with the specific intercom I have!

So I got one and tried setting it up, but could not get it to work.

This post documents how I have analyzed what goes over the intercom’s SCS bus. Perhaps the technique is interesting, or perhaps you want to learn more about SCS :)

Note that I have not yet used the Nuki Opener, so I can’t say anything about it yet. What I have seen so far makes a good impression, but it just does not seem to work at all with my intercom. I will update this article after working with the Nuki support to fix this.

Connecting the Nuki Opener to the bTicino 344212

First, I identified which wires are used for the bus: between BUS- and BUS+, the internet tells me that I would expect to measure ≈27V, and indeed a multimeter shows:

I then connected the Nuki Opener as described in “Connect the Nuki Opener to an unknown intercom”, Page 8, Bus intercoms → Basic setup without doorbell suppression:

I had previously tried the enhanced setup with doorbell suppression, as the Nuki app recommends, but switched to the simplest setup possible when capturing the signal.

Configuring the Nuki Opener

With the Nuki app, I configured the Opener either as:

• bTicino → 344212
• Generic → Bus (SCS)
• Unknown intercom

Unfortunately, with all configurations:

1. The app says it learned the door open signal successfully.
2. The device/app does react to door rings.
3. The device never successfully opens the door.

Capturing the SCS bus with sigrok

The logic analyzer that I have at home only works with signals under 5V. As the SCS bus is running at 27V, I’m capturing the signal with my Hantek 6022BE USB oscilloscope.

sigrok is a portable, cross-platform, free open source signal analysis software suite and supports the Hantek 6022BE out of the box, provided you have at least version 0.1.4 of the the sigrok fx2lafw package installed.

Check out sigrok’s “Getting started with a logic analyzer” if you’re new to sigrok!

The Nuki Opener has 3 different pin headers you can use, depending on where you want to attach it on your wall. These are connected straight through, so I used them to conveniently grab BUS+ and BUS- just like the Nuki sees it:

I set the oscilloscope probe head to its 10X divider setting, so that I had the full value range available, then started sampling 5M samples at 500 kHz:

You can see 10s worth of signal. The three bursts are transmissions on the SCS bus.

The labeling didn’t quite match for me: it shows e.g. 3.2V instead of 27V, but as long as the signal comes in clearly, it doesn’t matter if it is offset or scaled.

SCS bus decoding with sigrok: voltage levels

Let’s tell sigrok what voltage level corresponds to a low or high signal:

1. left-click on channel CH1
2. set “conversion” to “to logic via threshold”
3. set “conversion threshold” to 3.0V

Now you’ll see not only the captured signal, but also the logical signal below in green:

SCS bus decoding with sigrok: SCS decoder

Now that we have obtained a logical/digital signal (low/high), we can write a sigrok decoder for the SCS bus. See sigrok’s Protocol decoder HOWTO for an introduction.

In general, I strongly recommend investing into tooling, in particular when decoding protocols. Spending a few minutes to an hour at this stage will minimize mistakes and save lots of time later, and—when you contribute your tooling—enable others to do more interesting work!

I found it easy to write a sigrok decoder, having never used their API before. It was quick to get something onto the screen, mistakes were easy to correct, and the whole process was nicely iterative.

Until it is merged and released with a new version of libsigrokdecode, you can find my SCS decoder on GitHub.

The decoder looks at every layer of an SCS telegram: the start/stop bits, the data bits, the value and the value’s logical position/function in the SCS telegram.

Our SCS decoder displays the 3 bursts on the SCS bus when we ring the doorbell:

Only the middle burst sets a destination address of 0x3, the configured number of my intercom system. I am not sure what the first and last burst indicate!

The SCS bus activity when opening the door seems more clear:

These 2 bursts are sent one second apart, and only differ in the request parameter field: my guess is that 0xa4 means “start buzzing the door open” and 0xa0 means “stop buzzing the door open”.

I’m not sure why all these bursts repeat their SCS telegrams 3 times. My understanding was that SCS telegrams are repeated only when they are not acknowledged, and I indeed see no acknowledgement telegrams in my captures. Does that mean something is wrong with our intercom and it only works due to retransmissions?

SCS bus decoding with sigrok git: UART+SCS decoder

As Gerhard Sittig pointed out, in the git version of libsigrokdecode, one can use the existing UART decoder to decode SCS:

1. Set Baud rate to 9600
2. Set Sample point to 20%

This seems a little more robust than my cobbled-together SCS decoder from above :)

In addition to the UART decoder, we can still use a custom SCS decoder to label individual bytes within an SCS telegram according to their function, and do CRC checks.

Captured SCS telegrams

You can find my most recent captures in 2020-09-27-rohdaten-klingel-rev2.zip:

• 2020-09-27-anlern-01-open-PUR-filtered.srzip is the door buzzer
• 2020-09-27-anlern-02-klingel-PUR-filtered.srzip is the bell ringing

To extract the interesting parts from the sigrok files, I:

1. Click the Show Cursors icon in PulseView’s toolbar.
2. Position the left and right cursor edges such that the signal of interest is selected.
3. Click the drop-down next to the Save icon and select Save Selected Range As.

Further reading

I used the following sources; please let me know of any others!

• https://it.wikipedia.org/wiki/Bus_SCS
• http://guidopic.altervista.org/alter/eibscsgt.html
• https://www.mikrocontroller.net/topic/493823

Following the submission of our paper on a posteriori error estimation in the Kohn-Sham equations a few months ago I was recently invited to present our work at the Faraday Discussions on New horizons in density functional theory. Being amongst speakers such as Kieron Burke, Andreas Savin or Weitao Yang this was truly a great honour.

Even though the conference had to be virtual, I enjoyed it very much, especially because of its very unusual format. Unlike most other conferences where the presenting author typically does his thing for like 30 minutes, followed by just a few questions, the situation is completely reversed for the Faraday discussions. Since we had to submit our paper already months in advance (and this was shared with the other participants) the content of my talk was already known to the audience. The main chunk of the time at the conference was therefore allocated to the discussion and not to the presentation. My few slides therefore only briefly recap our work and hint at the general motivation and outlook. As I hoped our work did indeed stimulate an intense discussion with interested and stimulating questions, which I really appreciated (thanks to everyone who asked or commented). As far as I understand both the paper as well a transcript of the discussion will be part of the official Faraday Discussions conference proceedings, which will be published by the Royal Society of Chemistry soon. As per usual my slides are attached below.

Link	Licence
A posteriori error estimation for the non-self-consistent Kohn-Sham equations (Slides)

Kaum zu glauben, aber wahr: Die Suche nach einem neuen Standort ist beendet! Das RaumZeitLabor bleibt in Käfertal und zieht 1,3 Luftlinienkilometer weiter in die Weinheimer Straße 58–60.

Hier gibt es noch einiges tun. Wände müssen versetzt, beziehungsweise gestellt werden, der Keller gehört renoviert und werkstatttauglich hergerichtet, die neue Aufteilung aller Räume muss durchdacht werden. Aber auch das “alte” RZL und die eigentliche Umzugsplanung sollten vor lauter Vorfreude auf die neue Location nicht vernachlässigt werden. 

Wir freuen uns weiterhin über Beteiligung aller Art. Wie es die kommenden Wochen weitergeht, erfahrt ihr auf der Mailingliste und sicher auch über Twitter/Mastodon.

For the past half a year or so Antoine Levitt and myself have been looking at a particular tricky busyness for solid-state density-functional theory (DFT) calculations, namely how to design efficient self-consistent field (SCF) schemes for large inhomogeneous systems. I have already previously reported on this matter in a short talk at the seminar of our interdisciplinary working group, but now our results have reached a stage suitable for publication.

The underlying problem we are tackling in our work is that for large systems, meaning increased sizes of the unit cell, the SCF iterations become harder and harder to solve. Mathematically speaking the (spectral) condition number of the fixed point iterations underlying the SCF procedure increase rather drastically in such cases, leading to very slow convergence. For example in aluminium the number of iterations required to converge an SCF with a damped iteration scheme (the most simple one) increases quadratically with the system size. This quickly makes calculations intractable and multiple more sophisticated approaches have therefore been developed over the years. As is detailed in our work there are mainly two orthogonal directions of attack. The first is to black-box "accelerate" the convergence by using the so-called Anderson (or Pulay or DIIS) scheme. This reduces the growth of iterations with system size from quadratic to linear (in the aluminium example), which is a good start. The second approach is to use a carefully designed preconditioner for the SCF in order to tame the SCF iterations. Figuratively speaking this approach makes use of known physics to prevents the SCF from looking in the wrong direction for the solution. If done right, meaning that the physics modelled by the preconditioner fits the system at hand, this allows the SCF iteration count to become independent of system size. This latter approach is clearly the more important route to cure the problem, but both approaches are orthogonal and are therefore typically combined in order to get the fastest convergence.

Now what does it mean the preconditioner has to fit the system? As we detail in the paper, the convergence of an SCF is intimately linked to the dielectric behaviour of the material one models with the SCF. For homogeneous cases (i.e. bulk insulators, metals and semiconductors) people have devised very good models for their dielectric behaviour and have used them to construct preconditioners. As is well known (and confirmed by our study) these models show exactly the desirable property of a size-independent iteration count. The caveat is only that metals, insulators and semiconductors have deviating dielectric properties, meaning that each of these calls for a different preconditioning strategy. In return this means that heterogeneous cases where multiple of these materials are combined are difficult to treat in practice because none of the bulk recipes fully fit.

The main aim of our work was therefore to design a preconditioner which automatically and locally adapts to the system at hand, meaning that for heterogeneous cases it treats metallic regions like metals, insulating regions like insulators and so on. As we demonstrate with a number of test cases our preconditioner is able to do this completely black-box and parameter-free and performs well also for large heterogeneous systems. This is in contrast to previous approaches to tackle this problem, which were not as general as our approach and sometimes required complex hand-tuning of the involved parameters.

While our preconditioner solves the problem of efficiently treating cases like metallic slabs, metal clusters and basically any combination of metallic parts, insulators and vacuum, it is not fully capable of distinguishing insulators and semiconductors. We show that this can be cured at the expense of introducing another parameter to our algorithm. This works, but is not completely satisfactory to us. Part of our ongoing work is therefore to extend our scheme to treat mixed systems involving semiconductors as well. Another aspect we have so far neglected is spin, which is a constant annoyance for converging SCFs. Having a solid dielectric model as we propose it, also opens way to adapt preconditioning to each spin component differently. We hope to use this in the future to tackle convergence issues with spin in a hopefully more rigorous way than this is done to date.

The full abstract of our paper reads

We propose a new preconditioner for computing the self-consistent problem in Kohn-Sham density functional theory, based on the local density of states. This preconditioner is inexpensive and able to cure the long-range charge sloshing known to hamper convergence in large, inhomogeneous systems such as clusters and surfaces. It is based on a parameter-free and physically motivated approximation to the independent-particle susceptibility operator, appropriate for both metals and insulators. It can be extended to semiconductors by using the macroscopic electronic dielectric constant as a parameter in the model. We test our preconditioner successfully on inhomogeneous systems containing metals, insulators, semiconductors and vacuum.

Motivation

Despite using a FTTH internet connection since 2014, aside from the one fiber uplink, I had always used network gear with 1 Gbit/s links over regular old rj45 cat5(e) cables.

I liked the simplicity and uniformity of that setup, but decided it’s time to add at least one fiber connection, to get rid of a temporary ethernet cable that connected my kitchen with the rest of my network that is largely in the living room and office.

The temporary ethernet cable was an experiment to verify that running a server or two in my kitchen actually works (it does!). I used a flat ethernet cable, which is great for test setups like that, as you can often tape it onto the walls and still close the doors.

So, we will replace one ethernet cable with one fiber cable and converters at each end:

Why is it good to switch from copper ethernet cables to fiber in this case? Fiber cables are smaller and hence easier to fit into existing cable ducts. While regular ethernet cable is way too thick to fit into any of the existing ducts in my flat, I was hoping that fiber might fit!

When I actually received the cables, I was surprised how much thinner fiber cables actually can be: there are 0.9mm cables, which are so thin, they can be hidden in plain sight! I had only ever seen 2mm fiber cables before, and the 0.9mm cables are incredibly light, flexible and thin! Even pasta is typically thicker:

Preparing a delicious pot of glass noodles ;)

The cable shown above comes from the fiber store FS.COM, which different people have praised on multiple occasions, so naturally I was curious to give them a shot myself.

Also, for the longest time, it was my understanding that fiber connectors can only be put onto fiber cables using expensive (≫2000 CHF) machines. A while ago I heard about field assembly connectors so I wanted to verify that those indeed work.

Aside from practical reasons, playing around with fiber networking also makes for a good hobby during a pandemic :)

Hardware Selection

I ordered all my fiber equipment at FS.COM: everything they have is very affordable, and products in stock at their German warehouse arrive in Switzerland (and presumably other European countries) within the same week.

If you are in the luxurious position to have enough physical space and agility to pull through an entire fiber cable, without having to remove any connectors, you can make a new network connection with just a few parts:

I recommend buying an extra fiber cable or two so that you can accidentally damage a cable and still have enough spares.

Total cost thus far: just under 100 CHF. If you have existing switches with a free SFP slot, you can use those instead of the media converters and save most of the cost.

If you need to temporarily remove one or both of the fiber cable connector(s), you also need field assembly connectors and a few tools in addition:

I recommend buying twice the number of field assembly connectors, for practicing.

Personally, I screwed up two connectors before figuring out how the process goes.

Total cost: about 160 CHF for the field assembly equipment, so 260 CHF in total.

To boost your confidence in the resulting fiber, the following items are nice to have, but you can get by without, if you’re on a budget.

With the visual fault locator, you can shine a light through your fiber. You can verify correct connector assembly by looking at how the light comes out of the connector.

The fiber cleaning swabs are good to have in general, but for the field assembly connector, you need to use alcohol-soaked wipes anyway (which FS.COM does not stock).

The total cost for everything is just under 300 CHF.

Hardware Selection Process

The large selection at FS.COM can be overwhelming to navigate at first. My selection process went something like this:

My first constraint is using bi-directional (BiDi) fiber optics modules so that I only need to lay a single fiber cable, as opposed to two fiber cables.

The second constraint is to use field assembly connectors.

If possible, I wanted to use bend-insensitive fiber so that I wouldn’t need to pay so much attention to the bend radius and have more flexibility in where and how I can lay fiber.

With these constraints, there aren’t too many products left to combine. An obvious and good choice are 0.9mm fiber cable using LC/UPC connectors.

FS.COM details

As of 2020-08-05, FS.COM states they have 5 warehouses in 4 locations:

• Delaware (US)
• Munich (Germany)
• Melbourne (Australia)
• Shenzhen (China)

They recently built another, bigger (7 km²) warehouse in Shenzhen, and now produce inventory for the whole year.

By 2019, FS.COM had over 300,000 registered corporate customers, reaching nearly 200 million USD yearly sales.

Delivery times

As mentioned before, delivery times are quick when the products are in stock at FS.COM’s German warehouse.

In my case, I put in my order on 2020-Jun-26.

The items that shipped from the German warehouse arrived on 2020-Jul-01.

Some items had to be manufactured and/or shipped from Asia. Those items arrived after 3 more weeks, on 2020-Jul-24.

Unfortunately, FS.COM doesn’t stock any 0.9mm fiber cables in their German warehouse right now, so be prepared for a few weeks of waiting time.

Laying The Fiber

Use a cable puller to pull the fiber through existing cable ducts where possible.

• In general, buy the thinnest one you can find. I have this 4mm diameter cable puller, but a 3mm or even 2mm one would work in more situations.

• I found it worthwhile to buy a brand one. It is distinctly better to handle (less stiff, i.e. more flexible) than the cheap one I got, and thinner, too, which is always good.

In my experience, it generally did not work well to push the fiber into an existing duct or alongside an existing cable. I really needed a cable puller.

If you’re lucky and have enough space in your duct(s), you can leave the existing connectors on the fiber. I have successfully just used a piece of tape to fix the fiber connector on the cable puller, pushing down the nose temporarily:

Where there are no existing ducts, you may need to lay the fiber on top of the wall. Obviously, this is tricky as soon as you need to make a connection going through a wall: whereas copper ethernet cables can be bent and squeezed into door frames, you quickly risk breaking fiber cables.

Luckily, the fiber is very light, so it’s very easy to fix to the wall with a piece of tape:

You can see the upstream internet fiber in the top right corner, which is rather thick in comparison to my 0.9mm yellow fiber that’s barely visible in the middle of the picture.

Note how the fiber entirely disappears behind the existing duct atop the door!

Above, you can see the flat ethernet cable I have been using as a temporary experiment.

Where there is an existing cable that you can temporarily remove, it might be possible to remove it, put the fiber in, and put the old cable back in, too. This is possible because the 0.9mm fiber cable is so thin!

I’m using this technique to cross another wall where the existing cable duct is too full, but there is a cable that can be removed and put back after pulling the fiber through:

…and on the other side of the wall:

Note how the fiber is thin enough to fit between the socket and duct!

Note: despite measuring how long a fiber cable I would need, my cable turned out too short! While the cable was just as long as I had measured, with distances exceeding 10m, it is a good idea to add a few meters spare on each side of the connection.

Field assembly connectors

To give you an overview, these are the required steps at a high level:

1. Cut the fiber with the Fibre Optic Kevlar Cutter
2. Strip the fiber with the Fibre Optical Stripper
3. Put the field assembly jacket onto the fiber
4. Cut the stripped fiber cleanly with the High Precision Fibre Optic Cleaver FS-08C
5. Put the field assembly connector onto the fiber

I thought the following resources were useful:

1. Pictograms: PDF: FS.COM LC UPC field assembley connectors quick start guide
2. Pictures: Installation Procedure on FS.COM
3. Video: YouTube: Terminate Fiber in 5 Minutes: this video shows a different product, but I found it helpful to see any field assembly connector on video, and this is one of the better videos I could find.

Beware: the little paper booklet that comes with the field assembly connector contains measurements which are not to scale. I have suggested to FS.COM that they fix this, but until then, you’ll need to use e.g. a tape measure.

For establishing an intuition of their different sizes, here are the different connectors:

From left to right:

• 2.0mm fiber cable
• cat6 ethernet cable
• 0.9mm fiber cable (LC/UPC factory)
• 0.9mm fiber cable (LC/UPC field assembly connector)

The 0.9mm fiber cables come with smaller connectors than the 2.0mm fiber cables, and that alone might be a reason to prefer them in some situations.

The field assembly connectors are pretty bulky in comparison, but since you can attach them yourself after pulling only the cable through the walls and/or ducts, you usually don’t care too much about their size.

Conclusion

Modern fiber cables available at FS.COM are:

• thinner than I expected
• more robust than I expected
• cheaper than I expected
• survive tighter bend radiuses than I expected

Replacing this particular connection with a fiber connection was a smooth process overall, and I would recommend it in other situations as well.

I would claim that it is totally feasible for anyone with an hour of patience to learn how to put a field assembly connector onto a fiber cable.

If labor cost is expensive in your country or you just like doing things yourself, I can definitely recommend this approach. In case you mess the connector up and don’t want to fix it yourself, you can always call an electrician!

Stay tuned for the next part, where I upgrade the 1G link to a 10G link!

Since last Friday I have been attending JuliaCon, the annual conference for the Julia language. Naturally given the current situation the event did not take place "on location", but was instead converted into a virtual event. Albeit the different feel compared to a real-life conference the organisers did a very good job to maintain the social component into the event. Talks were pre-recorded and speakers available in a chat room to discuss during and after the presentation in written form. Birds of feather brainstorming sessions took place using audio discussions and at the end of every day there was a Gather Town virtual social, where one could videochat with fellow attendees by meeting up in a beautifully animated world, where each attendee was represented by a tiny avatar.

Apart from attending and listing to the great talks about the Julia language and its plenty of applications, I also had the chance to actively participate by giving a lecture about our package DFTK.jl. While I have presented on DFTK a few times before in front of expert audiences of the field, it was really the first time I presented DFTK as a released package to the broader Julia audience. That meant that I could, for once, give up on my usual storyline where I try and convince people into using Julia and instead focus on providing insight into the fascinating challenges of electronic-structure theory and how DFTK and Julia are ideal tools to tackle these.

In my talk I start easy by a general introduction into electronic-structure theory illustrating why an exact solution for electronic structures in molecules or solids is just not possible in realistic timeframes. Therefore one needs to live with approximate models, one example being density-functional theory (DFT), which we use in DFTK. As I detail in the talk an almost immediate consequence of the complexity of the problem is that advances in electronic-structure theory can typically only be realised if multiple disciplines join forces. An interdisciplinary project, however, brings some practical problems just quite frankly due to the fact that different fields have different approaches when tackling a problem. Being able to support such multidisciplinary motions in a common software platform for DFT, is one of the key aims of DFTK.

Related to this point we wanted DFTK to have a low entrance barrier for novel researchers. As time and money in research is tight programs should be easy to use and code simple and self-explanatory, such that new PhD students or researchers from foreign fields do not have a tough time to get started. In my talk I mention a few recent projects (an undergrad internship and a master project), where a noteworthy result could be achieved albeit students had little prior experience with neither Julia nor electronic-structure theory. A similar success story emphasising our ability to rapidly realise novel ideas in DFTK includes our recently published Faraday paper, where it only took 10 weeks from starting the project to submitting the paper.

Lastly, I discussed challenges arising from the so-called high-throughput screening methods, which are recently gaining popularity in computational materials design. In this particular research direction algorithms need to be particularly robust and tunable to find a sweet spot between accuracy and computational cost. This demands extremely stable and reliable algorithms, which poses interesting mathematical problems in numerical analysis and e.g. with respect to designing estimators for discretisation error. Especially in this area of application-oriented mathematical research we expect DFTK to be a handy tool in the future.

If you are interested in the full story a recording of the talk is available on youtube.

Link	Licence
DFTK: A Julian approach for simulating electrons in solids (Slides)
Youtube recording of the talk

tl;dr: Go's Context.Value is controversial because of a lack of type-safety. I design a solution for that based on the new generics design draft.

If you are following what's happening with Go, you are aware that recently an updated design draft for generics has dropped. What makes this particularly notable is that it comes with an actual prototype implementation of the draft, including a playground. This means for the first time, people get to actually try out how a Go with generics might feel, once they get in. It is a good opportunity to look at common Go code lacking type-safety and evaluate if and how generics can help address them.

One area I'd like to look at here is Context.Value. It is often criticized for not being explicit enough about the dependencies a function has and some people even go so far as to discourage its use altogether. On the other hand, I'm on record saying that it is too useful to ignore. Generics might be a way to bring together these viewpoints.

We want to be able to declare dependency on a functionality in context.Context via a function's signature and make it impossible to call it without providing that functionality, while also preserving the ability to pass it through APIs that don't know anything about it. As an example of such functionality, I will use logging. Let's start by creating a fictional little library to do that (the names are not ideal, but let's not worry about that):

package logctx

import (
    "context"
    "log"
)

type LogContext interface {
    // We embed a context.Context, to say that we are augmenting it with
    // additional functionality.
    context.Context

    // Logf logs the given values in the given format.
    Logf(format string, values ...interface{})
}

func WithLog(ctx context.Context, l *log.Logger) LogContext {
    return logContext{ctx, l}
}

// logContext is unexported, to ensure it can't be modified.
type logContext struct {
    context.Context
    l *log.Logger
}

func (ctx logContext) Logf(format string, values ...interface{}) {
    ctx.l.Printf(format, values...)
}

You might notice that we are not actually using Value() here. This is fundamental to the idea of getting compiler-checks - we need some compiler-known way to "tag" functionality and that can't be Value. However, we provide the same functionality, by essentially adding an optional interface to context.Context.

If we want to use this, we could write

func Foo(ctx logctx.LogContext, v int) {
    ctx.Logf("Foo(%v)", v)
}

func main() {
    ctx := logctx.WithLog(context.Background(), log.New(os.Stderr, "", log.LstdFlags))
    Foo(ctx, 42)
}

However, this has a huge problem: What if we want more than one functionality (each not knowing about the other)? We might try the same trick, say

package tracectx

import (
    "context"

    "github.com/opentracing/opentracing-go"
)

type TraceContext interface {
    context.Context
    Tracer() opentracing.Tracer
}

func WithTracer(ctx context.Context, t opentracing.Tracer) TraceContext {
    return traceContext{ctx, t}
}

type traceContext struct {
    context.Context
    t opentracing.Tracer
}

func (ctx traceContext) Tracer() opentracing.Tracer {
    return ctx.t
}

But because a context.Context is embedded, only those methods explicitly mentioned in that interface are added to traceContext. The Logf method is erased. After all, that is the trouble with optional interfaces.

This is where generics come in. We can change our wrapper-types and -functions like this:

type LogContext(type parent context.Context) struct {
    // the type-parameter is lower case, so the field is not exported.
    parent
    l *log.Logger
}

func WithLog(type Parent context.Context) (ctx Parent, l *log.Logger) LogContext(Parent) {
    return LogContext(parent){ctx, l}
}

By adding a type-parameter and embedding it, we actually get all methods of the parent context on LogContext. We are no longer erasing them. After giving the tracectx package the same treatment, we can use them like this:

// FooContext encapsulates all the dependencies of Foo in a context.Context.
type FooContext interface {
    context.Context
    Logf(format string, values ...interface{})
    Tracer() opentracing.Tracer
}

func Foo(ctx FooContext, v int) {
    span := ctx.Tracer().StartSpan("Foo")
    defer span.Finish()

    ctx.Logf("Foo(%v)", v)
}

func main() {
    l := log.New(os.Stderr, "", log.LstdFlags)
    t := opentracing.GlobalTracer()
    // ctx has type TraceContext(LogContext(context.Context)),
    //    which embeds a LogContext(context.Context),
    //    which embeds a context.Context
    // So it has all the required methods
    ctx := tracectx.WithTracer(logctx.WithLog(context.Background(), l), t)
    Foo(ctx, 42)
}

Foo has now fully declared its dependencies on a logger and a tracectx, without requiring any type-assertions or runtime-checks. The logging- and tracing-libraries don't know about each other and yet are able to wrap each other without loss of type-information. Constructing the context is not particularly ergonomic though. We require a long chained function call, because the values returned by the functions have no longer a unified type context.Context (so the ctx variable can't be re-used).

Another thing to note is that we exported LogContext as a struct, instead of an interface. This is necessary, because we can't embed type-parameters into interfaces, but we can embed them as struct-fields. So this is the only way we can express that the returned type has all the methods the parameter type has. The downside is that we are making this a concrete type, which isn't always what we want¹.

We have now succeeded in annotating context.Context with dependencies, but this alone is not super useful of course. We also need to be able to pass it through agnostic APIs (the fundamental problem Context.Value solves). However, this is easy enough to do.

First, let's change the context API to use the same form of generic wrappers. This isn't backwards compatible, of course, but this entire blog post is a thought experiment, so we are ignoring that. I don't provide the full code here, for brevity's sake, but the basic API would change into this:

package context

// CancelContext is the generic version of the currently unexported cancelCtx.
type CancelContext(type parent context.Context) struct {
    parent
    // other fields
}

func WithCancel(type Parent context.Context) (ctx Parent) (ctx CancelContext(Parent), cancel CancelFunc) {
    // ...
}

This change is necessary to enable WithCancel to also preserve methods of the parent context. We can now use this in an API that passes through a parametric context. For example, say we want to have an errgroup package, that passes the context through to the argument to (*Group).Go, instead of returning it from WithContext:

// Derived from the current errgroup code.

// A Group is a collection of goroutines working on subtasks that are part of the same overall task.
//
// A zero Group is invalid (as opposed to the original errgroup).
type Group(type Context context.Context) struct {
    ctx    Context
    cancel func()

    wg sync.WaitGroup

    errOnce sync.Once
    err     error
}

func WithContext(type C context.Context) (ctx C) *Group(C) {
    ctx, cancel := context.WithCancel(ctx)
    return &Group(C){ctx: ctx, cancel: cancel}
}

func (g *Group(Context)) Wait() error {
    g.wg.Wait()
    return g.err
}

func (g *Group(Context)) Go(f func(Context) error) {
    g.wg.Add(1)

    go func() {
        defer g.wg.Done()

        if err := f(g.ctx); err != nil {
            g.errOnce.Do(func() {
                g.err = err
            })
        }
    }()
}

Note that the code here has barely changed. It can be used as

func Foo(ctx FooContext) error {
    span := ctx.Tracer().StartSpan("Foo")
    defer span.Finish()
    ctx.Logf("Foo was called")
}

func main() {
    var ctx FooContext = newFooContext()
    eg := errgroup.WithContext(ctx)
    for i := 0; i < 20; i++ {
        eg.Go(Foo)
    }
    if err := eg.Wait(); err != nil {
        log.Fatal(err)
    }
}

After playing around with this for a couple of days, I feel pretty confident that these patterns make it possible to get a fully type-safe version of context.Context, while preserving the ability to have APIs that pass it through untouched or augmented.

A completely different question, of course, is whether all of this is a good idea. Personally, I am on the fence about it. It is definitely valuable, to have a type-safe version of context.Context. And I think it is impressive how small the impact of it is on the users of APIs written this way. The type-argument can almost always be inferred and writing code to make use of this is very natural - you just declare a suitable context-interface and take it as an argument. You can also freely pass it to functions taking a pure context.Context unimpeded.

On the other hand, I am not completely convinced the cost is worth it. As soon as you do non-trivial things with a context, it becomes a pretty "infectious" change. For example, I played around with a mock gRPC API to allow interceptors to take a parametric context and it requires almost all types and functions involved to take a type-parameter. And this doesn't even touch on the fact that gRPC itself might want to add annotations to the context, which adds even more types. I am not sure if the additional machinery is really worth the benefit of some type-safety - especially as it's not always super intuitive and easily understandable. And even more so, if it needs to be combined with other type-parameters, to achieve other goals.

I think this is an example of what I tend to dislike about generics and powerful type-systems in general. They tempt you to write a lot of extra machinery and types in a way that isn't necessarily semantically meaningful, but only used to encode some invariant in a way the compiler understands.

[1] One upside however, is that this could actually address the other criticism of context.Value: Its performance. If we consequently embed the parent-context as values in struct fields, the final context will be a flat struct. The interface-table of all the extra methods we add will point at the concrete implementations. There's no longer any need for a linear search to find a context value.

I don't actually think there is much of a performance problem with context.Value in practice, but if there is, this could solve that. ⬆

Back in 2013, I published a replacement controller for the Kinesis Advantage ergonomic keyboard. In the community, it is often referred to simply as the “stapelberg”, and became quite popular.

Many people like to use the feature-rich QMK firmware, which supports my replacement controller out of the box.

On eBay, you can frequently find complete stapelberg kits or even already-modified Kinesis keyboards including the stapelberg board for sale.

In 2017, Kinesis released the Kinesis Advantage 2, which uses a different connector (an FPC connector) for connecting the two thumb pad PCBs to the controller PCB, instead of the soldered cable the older Kinesis Advantage used. Aside from the change in connector and cable type, the newer keyboard uses the same pinout as the old one.

I wanted to at least update my project to support the Kinesis Advantage 2. While doing so, I decided to also make a bunch of improvements to make the project more approachable and usable for beginners. Among many other improvements, the project switched from Eagle to KiCad, which is FOSS and means no more costly license fees!

kinT (T for Teensy!)

I am hereby announcing the kinT kinesis keyboard controller: a replacement keyboard controller for your Kinesis Advantage or Advantage 2 ergonomic keyboards.

The Teensy footprint looks a bit odd, but it’s a combined footprint so that you can use the same board with many different Teensy microcontrollers, giving you full flexibility regarding cost and features. See “Compatibility: which Teensy to use?” for more details.

I originally replaced the controller of my Kinesis Advantage to work around a bug, but these days I do most of it just because I enjoy tinkering with keyboards.

You might consider to replace your keyboard controller for example…

• to build or modify your own keyboard
• to work around bugs in the standard controller
• because you prefer to run open source software such as the QMK firmware, even on your keyboard

Building your own kinT keyboard controller

1. Follow “Buying the board and components (Bill of materials)”. When ordering from OSH Park (board) and Digi-Key (components), you’ll get the minimum quantity of 3 boards for 72 USD (24 USD per board), and one set of components for 49 USD.

   • If you have any special requirements regarding which Teensy microcontroller to use, this is the step where you would replace the Teensy 3.6 with your choice.

2. Wait for the components to arrive. When ordering from big shops like Digi-Key or Mouser, this typically takes 2 days to many places in the world.

3. Wait for the boards to arrive. This takes 6 days in the best case when ordering from OSH Park with their Super Swift Service option. In general, the longer you are willing to wait, the cheaper it is going to get.

4. Follow the soldering guide. This will take about an hour.

5. Install the firmware

Improvements over the older replacement board

In case you’re familiar with the older replacement board and are wondering what changed, here is a complete list:

• The kinT supports both, the older Kinesis Advantage (KB500) and the newer Kinesis Advantage 2 (KB600) keyboards. They differ in how the thumb pads are connected. See the soldering instructions below.

• The kinT is made for the newer Teensy 3.x and 4.x series, which will remain widely available for years to come, whereas the future of the Teensy++ 2.0 is not as certain.

• The kinT is a smaller PCB (4.25 x 3.39 inches, or 108.0 x 86.1 mm), which makes it:

  • more compact: can be inserted/removed without having to unscrew a key well.

  • cheaper: 72 USD for 3 boards at oshpark, instead of 81 USD.

• The kinT silkscreen (front, back) and schematic are much much clearer, making assembly a breeze.

• The kinT is a good starting point for your own project:

  • kinT was designed in the open source KiCad program, meaning you do not need any license subscriptions.

  • The clear silkscreen and schematic make development and debugging easier.

• On the kinT, the Teensy no longer has to be soldered onto the board upside down.

• On the kinT, the FPC connectors have been moved for less strain on the cables.

• The kinT makes possible lower-cost builds: if you don’t need the scroll lock, num lock and keypad LEDs, you can use a Teensy LC for merely 11 USD.

Conclusion

I’m very excited to release this new keyboard controller, and I can’t wait to see all the custom builds and modifications!

By the way, there is also a (4-hour!) stream recording in case you are interested in some more history and context, and want to see me solder a kinT controller live on stream!

Das Jahr 2020 und seine schlechten Neuigkeiten hören nicht auf. Zu allem Übel sind uns jetzt auch noch unsere Räumlichkeiten zum 31. Oktober 2020 gekündigt worden.

Damit wir passend zu Halloween eine Einweihungsparty feiern können, brauchen wir eure Mithilfe bei der Suche nach einem neuen Zuhause für den Verein und seinen Maschinenpark.

Was wir uns für ein zukünftiges RaumZeitLabor wünschen würden:

• Büro- und Werkstatt-/Hallen-Kombination (wir brauchen wieder Arbeitsbereich und Platz für unsere Werkstätten)
• Mindestens 200m², besser ein paar mehr m² Platz oder eine Option auf später Erweiterung
• Küche oder Anschlüsse, um eine aufstellen zu können
• Sanitäre Anlagen und Heizung
• Einigermaßen guter ÖPNV-Anschluss, Parkmöglichkeiten in der Nähe
• Internet, schnell

Bonus:

• provisionsfrei
• wenige Nachbarn, oder Nachbarn, die nur tagsüber vor Ort sind
• barrierearmer Zugang

Solltet ihr einen Hinweis auf eine solche Immobilie haben, lasst es uns wissen und schreibt uns gern unter vorstand@raumzeitlabor.de an.

Die kommenden vier Monate heißt es jetzt „Auf die Kartons, fertig, los!“. Kommt bitte vorbei, nehmt eure privaten Sachen mit nach Hause und helft uns, alles umzugssicher zu verpacken.

Vielen Dank!

Socket activation is the idea of activating a daemon or service not by manually starting it, but by merely pre-exposing the socket that is used for communication with that service. This has several advantages:

• system boot speeds up as less things need to actually be started at boot time
• system resource usage is reduced as less services actually run¹
• you can restart services behind the socket invisible to the client (if the service gracefully takes up the connection on the socket again) without loosing messages²

The two requirements for this are:

• a daemon that manages these sockets and activates the corresponding process once communication happens
• the socket-activated daemon being able to work with a preestablished socket-connection

For the first part, systemd got us covered. The second part must be implemented in the daemon code, but we can regardless activate any socket based service, even if they don’t implement socket activation themselves with a little help. Let’s take a look at how things work in this order.

Systemd socket activation: The principle

This will be a brief rundown on how systemd’s socket activation works (see here for all the details):

To activate a service via its socket, we need to define two units: the service and the socket. A socket can be a network socket, a unix socket and several other connection types. In this post, however, we will exemplarily focus on network sockets. We place the service file of our socket-activatable service (we will come back to what this means in a moment) as /etc/systemd/system/myservice.service:

[Unit]
Description=Socket-activatable example service

[Service]
ExecStart=/usr/bin/myservice
NonBlocking=True

and we place the corresponding socket unit (in this case a TCP-network-socket) as /etc/systemd/system/myservice.socket:

[Unit]
Description=Socket for example service

[Socket]
# In this example we only listen on localhost
ListenStream=127.0.0.1:1234
NoDelay=true

[Install]
WantedBy=sockets.target

Note that the service does not require an install section, only the socket does³. We then enable our socket via systemctl enable --now myservice.socket.

Now we have an inactive service unit, but an active socket which is set up by systemd. The moment any activity occurs on this socket, systemd will start myservice.service and hand over the socket, buffering what is already written to it until the service takes over.

Leveraging socket activation in practice

In practice, we can distinguish three different cases:

1. Socket activation natively supported already

If the service in question already natively supports socket activation, simply activate the corresponding socket unit (or create one if it doesn’t exist yet). See systemctl list-units | grep socket for socket-units already available on the system.

2. Fault tolerance and boot time optimization desired only

If our service is intended to be started at boot and socket activation is intended only to provide transparent restarts and boot parallelization, we can simply use systemd’s systemd-socket-proxyd. See man systemd-socket-proxyd or here for details and usage examples.

Make sure you add an [Install]-section to the service file and enable the service itself as well then as systemd-socket-proxyd merely decouples the service unit from its socket, but doesn’t start it automatically.

3. On-demand starting and stopping of arbitrary services

If our service does not support socket activation natively and we want it to start not at boot time, but on-demand, we can use the tool socket-activate. For this, configure actual.service to listen on localhost on a different port such as 127.0.0.1:12345 and create two units: socket-activate-actual.service containing

[Unit]
Description=Socket-activate proxy for example service

[Service]
ExecStart=/usr/bin/socket-activate -u "actual.service" -a "127.0.0.1:12345"
NonBlocking=True

as well as a corresponding socket-activate-actual.socket as noted above, listening on the actual desired port. On the first connection to the socket, socket-activate-actual.service will be started and in turn start actual.service, proxying all traffic to it.

If socket-activate is invoked with an additional -t <timeout>, then both socket-activate-actual.service as well as actual.service are stopped again when no activity is detected for the specified timeout.

Fortunately the general lockdown due to the Corona pandemic slowly starts to ease around Paris as well. While basically all seminars are only virtual it is good to see old procedures and habits to slowly return. From my end I gave the first talk after the forced break today in the EMC2 group meeting.

Since it was the first time discussing DFTK in the EMC2 synergy group I decided to talk about it taking the angle of tackling an actual research problem. After presenting briefly DFT methods and DFTK in the first half of the talk, I therefore focused on one of my ongoing projects with Antoine Levitt, namely constructing better preconditioners for the self-consistent field (SCF) iterations in mixed systems. What is meant by mixed systems are systems where locally differing dielectric properties are found, i.e. where some parts of the material are insulating, others may be metallic or semiconductors. Since the dielectric properties are closely related to the spectrum of the SCF fixed-point map, they therefore also control the convergence properties of SCF procedures. For metals and (to a minor extent) semiconductors simple SCF procedures, where one just applies the SCF cycle over and over require extremely small step sizes (i.e. small damping values). As a result the SCF converges only very slowly. The remedy is to precondition the spectrum of the SCF map itself by using so-called mixing techniques. In state-of-the-art approaches these are usually material-specific, i.e. different mixings are used for insulators, metals or semiconductors. This is fine for bulk materials, but fails in case of mixed systems, since one has to globally select a single approach. Our recent work has been to investigate the spectrum of the SCF map and to try and construct a preconditioner which locally adapts and as a result is able to properly treat mixed systems as well. The results I presented today are, however, not yet final and more investigations are to be underdone for our approach to work as reliable as we want.

Link	Licence
SCF preconditioning for mixed systems: A DFTK case study (Slides)
A few DFTK examples (Jupyter notebook)
SCF preconditioners in 1D (Jupyter notebook)

For my programming stream at twitch.tv/stapelberg, I wanted to add an additional camera to show test devices, electronics projects, etc. I couldn’t find my old webcam, and new ones are hard to come by currently, so I figured I would try to include a phone camera somehow.

The setup that I ended up with is:

iPhone camera
→ Instant Webcam
→ WiFi
→ gstreamer
→ v4l2loopback
→ OBS

Disclaimer: I was only interested in a video stream! I don’t think this setup would be helpful for video conferencing, due to lacking audio/video synchronization.

iPhone Software: Instant Webcam app

I’m using the PhobosLab Instant Webcam (install from the Apple App Store) app on an old iPhone 8 that I bought used.

There are three interesting related blog posts by app author Dominic Szablewski:

1. MPEG1 Video Decoder in JavaScript (2013-May)
2. HTML5 Live Video Streaming via WebSockets (2013-Sep)
3. Decode it like it’s 1999 (2017-Feb)

As hinted at in the blog posts, the way the app works is by streaming MPEG1 video from the iPhone (presumably via ffmpeg?) to the jsmpeg JavaScript library via WebSockets.

After some git archeology, I figured out that jsmpeg was rewritten in commit 7bf420fd just after v0.2. You can browse the old version on GitHub.

Notably, the Instant Webcam app seems to still use the older v0.2 version, which starts WebSocket streams with a custom 8-byte header that we need to strip.

Linux Software

Install the v4l2loopback kernel module, e.g. community/v4l2loopback-dkms on Arch Linux or v4l2loopback-dkms on Debian. I used version 0.12.5-1 at the time of writing.

Then, install gstreamer and required plugins. I used version 1.16.2 for all of these:

• gstreamer
• gst-plugins-bad for mpegvideoparse
• gst-libav for avdec_mpeg2video

Lastly, install either websocat or wsta for accessing WebSockets. I successfully tested with websocat 1.5.0 and wsta 0.5.0.

Streaming

First, load the v4l2loopback kernel module:

% sudo modprobe v4l2loopback video_nr=10 card_label=v4l2-iphone

Then, we’re going to use gstreamer to decode the WebSocket MPEG1 stream (after stripping the custom 8-byte header) and send it into the /dev/video10 V4L2 device, to the v4l2loopback kernel module:

% websocat --binary ws://iPhone.lan/ws | \
  dd bs=8 skip=1 | \
  gst-launch-1.0 \
    fdsrc \
    ! queue \
    ! mpegvideoparse \
    ! avdec_mpeg2video \
    ! videoconvert \
    ! videorate \
    ! 'video/x-raw, format=YUY2, framerate=30/1' \
    ! v4l2sink device=/dev/video10 sync=false

Here are a couple of notes about individual parts of this pipeline:

• You must set websocat (or the alternative wsta) into binary mode, otherwise they will garble the output stream with newline characters, resulting in a seemingly kinda working stream that just displays garbage. Ask me how I know.

• The queue element uncouples decoding from reading from the network socket, which should help in case the network has intermittent troubles.

• Without enforcing framerate=30/1, you cannot cancel and restart the gstreamer pipeline: subsequent invocations will fail with streaming stopped, reason not-negotiated (-4)

• Setting format YUY2 allows ffmpeg-based decoders to play the stream. Without this setting, e.g. ffplay will fail with [ffmpeg/demuxer] video4linux2,v4l2: Dequeued v4l2 buffer contains 462848 bytes, but 460800 were expected. Flags: 0x00000001.

• The sync=false property on v4l2sink plays frames as quickly as possible without trying to do any synchronization.

Now, consumers such as OBS (Open Broadcaster Software), ffplay or mpv can capture from /dev/video10:

% ffplay /dev/video10
% mpv av://v4l2:/dev/video10 --profile=low-latency

Debugging

Hopefully the instructions above just work for you, but in case things go wrong, maybe the following notes are helpful.

To debug issues, I used the GST_DEBUG_DUMP_DOT_DIR environment variable as described on Debugging tools: Getting pipeline graphs. In these graphs, you can quickly see which pipeline elements negotiate which caps.

I also used the PL_MPEG example program to play the supplied MPEG test file. PL_MPEG is written by Dominic Szablewski as well, and you can read more about it in Dominic’s blog post MPEG1 Single file C library. I figured the codec and parameters might be similar between the different projects of the same author and used this to gain more confidence into the stream parameters.

I also used Wireshark to look at the stream traffic to discover that websocat and wsta garble the stream output by default unless the --binary flag is used.