Planet NoName e.V.

2021-08-28

michael-herbst.com

Q-Chem 5 paper

About two years ago I integrated my open-source ctx library into the Q-Chem quantum-chemistry software suite. Quickly ctx became part of the core stack for managing computational results inside Q-Chem. In particular inside the ccman and adcman modules, which are responsible for most of the coupled-cluster and algebraic-diagrammatic construction methods available in Q-Chem, ctx is widely used.

In a recently published paper by all the Q-Chem authors the developments inside the Q-Chem package leading up the major version 5 of the software are now summarised. The full abstract reads

This article summarizes technical advances contained in the fifth major release of the Q-Chem quantum chemistry program package, covering developments since 2015. A comprehensive library of exchange-correlation functionals, along with a suite of correlated many-body methods, continues to be a hallmark of the Q-Chem software. The many-body methods include novel variants of both coupled-cluster and configuration-interaction approaches along with methods based on the algebraic diagrammatic construction and variational reduced density-matrix methods. Methods highlighted in Q-Chem 5 include a suite of tools for modeling core-level spectroscopy, methods for describing metastable resonances, methods for computing vibronic spectra, the nuclear–electronic orbital method, and several different energy decomposition analysis techniques. High-performance capabilities including multithreaded parallelism and support for calculations on graphics processing units are described. Q-Chem boasts a community of well over 100 active academic developers, and the continuing evolution of the software is supported by an "open teamware" model and an increasingly modular design.

by Michael F. Herbst at 2021-08-28 22:30 under electronic structure theory, theoretical chemistry

2021-08-28

sECuREs website

Silent HP Z440 workstation: replacing noisy fans

Since March 2020, I have been using my work computer at home: an HP Z440 workstation.

When I originally took the machine home, I immediately noticed that it’s quite a bit louder than my other PCs, but only now did I finally decide to investigate what I could do about it.

Finding all the fans

I first identified all fans, both by opening the chassis and looking around, and by looking at the HP Z440 Maintenance and Service Guide, which contains this description:

chassis components

Specifically, I identified the following fans:

  • “1 Fan”, a 92mm rear fan, sucking air out of the back of the chassis.
  • “5 Memory fans”, two 60mm fans in a custom HP plastic enclosure that are positioned directly above the DIMM slots to the left and right of the CPU.
  • “6 CPU Heat sink”, a 92mm fan on top of a heat sink
  • “11 Rear System Fan”, a 92mm front (!) fan, pulling air into the front of the chassis.
  • My aftermarket nVidia GeForce GPU has 3 fans on a massive heat sink.
  • The power supply has a fan, too, which I will not touch.

Memory fans

The Z440 comes with a custom HP plastic enclosure that is put over the CPU cooler, fastened with two clips at opposite ends, and positions two small 60mm fans above the DIMM banks.

This memory fan plastic enclosure is a pain to find anywhere. It looks like HP is no longer producing it.

The enclosure plugs into the mainboard with a custom connector that is directly wired up to the fans, meaning it’s a pain to replace the fans.

memory fans

Luckily, while shopping around for an enclosure I could modify, I realized that memory fans are only required when installing more than 4 DIMM modules!

My machine “only” has 64 GB of RAM, in 4 DIMM modules, and I don’t intend to upgrade anytime soon, so I just unplugged the whole memory fan enclosure and removed it from the chassis.

The UEFI firmware does not complain about the memory fans missing (contrary to the rear fan!), and this simple change alone makes a noticeable difference in noise levels.

GPU fans

nVidia GPUs can be run at different “PowerMizer” performance levels:

nVidia PowerMizer

Many years ago, I ran into lag when using Chrome that went away as soon as I switched my nVidia GPU’s Preferred Mode to “Prefer Maximum Performance” instead of “Auto” or “Adaptive mode”.

It turns out that nowadays, that is no longer a problem, so running at Prefer Maximum Performance is no longer necessary.

Worse, pinning the GPU at the highest Performance Level means that it produces more heat, resulting in the fans having to spin up more often, and run for longer durations.

But, even after switching to Auto, resulting in Adaptive mode being chosen, I noticed that my GPU was stuck at a higher PowerMizer level than I thought it should be.

An easy fix is to limit the GPU to a certain PowerMizer level, and ideally not the lowest level (level 0). For me, one level after that (level 1) seems to result in no slow-down during my typical usage.

I followed this blog post to limit my GPU to PowerMizer level 1, i.e. I added /etc/modprobe.d/nvidia-power-save.conf with the following contents:

options nvidia NVreg_RegistryDwords="OverrideMaxPerf=0x2"

…followed by a rebuild of my initramfs (update-initramfs -u) and a reboot.

This way, the fans don’t typically need to spin up as the GPU stays below its temperature limit.

Rear and front fans

With the memory fans and GPU fans out of the way, two easy to check fans remain: the rear fan and front fan. These are 92mm in size, the model number is Foxconn PVA092G12S.

rear fan

I unplugged both of them to see what effect these fans have on the noise level, and the difference was significant!

Unfortunately, unplugging isn’t enough: the UEFI firmware complains on boot when the rear fan is not connected, requiring you to press Enter to boot. Also, the machine seems to get a few degrees Celsius hotter inside without the front and rear fans, so I don’t want to run the machine without these fans for an extended period of time.

I ordered two Noctua NF-A9x14 PWM fans (for about 25 CHF each) to replace the stock front and rear fans.

Unfortunately, HP uses a custom 4-pin fan connector on its Z440 mainboard! Luckily, modifying the connector of the Noctua Low-Noise Adapter cable to fit on the custom 4-pin connector is as simple as using a knife to remove the connector’s guard rails:

fan connector mod

CPU fan

For the CPU fan, HP again chose to use a custom (6-pin) connector.

On the web, I read that the Z440 CPU fan is quite efficient and not worth replacing. This matches my experience, so I kept the standard Z440 CPU cooler.

Conclusion

I was quite happy to discover that I could just unplug the memory fans, and configure my GPU to make less noise. Together with replacing the front/rear fans with Noctua ones, the machine is much quieter now than before!

One downside of workstation-class hardware is that manufacturers (at least HP) like to build custom parts and solutions. Using their own fan connectors instead of standard connectors is such a pain! I’ll be sure to stick to standard PC hardware :)

at 2021-08-28 13:16

2021-08-04

michael-herbst.com

JuliaCon BoF discussion session: Building a Chemistry and Materials Science Ecosystem

The second event I co-organised at this year's JuliaCon (see this article for the other) was a Birds of Feather (BoF) discussion session titled Building a Chemistry and Materials Science Ecosystem in Julia. In this session Rachel Kurchin and I wanted to gather the various stakeholders working on Julia codes for chemistry and materials simulations and discuss possible overlaps and plan future joint efforts.

This has been the first time a meeting dedicated to this scientific field has been conducted within the Julia community and so we were quite curious about who would turn up. In the end we had a pretty mixed crowd consisting of Julia users tackling research problems in chemistry and materials as well as plenty of maintainers of various Julia packages related to the field, but also some veteran Julia users joined the discussion. This mix of people provoked a rather rich and lively debate about the perspectives of Julia in this respective field and the 90 minutes which were given to us passed almost in an instance.

A central discussion point within the session was the need for joint interfaces shared amongst the key packages of the ecosystem both to leverage Julia's unique composability between the various packages and to furthermore enhance the interoperability and lead to a good user experience. As many have pointed out during the session, a good first step is the design of an interface for representing the structure of the chemical system or the material to be studied. In particular this would allow to deveop unified approches to share data between packages, setup calculations and plainly compare between different approaches. Additionally annoying aspects such as file parsing, data export, plotting or other post-processing could then be easily implemented once using the general interface and used by everyone in the Julia community. Naturally a time slot of 90 minutes is just about sufficient to get the discussion started and scratch the surface, so the session has not yet yielded anything conclusive. However, following up from the conference the debate has definitely intensified amongst participants and I would not be suprised if some progress will be made.

In case you are interested to participate in these developments or plainly want to get in touch with Julia users and developers from chemistry, molecular or materials science, here are a number of relevant resouces:

by Michael F. Herbst at 2021-08-04 10:00 under workshop, electronic structure theory, Julia

2021-08-03

michael-herbst.com

JuliaCon DFTK workshop: A mathematical look at electronic structure theory

From 13th July till 30th July this year's JuliaCon finally took place virtually. The first week (13th till 27th) hosted a number of three-hour live-streamed sessions of workshops, while the "regular" conference with a number of prerecorded talks started on 28th.

After my introductory talk to electronic structure theory and our DFTK code at last year's Juliacon, this year I participated at the conference with two events. One BoF discussion session gathering the people working on materials-science and electronic-structure codes in Julia about which I will write some more in a follow-up blog article.

My second event was a three-hour workshop titled A mathematical look at electronic structure theory in which I prepared a broadly accessible introduction into density-functional theory (DFT), the numerical procedures to solve DFT as well as some tools from numerical analysis to understand the convergence properties of these methods. As the tool to conduct the relevant calculations, code up and study the respective self-consistent field (SCF) algorithms we used our density-functional toolkit (DFTK). The workshop therefore also provides a great showcase for the merits of this code and how it leverages the broader Julia ecosystem to gain its unique features (arbitrary floating-point types, flexible and composable algorithms, automatic differentiation, numerical analysis techniques to investigate convergence failures, etc. ). For more details on the workshop see the dedicated teaching page.

What surprised me very positively during conducting the workshop was the large number of viewers that followed the workshop live and actively engaged by asking questions or posting comments on Youtube. Since the workshop was hosted at Juliacon I wouldn't have thought this topic would capture this many people, so in retrospect I am very happy I did it. In that sense also a big thanks to everyone who participated and provided me with feedback afterwards. (BTW: I'm still happy to take any in case you have some comments or suggesitons).

In case you missed the workshop the complete materials are available on github and the full recording of the workshop is available on Youtube.

by Michael F. Herbst at 2021-08-03 10:00 under workshop, electronic structure theory, Kohn-Sham, high-throughput, DFT, DFTK, solid state, Julia

2021-08-01

michael-herbst.com

Virtual materials design 2021: Black-box density-functional theory methods

On 20th and 21st July 2021 the Virtual Materials Design 2021 CECAM workshop took place virtually. I was excited about this workshop and the opportunity to get in touch with researchers working on high-throughput computational materials design. While I am not actively working in this field the special requirements of the multitude of calculations running in this field clearly have been one of the main motivations for my work on DFTK, error control and black-box SCF algorithms. In advance of the workshop I asked the organisers to participate with a contributed talk to present my work to this community for the first time, which thankfully got accepted.

Due to the virtual format the workshop it was unfortunately rather packed, which allowed for little time to engage in discussion during the presentation slot. However, the organisers arranged multiple longer poster sessions in a GatherTown virtual world, which allowed for almost realistic face-to-face discussions. In these GatherTown sessions I talked with a number of scientists working on high-throughput studies as well as designing the large software infrastructures, which are commonly used to conduct these. At the level of performing millions of individual calculations in a screening study this naturally poses especial demands on the workflow software as well and I was curious to learn about some of the details.

With my focus on advocating a more mathematical look at screening and DFT simulations I represented a minority viewpoint at the meeting and I was very curious about the general feedback and critique of the more applied scientists in response to our recently proposed ideas. In general people were indeed quite interested to learn about our work on reliable SCF methods for inhomogeneous systems, but being confronted with our recent error estimation perspectives, some had doubts about the required effort being really worth it for DFT simulations. I certainly understand that concern. However, I think one should keep in mind the successes and potential, which has been unlocked by error estimation techniques in other fields, such as finite-element modelling or aerospace design. In these fields simulation methods have both become more efficient due to the lessons learned from uncertainty quantification and error estimation and the nowadays well-established error estimation techniques have furthermore contributed to prevent accidents from trusting faulty simulation data (such as the Sleipner A oil rig collapse). While clearly not all aspects of macroscopic modelling apply in the microscopic world, it is not hard to imagine that error bars establishing a guaranteed trustworthiness can make screening decisions more robust, thus potentially preventing costly manufacture of less useful compounds. Furthermore I expect a careful introduction of numerical errors (e.g. by lowering the floating-point type) to balance numerical error against the (usually much larger) DFT model error to allow for notable computational savings when performing on the order of millions of DFT calculations.

Overall I have enjoyed the two afternoons with many discussions in the high-throughput design community. As usual my slides are attached below.

Link Licence
Towards error-controlled, black-box density-functional theory methods (Slides) Creative Commons License

by Michael F. Herbst at 2021-08-01 10:00 under talk, electronic structure theory, Kohn-Sham, high-throughput, DFT, DFTK, solid state

2021-07-15

michael-herbst.com

SSD Seminar: Accelerating the discovery of tomorrow's materials by robust and error-controlled simulations

A couple of days ago, on 12th July, I was invited to present my research in the SSD Seminar Series of RWTH Aachen. Being part of the research training group on modern inverse problems as well as the School for Simulation and Data Science (SSD) the SSD seminars are interdisciplinary and feature researchers as well as Master-level students from a couple of departments at RWTH (mathematics, computer science, simulation sciences, ...).

To make my recent work on error estimation and the design of robust algorithms for density-functional theory broadly accessible I started by motivating the need for density-functional theory (DFT) and high-throughput methods for the discovery and design of novel materials. Afterwards I briefly hinted at the mathematical structure of the equations, which need to be solved to obtain DFT properties. With this in mind I presented current research questions at the edge of mathematics and electronic-structure modelling and presented some of my recent results. As usual the slides are attached below.

Link Licence
Accelerating the discovery of tomorrow's materials by robust and error-controlled electronic-structure simulations (Slides) Creative Commons License

by Michael F. Herbst at 2021-07-15 10:01 under talk, electronic structure theory, Kohn-Sham, high-throughput, DFT, solid state

2021-07-15

michael-herbst.com

Talk at many-body seminar at RWTH

On 29th June I was invited to present a short summary of my research at the seminar of the research training group Quantum Many-Body Methods at RWTH Aachen University. In the talk I give a overview over my ongoing work about reliable black-box self-consistent field schemes for high-throughput DFT calculations. My slides are attached below.

Link Licence
Reliable black-box self-consistent field schemes for high-throughput DFT calculations (Slides) Creative Commons License

by Michael F. Herbst at 2021-07-15 10:00 under talk, electronic structure theory, Kohn-Sham, high-throughput, DFT, solid state

2021-07-10

sECuREs website

25 Gigabit Linux internet router PC build

init7 recently announced that with their FTTH fiber offering Fiber7, they will now sell and connect you with 25 Gbit/s (Fiber7-X2) or 10 Gbit/s (Fiber7-X) fiber optics, if you want more than 1 Gbit/s.

While this offer will only become available at my location late this year (or possibly later due to the supply chain shortage), I already wanted to get the hardware on my end sorted out.

After my previous disappointment with the MikroTik CCR2004, I decided to try a custom PC build.

An alternative to many specialized devices, including routers, is to use a PC with an expansion card. An internet router’s job is to configure a network connection and forward network packets. So, in our case, we’ll build a PC and install some network expansion cards!

router PC build

Goals

For this PC internet router build, I had the following goals, highest priority to lowest priority:

  1. Enough performance to saturate 25 Gbit/s, e.g. with two 10 Gbit/s downloads.
  2. Silent: no loud fan noise.
  3. Power-efficient: low power usage, as little heat as possible.
  4. Low cost (well, for a high-end networking build…).

Network Port Plan

The simplest internet router has 2 network connections: one uplink to the internet, and the local network. You can build a router without extra cards by using a mainboard with 2 network ports.

Because there are no mainboards with SFP28 slots (for 25 Gbit/s SFP28 fiber modules), we need at least 1 network card for our build. You might be able to get by with a dual-port SFP28 network card if you have an SFP28-compatible network switch already, or need just one fast connection.

I want to connect a few fast devices (directly and via fiber) to my router, so I’m using 2 network cards: an SFP28 network card for the uplink, and a quad-port 10G SFP+ network card for the local network (LAN). This leaves us with the following network ports and connections:

Network Card max speed cable effective Connection
Intel XXV710 25 Gbit/s fiber 25 Gbit/s Fiber7-X2 uplink
Intel XXV710 25 Gbit/s DAC 10 Gbit/s workstation
Intel XL710 10 Gbit/s RJ45 1 Gbit/s rest (RJ45 Gigabit)
Intel XL710 10 Gbit/s fiber 10 Gbit/s MikroTik 1
Intel XL710 10 Gbit/s fiber 10 Gbit/s MikroTik 2
Intel XL710 10 Gbit/s / 10 Gbit/s (unused)
onboard 2.5 Gbit/s RJ45 1 Gbit/s (management)
network connectors

Hardware selection

Now that we have defined the goals and network needs, let’s select the actual hardware!

Network Cards

My favorite store for 10 Gbit/s+ network equipment is FS.COM. They offer Intel-based cards:

Network cards

Both cards work out of the box with the i40e Linux kernel driver, no firmware blobs required.

For a good overview over the different available Intel cards, check out the second page (“Product View”) in the card’s User Manual.

CPU and Chipset

I read on many different sites that AMD’s current CPUs beat Intel’s CPUs in terms of performance per watt. We can better achieve goals 2 and 3 (low noise and low power usage) by using fewer watts, so we’ll pick an AMD CPU and mainboard for this build.

AMD’s current CPU generation is Zen 3, and current Zen 3 based CPUs can be divided into 65W TDP (Thermal Design Power) and 105W TDP models. Only one 65W model is available to customers right now: the Ryzen 5 5600X.

Mainboards are built for/with a certain so-called chipset. Zen 3 CPUs use the AM4 socket, for which 8 different chipsets exist. Our network cards need PCIe 3.0, so that disqualifies 5 chipsets right away: only the A520, B550 and X570 chipsets remain.

Ryzen 5

Mainboard: PCIe bandwidth

I originally tried using the ASUS PRIME X570-P mainboard, but I ran into two problems:

Too loud: X570 mainboards need an annoyingly loud chipset fan for their 15W TDP. Other chipsets such as the B550 don’t need a fan for their 5W TDP. With a loud chipset fan, goal 2 (low noise) cannot be achieved. Only the recently-released X570S variant comes without fans.

Not enough PCIe bandwidth/slots! This is how the ASUS tech specs describe the slots:

This means the board has 2 slots (1 CPU, 1 chipset) that are physically wide enough to hold a full-length x16 card, but only the first port can electronically be used as an x16 slot. The other port only has PCIe lanes electronically connected for x4, hence “x16 (max at x4 mode)”.

Unfortunately, our network cards need electrical connection of all their PCIe x8 lanes to run at full speed. Perhaps Intel/FS.COM will one day offer a new generation of network cards that use PCIe 4.0, because PCIe 4.0 x4 achieves the same 7.877 GB/s throughput as PCIe 3.0 x8. Until then, I needed to find a new mainboard.

Searching mainboards by PCIe capabilities is rather tedious, as mainboard block diagrams or PCIe tree diagrams are not consistently available from all mainboard vendors.

Instead, we can look explicitly for a feature called PCIe Bifurcation. In a nutshell, PCIe bifurcation lets us divide the PCIe bandwidth from the Ryzen CPU from 1 PCIe 4.0 x16 into 1 PCIe 4.0 x8 + 1 PCIe 4.0 x8, definitely satisfying our requirement for two x8 slots at full bandwidth.

I found a list of (only!) three B550 mainboards supporting PCIe Bifurcation in an Anandtech review. Two are made by Gigabyte, one by ASRock. I read the Gigabyte UEFI setup is rather odd, so I went with the ASRock B550 Taichi mainboard.

Case

For the case, I needed a midi case (large enough for the B550 mainboard’s ATX form factor) with plenty of options for large, low-spinning fans.

I stumbled upon the Corsair 4000D Airflow, which is available for 80 CHF and achieved positive reviews. I’m pleased with the 4000D: there are no sharp corners, installation is quick, easy and clean, and the front and top panels offer plenty of space for cooling behind large air intakes:

Airflow case (from the top)

Inside, the case offers plenty of space and options for routing cables on the back side:

Airflow case (back)

Which in turn makes for a clean front side:

Airflow case (front)

Fans

I have been happy with Noctua fans for many years. In this build, I’m using only Noctua fans so that I can reach goal 2 (silent, no loud fan noise):

Noctua fans

These fans are large (140mm), so they can spin on slow speeds and still be effective.

The specific fan configuration I ended up with:

  • 1 Noctua NF-A14 PWM 140mm in the front, pulling air out of the case
  • 1 Noctua NF-A14 PWM 140mm in the top, pulling air into the case
  • 1 Noctua NF-A12x25 PWM 120mm in the back, pulling air into the case
  • 1 Noctua NH-L12S CPU fan

Note that this is most likely overkill: I can well imagine that I could turn off one of these fans entirely without a noticeable effect on temperatures. But I wanted to be on the safe side and have a lot of cooling capacity, as I don’t know how hot the Intel network cards run in practice.

Fan Controller

The ASRock B550 Taichi comes with a Nuvoton NCT6683D-T fan controller.

Unfortunately, ASRock seems to have set the Customer ID register to 0 instead of CUSTOMER_ID_ASROCK, so you need to load the nct6683 Linux driver with its force option.

Once the module is loaded, lm-sensors lists accurate PWM fan speeds, but the temperature values are mislabeled and don’t quite match the temperatures I see in the UEFI H/W Monitor:

nct6683-isa-0a20
Adapter: ISA adapter
fan1:              471 RPM  (min =    0 RPM)
fan2:                0 RPM  (min =    0 RPM)
fan3:                0 RPM  (min =    0 RPM)
fan4:                0 RPM  (min =    0 RPM)
fan5:                0 RPM  (min =    0 RPM)
fan6:                0 RPM  (min =    0 RPM)
fan7:                0 RPM  (min =    0 RPM)
Thermistor 14:     +45.5 C  (low  =  +0.0 C)
                            (high =  +0.0 C, hyst =  +0.0 C)
                            (crit =  +0.0 C)  sensor = thermistor
AMD TSI Addr 98h:  +40.0 C  (low  =  +0.0 C)
                            (high =  +0.0 C, hyst =  +0.0 C)
                            (crit =  +0.0 C)  sensor = AMD AMDSI
intrusion0:       OK
beep_enable:      disabled

At least with the nct6683 Linux driver, there is no way to change the PWM fan speed: the corresponding files in the hwmon interface are marked read-only.

At this point I accepted that I won’t be able to work with the fan controller from Linux, and tried just configuring static fan control settings in the UEFI setup.

But despite identical fan settings, one of my 140mm fans would end up turned off. I’m not sure why — is it an unclean PWM signal, or is there just a bug in the fan controller?

Controlling the fans to reliably spin at a low speed is vital to reach goal 2 (low noise), so I looked around for third-party fan controllers and found the Corsair Commander Pro, which a blog post explains is compatible with Linux.

Server Disk

This part of the build is not router-related, but I figured if I have a fast machine with a fast network connection, I could add a fast big disk to it and retire my other server PC.

Specifically, I chose the Samsung 970 EVO Plus M.2 SSD with 2 TB of capacity. This disk can deliver 3500 MB/s of sequential read throughput, which is more than the ≈3000 MB/s that a 25 Gbit/s link can handle.

Graphics Card

An important part of computer builds for me is making troubleshooting and maintenance as easy as possible. In my current tech landscape, that translates to connecting an HDMI monitor and a USB keyboard, for example to boot from a different device, to enter the UEFI setup, or to look at Linux console messages.

Unfortunately, the Ryzen 5 5600X does not have integrated graphics, so to get any graphics output, we need to install a graphics card. I chose the Zotac GeForce GT 710 Zone Edition, because it was the cheapest available card (60 CHF) that’s passively cooled.

An alternative to using a graphics card might be to use a PCIe IPMI card like the ASRock PAUL, however these seem to be harder to find, and more expensive.

Longer-term, I think the best option would be to use the Ryzen 5 5600G with integrated graphics, but that model only becomes available later this year.

Component List

I’m listing 2 different options here. Option A is what I built (router+server), but Option B is a lot cheaper if you only want a router. Both options use the same base components:

Price Type Article
347 CHF Network card FS.COM Intel XXV710, 2 × 25 Gbit/s (#75603)
329 CHF Network card FS.COM Intel XL710, 4 × 10 Gbit/s (#75602)
314 CHF CPU Ryzen 5 5600X
290 CHF Mainboard ASRock B550 Taichi
92 CHF Case Corsair 4000D Airflow (Midi Tower)
67 CHF Fan control Corsair Commander Pro
65 CHF Case fan 2 × Noctua NF-A14 PWM (140mm)
62 CHF CPU fan Noctua NH-L12S
35 CHF Case fan 1 × Noctua NF-A12x25 PWM (120mm)
60 CHF GPU Zotac GeForce GT 710 Zone Edition (1GB)

Base total: 1590 CHF

Option A: Server extension. Because I had some parts lying around, and because I wanted to use my router for serving files (from large RAM cache/fast disk), I went with the following parts:

Price Type Article
309 CHF Disk Samsung 970 EVO Plus 2000GB, M.2 2280
439 CHF RAM 64GB HyperX Predator RAM (4x, 16GB, DDR4-3600, DIMM 288)
127 CHF Power supply Corsair SF600 Platinum (600W)
14 CHF Power ext Silverstone ATX 24-24Pin Extension (30cm)
10 CHF Power ext Silverstone ATX Extension 8-8(4+4)Pin (30cm)

The Corsair SF600 power supply is not server-related, I just had it lying around. I’d recommend going for the Corsair RM650x *2018* (which has longer cables) instead.

Server total: 2770 CHF

Option B: Non-server (router only) alternative. If you’re only interested in routing, you can opt for cheaper low-end disk and RAM, for example:

Price Type Article
112 CHF Power supply Corsair RM650x *2018*
33 CHF Disk Kingston A400 120GB M.2 SSD
29 CHF RAM Crucial CT4G4DFS8266 4GB DDR4-2666 RAM

Non-server total: 1764 CHF

ASRock B550 Taichi Mainboard UEFI Setup

To enable PCIe Bifurcation for our two PCIe 3.0 x8 card setup:

  1. Set Advanced > AMD PBS > PCIe/GFX Lanes Configuration
    to x8x8.

To always turn on the PC after power is lost:

  1. Set Advanced > Onboard Devices Configuration > Restore On AC Power Loss
    to Power On.

To PXE boot (via UEFI) on the onboard ethernet port (management), but disable slow option roms for PXE boot on the FS.COM network cards:

  1. Set Boot > Boot From Onboard LAN
    to Enabled.
  2. Set Boot > CSM (Compatibility Support Module) > Launch PXE OpROM Policy
    to UEFI only.

Fan Controller Setup

The Corsair Commander Pro fan controller is well-supported on Linux.

After enabling the Linux kernel option CONFIG_SENSORS_CORSAIR_CPRO, the device shows up in the hwmon subsystem.

You can completely spin up (100% PWM) or turn off (0% PWM) a fan like so:

# echo 255 > /sys/class/hwmon/hwmon3/pwm1
# echo 0 > /sys/class/hwmon/hwmon3/pwm1

I run my fans at 13% PWM, which translates to about 226 rpm:

# echo 33 > /sys/class/hwmon/hwmon3/pwm1
# cat /sys/class/hwmon/hwmon3/fan1_input
226

Conveniently, the Corsair Commander Pro stores your settings even when power is lost. So you don’t even need to run a permanent fan control process, a one-off adjustment might be sufficient.

Power Usage

The PC consumes about 48W of power when idle (only management network connected) by default without further tuning. Each extra network link increases power usage by ≈1W:

graph showing power consumption when enabling network links

Enabling all Ryzen-related options in my Linux kernel and switching to the powersave CPU frequency governor lowers power usage by ≈1W.

On some mainboards, you might need to force-enable Global C-States to save power. Not on the B550 Taichi, though.

I tried undervolting the CPU, but that didn’t even make ≈1W of difference in power usage. Potentially making my setup unreliable is not worth that little power saving to me.

I measured these values using a Homematic HM-ES-PMSw1-Pl-DN-R5 I had lying around.

Performance

Goal 1 is to saturate 25 Gbit/s, for example using two 10 Gbit/s downloads. I’m talking about large bulk transfers here, not many small transfers.

To get a feel for the performance/headroom of the router build, I ran 3 different tests.

Test A: 10 Gbit/s bridging throughput

For this test, I connected 2 PCs to the router’s XL710 network card and used iperf3(1) to generate a 10 Gbit/s TCP stream between the 2 PCs. The router doesn’t need to modify the packets in this scenario, only forward them, so this should be the lightest load scenario.

bridging throughput

Test B: 10 Gbit/s NAT throughput

In this test, the 2 PCs were connected such that the router performs Network Address Translation (NAT), which is required for downloads from the internet via IPv4.

This scenario is slightly more involved, as the router needs to modify packets. But, as we can see below, a 10 Gbit/s NAT stream consumes barely more resources than 10 Gbit/s bridging:

NAT throughput

Test C: 4 × 10 Gbit/s TCP streams

In this test, I wanted to max out the XL710 network card, so I connected 4 PCs and started an iperf3(1) benchmark between each PC and the router itself, simultaneously.

This scenario consumes about 16% CPU, meaning we’ll most likely have plenty of headroom even when all ports are maxed out!

four 10 Gbit/s streams

Tip: make sure to enable the CONFIG_IRQ_TIME_ACCOUNTING Linux kernel option to include IRQ handlers in CPU usage numbers for accurate measurements.

Alternatives considered

The passively-cooled SuperServer E302-9D comes with 2 SFP+ ports (10 Gbit/s). It even comes with 2 PCIe 3.0 x8 capable slots. Unfortunately it seems impossible to currently buy this machine, at least in Switzerland.

You can find a few more suggestions in the replies of this Twitter thread. Most are either unavailable, require a lot more DIY work (e.g. a custom case), or don’t support 25 Gbit/s.

Router software: router7 porting

I wrote router7, my own small home internet router software in Go, back in 2018, and have been using it ever since.

I don’t have time to support any users, so I don’t recommend anyone else use router7, unless the project really excites you, and the lack of support doesn’t bother you! Instead, you might be better served with a more established and supported router software option. Popular options include OPNsense or OpenWrt. See also Wikipedia’s List of router and firewall distributions.

To make router7 work for this 25 Gbit/s router PC build, I had to make a few adjustments.

Because we are using UEFI network boot instead of BIOS network boot, I first had to make the PXE boot implementation in router7’s installer work with UEFI PXE boot.

I then enabled a few additional kernel options for network and storage drivers in router7’s kernel.

To router7’s control plane code, I added bridge network device configuration, which in my previous 2-port router setup was not needed.

During development, I compiled a few Linux programs statically or copied them with their dependencies (→ gokrazy prototyping) to run them on router7, such as sensors(1) , ethtool(8) , as well as iproute2’s ip(8) and bridge(8) implementation.

Next Steps

Based on my tests, the hardware I selected seems to deliver enough performance to use it for distributing a 25 Gbit/s upstream link across multiple 10 Gbit/s devices.

I won’t know for sure until the fiber7 Point Of Presence (POP, German Anschlusszentrale) close to my home is upgraded to support 25 Gbit/s “Fiber7-X2” connections. As I mentioned, unfortunately the upgrade plan is delayed due to the component shortage. I’ll keep you posted!

Appendix A: DPDK test

Pim ran a DPDK based loadtester called T-Rex on this machine. Here’s his summary of the test:

For DPDK, this hardware does 4x10G at 64b frames. It does not do 6x10G as it tops out at 62Mpps using 4 cores (of 15.5Mpps per core).

I couldn't test 25G symmetric [because we lacked a 25G DAC cable], but extrapolating from the numbers, 3 CPUs source and sink ~24.6Gbit per core, so we'd probably make it, leaving 1 core for OS and 2 cores for controlplane.

If the machine had a 12 core Ryzen, it would saturate all NICs with room to spare. So that's what I'll end up buying :)

DPDK test

at 2021-07-10 11:43

2021-06-21

michael-herbst.com

Errors and uncertainty quantification in density-functional theory

On 8th June I was invited to the seminar of the Uncertainty Quantification (UQ) group of Prof. Youssef Marzouk at MIT. Youssef and I planned to have this seminar since my involvement with MIT's CESMIX project last February (see also this blog article), but it took use quite some time to get it arranged. Finally I managed to present my point of view on UQ in density-functional theory (DFT), sneakily re-using most of the slides I had already prepared for my recent UQ-in-DFT talk at RWTH Aachen's UQ group the week earlier.

Similar to the Aachen talk I've put strong emphasis on engaging audience participation and discussion. I first introduced the UQ group to electronic structure theory and DFT, allowing for enough time to discuss the key ideas of the physics. Then I pointed out current research in error estimation and UQ in DFT and provided a number of opportunities for interesting future UQ-related research. The discussion was very lively and I hardly made it beyond a slide without a question, which was just great. Since a lot could be gained from stronger uncertainty quantification tools in DFT in my opinion, I hoped this talk made DFT more accessible to the UQ group and made some people curious to look into the details. On my end I would definitely enjoy to learn more about UQ in the future and look forward to my future UQ-related involvements in the CESMIX project. As usual my slides are attached below.

Link Licence
Errors and uncertainty quantification in electronic-structure theory (Slides) Creative Commons License

by Michael F. Herbst at 2021-06-21 10:00 under talk, electronic structure theory, Kohn-Sham, uncertainty quantification, DFT, solid state

2021-06-15

michael-herbst.com

Talk at MATH4UQ seminar series at RWTH

On 1st June I was invited to the MATH4UQ seminar series of the Mathematics of Uncertainty Quantification chair of Prof. Raul Tempone at RWTH Aachen University.

Over the past months I got more and more interested in mathematical methods for uncertainty quantification (UQ) as an opportunity to estimate and understand errors in density-functional theory (DFT) calculations. In particular I imagine UQ methods to be useful to estimate the model error of a DFT model itself. At this level statistical approaches are likely the only feasible option for a practical error estimation, since the mathematical complexity of modern DFT models beyond the local density approximations very likely make a posteriori error analysis strategies extremely infeasible.

In my talk I explain the basics of DFT and provide a rough overview of present UQ developments in this method. Since I know very little about UQ and my audience knew very little about DFT, I intended the talk to be more of a Q&A session, where the slides are around to stimulate discussion. This turned out to work very well and I am very grateful to the many interesting questions from the audience and the enjoyful discussion. As usual my slides are attached below. Additionally a recording of my talk can be found on youtube.

Link Licence
Errors in electronic-structure theory: Status and directions for future research (Slides) Creative Commons License
Youtube recording of the talk

by Michael F. Herbst at 2021-06-15 10:00 under talk, electronic structure theory, Kohn-Sham, uncertainty quantification, DFT, solid state

2021-06-05

sECuREs website

Laptop review: ThinkPad X1 Extreme (Gen 2)

ThinkPad X1 Extreme Gen 2, pear for scale

For many of my school and university years, I used and liked my ThinkPad X200 ultraportable laptop. But now that these years are long gone, I realized my use-case for laptops had changed: instead of carrying my laptop with me every day, I am now only bringing it on occasion, for example when I travel to conferences, visit friends, or do volunteer work.

After the ThinkPad X200, I used a few different laptops:

  • MacBook Pro 13" Retina, bought for its screen
  • ThinkPad X1 Carbon, which newly introduced a hi-dpi screen to ThinkPads
  • Dell XPS 9360, for a change, to try a device that ships with Linux

With each of these devices, I have felt limited by the lack of connectors and slim compute power that comes with the Ultrabook brand, even after years of technical progress.

More compute power is nice to be able to work on projects with larger data sets, for example debiman (scanning and converting all manpages in Debian), or distri (building Linux packages).

More peripheral options such as USB ports are nice when connecting a keyboard, trackball, USB-to-serial adapter, etc., to work on a micro controller or Raspberry Pi project, for example.

So, I was ready to switch from the heaviest Ultrabooks to the lightest of the “mobile workstation” category, when I stumbled upon Lenovo’s ThinkPad X1 Extreme (Gen 2), and it piqued my curiosity.

Peripherals

Let me start by going into the key peripherals of a laptop: keyboard, touchpad and screen. I will talk about these independently from the remaining hardware because they define the experience of using the computer.

Keyboard

After having used the Dell XPS 9360 for a few years, I can confidently say that the keyboard of the ThinkPads is definitely much better, and in a noticeable way.

It’s not that the Dell keyboards are bad. But comparing the Dell and ThinkPad side-by-side makes it really clear that the ThinkPad keyboards are the best notebook keyboards.

On the ThinkPad keyboard, every key press lands exactly as I imagine. Never do I need to hit a key twice because I didn’t press it just-right, and never do I notice additional ghost key presses.

Even though I connect my external Kinesis Advantage keyboard when doing longer stretches of work, the quality of the built-in keyboard matters: a good keyboard enables using the laptop on the couch.

Touchpad

Unfortunately, while the keyboard is great, I can’t say the same about the touchpad. I mean, it’s not terrible, but it’s also not good by any stretch.

This seems to be the status quo with PC touchpads for decades. It really blows my mind that Apple’s touchpads are consistently so much better!

My only hope is that Bill Harding (GitClear), who is working on improving the Linux touchpad experience, will eventually find a magic software tweak or something…

As mentioned on the ArchWiki, I also had to adjust the sensitivity like so:

% xinput set-prop 'SynPS/2 Synaptics TouchPad' 'libinput Accel Speed' 0.5

Display

I have high demands regarding displays: since 2013, every device of mine has a hi-dpi display.

The industry hasn’t improved displays across the board as fast as I’d like, so non-hi-dpi displays are still quite common. The silver lining is that it makes laptop selection a little easier for me: anything without a decent display I can discard right away.

I’m glad to report that the 4K display in the ThinkPad X1 Extreme with its 3840x2160 pixels is sharp, bright, and generally has good viewing angles.

It’s also a touchscreen, which I don’t strictly need, but it’s nice to use it from time to time.

I use the display in 200% scaling mode, i.e. I set Xft.dpi: 192. See also HiDPI in ArchWiki.

Hardware

Spec-wise, the ThinkPad X1 Extreme is a beast!

ThinkPad X1 Extreme Specs

The build quality seems very robust to me.

Another big plus of the ThinkPad series over other laptop series is the availability of the official Hardware Maintenance Manual: you can put “ThinkPad X1 Extreme Gen 2 Hardware Maintenance Manual” into Google and will find p1_gen2_x1extreme_hmm_v1.pdf as the first hit. This manual describes in detail how to repair or upgrade your device if you want to (or have to) do it yourself.

WiFi

The built-in Intel AX200 WiFi interface works fine, provided you have a new-enough linux-firmware package and kernel version installed.

I had trouble with Linux 5.6.0, and Linux 5.6.5 fixed it. Luckily, at the time of writing, Linux 5.11 is the most recent release, so most distributions should be recent enough for things to just work.

The WiFi card reaches almost the same download speed as the most modern WiFi device I can test: a MacBook Air M1. Both are connected to my UniFi UAP-AC-HD access point.

Laptop Download Upload
ThinkPad X1 Extreme 500 Mbit/s 150 Mbit/s
MacBook Air M1 600 Mbit/s 500 Mbit/s

I’m not sure why the upload speed is so low in comparison.

GPU

The GPU in this machine is by far the most troublesome bit of hardware.

I had hoped that after many years of laptops containing Intel/nVidia hybrid graphics, this setup would largely work, but was disappointed.

Both the proprietary nVidia driver and the nouveau driver would not work reliably for me. I ran into kernel error messages and hard-freezes, with even SSH sessions to the machine breaking.

In the end, I blacklisted the nouveau driver to use Intel graphics only:

% echo blacklist nouveau | sudo tee /etc/modprobe.d/blacklist.conf 

Without the nVidia driver, the GPU will not go into powersave mode, so I remove it from the PCI bus entirely to save power:

#!/bin/zsh

sudo tee /sys/bus/pci/devices/0000\:01\:00.0/remove <<<1
sudo tee /sys/bus/pci/devices/0000\:01\:00.1/remove <<<1

You can only re-awaken the GPU with a reboot.

Obviously this isn’t a great setup — I would prefer to be able to actually use the GPU. If you have any tips or a better experience, please let me know.

Also note that the HDMI port will be unusable if you go this route, as the HDMI port is connected to the nVidia GPU only.

Battery life

The 80 Wh battery lasts between 5 to 6 hours for me, without any extra power saving tuning beyond what the Linux distribution Fedora 33 comes with by default.

This is good enough for using the laptop away from a power socket from time to time, which matches my expectation for this kind of mobile workstation.

Software support

Linux support is generally good on this machine! Yes, I provide a few pointers in this article regarding problems, patches and old software versions. But, if you use a newer Linux distribution, all of these fixes are included and things just work out of the box. I tested with Fedora 33.

For a few months, I was using this laptop exclusively with my research Linux distribution distri, so even if you just track upstream software closely, the machine works well.

Firmware updates

Lenovo partnered with the Linux Vendor Firmware Service Project (LVFS), which means that through fwupd, ThinkPad laptops such as this X1 Extreme can easily receive firmware updates!

This is a huge improvement in comparison to earlier ThinkPad models, where you had to jump through hoops with Windows-only software, or CD images that you needed to boot just right.

If your laptop has a very old firmware version (before 1.30), you might be affected by the skipping keystrokes issues. You can check using the always-handy lshw(1) tool.

Performance

The specific configuration of my ThinkPad is:

ThinkPad X1 Extreme Spec (2020)
CPU Intel Core i7-9750H CPU @ 2.60GHz
RAM 2 × 32 GB Samsung M471A4G43MB1-CTD
Disk 2 × SAMSUNG MZVLB2T0HALB-000L7 NVMe disk

You can google for CPU benchmarks and comparisons yourself, and those likely are more scientific and carefully done than I have time for.

What I can provide however, is a comparison of working on one of my projects on the ThinkPad vs. on my workstation, an Intel Core i9-9900K that I bought in 2018:

Workstation Spec (2018)
CPU Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
RAM 4 × Corsair CMK32GX4M2A2666C16
Disk Corsair Force MP600 M.2 NVMe disk

Specifically, I am comparing how long my manpage static archive generator debiman takes to analyze and render all manpages in Debian unstable, using the following command:

ulimit -n 8192; time ~/go/bin/debiman \
  -keyring=/usr/share/keyrings/debian-archive-keyring.gpg \
  -sync_codenames=, \
  -sync_suites=unstable \
  -serving_dir=/srv/man/benchmark \
  -inject_assets=~/go/src/github.com/Debian/debiman/debian-assets \
  -concurrency_render=20 \
  -alternatives_dir=~/go/src/github.com/Debian/debiman/piuparts

On both machines, I ensured that:

  1. The CPU performance governor was set to performance
  2. A warm apt-cacher-ng cache was present, i.e. network download was not part of the test.
  3. Linux kernel caches were dropped using echo 3 | sudo tee /proc/sys/vm/drop_caches
  4. I was using debiman git revision f78c160

Here are the results:

Machine Time
i9-9900K Workstation 4:57,10 (100%)
ThinkPad X1 Extreme (Gen 2) 7:19,56 (147%)

This reaffirms my impression that even high-end laptop hardware just cannot beat a workstation setup (which has more space and better thermals), but it comes close enough to be useful.

Conclusion

Positives:

  • The ergonomics of the device really are great. It is a pleasure to type on a first-class, full-size ThinkPad keyboard. The screen has good quality and a high resolution.

  • Performance-wise, this machine can almost replace a proper workstation.

Negatives are:

  • the mediocre battery life
  • an annoyingly loud fan that spins up too frequently
  • poor software/driver support for hybrid nVidia GPUs.

Notably, all of these could be improved by better power saving, so perhaps it’s just a matter of time until Linux kernel developers land some improvements…? :)

at 2021-06-05 18:43

2021-05-30

michael-herbst.com

SIAM LA: Robust and efficient accelerated methods for density-functional theory

Just one day after my talk at the SIAM Materials Science conference (blog article) I gave another talk at a SIAM meeting, this time at SIAM Linear Algebra. I was very much looking forward to participate in SIAM LA, firstly because it was the first time I attended this conference, but also secondly because it was a good opportunity to talk about our recent algorithmic work on robust DFT methods to an international crowd of mathematicians.

I presented as part of the minisymposium Theory and Practice of Extrapolation and Acceleration Methods, which consisted of three interesting sessions of historic and recent talks about extrapolation and convergence acceleration in the broadest sense of the word. Both topics about iterative methods as well as summation theory and sequence summation were discussed, which turned out to be a very enjoyful mix. In that sense I am really grateful for the mini organisers, Agnieszka Miedlar and Yousef Saad, for the invitation and for allowing me to be part of the great sessions.

Beyond the mini I enjoyed a number of talks about emerging topics in numerical linear algebra such as mixed-precision computation, low-rank tensor approximations or randomised methods. Even though the time zone difference meant that the conference was mostly running during the afternoon and late evening for me and even though the collision with SIAM Materials Science made it quite a busy week, I took a lot from SIAM LA and I'm already looking forward to next time.

Link Licence
Robust and Efficient Accelerated Methods for Kohn-Sham Density-Functional Theory Creative Commons License

by Michael F. Herbst at 2021-05-30 16:01 under talk, electronic structure theory, Julia, DFTK, numerical analysis, Kohn-Sham, high-throughput, DFT, solid state

2021-05-30

michael-herbst.com

SIAM MS: Using the density-functional toolkit to design black-box DFT methods

After being moved by one year due to the pandemic, the last two weeks (from 17th to 28th May) the SIAM materials science conference finally took place in virtual form. Unfortunately this meant that the conference was scheduled in parallel to the SIAM Linear Algebra virtual conference, where I also presented (blog article), which made my past two weeks rather busy.

At SIAM Materials I was invited to talk in the Minisymposium Numerics of electronic structure calculations, which was organised by the steering committee of the GAMM activity group moansi (Modelling, Analysis and Simulation of Molecular Systems). Besides interesting sessions about some mathematical insights to electronic-structure methods, this gave the mini the additional feature of a spring gathering for the usual crowd of the activity group, which I already had the pleasure to meet at previous moansi workshops.

In my talk I gave a broad overview of the recent projects we realised with the density-functional toolkit for making self-consistent field calculations for density-functional theory more robust and reliable. Apart from our work on preconditioning inhomogeneous systems with the LDOS preconditioner I also presented first work-in-progress results on an adaptive damping strategy we recently came up with. The idea of our method is to use a line search based on an approximate quadratic model for cases where a proposed SCF step is not successful (i.e. increases energy and SCF residual). This firstly allows to automatically choose the damping parameter (instead of requiring the user to choose one by trial and error). Secondly it makes the SCF procedure more robust, especially for tricky cases. For example in our experiments on Heusler alloys our adaptive damping approach was the only method that managed to converge on some cases.

Link Licence
Using the density-functional toolkit (DFTK) to design black-box methods in density-functional theory Creative Commons License

by Michael F. Herbst at 2021-05-30 16:00 under talk, electronic structure theory, Julia, DFTK, theoretical chemistry, numerical analysis, Kohn-Sham, high-throughput, DFT, solid state

2021-05-28

sECuREs website

How I configured and then promptly returned a MikroTik CCR2004 router for Fiber7

init7 recently announced that with their FTTH fiber offering Fiber7, they will now sell and connect you with 25 Gbit/s (Fiber7-X2) or 10 Gbit/s (Fiber7-X) fiber optics, if you want more than 1 Gbit/s.

This is possible thanks to the upgrade of their network infrastructure as part of their “lifecycle management”, meaning the old networking gear was declared as end-of-life. The new networking gear supports not only SFP+ modules (10 Gbit/s), but also SFP28 modules (25 Gbit/s).

Availability depends on the POP (Point Of Presence, German «Anschlusszentrale») you’re connected to. My POP is planned to be upgraded in September.

Nevertheless, I wanted to already prepare my end of the connection, and ordered the only router that init7 currently lists as compatible with Fiber7-X/X2: the MikroTik CCR2004-1G-12S+2XS.

MikroTik CCR2004-1G-12S+2XS

The rest of this article walks through what I needed to configure (a lot, compared to Ubiquiti or OpenWRT) in the hope that it helps other MikroTik users, and then ends in Why I returned it.

Configuration

Connect an Ethernet cable to the management port on the MikroTik and:

  1. log into the system using ssh admin@192.168.88.1
  2. point a web browser to “Webfig” at http://192.168.88.1/ (no login required)

Update firmware

Update the CCR2004 to the latest firmware version. At the time of writing, the Long-term RouterOS track is at version 6.47.9 for the CCR2004 (ARM64):

  1. Use /system package print to display the current version.
  2. Upload routeros-arm64-6.47.9.npk using Webfig.
  3. /system reboot and verify that /system package print shows 6.47.9 now.

Set up auth

Set a password to prevent others from logging into the router:

/user set admin password=secret

Additionally, you can enable passwordless SSH key login, if you want.

  1. Create an RSA key, because ed25519 keys are not supported:

    % ssh-keygen -t rsa
    Generating public/private rsa key pair.
    Enter file in which to save the key: /home/michael/.ssh/id_mikrotik
    
  2. Upload the id_mikrotik.pub file in Webfig

  3. Import the SSH public key for the admin user:

    /user ssh-keys import user=admin public-key-file=id_mikrotik.pub
    

Lock down the router

  1. Enable HTTPS in Webfig.

  2. Disable all remote access except for SSH and HTTPS:

    /ip service disable telnet,ftp,www,api,api-ssl,winbox
    
  3. Follow MikroTik Securing Your Router recommendations:

    /tool mac-server set allowed-interface-list=none
    /tool mac-server mac-winbox set allowed-interface-list=none
    /tool mac-server ping set enabled=no
    /tool bandwidth-server set enabled=no
    /ip ssh set strong-crypto=yes
    /ip neighbor discovery-settings set discover-interface-list=none
    

Enable DHCPv6 Client

For some reason, you need to explicitly enable IPv6 in 2021:

/system package enable ipv6
/system reboot

MikroTik says this is a precaution so that users don’t end up with default-open firewall settings for IPv6. But then why don’t they just add some default firewall rules?!

Anyway, to configure and immediately enable the DHCPv6 client, use:

/ipv6 dhcp-client add pool-name=fiber7 pool-prefix-length=64 interface=sfp28-1 add-default-route=yes use-peer-dns=no request=address,prefix

Modify the IPv6 DUID

Unfortunately, MikroTik does not offer any user interface to set the IPv6 DUID, which I need to configure to obtain my static IPv6 network prefix from my provider’s DHCPv6 server.

Luckily, the DUID is included in backup files, so we can edit it and restore from backup:

  1. Run /system backup save
  2. Download the backup file in Webfig by navigating to Files → Backup → Download.

  3. Convert the backup file to hex in textual form, edit the DUID and convert it back to binary:

    % xxd MikroTik-19700102-0111.backup MikroTik-19700102-0111.backup.hex
    
    % emacs MikroTik-19700102-0111.backup.hex
    # Search for “dhcp/duid” in the file and edit accordingly:
    # got:  00030001085531dfa69e
    
    % xxd -r MikroTik-19700102-0111.backup.hex MikroTik-19700102-0111-patched.backup
    
  4. Upload the file in Webfig, then restore the backup:

    /system backup load name=MikroTik-19700102-0111-patched.backup

Enable IPv6 Router Advertisements

To make the router assign an IPv6 address from the obtained pool for itself, and then send IPv6 Router Advertisements to the network, set:

/ipv6 address add address=::1 from-pool=fiber7 interface=bridge1
/ipv6 nd add interface=bridge1 managed-address-configuration=yes other-configuration=yes

Enable DHCPv4 Client

To configure and immediately enable the DHCPv4 client on the upstream port, use:

/ip dhcp-client add interface=sfp28-1 disabled=no

I also changed the MAC address to match my old router’s address, just to take maximum precaution to avoid any Port Security related issues with my provider’s DHCP server:

/interface ethernet set sfp28-1 mac-address=00:0d:fa:4c:0c:31

Enable DNS Server

By default, only the MikroTik itself can send DNS queries. Enable access for network clients:

/ip dns set allow-remote-requests=yes

Enable DHCPv4 Server

First, let’s bundle all SFP+ ports into a single bridge interface:

/interface bridge add name=bridge1
/interface bridge port add bridge=bridge1 interface=sfp-sfpplus1 hw=yes
/interface bridge port add bridge=bridge1 interface=sfp-sfpplus2 hw=yes
/interface bridge port add bridge=bridge1 interface=sfp-sfpplus3 hw=yes
/interface bridge port add bridge=bridge1 interface=sfp-sfpplus4 hw=yes
/interface bridge port add bridge=bridge1 interface=sfp-sfpplus5 hw=yes
/interface bridge port add bridge=bridge1 interface=sfp-sfpplus6 hw=yes
/interface bridge port add bridge=bridge1 interface=sfp-sfpplus7 hw=yes
/interface bridge port add bridge=bridge1 interface=sfp-sfpplus8 hw=yes
/interface bridge port add bridge=bridge1 interface=sfp-sfpplus9 hw=yes
/interface bridge port add bridge=bridge1 interface=sfp-sfpplus10 hw=yes
/interface bridge port add bridge=bridge1 interface=sfp-sfpplus11 hw=yes
/interface bridge port add bridge=bridge1 interface=sfp-sfpplus12 hw=yes

This means we’ll use the device like a big switch with routing between the switch and the uplink port sfp28-1.

To configure the DHCPv4 Server, configure an IP address, then start the setup wizard:

/ip address add address=10.0.0.1/24 interface=bridge1
/ip dhcp-server setup
Select interface to run DHCP server on

dhcp server interface: bridge1
Select network for DHCP addresses

dhcp address space: 10.0.0.0/24
Select gateway for given network

gateway for dhcp network: 10.0.0.1
Select pool of ip addresses given out by DHCP server

addresses to give out: 10.0.0.2-10.0.0.240
Select DNS servers

dns servers: 10.0.0.1
Select lease time

lease time: 20m

Enable IPv4 NAT

We need NAT to route all IPv4 traffic over our single public IP address:

/ip firewall nat add action=masquerade chain=srcnat out-interface=sfp28-1 to-addresses=0.0.0.0

Disable NAT services for security, e.g. to mitigate against NAT slipstreaming attacks:

/ip firewall service-port disable ftp,tftp,irc,h323,sip,pptp,udplite,dccp,sctp

I can observe ≈10-20% CPU load when doing a Gigabit speed test over IPv4.

TODO list

The following features I did not get around to configuring, but they were on my list:

Why I returned it

Initially, I thought the device’s fan spins up only at boot, and then the large heatsink takes care of all cooling needs. Unfortunately, after an hour or so into my experiment, I noticed that the MikroTik would spin up the fan for a whole minute or so occasionally! Very annoying.

I also ran into weird DNS slow-downs, which I didn’t fully diagnose. In Wireshark, it looked like my machine sent 2 DNS queries but received only 1 DNS result, and then waited for a timeout.

I also noticed that I have a few more unexpected dependencies such as my home automation using DHCP lease state by subscribing to an MQTT topic. Addressing this issue and other similar little problems would have taken a bunch more time and would have resulted in a less reliable system than I have today.

Since I last used MikroTik in 2014 the software seems to have barely changed. I wish they finally implemented some table-stakes features like DNS resolution for DHCP hostnames.

Given all the above, I no longer felt like getting enough value for the money from the MikroTik, and found it easier to just switch back to my own router7 and return the MikroTik.

I will probably stick with the router7 software, but exchange the PC Engines APU with the smallest PC that has enough PCI-E bandwidth for a multi-port SFP28 network card.

Appendix A: Full configuration

# may/28/2021 11:40:15 by RouterOS 6.47.9
# software id = 6YZE-HKM8
#
# model = CCR2004-1G-12S+2XS
/interface bridge
add name=bridge1
/interface ethernet
set [ find default-name=sfp28-1 ] auto-negotiation=no mac-address=00:0d:fa:4c:0c:31
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool0 ranges=10.0.0.2-10.0.0.240
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=bridge1 lease-time=20m name=dhcp1
/interface bridge port
add bridge=bridge1 interface=sfp-sfpplus1
add bridge=bridge1 interface=sfp-sfpplus2
add bridge=bridge1 interface=sfp-sfpplus3
add bridge=bridge1 interface=sfp-sfpplus4
add bridge=bridge1 interface=sfp-sfpplus5
add bridge=bridge1 interface=sfp-sfpplus6
add bridge=bridge1 interface=sfp-sfpplus7
add bridge=bridge1 interface=sfp-sfpplus8
add bridge=bridge1 interface=sfp-sfpplus9
add bridge=bridge1 interface=sfp-sfpplus10
add bridge=bridge1 interface=sfp-sfpplus11
add bridge=bridge1 interface=sfp-sfpplus12
/ip neighbor discovery-settings
set discover-interface-list=none
/ip address
add address=192.168.88.1/24 comment=defconf interface=ether1 network=192.168.88.0
add address=10.0.0.1/24 interface=bridge1 network=10.0.0.0
/ip dhcp-client
add disabled=no interface=sfp28-1 use-peer-dns=no
/ip dhcp-server lease
add address=10.0.0.54 mac-address=DC:A6:32:02:AA:10
/ip dhcp-server network
add address=10.0.0.0/24 dns-server=10.0.0.1 domain=lan gateway=10.0.0.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4,2001:4860:4860::8888,2001:4860:4860::8844
/ip firewall nat
add action=masquerade chain=srcnat out-interface=sfp28-1 to-addresses=0.0.0.0
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set www-ssl certificate=webfig disabled=no
set api disabled=yes
set winbox disabled=yes
set api-ssl disabled=yes
/ip ssh
set strong-crypto=yes
/ipv6 address
add address=::1 from-pool=fiber7 interface=bridge1
/ipv6 dhcp-client
add add-default-route=yes interface=sfp28-1 pool-name=fiber7 request=address,prefix use-peer-dns=no
/ipv6 nd
add interface=bridge1 managed-address-configuration=yes other-configuration=yes
/system clock
set time-zone-name=Europe/Zurich
/system logging
add topics=dhcp
/tool bandwidth-server
set enabled=no
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none
/tool mac-server ping
set enabled=no

at 2021-05-28 12:57

2021-05-20

RaumZeitLabor

GnOnlinePN – Workadventure Edition

Hey, ihr Aillioliebhaber!

Das wohlriechendste Ereignis des Jahres steht an: Am 5. Juni wird es garlicious bei der GnOnlinePN21!

Dieses Jahr wollen wir uns ab 20 Uhr im Workadventure treffen und gemeinsam dem weißen Knollengold huldigen. Untermalt wird das ganze mit den Klängen von seiner Durchlaucht, RZL Resident-DJ Asthma!

Genug mundgeruchgerechten Abstand bietet das Digit-ail-Event auch, sodass ihr euch mit dem Genuss knoblauchhaltiger Speisen und Getränke wirklich nicht zurückhalten müsst.

Lasst uns in der World alliumiteinander Garlic Bread grillen, Rezepte tauschen und mit Knoblauchtschunk anstoßen.

Den Link zur Teilnahme werden wir im Laufe des Veranstaltungsnachmittages über Twitter verbreiten. Bei Fragen oder Ideen könnt ihr uns via Mail oder Twitter erreichen.

Bis dann!
Man riecht sich!

Gnoblauchgnollen

by flederrattie at 2021-05-20 00:00

2021-05-16

sECuREs website

Home network 10 Gbit/s upgrade

After adding a fiber link to my home network, I am upgrading that link from 1 Gbit/s to 10 Gbit/s.

As a reminder, conceptually the fiber link is built using two media converters from/to ethernet:

0.9mm thin fiber cables

Schematically, this is what’s connected to both ends:

1 Gbit/s bottleneck

All links are 1 Gbit/s, so it’s easy to see that, for example, transfers between chuchi↔router7 and storage2↔midna cannot both use 1 Gbit/s at the same time.

This upgrade serves 2 purposes:

  1. Raise the floor to 1 Gbit/s end-to-end: Ensure that serving large files (e.g. distri Linux images and packages) does no longer impact, and is no longer impacted by, other bandwidth flows that also use this transfer link in my home network, e.g. daily backups.

  2. Raise the ceiling to 10 Gbit/s: Make it possible to selectively upgrade Linux PCs on either end of the link to 10 Gbit/s peak bandwidth.

Note that the internet uplink remains untouched at 1 Gbit/s — only transfers within the home network can happen at 10 Gbit/s.

Replacing the media converters with Mikrotik switches

We first replace both media converters and switches with a Mikrotik CRS305-1G-4S+IN.

Mikrotik CRS305-1G-4S+IN

This device costs 149 CHF on digitec and comes with 5 ports:

  • 1 × RJ45 Ethernet port for management, can be used as a regular 1 Gbit/s port.
  • 4 × SFP+ ports

Each SFP+ port can be used with either an RJ-45 Ethernet or a fiber SFP+ module, but beware! As Nexus2kSwiss points out on twitter, the Mikrotik supports at most 2 RJ-45 SFPs at a time!

Fiber module upgrade

I’m using 10 Gbit/s fiber SFP+ modules for the fiber link between my kitchen and living room.

To make use of the 10 Gbit/s link between the switches, all devices that should get their guaranteed 1 Gbit/s end-to-end connection need to be connected directly to a Mikrotik switch.

I’m connecting the PCs to the switch using Direct Attach Cables (DAC) where possible. The advantage of DAC cables over RJ45 SFP+ modules is their lower power usage and heat.

The resulting list of SFP modules used in the two Mikrotik switches looks like so:

Mikrotik 1 SFP speed speed Mikrotik 2 SFP
chuchi 10 Gbit/s DAC 10 Gbit/s DAC midna
storage2 1 Gbit/s RJ45 1 Gbit/s RJ45 router7
10 Gbit/s BiDi ⬅ BiDi fiber link ➡ 10 Gbit/s BiDi

Hardware sourcing

The total cost of this upgrade is 676 CHF, with the biggest chunk spent on the Mellanox ConnectX-3 network cards and MikroTik switches.

FS (Fiber Store) order

FS.COM was my go-to source for anything fiber-related. Everything they have is very affordable, and products in stock at their German warehouse arrive in Switzerland (and presumably other European countries, too) within the same week.

num price name
1 × 34 CHF Generic Compatible 10GBASE-BX BiDi SFP+ 1270nm-TX/1330nm-RX 10km DOM Transceiver Module, FS P/N: SFP-10G-BX #74681
1 × 34 CHF Generic Compatible 10GBASE-BX BiDi SFP+ 1330nm-TX/1270nm-RX 10km DOM Transceiver Module, FS P/N: SFP-10G-BX #74682
2 × 14 CHF 3m Generic Compatible 10G SFP+ Passive Direct Attach Copper Twinax Cable
0 × 56 CHF SFP+ Transceiver Modul - Generisch kompatibel 10GBASE-T SFP+ Kupfer RJ-45 30m, FS P/N: SFP-10G-T #74680

digitec order

There are a few items that FS.COM doesn’t stock. These I bought at digitec, a big and popular electronics store in Switzerland. My thinking is that if products are available at digitec, they most likely are available at your preferred big electronics store, too.

num price name
2 × 149 CHF Mikrotik CRS305-1G-4S+IN switch

misc order

The Mellanox cards are not as widely available as I’d like.

I’m waiting for an FS.COM card to arrive, which might be a better choice.

num price name
2 × 129 EUR Mellanox ConnectX-3 MCX311A-XCAT

Mikrotik switch setup

I want to use my switches only as switches, not for any routing or other layer 3 features that might reduce bandwidth, so I first reboot the MikroTik CRS305-1G-4S+ into SwOS:

  1. In the web interface menu, navigate to System → Routerboard → Settings, open the Boot OS drop-down and select option SwOS.

  2. In the web interface menu, navigate to System → Reboot.

  3. After the device rebooted, change the hostname which was reset to MikroTik.

Next, upgrade the firmware to 2.12 to fix a weird issue with certain combinations of SFP modules (SFP-10G-BX in SFP1, SFP-10G-T in SFP2):

  1. In the SwOS web interface, select the Upgrade tab, then click Download & Upgrade.

Network card setup (Linux)

After booting with the Mellanox ConnectX3 in a PCIe slot, the card should show up in dmesg(8) :

mlx4_core: Mellanox ConnectX core driver v4.0-0
mlx4_core: Initializing 0000:03:00.0
mlx4_core 0000:03:00.0: DMFS high rate steer mode is: disabled performance optimized steering
mlx4_core 0000:03:00.0: 31.504 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x4 link)
mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
mlx4_en 0000:03:00.0: Activating port:1
mlx4_en: 0000:03:00.0: Port 1: Using 16 TX rings
mlx4_en: 0000:03:00.0: Port 1: Using 16 RX rings
mlx4_en: 0000:03:00.0: Port 1: Initializing port
mlx4_en 0000:03:00.0: registered PHC clock
mlx4_core 0000:03:00.0 enp3s0: renamed from eth0
<mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
<mlx4_ib> mlx4_ib_add: counter index 1 for port 1 allocated 1
mlx4_en: enp3s0: Steering Mode 1
mlx4_en: enp3s0: Link Up

Another way to verify the device is running at maximum speed on the computer’s PCIe bus, is to ensure LnkSta matches LnkCap in the lspci(8) output:

% sudo lspci -vv
03:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
	Subsystem: Mellanox Technologies Device 0055
[…]
	Capabilities: [60] Express (v2) Endpoint, MSI 00
[…]
		LnkCap:	Port #8, Speed 8GT/s, Width x4, ASPM L0s, Exit Latency L0s unlimited
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (ok), Width x4 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
[…]

You can verify your network link is running at 10 Gbit/s using ethtool(8) :

% sudo ethtool enp3s0
Settings for enp3s0:
	Supported ports: [ FIBRE ]
	Supported link modes:   1000baseKX/Full
	                        10000baseKR/Full
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: No
	Supported FEC modes: Not reported
	Advertised link modes:  1000baseKX/Full
	                        10000baseKR/Full
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: No
	Advertised FEC modes: Not reported
	Speed: 10000Mb/s
	Duplex: Full
	Auto-negotiation: off
	Port: Direct Attach Copper
	PHYAD: 0
	Transceiver: internal
	Supports Wake-on: d
	Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
	Link detected: yes

Benchmarking batch transfers

As mentioned in the introduction, routing 10 Gbit/s is out of scope in this article. If you’re interested in routing performance, check out Andree Toonk’s post which confirms that Linux can route 10 Gbit/s at line rate.

The following sections cover individual batch transfers of large files, not many small flows.

iperf3 speed test

Out of the box, the speeds that iperf3(1) measures are decent:

chuchi % iperf3 --version
iperf 3.6 (cJSON 1.5.2)
Linux chuchi 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64
Optional features available: CPU affinity setting, IPv6 flow label, SCTP, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication

chuchi % iperf3 --server
[…]

midna % iperf3 --version          
iperf 3.9 (cJSON 1.7.13)
Linux midna 5.12.1-arch1-1 #1 SMP PREEMPT Sun, 02 May 2021 12:43:58 +0000 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting, sendfile / zerocopy, socket pacing, authentication

midna % iperf3 --client chuchi.lan
Connecting to host 10.0.0.173, port 5201
[  5] local 10.0.0.76 port 43168 connected to 10.0.0.173 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.62 MBytes       
[  5]   1.00-2.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.70 MBytes       
[  5]   2.00-3.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.70 MBytes       
[  5]   3.00-4.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.78 MBytes       
[  5]   4.00-5.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.87 MBytes       
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.87 MBytes       
[  5]   6.00-7.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.87 MBytes       
[  5]   7.00-8.00   sec  1.10 GBytes  9.41 Gbits/sec    0   1.87 MBytes       
[  5]   8.00-9.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.96 MBytes       
[  5]   9.00-10.00  sec  1.09 GBytes  9.38 Gbits/sec  402   1.52 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec  402             sender
[  5]   0.00-10.00  sec  11.0 GBytes  9.40 Gbits/sec                  receiver

iperf Done.

HTTP speed test

Downloading a file from an nginx(1) web server using curl(1) is fast, too:

% curl -o /dev/null http://chuchi.lan/distri/supersilverhaze/img/distri-disk.img.zst
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  934M  100  934M    0     0  1118M      0 --:--:-- --:--:-- --:--:-- 1117M

Note that this download was served from RAM (Linux page cache). The next upgrade I need to do in this machine is replace the SATA SSD with an NVMe SSD, because the disk is now the bottleneck.

Conclusion

This was a pleasantly simple upgrade: plug in a bunch of new hardware and batch transfers become faster.

The Mikrotik switch provides great value for money, and the Mellanox ConnectX-3 cards work well, provided you can find them.

Appendix A: Switching from RJ45 SFP+ modules to Direct Attach Cables

Originally, I connected all PCs to the MikroTik switches with RJ45 SFP+ modules for two reasons:

  1. I bought Intel X550-T2 PCIe 10 Gbit/s network cards that RJ45 as my first choice.
  2. The SFP+ modules are backwards-compatible and can be used with 1 Gbit/s RJ45 devices, too, which makes for a nice incremental upgrade path.

However, I later was made aware that the RJ45 SFP+ modules use significantly more power and run significantly hotter than Direct Attach Cables (DAC).

I measured it: each RJ45 SFP+ module was causing my BiDi SFP+ module to run 5℃ hotter!

Around 06/02 I replaced one RJ45 SFP+ module with a Direct Attach Cable.

Around 06/06 I replaced the remaining RJ45 SFP+ module with another Direct Attach Cable.

As you can see, this caused a 10℃ drop in temperature of the BiDi SFP+ module.

The MikroTik is still uncomfortably hot, making it hard to work with when it’s powered on.

Appendix B: Network card setup (Linux) with Intel X550-T2

For reference, here is the Network card setup (Linux) section, but with the Intel X550-T2 that I previously used.

After booting with the Intel X550-T2 in a PCIe slot, the card should show up in dmesg(8) :

ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16 XDP Queue count = 0
ixgbe 0000:03:00.0: 31.504 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x4 link)
ixgbe 0000:03:00.0: MAC: 4, PHY: 0, PBA No: H86377-006
ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection
libphy: ixgbe-mdio: probed
ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 16, Tx Queue count = 16 XDP Queue count = 0
ixgbe 0000:03:00.1: 31.504 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x4 link)
ixgbe 0000:03:00.1: MAC: 4, PHY: 0, PBA No: H86377-006
tun: Universal TUN/TAP device driver, 1.6
ixgbe 0000:03:00.1: Intel(R) 10 Gigabit Network Connection
libphy: ixgbe-mdio: probed
ixgbe 0000:03:00.0 enp3s0f0: renamed from eth0
ixgbe 0000:03:00.1 enp3s0f1: renamed from eth1
pps pps0: new PPS source ptp1
ixgbe 0000:03:00.0: registered PHC device on enp3s0f0
pps pps1: new PPS source ptp2
ixgbe 0000:03:00.1: registered PHC device on enp3s0f1

I think if you only use 1 of the card’s 2 network ports, you might not hit any bottlenecks even when running the card only at PCIe 3.0 ×2 link speed, but I haven’t verified this!

Another way to verify the device is running at maximum speed on the computer’s PCIe bus, is to ensure LnkSta matches LnkCap in the lspci(8) output:

% sudo lspci -vv
[…]
03:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
        Subsystem: Intel Corporation Ethernet Converged Network Adapter X550-T2
[…]
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
[…]
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x4 (ok)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
[…]

You can verify your network link is running at 10 Gbit/s using ethtool(8) :

% sudo ethtool enp3s0f1 
Settings for enp3s0f1:
	Supported ports: [ TP ]
	Supported link modes:   100baseT/Full
	                        1000baseT/Full
	                        10000baseT/Full
	                        2500baseT/Full
	                        5000baseT/Full
	Supported pause frame use: Symmetric
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  100baseT/Full
	                        1000baseT/Full
	                        10000baseT/Full
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 10000Mb/s
	Duplex: Full
	Auto-negotiation: on
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	MDI-X: Unknown
	Supports Wake-on: d
	Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
	Link detected: yes

Appendix C: BIOS update for Mellanox ConnectX-3

On my Supermicro X11SSZ-QF mainboard, the Mellanox ConnectX-3 would not establish a link. The Mellanox Linux kernel driver logged a number of errors:

kernel: mlx4_en: enp1s0: CQE error - cqn 0x8e, ci 0x0, vendor syndrome: 0x57 syndrome: 0x4
kernel: mlx4_en: enp1s0: Related WQE - qpn 0x20d, wqe index 0x0, wqe size 0x40
kernel: mlx4_en: enp1s0: Scheduling port restart
kernel: mlx4_core 0000:01:00.0: Internal error detected:
kernel: mlx4_core 0000:01:00.0: device is going to be reset
kernel: mlx4_core 0000:01:00.0: crdump: devlink snapshot disabled, skipping
kernel: mlx4_core 0000:01:00.0: device was reset successfully
kernel: mlx4_en 0000:01:00.0: Internal error detected, restarting device
kernel: <mlx4_ib> mlx4_ib_handle_catas_error: mlx4_ib_handle_catas_error was started
kernel: <mlx4_ib> mlx4_ib_handle_catas_error: mlx4_ib_handle_catas_error ended
kernel: mlx4_core 0000:01:00.0: command 0x21 failed: fw status = 0x1
kernel: pcieport 0000:00:1c.0: AER: Uncorrected (Fatal) error received: 0000:00:1c.0
kernel: pcieport 0000:00:1c.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Transaction Layer, (Receiver ID)
kernel: mlx4_core 0000:01:00.0: command 0x43 failed: fw status = 0x1
kernel: infiniband mlx4_0: ib_query_port failed (-5)
kernel: pcieport 0000:00:1c.0:   device [8086:a110] error status/mask=00040000/00010000
kernel: pcieport 0000:00:1c.0:    [18] MalfTLP                (First)
kernel: pcieport 0000:00:1c.0: AER:   TLP Header: 4a000001 01000004 00000000 00000000
kernel: mlx4_core 0000:01:00.0: mlx4_pci_err_detected was called
kernel: mlx4_core 0000:01:00.0: Fail to set mac in port 1 during unregister
systemd-networkd[313]: enp1s0: Link DOWN
kernel: mlx4_en: enp1s0: Failed activating Rx CQ
kernel: mlx4_en: enp1s0: Failed restarting port 1
kernel: mlx4_en: enp1s0: Link Down
kernel: mlx4_en: enp1s0: Close port called
systemd-networkd[313]: enp1s0: Lost carrier
kernel: mlx4_en 0000:01:00.0: removed PHC
kernel: mlx4_core 0000:01:00.0: mlx4_restart_one_up: ERROR: mlx4_load_one failed, pci_name=0000:01:00.0, err=-5
kernel: mlx4_core 0000:01:00.0: mlx4_restart_one was ended, ret=-5
systemd-networkd[313]: enp1s0: DHCPv6 lease lost
kernel: pcieport 0000:00:1c.0: AER: Root Port link has been reset
kernel: mlx4_core 0000:01:00.0: mlx4_pci_resume was called
kernel: mlx4_core 0000:01:00.0: Multiple PFs not yet supported - Skipping PF
kernel: mlx4_core 0000:01:00.0: mlx4_pci_resume: mlx4_load_one failed, err=-22
kernel: pcieport 0000:00:1c.0: AER: device recovery successful

What helped was to update the X11SSZ-QF BIOS to the latest version.

at 2021-05-16 15:33

2021-05-11

michael-herbst.com

Infomath seminar: A one-hour introduction to Julia

Just one day after my talk at the Lüchow group in Aachen, on 6th May I was asked to give a short introduction to Julia at the Infomath seminar series at Sorbonne Université. While virtual seminars certainly don't share the same spirit as in-person ones do, the ability to quickly hop between seminar series organised all across the world has advantages, too.

In my one-hour talk I gave a short introduction into Julia, focusing on the perspective of applied mathematicians. I gave a short speed comparsion of Julia, python and C on a simple example and presented some of its strengths in different application scenarios (numerical linear algebra, numerical methods for solving PDEs, data science and statistical learning). For future reference I have put the Jupyter notebooks I used during the lecture on github.

Link Licence
An introductory hour to the Julia programming language (Github repository) Creative Commons License

by Michael F. Herbst at 2021-05-11 10:01 under talk, Julia, programming and scripting

2021-05-11

michael-herbst.com

Talk at Lüchow group seminar at RWTH

On 5th May I was invited to present a short summary of my research at the local theoretical chemistry research group of Prof. Dr. Arne Lüchow at RWTH Aachen. Because I wanted to give a broad overview of topics that I worked on over the past few years, I did not really go into many details. Nevertheless my talk lead to interesting and lively discussions, which I enjoyed very much. Clearly the pandemic makes it difficult to get in touch with other researchers, such that I was pretty happy to be able to get to get in touch with some more chemists from the RWTH during the seminar.

Link Licence
High-throughput electronic-structure simulations: Where reliability really matters (Slides) Creative Commons License

by Michael F. Herbst at 2021-05-11 10:00 under talk, electronic structure theory, Kohn-Sham, high-throughput, DFT, solid state

2021-05-08

sECuREs website

Measure and reduce keyboard input latency with QMK on the Kinesis Advantage

Over the last few years, I worked on a few projects around keyboard input latency:

In 2018, I introduced the kinX keyboard controller with 0.2ms of input latency.

In 2020, I introduced the kinT keyboard controller, which works with a wide range of Teensy micro controllers, and both the old KB500 and the newer KB600 Kinesis Advantage models.

While the 2018 kinX controller had built-in latency measurement, I was starting from scratch with the kinT design, where I wanted to use the QMK keyboard firmware instead of my own firmware.

That got me thinking: instead of adjusting the firmware to self-report latency numbers, is there a way we can do latency measurements externally, ideally without software changes?

This article walks you through how to set up a measurement environment for your keyboard controller’s input latency, be it original or self-built. I’ll use a Kinesis Advantage keyboard, but this approach should generalize to all keyboards.

I will explain a few common causes for extra keyboard input latency and show you how to fix them in the QMK keyboard firmware.

Measurement setup

The idea is to connect a Teensy 4.0 (or similar), which simulates pressing the Caps Lock key and measures the duration until the keypress resulted in a Caps Lock LED change.

We use the Caps Lock key because it is one of the few keys that results in an LED change.

Here you can see the Teensy 4.0 connected to the kinT controller, connected to a laptop:

measurement setup

Enable the debug console in QMK

Let’s get our QMK working copy ready for development! I like to work in a separate QMK working copy per project:

% docker run -it -v $PWD:/usr/src archlinux
# pacman -Sy && pacman -S qmk make which diffutils python-hidapi python-pyusb
# cd /usr/src
# qmk clone -b develop qmk/qmk_firmware $PWD/qmk-input-latency
# cd qmk-input-latency

I compile the firmware for my keyboard like so:

# make kinesis/kint36:stapelberg

To enable the debug console, I need to edit my QMK keymap stapelberg by updating keyboards/kinesis/keymaps/stapelberg/rules.mk to contain:

CONSOLE_ENABLE = yes

After compiling and flashing the firmware, the hid_listen tool will detect the device and listen for QMK debug messages:

% sudo hid_listen
Waiting for device:...
Listening:

Finding the pins

Let’s locate the Caps Lock key’s corresponding row and column in our keyboard matrix!

We can make QMK show which keys are recognized after each scan by adding to keyboards/kinesis/keymaps/stapelberg/keymap.c the following code:

void keyboard_post_init_user() {
  debug_config.enable = true;
  debug_config.matrix = true;
}

Now we’ll see in the hid_listen output which key is active when pressing Caps Lock:

r/c 01234567
00: 00100000
01: 00000000
[…]

For our kinT controller, Caps Lock is on QMK matrix row 0, column 2.

In the kinT schematic, the corresponding signals are ROW_EQL and COL_2.

To hook up the Teensy 4.0 latency measurement driver, I am making the following GPIO connections to the kint36, kint41 or kint2pp (with voltage converter!) keyboard controllers:

driver 4.0 signal kint36, kint41 kint2pp (5V!)
GND GND GND GND
pin 10 ROW_EQL pin 8 D7
pin 11 COL_2 pin 15 F7
pin 12 LED_CAPS_LOCK pin 12 C1

Eager Caps Lock LED

When the host signals to the keyboard that Caps Lock is now turned on, the QMK firmware first updates a flag in the USB interrupt handler, but only updates the Caps Lock LED pin after the next matrix scan has completed.

This is fine in normal usage, but our measurement readings will get more precise if we immediately update the Caps Lock LED pin. We can do this in set_led_transfer_cb in tmk_core/protocol/chibios/usb_main.c, which is called from the USB interrupt handler:

#include "gpio.h"

static void set_led_transfer_cb(USBDriver *usbp) {
    if (usbp->setup[6] == 2) { /* LSB(wLength) */
        uint8_t report_id = set_report_buf[0];
        if ((report_id == REPORT_ID_KEYBOARD) || (report_id == REPORT_ID_NKRO)) {
            keyboard_led_state = set_report_buf[1];
        }
    } else {
        keyboard_led_state = set_report_buf[0];
    }
    if ((keyboard_led_state & 2) != 0) {
      writePinLow(C7); // turn on CAPS_LOCK LED
    } else {
      writePinHigh(C7); // turn off CAPS_LOCK LED
    }
}

Host side (Linux)

On the USB host, i.e. the Linux computer, I switch to a Virtual Terminal (VT) by stopping my login manager (killing my current graphical session!):

% sudo systemctl stop gdm

With the Virtual Terminal active, we know that the Caps Lock key press will be handled entirely in kernel driver code without having to round-trip to userspace.

We can verify this by collecting stack traces with bpftrace(8) when the kernel executes the kbd_event function in drivers/tty/vt:

% sudo bpftrace -e 'kprobe:kbd_event { @[kstack] = count(); }'

After pressing Caps Lock and cancelling the bpftrace process, you should see a stack trace.

I then measured the baseline end-to-end latency, using my measure-fw firmware running on the FRDM-K66F eval kit, a cheap and widely available USB 2.0 High Speed device. The firmware measures the latency between a button press and the USB HID report for the Caps Lock LED, but without any additional matrix scanning delay or similar:

% cat /dev/ttyACM0
sof=74 μs	report=393 μs
sof=42 μs	report=512 μs
sof=19 μs	report=512 μs
sof=39 μs	report=488 μs
sof=20 μs	report=518 μs
sof=90 μs	report=181 μs
sof=42 μs	report=389 μs
sof=7 μs	report=319 μs

This is the quickest reaction we can get out of this computer. Anything on top (e.g. X11, application) will be slower, so this measurement establishes a lower bound.

Code to simulate key presses and take measurements

I’m running the latencydriver Arduino sketch, with the Arduino IDE configured for:

Teensy 4.0 (USB Type: Serial, CPU Speed: 600 MHz, Optimize: Faster)

Here’s how we set up the pins in the measurement driver Teensy 4.0:

void setup() {
  Serial.begin(9600);

  // Connected to kinT pin 15, COL_2
  pinMode(11, OUTPUT);
  digitalWrite(11, HIGH);

  // Connected to kinT pin 8, ROW_EQL.
  // Pin 11 will be high/low in accordance with pin 10
  // to simulate a key-press, and always high (unpressed)
  // otherwise.
  pinMode(10, INPUT_PULLDOWN);
  attachInterrupt(digitalPinToInterrupt(10), onScan, CHANGE);

  // Connected to the kinT LED_CAPS_LOCK output:
  pinMode(12, INPUT_PULLDOWN);
  attachInterrupt(digitalPinToInterrupt(12), onCapsLockLED, CHANGE);
}

In order to make a key read as pressed, we need to connect the column with the row in the keyboard matrix, but only when the column is scanned. We do that in the interrupt handler like so:

bool simulate_press = false;

void onScan() {
  if (simulate_press) {
    // connect row scan signal with column read
    digitalWrite(11, digitalRead(10));
  } else {
    // always read not pressed otherwise
    digitalWrite(11, HIGH);
  }
}

In our text interface, we can now start a measurement like so:

caps_lock_on_to_off = capsLockOn();
Serial.printf("# Caps Lock key pressed (transition: %s)\r\n",
  caps_lock_on_to_off ? "on to off" : "off to on");
simulate_press = true;
t0 = ARM_DWT_CYCCNT;
emt0 = 0;
eut0 = 0;

The next keyboard matrix scan will detect the key as pressed, send the HID report to the OS, and when the OS responds with its HID report containing the Caps Lock LED status, our Caps Lock LED interrupt handler is called to finish the measurement:

void onCapsLockLED() {
  const uint32_t t1 = ARM_DWT_CYCCNT;
  const uint32_t elapsed_millis = emt0;
  const uint32_t elapsed_micros = eut0;
  uint32_t elapsed_nanos = (t1 - t0) / cycles_per_ns;

  Serial.printf("# Caps Lock LED (pin 12) is now %s\r\n", capsLockOn() ? "on" : "off");
  Serial.printf("# %u ms == %u us\r\n", elapsed_millis, elapsed_micros);
  Serial.printf("BenchmarkKeypressToLEDReport 1 %u ns/op\r\n", elapsed_nanos);
  Serial.printf("\r\n");
}

Running measurements

Connect the Teensy 4.0 to your computer and open its USB serial console:

% screen /dev/ttyACM0 115200

You should be greeted by a welcome message:

# kinT latency measurement driver
#   t  - trigger measurement

To save your measurements to file, use C-a H in screen to make it write to file screenlog.0.

Press t a few times to trigger a few measurements and close screen using C-a k.

You can summarize the measurements using benchstat:

% benchstat screenlog.0
name                 time/op
KeypressToLEDReport  1.82ms ±20%

Scan-to-scan delay

The measurement output on the USB serial console also contains the matrix scan-to-scan delay:

# scan-to-scan delay: 422475 ns

Each keyboard matrix scan turns on each row one-by-one, then reads all the columns.

This means that in each matrix scan, ROW_EQL will be set high once, then low again.

The Teensy 4.0 measures scan-to-scan delay by timing the activations of ROW_EQL.

We can verify this approach by making QMK self-report its scan rate. Enable the matrix scan rate debug option in keyboards/kinesis/keymaps/stapelberg/config.h like so:

#pragma once

#define DEBUG_MATRIX_SCAN_RATE

Using hid_listen we can now see the following QMK debug messages:

% sudo hid_listen
Waiting for new device:..
Listening:
matrix scan frequency: 2300
matrix scan frequency: 2367
matrix scan frequency: 2367

A matrix scan rate/frequency of 2367 scans per second corresponds to 422μs per scan:

1000000 μs / 2367 scans/second = 422μs

Yet another way of verifying the approach is by short-circuiting an end-to-end measurement with a one-line change in our QMK keyboard code:

bool process_action_kb(keyrecord_t *record) {
#define LED_CAPS_LOCK LINE_PIN12
#define ledTurnOn writePinLow
  ledTurnOn(LED_CAPS_LOCK);
  return true;
}

Repeating the measurements, this gives us:

% benchstat screenlog.0     
name                 time/op
KeypressToLEDReport  693µs ±26%

This value is between [0, 2 * 422μs] because a key might be pressed after it was already scanned by the in-progress matrix scan, meaning it will need to wait until the next scan completed (!) before it can be registered as pressed.

Measurement harness

Now that we have our general measurement environment all set up, it’s time to connect our Teensy 4.0 to a few different keyboard controllers!

kint36, kint41: GPIO

If you have an un-soldered micro controller you want to measure, setup is easy: just connect all GPIOs to the Teensy 4.0 latency test driver directly! I’m using this for the kint36 and kint41:

GPIO measurement

(build in /home/michael/kinx/kintpp/rebased, last results in screenlog-kint36-eager-caps.0)

kint2pp: 5V

Because the Teensy++ uses 5V logic levels, we need to convert the levels from/to 3.3V. This is easily done using e.g. the SparkFun Logic Level Converter (Bi-Directional) on a breadboard:

kint2pp with level shifter

kinX: FPC

But what if you have a design where the micro controller doesn’t come standalone, only soldered to a keyboard controller board, such as my earlier kinX controller?

You can use a spare FPC connector (Molex 39-53-2135) and solder jumper wires to the pins for COL_2 and ROW_EQL. For Caps Lock and Ground, I soldered jumper wires to the board:

kinX measurement

Original Kinesis controller

But what if you don’t want to solder jumper wires directly to the board?

The least invasive method is to connect the FPC connector break-out, and hold probe heads onto the contacts while doing your measurements:

kinesis original controller measurement

QMK input latency

Now that the measurement hardware is set up, we can go through the code.

The following sections each cover one possible contributor to input latency.

Eager debounce

Key switches don’t generate a clean signal when pressed, instead they show a ripple effect. Getting rid of this ripple is called debouncing, and every keyboard firmware does it.

See QMK’s documentation on the Debounce API for a good explanation of the differences between the different debounce approaches.

QMK’s default debounce algorithm sym_defer_g is chosen very cautiously. I don’t know what the criteria are specifically for which types of key switches suffer from noise and therefore need the sym_defer_g algorithm, but I know that Cherry MX key switches with diodes like used in the Kinesis Advantage don’t have noise and hence can use the other debounce algorithms, too.

While the default sym_defer_g debounce algorithm is robust, it also adds 5ms of input latency:

% benchstat screenlog-kint36.0
name                 time/op
KeypressToLEDReport  7.61ms ± 8%

For lower input latency, we need an eager algorithm. Specifically, I am chosing the sym_eager_pk debounce algorithm by adding to my keyboards/kinesis/kint36/rules.mk:

DEBOUNCE_TYPE = sym_eager_pk

Now, the extra 5ms are gone:

% benchstat screenlog-kint36-eager.0
name                 time/op
KeypressToLEDReport  2.12ms ±16%

Example change: https://github.com/qmk/qmk_firmware/pull/12626

Quicker USB polling interval

The USB host (computer) divides time into fixed-length segments called frames:

  • USB Full Speed (USB 1.0) uses frames that are 1ms each.
  • USB High Speed (USB 2.0) introduces micro frames, which are 125μs.

Each USB device specifies in its device descriptor how frequently (in frames) the device should be polled. The quickest polling rate for USB 1.0 is 1 frame, meaning the device can send data after at most 1ms. Similarly, for USB 2.0, it’s 1 micro frame, i.e. send data every 125μs.

Of course, a quicker polling rate also means occupying resources on the USB bus which are then no longer available to other devices. On larger USB hubs, this might mean fewer devices can be used concurrently. The specifics of this limitation depend on a lot of other factors, too. The polling rate plays a role, in combination with the max. packet size and the number of endpoints.

Note that we are only talking about concurrent device usage, not about hogging bandwidth: the bulk transfers that USB mass storage devices use are not any slower in my tests. I achieve about 37 MiB/s with or without the kint41 USB 2.0 High Speed controller with bInterval=1 present.

Even connecting two kint41 controllers at the same time still leaves enough resources to use a Logitech C920 webcam in its most bandwidth-intensive pixel format and resolution. The same cannot be said for e.g. NXP’s LPC-Link2 debug probe.

To display the configured interval, the Linux kernel provides a debug pseudo file:

% sudo cat /sys/kernel/debug/usb/devices

[…]
T:  Bus=01 Lev=02 Prnt=09 Port=02 Cnt=02 Dev#= 53 Spd=480  MxCh= 0
D:  Ver= 2.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=1209 ProdID=345c Rev= 0.01
S:  Manufacturer="https://github.com/stapelberg"
S:  Product="kinT (kint41)"
C:* #Ifs= 3 Cfg#= 1 Atr=a0 MxPwr=500mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=03(HID  ) Sub=01 Prot=01 Driver=usbhid
E:  Ad=81(I) Atr=03(Int.) MxPS=   8 Ivl=125us
I:* If#= 1 Alt= 0 #EPs= 1 Cls=03(HID  ) Sub=00 Prot=00 Driver=usbhid
E:  Ad=82(I) Atr=03(Int.) MxPS=  32 Ivl=125us
I:* If#= 2 Alt= 0 #EPs= 2 Cls=03(HID  ) Sub=00 Prot=00 Driver=usbhid
E:  Ad=83(I) Atr=03(Int.) MxPS=  32 Ivl=125us
E:  Ad=04(O) Atr=03(Int.) MxPS=  32 Ivl=125us
[…]

Alternatively, you can display the USB device descriptor using e.g. sudo lsusb -v -d 1209:345c and interpret the bInterval setting yourself.

The above shows the best case: a USB 2.0 High Speed device (Spd=480) with bInterval=1 in its device descriptor (Iv=125us).

The original Kinesis Advantage 2 keyboard controller (KB600) uses USB 2.0, but in Full Speed mode (Spd=12), i.e. no faster than USB 1.1. In addition, they specify bInterval=10, which results in a 10ms polling interval (Ivl=10ms):

T:  Bus=01 Lev=02 Prnt=09 Port=02 Cnt=02 Dev#= 52 Spd=12   MxCh= 0
D:  Ver= 2.00 Cls=00(>ifc ) Sub=00 Prot=00 MxPS=64 #Cfgs=  1
P:  Vendor=29ea ProdID=0102 Rev= 1.00
S:  Manufacturer=Kinesis
S:  Product=Advantage2 Keyboard
C:* #Ifs= 3 Cfg#= 1 Atr=a0 MxPwr=100mA
I:* If#= 0 Alt= 0 #EPs= 1 Cls=03(HID  ) Sub=01 Prot=02 Driver=usbhid
E:  Ad=83(I) Atr=03(Int.) MxPS=   8 Ivl=10ms
I:* If#= 1 Alt= 0 #EPs= 1 Cls=03(HID  ) Sub=01 Prot=01 Driver=usbhid
E:  Ad=84(I) Atr=03(Int.) MxPS=   8 Ivl=2ms
I:* If#= 2 Alt= 0 #EPs= 1 Cls=03(HID  ) Sub=00 Prot=00 Driver=usbhid
E:  Ad=85(I) Atr=03(Int.) MxPS=   8 Ivl=2ms

My recommendation:

  • With USB 1.1 Full Speed, definitely specify bInterval=1. I’m not aware of any downsides.
  • With USB 2.0 High Speed, I also think bInterval=1 is a good choice, but I am less certain. If you run into trouble, reduce to bInterval=3 and send me a message :)

For details on measuring, see Appendix B: USB polling interval (device side).

Example change: https://github.com/qmk/qmk_firmware/pull/12625

Faster matrix scan

The purpose of a keyboard controller is reporting pressed keys after scanning the key matrix. The more scans a keyboard controller can do per second, the faster it can react to your key press.

How many scans your controller does depends on multiple factors:

  • The clock speed of your micro controller. It’s worth checking if your micro controller model supports running at faster clock speeds, or upgrading your keyboard to a faster model to begin with. There is a point of diminishing returns, which I would guess is at ≈100 MHz. Comparing e.g. the kint36 at 120 MHz vs. 180 MHz, the difference in scan-to-scan is 5μs.

  • How much other code your firmware runs aside from matrix scanning. If you enable any non-standard QMK features, or even self-written code, it’s worth disabling and measuring.

  • Whether you run scans back-to-back or e.g. synchronized with USB start-of-frame interrupts. QMK runs scans back-to-back, so this point is only relevant for other firmwares.

  • How long you need to sleep to let the signal settle. Reducing your sleep times results in more scans per second, but if you don’t sleep long enough, you’ll see ghost key presses. See also the next section about Shorter sleeps.

For details on measuring, see the Scan-to-scan delay section above.

I also tried configuring the GPIOs to be faster to see if that would reduce the required unselect delay, but unfortunately there was no difference between the default setting and the fastest setting: drive strength 6 (DSE=6), fast slew rate (SRE=1), 200 MHz (SPEED=3).

Shorter sleeps

QMK calls ChibiOS’s chThdSleepMicroseconds function in its matrix scanning code. This function unfortunately has a rather long shortest sleep duration of 1 ChibiOS tick: if you tell it to sleep less than 100μs, it will still sleep at least 100μs!

This is a problem on controllers such as the kint41, where we want to sleep for only 10μs.

The length of a ChibiOS tick is determined by how the ARM SysTick timer is set up on the specific micro controller you’re using. While the SysTick timer itself could be configured to fire more frequently, it is not advisable to shorten ChibiOS ticks: chSysTimerHandlerI() must be executable in less than one tick.

Instead, I found it easier to implement short delays by busy-looping until the ARM Cycle Counter Register (CYCCNT) indicates enough time has passed. Here’s an example from keyboards/kinesis/kint41/kint41.c:

// delay_inline sleeps for |cycles| (e.g. sleeping for F_CPU will sleep 1s).
//
// delay_inline assumes the cycle counter has already been initialized and
// should not be modified, i.e. is safe to call during keyboard matrix scan.
//
// ChibiOS enables the cycle counter in chcore_v7m.c.
static void delay_inline(const uint32_t cycles) {
  const uint32_t start = DWT->CYCCNT;
  while ((DWT->CYCCNT - start) < cycles) {
    // busy-loop until time has passed
  }
}

void matrix_output_unselect_delay(void) {
  // 600 cycles at 0.6 cycles/ns == 1μs
  const uint32_t cycles_per_us = 600;
  delay_inline(10 * cycles_per_us);
}

Of course, the cycles/ns value is specific to the frequency at which your micro controller runs, so this code needs to be adjusted for each platform.

Results

With the QMK keyboard firmware configured for lowest input latency, how do the different Kinesis keyboard controller compare? Here are my measurements:

model CPU speed USB poll interval scan-to-scan scan rate caps-to-report
kint41 600 MHz 125μs 181μs 5456 scans/s 930µs ±17%
kinX 120 MHz 125μs 213μs 4694 scans/s 953µs ±15%
kint36 180 MHz 1000μs 444μs 2252 scans/s 1.97ms ±15%
kint2pp 16 MHz 1000μs 926μs 1078 scans/s 3.27ms ±32%
original 60 MHz 10000μs 1936μs 516 scans/s 13.6ms ±21%

The changes required to obtain these results are included since QMK 0.12.38 (2021-04-20).

kint41 support is being added with all required changes to begin with, but still in progress.

The following sections go into detail about the results.

kint41

I am glad that the most recent Teensy 4.1 micro controller takes the lead! The kinX controller achieved similar numbers, but was quite difficult to build, so few people ended up using it.

The key improvement compared to the Teensy 3.6 is the now-available USB 2.0 High Speed, and the powerful clock speed of 600 MHz allows for an even faster matrix scan rate.

kinX

In my previous article about the kinX controller, I measured the kinX scan delay as ≈100μs. During my work on this article, I learnt that the ≈100μs figure was misleading: the measurement code turned off interrupts to measure only the scan function. While that is technically correct, it is not a useful measure, as in practice, interrupts should not be disabled, and the scanning function is interrupted frequently enough that it comes in at ≈208μs.

I also fixed the USB polling interval in the kinX firmware, which wasn’t set to bInterval=1.

Original Kinesis

The original keyboard controller that the Kinesis Advantage 2 (KB600) keyboard comes with uses an AT32UC3B0256 micro controller which is clocked at 60 MHz, but the measured input latency is much higher than even the slowest kint controller (kint2pp at 16 MHz). What gives?

Here’s what we can deduce without access to their firmware:

  1. They seem to be using an eager debounce algorithm (good!), otherwise we would observe even higher latency.
  2. Their USB polling interval setting (bInterval=10) is excessively high, even more so because they are using USB Full Speed with longer USB frames. I would recommend they change it to bInterval=1 for up to 10ms less input latency!
  3. The matrix scan rate is twice as slow as with my kint2pp. I can’t say for sure why this is. Perhaps their firmware does a lot of other things between matrix scans.

Note that we could not apply the Eager Caps Lock LED firmware change to the original controller, which is why the measurement variance is ±21%. This variance includes ± 1.9ms for finishing a matrix scan before updating the LED state.

Conclusion

After analyzing the different controllers in my measurement environment, I think the following factors play the largest role in keyboard input latency, ordered by importance:

  1. Does the firmware use an eager debounce algorithm?
  2. Does the device specify a quick USB polling rate (bInterval setting)?
  3. Is the matrix scan frequency in the expected range, or are there unexpected slow-downs?

Hopefully, this article gives you all the tools you need to measure and reduce keyboard input latency of your own keyboard controller!

Appendix A: isitsnappy

The iPhone app Is It Snappy? records video using the iPhone’s 240 fps camera and allows you to mark the frame that starts respectively ends the measurement.

The app does a good job of making this otherwise tedious process of navigating a video frame by frame much more pleasant.

However, for measuring keyboard input latency, I think this approach is futile:

  • The resolution is too imprecise. At 240 fps, that means each frame represents 4.6ms of time, which is already higher than the input latency of our slowest micro controller.
  • Visually deciding whether a key switch is pressed or not pressed, at frame-perfect precision, seems impossible to me.

I believe the app can work, provided the latency you want to measure is really high. But with the devices covered in this article, the app couldn’t measure even 10ms of injected input latency.

Appendix B: USB polling interval (device side)

You can also verify the USB polling interval on the device side. In the SOF (Start Of Frame) interrupt in tmk_core/protocol/chibios/usb_main.c, we can print the cycle delta to the previous SOF callback, every second:

#include "timer.h"

static uint32_t last_sof = 0;
static uint32_t sof_timer = 0;
void kbd_sof_cb(USBDriver *usbp) {
  (void)usbp;

  uint32_t now = DWT->CYCCNT;
  uint32_t delta = now - last_sof;
  last_sof = now;

  uint32_t timer_now = timer_read32();
  if (TIMER_DIFF_32(timer_now, sof_timer) > 1000) {
    sof_timer = timer_now;
    dprintf("sof delta: %u cycles", delta);
  }
}

Using hid_listen, we expect to see ≈75000 cycles of delta, which corresponds to the 125μs microframe latency of USB 2.0 High Speed with bInterval=1 in the USB device descriptor:

125μs * 1000 * 0.6 cycles/ns = 75000 cycles

at 2021-05-08 13:57

2021-05-07

michael-herbst.com

DFTK: A Julian approach for simulating electrons in solids

Following my talk at Juliacon about our DFTK code last year (slides, recording, blog article), we have now published an extended abstract in the JuliaCon proceedings, which you can find below. The JuliaCon proceedings use the same open journals software stack to manage their publication infrastructure as the Journal of Open-source Software. This stack is actually pretty impressive since it reduces the effort both on the reviewer as well as on the author side to comments within github issues. Since thus the complete exchange (including the review process) is public, this is not only convenient, but also leads to truly transparent publication process. I wish publishing with all journals was like that ...

by Michael F. Herbst at 2021-05-07 22:30 under talk, electronic structure theory, Julia, HPC, DFTK, theoretical chemistry, SCF, high-throughput

2021-05-01

michael-herbst.com

Thoughts on initial guess methods for DFT

On Thursday I gave a brief talk in our weekly ACED differentiate group meeting about initial guess methods for starting self-consistent field calculations in methods such as density-functional theory. For preparing the talk I did a little digging into both the standard approaches used by many molecular and solid-state codes and did a literature review of some recent ideas motivated from reduced-order modelling or data science. The slides of my talk (which include most references I found) are attached below.

Link Licence
Thoughts on initial guess methods for DFT (Slides) Creative Commons License

by Michael F. Herbst at 2021-05-01 10:00 under talk, electronic structure theory, Kohn-Sham, high-throughput, DFT, solid state

2021-04-30

RaumZeitLabor

Halbjähriges im RaumZweitLabor – Aufbauliebe in den Zeiten des Corona

Im neuen RZL feiern wir aktuell noch, wie ein verliebtes Teenagerpärchen, jedes Wochen- und Monatsjubiläum. Unfassbar, dass jetzt schon unser „Halbjähriges“ ansteht. <3

Um euch an unserem Glück teilhaben zu lassen, gibt es hier ein kleines Update, was in der letzten Zeit alles so Aufregendes unter Einhaltung der Corona-Auflagen passiert ist:

Das erste Mal Kisten auspacken, neue Räume umbauen, aufteilen und einräumen; das erste Mal 5 Kubikmeter KMF und andere Altlasten entsorgen; das erste Mal Personenfahrstuhl kaputt, Wasser im Lastenaufzugsschacht im Keller und Kommunikationsprobleme mit dem Vermieter; das erste Mal Rechnungen bei der Versicherung einreichen; das erste Mal Adressänderungen überall; das erste Mal Brandverhütungsschau der Feuerwehr im neuen Raum; das erste Mal Winter mit funktionierender Heizung; das erste Mal schöne, bunte Deko an die neuen Wände packen; das erste Mal Labortische selbst bauen; und vieles, vieles mehr…

Außerdem sind wir Teil des Rats für Kunst und Kultur Mannheim in der Sektion Kulturelle Bildung und Soziokultur geworden, arbeiten im Hintergrund an spannenden Projekten für die Nach-Corona-Zeit und haben die neuen Räume auch endlich mit dem Siegel „Inte approved“ zertifizieren lassen!

Damit aus der frischen Beziehung aber keine schnell verflossene Romanze wird, sind wir weiterhin auf eure Beteiligung bei (Aufbau-)Aktionen angewiesen und freuen uns mehr denn je über einmalige und dauerhafte Zeichen der Zuneigung.

Halbjahr21-Collage

by flederrattie at 2021-04-30 00:00

2021-04-27

sECuREs website

Linux and USB virtual serial devices (CDC ACM)

During my work on Teensy 4.1 support in ChibiOS for the QMK keyboard firmware, I noticed that ChibiOS’s virtual serial device USB demo would sometimes print garbled output, and that I would never see the ChibiOS shell prompt.

This article walks you through diagnosing and working around this issue, in the hope that it helps others who are working with micro controllers and USB virtual serial devices.

Background

Serial interfaces are often the easiest option when working with micro controllers to print text: you only connect GND and the micro controller’s serial TX pin to a USB-to-serial converter. The RX pin is only needed when you want to send text to the micro controller as well.

While conceptually simple, the requirement for an extra piece of hardware (USB-to-serial adapter) is annoying. If your micro controller has a working USB interface and USB stack, a popular alternative is for the micro controller to provide a virtual serial device via USB.

This way, you just need one USB cable between your micro controller and computer, reusing the same connection you already use for programming the device.

A popular choice within this solution is to provide a device conforming to the USB Communications Device Class (CDC) standard, specifically its Abstract Control Model (ACM), which is typically used for modem hardware.

On Linux, these devices show up as e.g. /dev/ttyACM0. In case you’re wondering: /dev/ttyUSB0 device names are used by more specific drivers (vendor-specific). The blog post What is the difference between /dev/ttyUSB and /dev/ttyACM? goes into a lot more detail.

ModemManager

One unfortunate side-effect of using a modem standard to provide a generic serial device is that modem-related software might mistake our micro controller for a modem.

Use the following command to disable ModemManager until the next reboot, which otherwise might open and probe any new serial devices:

% sudo systemctl mask --runtime --now ModemManager

Problem statement

With a regular, non-USB serial interface, you can send data at any time. If nobody is receiving the data on the other end, the micro controller doesn’t care and still writes serial data.

When using the ChibiOS shell with a regular serial interface, this means that if you open the serial interface too late, you will not see the ChibiOS shell prompt. But, if you have the serial interface already opened when powering on your device, you will be greeted by ChibiOS’s shell prompt:

ChibiOS/RT Shell
ch> 

With a USB serial, however, the host will not transfer data from the device until the serial interface is opened. This means that writes to the USB serial can block, whereas writes to the UART serial will not block but may go ignored if nobody is listening.

So when I open the USB serial interface, I would expect to see the ChibiOS shell prompt like above. Instead, I would often not see any prompt at all, and I would even sometimes see garbled output like this:

cch> biOS/RT She

USB analysis with Wireshark

Wireshark allows us to analyze USB traffic in combination with the usbmon Linux kernel module.

Looking through the captured packets, I noticed unexpected packets from the host (computer) to the device (micro controller), specifically containing the following bytes:

  1. hex 0xa = ASCII \n
  2. hex 0xd = ASCII \r

Seeing any packets in this direction is unexpected, because I am only opening the serial interface for reading, and I am not consciously sending anything. So where do the packets come from?

To verify I am not missing any nuance of the CDC protocol, I added debug statements to the ChibiOS shell to log any incoming data. The \n\r bytes indeed make it to the ChibiOS shell.

When the shell receives a line break, it prints a new prompt. This seems to be the reason why I’m seeing garbled data: while the output is transferred to the host, line breaks are received, causing more data transfers. It’s as if somebody was hammering the return key really quickly.

Linux tty echo vs. ChibiOS shell banner

The unexpected \n\r bytes turn out to come from the Linux USB CDC ACM driver, or its interplay with the Linux tty driver, to be specific. The CDC ACM driver is a kind of tty driver, so it is built atop the Linux tty infrastructure, whose standard settings include various ECHO flags.

When echoing is enabled, the ChibiOS shell banner triggers echo characters, which in turn are interpreted as input to the shell, causing garbled output.

So why is echoing enabled? Wouldn’t a terminal emulator turn off echoing first thing?

Yes. But, when the CDC ACM driver receives the first data transfer via USB (already queued), the standard tty settings are still in effect, because the application did not yet have a chance to set its tty configuration up!

This can be verified by running the following command on a Linux host:

% stty -F /dev/ttyACM0 115200 -echo -echoe -echok

Even though the command’s sole purpose is to configure the tty, its opening of the device still causes the banner to print, and echoing to happen, and garbled output is the result.

It turns out this is a somewhat common problem. Hence, the Linux USB CDC ACM driver has a quirks table, in which devices that print a banner select the DISABLE_ECHO quirk, which results in the CDC ACM driver turning off the echoing termios flag early:

static const struct usb_device_id acm_ids[] = {
	/* quirky and broken devices */
	{ USB_DEVICE(0x0424, 0x274e), /* Microchip Technology, Inc. */
	  .driver_info = DISABLE_ECHO, }, /* DISABLE ECHO in termios flag */
// 

So, a quick solution to turn off echoing early is to change your USB vendor and product id (VID/PID) to an ID for which the Linux kernel applies the DISABLE_ECHO quirk, e.g.:

#define USB_DEVICE_VID 0x0424
#define USB_DEVICE_PID 0x274e

Flushing in Screen

With tty echo disabled, I don’t see garbled output anymore, but still wouldn’t always see the ChibiOS shell prompt!

This issue turned out to be specific to the terminal emulator program I’m using. For many years, I have been using Screen for serial devices of any sort.

I was surprised to learn during this investigation that Screen flushes any pending output when opening the device. This typically isn’t a problem because adapter-backed serial devices are opened once and then stay open. USB virtual serial devices however are only opened when used, and disappear when loading new program code onto your micro controller.

I verified this is the problem by using cat(1) instead, with which I can indeed see the prompt:

% cat /dev/ttyACM0

ChibiOS/RT Shell
                
                ch> 

After commenting out the flush call in Screen’s sources, I could see the prompt in Screen as well.

Line ending conversion

Now that we no longer flush the prompt away, why is the spacing still incorrect, and where does it go wrong?


ChibiOS/RT Shell
                
                ch> 

If we use strace(1) to see what screen(1) or cat(1) read from the driver, we see:

797270 read(7, "\n\nChibiOS/RT Shell\n\nch> ", 4096) = 24

We would have expected "\r\nChibiOS/RT Shell\r\nch> " instead, meaning all Carriage Returns (\r) have been translated to Newlines (\n).

This is again due to the Linux tty driver’s default termios settings: c_iflag enables option ICRNL by default, which translates CR (Carriage Return) to NL (Newline).

Unfortunately, contrary to the DISABLE_ECHO quirk, there is no corresponding quirk in the Linux ACM driver to turn off line ending conversion, so a fix would need a Linux kernel driver change!

Device-side workaround: wait until opened

At this point, we have covered a few problems that would need to be fixed:

  1. Change USB VID/PID to get the DISABLE_ECHO quirk in the driver.
  2. Recompile terminal emulator programs to remove flushing, if needed.
  3. Modify kernel driver to add quirk to disable Carriage Return (\r) conversion.

Time for a quick reality check: this seems too hard and too long a time for all parts of the stack to be fixed. Is there an easier way, and why don’t others run into this problem? If only the device didn’t print its banner so early, that would circumvent all of the problems above, too!

Luckily, the host actually notifies the device when a terminal emulator program opens the USB serial device by sending a CDC_SET_CONTROL_LINE_STATE request. I verified this behavior on Linux, Windows and macOS.

So, let’s implement a workaround in our device code! We will delay starting the shell until:

  1. The USB serial device was opened (not just configured).
  2. An additional delay of 100ms has passed to give the terminal emulator application a chance to configure the serial device.

In our main.c loop, we wait until USB is active, and until we receive the first CDC_SET_CONTROL_LINE_STATE request because the serial port was opened:

  while (true) {
    if (SDU1.config->usbp->state == USB_ACTIVE) {
      chSemWait(&scls);
      chThdSleepMilliseconds(100);

      thread_t *shelltp = chThdCreateFromHeap(NULL, SHELL_WA_SIZE, "shell", NORMALPRIO + 1, shellThread, (void *)&shell_cfg1);
      chThdWait(shelltp);
    }
  }

And in our usbcfg.c, when receiving a CDC_SET_CONTROL_LINE_STATE request, we will reset the semaphore to non-blockingly wake up all waiters:

extern semaphore_t scls;

bool requests_hook(USBDriver *usbp) {
  const bool result = sduRequestsHook(usbp);

  if ((usbp->setup[0] & USB_RTYPE_TYPE_MASK) == USB_RTYPE_TYPE_CLASS &&
      usbp->setup[1] == CDC_SET_CONTROL_LINE_STATE) {
    osalSysLockFromISR();
    chSemResetI(&scls, 0);
    osalSysUnlockFromISR();
  }

  return result;
}

Screenshots: Mac and Windows

Aside from Linux, I also verified the workaround works on a Mac (with Screen):

USB virtual serial device on macOS

…and that it works on Windows (with PuTTY):

USB virtual serial device on Windows 10

at 2021-04-27 06:18

2021-04-09

michael-herbst.com

A novel black-box preconditioning strategy for high-throughput density-functional theory

A couple of weeks ago, from 15th to 19th March, I participated in the virtual annual meeting of the German Association of Applied Mathematics and Mechanics (GAMM). For me this meeting was the first time I presented my work to an audience of applied mathematicians with a broad background and no inherent interest in quantum chemistry. With only 15 minutes for my talk in the "scientific computing" track preparing the material was quite a challenge. I hope I still managed to convey the main ideas of our recently published LDOS preconditioner in a broadly accessible way. My slides are attached below.

Link Licence
A novel black-box preconditioning strategy for high-throughput density-functional theory (Slides) Creative Commons License

by Michael F. Herbst at 2021-04-09 16:00 under talk, electronic structure theory, Julia, DFTK, theoretical chemistry, numerical analysis, Kohn-Sham, high-throughput, DFT, solid state

2021-04-02

sECuREs website

Emacs: overriding the project.el project directory

I recently learnt about the Emacs package project.el, which is used to figure out which files and directories belong to the same project. This is used under the covers by Eglot, for example.

In practice, a project is recognized by looking for Git repositories, which is a decent first approximation that often just works.

But what if the detection fails? For example, maybe you want to anchor your project-based commands in a parent directory that contains multiple Git repositories.

Luckily, we can provide our own entry to the project-find-functions hook, and look for a .project.el file in the parent directories:

;; Returns the parent directory containing a .project.el file, if any,
;; to override the standard project.el detection logic when needed.
(defun zkj-project-override (dir)
  (let ((override (locate-dominating-file dir ".project.el")))
    (if override
      (cons 'vc override)
      nil)))

(use-package project
  ;; Cannot use :hook because 'project-find-functions does not end in -hook
  ;; Cannot use :init (must use :config) because otherwise
  ;; project-find-functions is not yet initialized.
  :config
  (add-hook 'project-find-functions #'zkj-project-override))

Now, we can use touch .project.el in any directory to make project.el recognize the directory as project root!

By the way, in case you are unfamiliar, the configuration above uses use-package, which is a great way to (lazily, i.e. quickly!) load and configure Emacs packages.

at 2021-04-02 12:08

2021-04-01

sECuREs website

Eclipse: Enabling Compilation Database (CDB, compile_commands.json) in NXP MCUXpresso v11.3

NXP’s Eclipse-based MCUXpresso IDE is the easiest way to make full use of the hardware debugging features of modern NXP micro controllers such as the i.MX RT1060 found on the NXP i.MX RT1060 Evaluation Kit (MIMXRT1060-EVK), which I use for Teensy 4 development.

For projects that are fully under your control, such as imported SDK examples, or anything you created within Eclipse, you wouldn’t necessarily need Compilation Database support.

When working with projects of type Makefile Project with Existing Code, however, Eclipse doesn’t know about preprocessor definition flags and include directories, unless you would manually duplicate them. In large and fast-changing projects, this is not an option.

The lack of compiler configuration knowledge (defines and include directories) breaks various C/C++ tooling features, such as Macro Expansion or the Open Declaration feature, both of which are an essential tool in my toolbelt, and particularly useful in large code bases such as micro controller projects with various SDKs etc.

In some configurations, Eclipse might be able to parse GCC build output, but when I was working with the QMK keyboard firmware, I couldn’t get the QMK makefiles to print commands that Eclipse would understand, not even with VERBOSE=true.

Luckily, there is a solution! Eclipse CDT 9.10 introduced Compilation Database support in 2019. MCUXpresso v11.3.0 ships with CDT 9.11.1.202006011430, meaning it does contain Compilation Database support.

In case you want to check which version your installed IDE has, open HelpAbout MCUXpresso IDE, click Installation Details, open the Features tab, then locate the Eclipse CDT, C/C++ Development Platform line.

For comparison, Eclipse IDE 2021-03 contains 10.2.0.202103011047, if you want to verify that the issues I reference below are indeed fixed.

Bug: command vs. arguments

Before we can enable Compilation Database support, we need to ensure we have a compatible compile_commands.json database file. Eclipse CDT’s Compilation Database support before version CDT 10 suffered from Bug 563006: it only understood the command JSON property, not the arguments property.

Depending on your build system, this isn’t a problem. For example, Meson/ninja’s compile_commands.json uses command and will work fine.

But, when using Make with Bear, you will end up with arguments by default.

Bear 3.0 allows generating a compile_commands.json Compilation Database with command, but requires multiple commands and config files, which is a bit inconvenient with Eclipse.

So, let’s put the extra commands into a commandbear.sh script:

#!/bin/sh

set -eux

intercept --output commands.json -- "$@"
citnames \
  --input commands.json \
  --output compile_commands.json \
  --config config.json

The "command_as_array": false option goes into config.json:

{
  "compilation": {
  },
  "output": {
    "content": {
      "include_only_existing_source": true
    },
    "format": {
      "command_as_array": false,
      "drop_output_field": false
    }
  }
}

Don’t forget to make the script executable:

chmod +x commandbear.sh

Then configure Eclipse to use the commandbear.sh script to build:

  1. Open Project Properties by right-clicking your project in the Project Explorer panel.
  2. Select C/C++ Build and open the Builder Settings tab
  3. In the Builder group, set the Build command text field to: ${workspace_loc:/qmk_firmware}/commandbear.sh make -j16

Verify your build is working by selecting ProjectClean… and triggering a build.

Enabling Compilation Database support

  1. Open Project Properties by right-clicking your project in the Project Explorer panel.
  2. Expand C/C++ General, select Preprocessor Include Paths, Macros etc. and open the Providers tab.
  3. Untick everything but:
    • MCU GCC Built-in Compiler Parser
    • MCU GCC Build Output Parser
    • Compilation Database Parser
  4. Select Compilation Database Parser, click Apply to make the Compilation Database text field editable.
  5. Put a full path to your compile_commands.json file into the text field, e.g. /home/michael/kinx/workspace/qmk_firmware/compile_commands.json. Note that variables will not be expanded! Support for using variables was added later in Bug 559186.
  6. Select MCU GCC Build Output Parser as Build parser.
  7. Tick the Exclude files not in the Compilation Database checkbox.
  8. Click Apply and Close.
Compilation Database Parser settings

You will know Compilation Database support works when its progress view shows up:

Compilation Database progress

If you have an incompatible or empty compile_commands.json, nothing visible will happen (no progress indicator or error messages).

After indexing completes, you should see:

  1. Files that were not used as greyed out in the Project Explorer
  2. Open Declaration in the context menu of a selected identifier (or F3) should jump to the correct file. For example, my test sequence for this feature in the QMK repository is:
    • in tmk_core/protocol/chibios/main.c, open init_usb_driver
    • open usbStart, should bring up lib/chibios git submodule
    • open usb_lld_start, should bring up MIMXRT1062 port
  3. Macros expanded correctly, e.g. MIMXRT1062_USB_USE_USB1 in the following example
Compilation Database in effect: files greyed out and macros expanded

Slow file exclusion in projects with many files

Bug 565457 explains an optimization in the algorithm used to generate the list of excluded paths, which I would summarize as “use whole directories instead of individual files”.

This optimization was introduced later, so in MCUXpresso v11.3, we still have to endure watching the slow algorithm for a few seconds:

Compilation Database exclusion slow

Conclusion

NXP, please release a new MCUXpresso IDE with a more recent CDT version!

The improvements in the newer version would make the setup so much simpler.

at 2021-04-01 09:59

2021-03-13

sECuREs website

Make your intercom smarter with an MQTT backpack

I bought the cheapest compatible BTicino intercom device (BT 344232 for 32 €) that I could find on eBay, then soldered in 4 wires and added microcontrollers to make it smart. It now connects to my Nuki Opener Smart Intercom IOT device, and to my local MQTT Pub/Sub bus (why not?).

modified BTicino

Background

In my last post about the BTicino intercom from November, I described how to use a Teensy microcontroller to reliably interpret SCS bus signals and drive a Nuki Opener (Smart Intercom).

Originally, I had hoped the Nuki developers would be able to fix their device based on my SCS bus research, but they don’t seem to be interested. Instead, their support actually suggested I run my microcontroller workaround indefinitely!

Hence, I decided to work on the next revision to clean up my setup in terms of cable clutter. I also figured: if I already need to run my own microcontroller, I also want to connect it to my local MQTT Pub/Sub bus for maximum flexibility.

Unfortunately, the Teensy microcontroller lacks built-in WiFi, or any kind of networking.

I switched to an ESP32-based microcontroller, but powering those from the SCS bus seems like a bad idea: they draw a lot of power, and building small high-quality power supplies is hard.

This made me scrap my previous plans to make my own SCS send/receive hardware.

Instead, I wondered what the easiest yet most reliable approach might be to make this intercom unit smart. Instead of building my own SCS hardware, could I use the intercom unit itself to send the door unlock signal, and could I obtain the unit’s already-decoded SCS bus signal?

Finding the signals

Based on my previous research, I roughly knew what to expect: closest to the bus terminals, there will be some components that filter the bus signal and convert the 27V into a lower voltage. Connected to that power supply is a microcontroller which deals with all user interface.

To learn more about the components, I first identified all ICs (Integrated Circuits) based on their labeling. The following are relevant:

I connected my development intercom unit to my SCS bus lab setup and used my oscilloscope to confirm expected signal levels based on the pinout from the IC datasheets.

I settled on the following 4 relatively easily accessible signals and soldered jumper wires to them:

  • 5V and GND: 5V, 100mA. Our QT Py microcontroller uses 7mA.
  • OPEN5V: activates the button which unlocks the door
  • SCSRX5V: converted SCS signal
BTicino signals

Converting the signals

Because the BTicino intercom units runs at 5V, but more modern microcontrollers run at 3.3V, we need to convert between the two voltages:

  1. We need to convert a 3.3V signal to OPEN5V to trigger opening the door.

  2. We need to convert SCSRX5V signal to 3.3V so that I can use an ESP32 microcontroller to read the signal and place it on MQTT.

Here’s the corresponding schematic:

schematic

Microcontroller selection

I eventually decided to task a dedicated microcontroller with the signal conversion, instead of having the WiFi-enabled microcontroller do everything, for multiple reasons:

  • Reliability. It turns out that using a hardware analog comparator results in a much higher signal quality than continuously sampling an ADC yourself, even when using the ESP32’s ULP (Ultra Low Power) co-processor to do the sampling.

  • Easy implementation. Converting an SCS signal to a serial signal is literally a single delayMicroseconds(20); call in the right place. Having a whole microcontroller for only this task eliminates any concurrency concerns. I have not had to debug or change the software even once in the last few months.

  • Easy debugging/introspection. I can connect a standard USB-to-serial adapter and verify the signal is read correctly. This quickly narrows down issues on either side of the serial interface. Issues with the microcontroller side can be reproduced by sending serial data.

Here are the 2 microcontrollers I’m using in this project, plus the Teensy I used previously:

Microcontroller WiFi Analog Comparator Price
Teensy 4.0 no yes 19 USD
Adafruit QT Py no yes 6 USD
TinyPICO yes no 20 USD

If ESP32 boards such as the TinyPICO had a hardware Analog Comparator, I would likely use just one microcontroller, but keep the serial interface on a GPIO for easy debugging.

Why the Adafruit QT Py?

The minimal function we need for our signal conversion device is to convert an SCS signal (5V) to a serial signal (3.3V). For this conversion, we need a hardware analog comparator and an output GPIO that we can drive independently, so that we can modify the signal.

Additionally, the device should use as little power as possible so that it can comfortably fit in the left-over energy budget of the intercom unit’s power supply.

The smallest microcontroller I know of that comes with a hardware analog comparator is the Adafruit QT Py. It’s a 32-bit Cortex M0+ (SAMD21) that can be programmed using the Arduino IDE, or MicroPython (hence the name).

Adafruit QT Py

There are other SAMD21 boards with the same form factor, such as the Seeeduino XIAO.

Why the TinyPICO ESP32 board?

When looking for a WiFi-enabled microcontroller, definitely go with something ESP32-based!

The community around the Espressif ESP32 (and its predecessor ESP8266) is definitely one of its biggest pluses: there are tons of Arduino sketches, troubleshooting tips, YouTube videos, reference documentation, forum posts, and so on.

The ESPs have been around since ≈2014, so many (largely-compatible) boards are available. In fact, I started this project on an M5Stack ESP32 Basic Core IoT Development Kit, deployed it on an Adafruit HUZZAH32 Breakout Board and ultimately ported it to the TinyPICO. Porting between the different microcontrollers was really smooth: the only adjustments were pin numbers and dropping in a TinyPICO helper library for its RGB LED, which I chose to use as a power LED.

I chose the TinyPICO ESP32 board specifically for its small form factor and convenience:

TinyPICO comparison with Adafruit Huzzah32 and Teensy 4.0

The TinyPICO is only 18mm × 32mm, slightly smaller than the Teensy 4.0’s 18mm × 35mm.

In comparison, the Adafruit HUZZAH32 breakout board is gigantic with its 25mm × 44mm. And that’s without the extra USB-to-serial adapter (FT232H in the picture above) you need for programming, serial console and powering the board!

The TinyPICO does not need an extra adapter. You can plug it in and program it immediately, just like the Teensy!

I’d like it if the next revision of the TinyPICO switched from Micro USB to USB C.

If the TinyPICO is not for you (or unavailable), search for other boards that contain the ESP32-PICO-D4 chip. For example, DFRobot’s ESP32-PICO-KIT or Espressif’s own ESP32-PICO-KIT.

Prototype

After testing everything on a breadboard, I soldered a horizontal pin header onto the QT Py, connected it to my Sparkfun level shifter board and soldered the remaining voltage divider components “flying”. The result barely fit into the case, but worked flawlessly for weeks:

prototype

Backpack PCB for the QT Py

After verifying this prototype works well in practice, I miniaturized it into a “backpack” PCB.

The backpack contains all the same parts as the prototype, but with fewer bulky wires and connectors, and using only SMD parts. The build you see below uses 0602 SMD parts, but if I made another revision I would probably chose the larger 0805 parts for easier soldering.

QT Py with backpack QT Py with backpack PCB

Assembly

To save some space in the intercom unit case, I decided to solder the jumper wires directly onto the TinyPICO instead of using a pin header. I could have gone one step further by cutting the wires at length and soldering them directly on both ends, without any connectors, but I wanted to be able to easily unplug and re-combine the parts of this project.

wires soldered directly into the TinyPICO

From top to bottom, I made the following connections:

Pin Color Function
25 red SCSRX_3V3
27 green OPEN_3V3
15 blue Nuki Opener blue cable
14 yellow Nuki Opener yellow cable
4 purple floor ring button pushed
3V3 white 3.3V for the floor ring button
5V orange power for the TinyPICO
GND brown ground for the TinyPICO
GND brown ground to the QT Py
GND brown ground to the Nuki Opener

The TinyPICO USB port is still usable for updating the software and serial console debugging.

Here’s the TinyPICO connected to the QT Py inside the intercom unit:

modified BTicino

The QT Py is powered by the intercom unit’s supply, and the TinyPICO I’m powering with an external USB power supply and a cut-open USB cable. This allows me to route the jumper wires through the intercom unit’s hole in the back, through which a USB plug doesn’t fit:

final installation

Software / Artifacts

You can find the Arduino sketches and KiCad files for this project at https://github.com/stapelberg/intercom-backpack

For debugging, I found it useful to publish every single byte received from the SCS bus on the doorbell/debug/scsrx MQTT topic. Full SCS telegrams are published to doorbell/events/scs, so by observing both, you can verify that retransmission suppression and SCS decoding work correctly.

Similarly, signaling a doorbell ring to the Nuki Opener can be debugged by sending a message to MQTT topic doorbell/debug/cmd/ring.

Initially, it wasn’t clear to me whether the WiFi library would maintain the connection indefinitely. After observing my microcontroller eventually disappearing from my network, I added the taskreconnect FreeRTOS task, and things have been stable since.

Nuki Opener: verdict

I now have a Nuki Opener running next to my own microcontroller, so I can see how well it works.

setup

Setting up the Nuki is the worst part: their colorful cable is super flimsy and loose, often losing contact. They should definitely switch to a cable with a mechanical lock.

The software part of the setup is okay, but the compatibility with the SCS bus is poor: I couldn’t get the device to work at all (see my initial post), and had to resort to using my own microcontroller to drive the Nuki in analogue mode.

I’m disappointed that the Nuki developers aren’t interested in improving their device’s compatibility and reliability with the SCS bus. They seem to capture/replay the entire signal (including re-transmissions) instead of actually decoding the signal.

in my day-to-day

The push notifications I get on my iPhone from the Nuki are often delayed. Usually the delay is a few seconds, but sometimes notifications arrive hours later or just don’t arrive at all!

While the push notifications are sent from a Nuki server and hence need the internet to function, the Nuki Bridge (translating Bluetooth Low Energey from the Nuki Opener to WiFi) allows configuring notifications in the local network via web hooks.

The Nuki Bridge’s notifications are much more reliable in my experience.

People sometimes ask why I use the Nuki Opener at all, given that I have some infrastructure of my own, too. While opening the door and receiving notifications is something I can do without the Nuki, too, I don’t want to spend my spare time re-implementing the Nuki app (on multiple platforms) with its geo fencing, friend invitations, ring to open, etc. In addition, the Nuki Opener physical device has a nice ring sound and large push button to open the door, both of which are convenient.

Conclusion

My intercom is now much smarter! Doorbell notifications make their way to my various devices via MQTT, and I can conveniently open the door from any device, as opposed to rushing to the intercom unit in the hallway.

Compared to the previous proof-of-concepts and development installations, I feel more confident in the current solution because it re-uses the intercom unit for the nitty-gritty SCS bus communication details.

The overall strategy should be widely applicable regardless of the specific intercom vendor/unit you have. Be sure to buy your own unit (don’t solder into your landlord’s intercom unit!) and test in a separate lab setup first, of course!

Appendix A: Troubleshooting

To debug the problem of ring detection no longer working, check:

  • Is the ESP32 still working?
    • ping doorbelltp
    • mosquitto_pub -h dr -t 'doorbell/debug/cmd/ring' -m '3' should signal a ring to the Nuki Opener and result in events on the MQTT bus
  • Is the QT Py still working?
    • Its power LED should be off. If the LED is on, the QT Py is in the bootloader.
    • Unplug and replug the +5V wire to the QT Py, see if that fixes it.
    • Connect a USB-to-serial adapter and see if triggering a door open results in SCS bytes on the serial interface.
    • See if ringing the bell results in SCS bytes on the serial interface. If no, re-solder cable to SCSRX5V.

To debug the problem of door opening no longer working, check:

  • Does it work when triggering it via the button on the BTicino? If yes, re-solder cable to OPEN5V.

at 2021-03-13 15:54

2021-03-06

sECuREs website

Debian Code Search: OpenAPI now available

Debian Code Search now offers an OpenAPI-based API!

Various developers have created ad-hoc client libraries based on how the web interface works.

The goal of offering an OpenAPI-based API is to provide developers with automatically generated client libraries for a large number of programming languages, that target a stable interface independent of the web interface’s implementation details.

Getting started

  1. Visit https://codesearch.debian.net/apikeys/ to download your personal API key. Login via Debian’s GitLab instance salsa.debian.org; register there if you have no account yet.

  2. Find the Debian Code Search client library for your programming language. If none exists yet, auto-generate a client library on editor.swagger.io: click “Generate Client”.

  3. Search all code in Debian from your own analysis tool, migration tracking dashboard, etc.

curl example

curl \
  -H "x-dcs-apikey: $(cat dcs-apikey-stapelberg.txt)" \
  -X GET \
  "https://codesearch.debian.net/api/v1/search?query=i3Font&match_mode=regexp" 

Web browser example

You can try out the API in your web browser in the OpenAPI documentation.

Code example (Go)

Here’s an example program that demonstrates how to set up an auto-generated Go client for the Debian Code Search OpenAPI, run a query, and aggregate the results:

func burndown() error {
	cfg := openapiclient.NewConfiguration()
	cfg.AddDefaultHeader("x-dcs-apikey", apiKey)
	client := openapiclient.NewAPIClient(cfg)
	ctx := context.Background()

	// Search through the full Debian Code Search corpus, blocking until all
	// results are available:
	results, _, err := client.SearchApi.Search(ctx, "fmt.Sprint(err)", &openapiclient.SearchApiSearchOpts{
		// Literal searches are faster and do not require escaping special
		// characters, regular expression searches are more powerful.
		MatchMode: optional.NewString("literal"),
	})
	if err != nil {
		return err
	}

	// Print to stdout a CSV file with the path and number of occurrences:
	wr := csv.NewWriter(os.Stdout)
	header := []string{"path", "number of occurrences"}
	if err := wr.Write(header); err != nil {
		return err
	}
	occurrences := make(map[string]int)
	for _, result := range results {
		occurrences[result.Path]++
	}
	for _, result := range results {
		o, ok := occurrences[result.Path]
		if !ok {
			continue
		}
		// Print one CSV record per path:
		delete(occurrences, result.Path)
		record := []string{result.Path, strconv.Itoa(o)}
		if err := wr.Write(record); err != nil {
			return err
		}
	}
	wr.Flush()
	return wr.Error()
}

The full example can be found under burndown.go.

Feedback?

File a GitHub issue on github.com/Debian/dcs please!

Migration status

I’m aware of the following third-party projects using Debian Code Search:

Tool Migration status
Debian Code Search CLI tool Updated to OpenAPI
identify-incomplete-xs-go-import-path Update pending
gnome-codesearch makes no API queries

If you find any others, please point them to this post in case they are not using Debian Code Search’s OpenAPI yet.

at 2021-03-06 10:15

2021-03-04

michael-herbst.com

PostDoc position at Appl. & Comput. Mathematics lab, RWTH Aachen University

This week I started my new position as a postdocotoral researcher at the Applied and Computational Mathematics (ACoM) research lab at RWTH Aachen University. The lab consists of two interdisciplinary research groups, namely the group of Prof. Dr. Manuel Torrilhon, who works on the mathematical modelling and simulation of technical processes (e.g. plasma or gas flow processes) as well as the group of Prof. Dr. Benjamin Stamm, which I am now joining. Ben's research focus is the numerical analysis of PDEs and linear algebra problems, which arise e.g. in electrostatics or quantum chemistry. This includes principle questions related to eigenvalue problems, but also concrete applications such as improving the performance of the polarisable continuum model, a standard solvation model in electronic structure theory. I see a good fit between our respective research backgrounds and I am happy for this opportunity to extend my research horizon and contribute to the research in Ben's group and the ACoM over the next years.

During my time in Aachen Ben and I want to continue to work on the numerical analysis and the development of mathematically-motivated methods for density-functional theory (DFT). One aspect we have in mind, for example, is to port Ben's recent work for constructing good initial guesses for the self-consistent iterations in molecular DFT (and Gaussian basis functions) to plane-wave DFT. As part of this research we will make use and extend the density-functional toolkit (DFTK), the density-functional theory code I started in Paris. Beyond our work at the ACoM I expect DFTK and its suitability for multidisciplinary research also to be helpful for reaching out to other researchers in the mathematics, computer science and physics departments in Aachen. In particular I see a good fit of DFTK within the JARA-CSD, a joint research initiative between RWTH Aachen and the Jülich research centre.

Having known Aachen already a little from my previous visits in Ben's group I am very much looking forward to work here. Not only is the city very pretty and welcoming, but also the interdisciplinary orientation of RWTH Aachen resonates well with me. I'm looking forward to the many interesting discussions to come and to becoming part of strengthening the interdisciplinary links in Aachen, while at the same time continuing to work at the boundary of chemistry, physics and mathematics.

by Michael F. Herbst at 2021-03-04 23:00 under electronic structure theory, DFT, solid state

2021-03-02

Insanity Industries

Pareto-optimal compression

Data compression is an incredibly useful tool in many situations, be it for backups, archiving, or even filesystems. But of the many compressors that there are, which of them is the one to be preferred? To properly answer this question, we first have to answer a different, but related one: What exactly makes a compression algorithm good?

Time for a closer investigation to find the best of the best of the best!

Pareto optimality or what makes a compression algorithm good

Pareto optimality is a situation where no […] preference criterion can be better off without making at least one […] preference criterion worse off […].

The concept of Pareto Optimality is tremendously useful, far beyond the scope of this blogpost, by acknowledging competing interests.

In our particular case of optimal compression, one of the obvious candidates is compression size. The smaller the resulting file is, the better the compressor, right?

But what if we are just compressing so we can speed up upload to a remote location? At this point, it doesn’t matter if the compression is supremely superior to everything else if we have to wait twice as long for it to happen compared to simply upload the uncompressed file.

So for our algorithms, we have at least two criteria that come into play: actual achievable compression and compression cost, which we will measure as “how long does it take to compress our content?"1. These two are the criteria this blogpost is focused on.

A practical example

Practically speaking, let’s assume two compression tools A and B, having both two compression levels 1 and 2, which we use on a sample file. The results might look like the following:

algorithm level file size time
A 1 60% 2s
A 2 50% 4s
B 1 40% 3s
B 2 30% 11s

We find that while both algorithms increase their compression size with level, B.1 is unconditionally better than A.2, so there is no reason to ever use A.2 if B.1 is available. Besides that, A.1, B.1 and B.2 continuously increase in compression effectiveness as well as time taken for that. For easier digestion, we can visualize these results:

Compression results for different hypothetical compression algorithms, including the Pareto frontier indicated in blue.

Compression results for different hypothetical compression algorithms, including the Pareto frontier indicated in blue.

Here we clearly see what could already be taken from the table above, but significantly more intuitive. Remembering our definition of Pareto optimality from before, we see that A.2 is not Pareto optimal, as B.1 is both better in time taken as well as in compression effect. This shows more intuitively that in this scenario there is no reason to use A.2 if B.1 is available. For A.1, B.1 and B.2 the choice is not so clear-cut, as the resulting file size can only be reduced further by investing more time into the compression. Hence, all three of them are Pareto optimal and constitute the Pareto frontier or the Pareto set.

One wants to always strive for choosing a Pareto optimal solution whenever possible, as non-Pareto-optimal solutions are always to some degree wasteful.

With this insight into Pareto optimality, we can now put our knowledge into practice and begin our journey to the Pareto frontiers of compression algorithms.

Setup and test procedure for real-world measurements

Data was gathered on a Linux system with a given sample file for the resulting filesize compared to the original as well as the time the process took for compressing. First, the sample file was read from disk entirely, ensuring it would be present in Linux' filesystem cache, then compressed with each compression program at each level 15 times for a decent statistic, the compressed result is routed directly into /dev/null. All presented compression times presented are the median of these 15 runs, compression size was only measured once after verifying that it was deterministic2.

Unless explicitly denoted otherwise, all compressors were run in their default configuration with a single compression thread. Furthermore, all applications on the machine used for benchmarking except the terminal holding the benchmarking process were closed to reduce interference by other processes as much as possible.

The tests for decompression were done analogously (with the obvious exception that the sample file is compressed according to the testcase in play beforehand), single-threaded decompression as well as routing the decompressed result to /dev/null.

All tests were run on several different CPUs: Intel i7-8550U, Intel i5-3320M3 (both mobile CPUs), Intel i7-7700K and AMD Ryzen 7 3800X (both workstation CPUs). While the absolute compression times changed, the Pareto frontiers did not, hence in this blogpost only the plots for the i7-8550U are shown exemplarily.

Case study: initramfs-compression

Just recently, the current maintainer of mkinitcpio, Giancarlo Razzolini, announced the transition from gzip to zstd as the default for initramfs-compression. The initramfs is a little mini-system, whose only objective is to prepare the hardware and assemble all disks so that the main system is readily prepared to take over operation. The initramfs needs to be loaded (and hence decompressed) at every boot as well as recreated occasionally, when components contained in it are updated.

mkinitcpio supports a variety of compression algorithms: none, gzip, bzip2, lzma, lzop, lz4 and most recently zstd.

Quantifying these algorithms and plotting the Pareto frontier for compression yields:

Compression results for a standard mkinitcpio-generated initramfs and the corresponding Pareto frontier. The difficult-to-decipher black culmination at about 34% resulting file size are the bzip2 results.

Compression results for a standard mkinitcpio-generated initramfs and the corresponding Pareto frontier. The difficult-to-decipher black culmination at about 34% resulting file size are the bzip2 results.

We find that the change of defaults from gzip to zstd was well justified, as gzip can no longer be considered Pareto optimal for this type of file. Choosing lzma as the default would make it even smaller, but this would be paid by a noticable higher resource usage for compression (which has to be invested on every update affecting the initramfs), so from the data zstd is certainly the wiser choice4.

This can also be seen when we take a look of Pareto optimality of decompression (after all, this decompression needs to happen on every single system boot):

Decompression results for a standard mkinitcpio-generated initramfs and the corresponding Pareto frontier.

Decompression results for a standard mkinitcpio-generated initramfs and the corresponding Pareto frontier.

We clearly see that zstd is blasting all other algorithms out of the water when it comes to decompression speed, making it even more of a good choice for this use case. Given these numbers, it is even more of a good choice for an initramfs, not only does it compress fast, it also decompresses impressively fast, six times faster than lzma which was previously known for its quick decompression speed despite high compression factors.

Given the data for zstd, it cannot be ruled out completely that zstd simply hit a non-CPU-bound on decompression, but even if it did, the conclusion for the choice of algorithm does not change.

dracut

If you happen to be on a Linux distribution that uses dracut for initramfs-generation, the conclusions that can be drawn for dracut-initramfs-compressions are the almost identical given the compression and decompression data for a dracut-initrd, the Pareto frontier remains mostly unchanged with just some more levels of zstd in it.

Real world usage recommendation

To use zstd in mkinitcpio, simply use version 30 and above. You can modify the compression level (default is -3) by adding a specific level to COMPRESSION_OPTIONS, but given the data, this doesn’t seem to provide much of a benefit.

For dracut, add compress="zstd" to /etc/dracut.conf to get zstd compression at the default level.

Case study: borgbackup

In the next scenario we will investigate the impact of compression when backing up data with borgbackup. Borg comes with an integrated option to compress the backupped data, with algorithms available being lz4, zstd, zlib/gzip and lzma. In addition to this, borg has an automated detection routine to see if the files backupped do actually compress enough to spend CPU cycles on compressing (see borg help compression for details).

For this scenario, we do not only have one definite sample, but will consider three different samples: A simple text file, being a dump of the Kernel log via the dmesg command, representing textual data. A binary, in this particular case the dockerd ELF binary, (rather arbitrarily) representing binary data. Finally, an mkinitcpio-image, intended as somewhat of a mixed-data sample. We do not consider media files, as these are typically already stored in compressed formats, hence unlikely to compress further and thus dealt with by borg’s compressibility heuristics.

The resulting Pareto frontiers for compression are, starting with least effective compression:

text binary initramfs
lz4.2 lz4.1 lz4.1
zstd.1 zstd.1 zstd.1
zstd.2 zstd.2 zstd.2
zstd.4 zstd.3 zstd.3
zstd.3 zstd.4 zstd.4
zstd.5 zstd.5 zstd.5
zstd.6 zstd.6 zstd.6
zstd.7 zstd.7 zstd.7
zstd.8 zstd.8 zstd.8
zstd.9 zstd.9 zstd.9
zstd.10 lzma.0 lzma.0
zstd.11 lzma.1 lzma.1
zstd.12 lzma.2 lzma.2
lzma.2 lzma.3 lzma.3
lzma.3 lzma.4 lzma.4
lzma.6 lzma.5 lzma.5
zstd.19 lzma.6 lzma.6
. lzma.7 lzma.7
. lzma.8 lzma.8
. lzma.9 lzma.9

We see that effectively, except for some brief occurrance of lz4 at the top, the relevant choices are lzma and zstd. More details can be seen in the plots linked in the column headers. Hence, as backups should be run often (with a tool as borg there is little reason for anything else but daily), zstd with a level slightly below 10 seems to be a good compromise of speed and resulting data size.

Real world usage recommendation

Adding --compression=auto,zstd,7 to the borg command used to create a backup will use zstd on level 7 if borgs internal heuristics considers the file in question to compress well, otherwise no compression will be used.

This flag can be added on-the fly, without affecting existing repositories or borgs deduplication. Already backupped data is not recompressed, meaning that adding this flag for use with an existing repository does not require a reupload of everything. Consequentially, it also means that to recompress the entire repository with zstd one effectively has to start from scratch.

Case study: Archiving with tar

Compression can not only be used for backup-tools like borg, it can also be used to archive files with tar. Some compressors have explicit flags in tar, such as gzip (-z), lzma (-J) or bzip2 (-j), but any compression algorithm can be used via tar’s -I flag.

Working with tar poses two challenges with regard to the compressor:

  • streaming: tar concatenates data and streams the result through the compressor. Hence, to extract files at a certain position of a tarball, the entirety of the data before that file needs to be decompressed as well.
  • compress-only: as a further consequence of this, tar lacks a feature testing if something is well compressible, so uncompressible data will also be sent through the compressor

If we want to investigate a good compressor without consideration for the input data, we aim for picking a Pareto optimal compressor that takes two properties into consideration:

  • fast decompression, to be able to easily and quickly extract data from the archive again if need be
  • good performance on non-compressible data (this particularly means that incompressible data should especially not increase in size).

Compression capabilities

To investigate the decompression capabilities of certain compressors, we can reuse the dataset used on the borg case study and add some incompressible data in form of a flac music file to the mix. As tar has a larger variety of usable algorithms, we include lz4, gzip, lzma, zstd, lzop and brotli as well.

We can exemplarily see the effectiveness of these on the dockerd elf binary (other datasets can be found below) first:

Compression results and the corresponding Pareto frontier for the dockerd elf binary. The unreadable black cluster at 25% compressed size is again bzip2, the one at 34% is predominantly lz4.

Compression results and the corresponding Pareto frontier for the dockerd elf binary. The unreadable black cluster at 25% compressed size is again bzip2, the one at 34% is predominantly lz4.

Decompression results and the corresponding Pareto frontier for the dockerd elf binary. The clusters are lzma for 20% resulting size, zstd at 25%, brotli at 24% and lz4 at 33%.

Decompression results and the corresponding Pareto frontier for the dockerd elf binary. The clusters are lzma for 20% resulting size, zstd at 25%, brotli at 24% and lz4 at 33%.

In summary, the Pareto frontiers for the different types of data overall turn out to be (with c for compression and d for decompression):

text (c) text (d) binary (c) binary (d) initramfs (c) initramfs (d) flac (c) flac (d)
lz4.2 lz4.9 lz4.1 lz4.8 lz4.1 lz4.7 lz4.2 zstd.1
zstd.1 zstd.13 lzop.1 lz4.9 lzop.6 lz4.9 brotli.1 zstd.5
zstd.2 zstd.11 lzop.3 zstd.15 zstd.1 zstd.1 brotli.2 zstd.7
zstd.4 zstd.12 zstd.1 zstd.16 zstd.2 zstd.9 brotli.3 zstd.19
zstd.3 zstd.14 zstd.2 zstd.17 zstd.3 zstd.14 zstd.18 .
zstd.5 zstd.15 zstd.3 zstd.19 zstd.4 zstd.15 zstd.19 .
zstd.6 zstd.19 zstd.4 lzma.8 zstd.5 zstd.16 . .
zstd.7 . zstd.5 lzma.9 zstd.6 zstd.17 . .
zstd.8 . zstd.6 . zstd.7 zstd.18 . .
zstd.9 . zstd.7 . zstd.8 zstd.19 . .
zstd.10 . zstd.8 . zstd.9 lzma.8 . .
brotli.5 . zstd.9 . brotli.5 lzma.9 . .
brotli.6 . brotli.5 . lzma.0 . . .
brotli.7 . lzma.1 . lzma.1 . . .
lzma.3 . lzma.2 . lzma.2 . . .
lzma.6 . lzma.3 . lzma.3 . . .
zstd.19 . lzma.4 . lzma.4 . . .
. . lzma.5 . lzma.5 . . .
. . lzma.6 . lzma.6 . . .
. . lzma.7 . lzma.7 . . .
. . lzma.8 . lzma.8 . . .
. . lzma.9 . lzma.9 . . .

As can be seen in the linked plots, we again find that while lzma still achieves the highest absolute compression, zstd dominates the sweet spot right before computational cost skyrockets. We also find brotli to be an interesting contender here, making it into the Pareto frontier as well. However, with only sometimes making it into the Pareto frontier, whereas lzma and zstd robustly defend their inclusion in it, it seems more advisable to resort to either lzma or zstd as this only provides a sample binary and actual data might vary. Furthermore, when it comes to decompression brotli is not Pareto optimal anymore at all, also indicating lzma and zstd as being the better choice.

Impact on incompressible files in detail

We will take another closer look at the incompressible case, represented by a flac file and strip away bzip2 and lzma, as we could tell from the linked plots that these two clearly increase the size of the result and are hence not Pareto optimal (as they are already beaten by the Pareto optimal case “no compression”).

The results have a clear indication:

Compression results on incompressible data.

Compression results on incompressible data.

Decompression results on incompressible data. The unreadable black cluster at 99.975% size is zstd, the one at 99.988% contains lzop and brotli.

Decompression results on incompressible data. The unreadable black cluster at 99.975% size is zstd, the one at 99.988% contains lzop and brotli.

The recommended choice of algorithm for compression is either brotli or zstd, but when it comes to decompression, zstd takes the lead again. This is of course a cornercase, the details of this might change with the particular choice of incompressible data. However, I do not expect the overall impression to significantly change.

Real world usage recommendation

Concluding this section, the real-world recommendation resulting from this seems to simply use zstd for any tarball compression if available. To do so, tar "-Izstd -10 -T0" can be a good choice, with -T0 telling zstd to parallelize the compression onto all available CPU cores, speeding things up even more beyond our measurements. Depending on your particular usecase it might be interesting to use an alias like

alias archive='tar "-Izstd -19 -T0" -cf'

which allows quickly taring data into a compressed archive via archive myarchive.tar.zst file1 file2 file3 ….

Case study: filesystems

Another usecase for compression is filesystem compression. Conceptually similar to what borg does, files are transparently compressing and decompressed when written to or read from disk.

Among the filesystems capable of such inline-compression are ZFS and btrfs. btrfs supports ZLIB (gzip), LZO (lzop) and zstd, whereas ZFS supports lz4, LZJB (which was not included in these benchmarks as no appropriate binary compressor was found), gzip and ZLE (zero-length-encoding, only compressing zeroes, hence also not tested). zstd support for OpenZFS has been merged, but apparently hasn’t made it into any stable version yet at time of writing according to the OpenZFS documentation.

This case is situated similar to the tar-case study, and as all compressors available for ZFS and btrfs have already been covered in the section above, there is no reason to reiterate these results here.

It shall, however, be noted, that at least for btrfs, the standard flag for filesystem compression adopts a similar heuristic as borg and hence the case of incompressible data might not be so relevant for a btrfs installation. That being said, the conclusion here is a recommendation of zstd, and as we have seen in the last section, the question of incompressible files doesn’t change the overall recommendation.

Real world usage recommendation

If you want to save diskspace by using compression, the mount option compress in combination with zstd is generally a good choice for btrfs. This also includes the compressibility-heuristics (compress-force would be the option that compresses without this heuristics). For ZFS, the general recommendation is consequentially also zstd once it makes its way into a release.

Conclusion

Concluding the experiments laid out in this blogpost, we can effectively state an almost unconditional and surprisingly clear recommendation to simply use zstd for everything. The exact level might depend on the usecase, but overall it has demonstrated to be the most versatile, yet effective compressor around when it comes to effectiveness and speed, both for compression and especially decompression. Furthermore, it has, in contrast to most other contenders, flags for built-in parallelization, which not used in this blogpost at all, and yet zstd still stomped almost the entire competition.

Only if resulting filesize should be pushed down as much as possible, without any regard for computational cost, lzma retains an edge for most kinds of data. In practice, however, the conclusion is to simply use zstd.

Thanks to Joru and corvus for proofreading and helpful comments.


  1. Which is a easy though little bit cheated way of asking “how much CPU-time do I have to burn on this?”. Obviously, there are plenty of other criteria that might be relevant, depending on the particular usecase, such as memory consumption. Furthermore, all these criteria also apply for decompression as well as compression, as we will investigate later, technically doubling the amount of criteria we can take into consideration. ↩︎

  2. For algorithms that allow parallelized compressions, this might no longer necessarily be the case, but all data in this blogpost was gathered with non-parallelized compression for all tested algorithms. ↩︎

  3. Fun fact on the site: the entire benchmarking suite (including some more data that is not included in this blogpost) runs 61 days straight on the i5-3320M. Fortunately it’s a bit faster on newer CPUs. :D ↩︎

  4. Furthermore, mkinitcpio runs zstd with -T0 by default, which parallelizes compression to all available cores. This accellerates compression even further, but was not tested in this particular scenario and hence not included in the plot, as most compressors do not support parallelization. But even without parallelization, zstd still makes it to the pareto frontier. There might be another blogpost upcoming to take a look at parallelization at some point, though… ↩︎

by Jonas Große Sundrup at 2021-03-02 22:39