Planet NoName e.V.

21. January 2018

Mero’s Blog

What even is error handling?

tl;dr: Error handling shouldn't be about how to best propagate an error value, but how to make it destroy it (or make it irrelevant). To encourage myself to do that, I started removing errors from function returns wherever I found it at all feasible

Error handling in Go is a contentious and often criticized issue. There is no shortage on articles criticizing the approach taken, no shortage on articles giving advice on how to deal with it (or defending it) and also no shortage on proposals on how to improve it.

During these discussion, I always feel there is something missing. The proposals for improvement usually deal with syntactical issues, how to avoid boilerplate. Then there is the other school of thought - where it's not about syntax, but about how to best pass errors around. Dave Chaney wrote an often quoted blog post on the subject, where he lists all the ways error information can be mapped into the Go type system, why he considers them flawed and what he suggests instead. This school of thought regularly comes up with helper packages, to make wrapping or annotating errors easier. pkg/errors is very popular (and is grown out of the approach of above blog post) but upspin's incarnation also gathered some attention.

I am dissatisfied with both schools of thought. Overall, neither seems to explicitly address, what to me is the underlying question: What is error handling? In this post, I'm trying to describe how I interpret the term and why, to me, the existing approaches and discussions mostly miss the mark. Note, that I don't claim this understanding to be universal - just how I would put into words my understanding of the topic.


Let's start with a maybe weird question: Why is the entry point into the program func main() and not func main() error? Personally, I start most of my programs writing

func main() {
  if err := run(); err != nil {
    log.Fatal(err)
  }
}

func run() error {
  // …
}

This allows me to use defer, pass on errors and all that good stuff. So, why doesn't the language just do that for me?

We can find part of the answer in this old golang-nuts thread. It is about return codes, instead of an error, but the principle is the same. And the best answer - in my opinion - is this:

I think the returned status is OS-specific, and so Go the language should not define its type (Maybe some OS can only report 8-bit result while some other OS support arbitrary string as program status, there is considerable differences between that; there might even be environment that don't support returning status code or the concept of status code simply doesn't exist)

I imagine some Plan 9 users might be disagree with the signature of os.Exit().

So, in essence: Not all implementations would necessarily be able to assign a reasonable meaning to a return code (or error) from main. For example, an embedded device likely couldn't really do anything with it. It thus seems preferable to not couple the language to this decision which only really makes semantic sense on a limited subset of implementations. Instead, we provide mechanisms in the standard library to exit the program or take any other reasonable action and then let the developer decide, under what circumstances they want to exit the program and with what code. Being coupled to a decision in the standard library is better than being coupled in the language itself. And a developer who targets a platform where an exit code doesn't make sense, can take a different action instead.

Of course, this leaves the programmer with a problem: What to do with errors? We could write it to stderr, but fmt.Fprintf also returns an error, so what to do with that one? Above I used log.Fatal, which does not return an error. What happens if the underlying io.Writer fails to write, though? What does log do with the resulting error? The answer is, of course: It ignores any errors.

The point is, that passing on the error is not a solution. Eventually every program will return to main (or os.Exit or panic) and the buck stops there. It needs to get handled and the signature of main enforces that the only way to do that is via side-effects - and if they fail, you just have to deal with that one too.


Let's continue with a similar question, that has a similar answer, that occasionally comes up: Why doesn't ServeHTTP return an error? Sooner or later, people face the question of what to do with errors in their HTTP Handlers. For example, what if you are writing out a JSON object and Marshal fails? In fact, a lot of HTTP frameworks out there will define their own handler-type, which differs from http.Handler in exactly that way. But if everyone wants to return an error from their handler, why doesn't the interface just add that error return itself? Was that just an oversight?

I'm strongly arguing that no, this was not an oversight, but the correct design decision. Because the HTTP Server package can not handle any errors. An HTTP server is supposed to stay running, every request demands a response. If ServeHTTP would return an error, the server would have to do something with it, but what to do is highly application-specific. You might respond that it should serve a 500 error code, but in 99% of cases, that is the wrong thing to do. Instead you should serve a more specific error code, so the client knows (for example) whether to retry or if the response is cacheable. http.Server could also just ignore the error and instead drop the request on the floor, but that's even worse. Or it could propagate it up the stack. But as we determined, eventually it would have to reach main and the buck stops there. You probably don't want your server to come down, every time a request contains an invalid parameter.

So, given that a) every request needs an answer and b) the right answer is highly application-specific, the translation from errors into status codes has to happen in application code. And just like main enforces you to handle any errors via side-effects by not allowing you to return an error, so does http force you to handle any errors via writing a response by not allowing you to return an error.¹

So, what are you supposed to do, when json.Marshal fails? Well, that depends on our application. Increment a metric. Log the error. panic. Write out a 500. Ignore it and write a 200. Commit to the uncomfortable knowledge, that sometimes, you can't just pass the decision on what to do with an error to someone else.


These two examples distill, I think, pretty well, what I view as error handling: An error is handled, when you destroy the error value. In that parlance, log.Error handles any errors of the underlying writer by not returning them. Every program needs to handle any error in some way, because main can't return anything and the values need to go somewhere. Any HTTP handler needs to actually handle errors, by translating them into HTTP responses.

And in that parlance, packages like pkg/errors have little, really, to do with error handling - instead, they provides you with a strategy for the case where you are not handling your errors. In the same vein, proposals that address the repetitive checking of errors via extra syntax do not really simplify their handling at all - they just move it around a bit. I would term that error propagation, instead - no doubt important, but keep in mind, that an error that was handled, doesn't need to be propagated at all. So to me, a good approach to error handling would be characterized by mostly obviating the need for convenient error propagation mechanisms.

And to me, at least, it seems that we talk too little about how to handle errors, in the end.


Does Go encourage explicit error handling? This is the phrasing very often used to justify the repetitive nature, but I tend to disagree. Compare, for example, Go's approach to checked exceptions in Java: There, errors are propagated via exceptions. Every exception that could be thrown (theoretically) must be annotated in the method signature. Any exception that you handle, has to be mentioned in a try-catch-statement. And the compiler will refuse to compile a program which does not explicitly mention how exceptions are handled. This, to me, seems like the pinnacle of explicit error handling. Rust, too, requires this - it introduces a ? operator to signify propagating an error, but that, still, is an explicit annotation. And apart from that, you can't use the return value of a function that might propagate an error, without explicitly handling that error first.

In Go, on the other hand, it is not only perfectly acceptable to ignore errors when it makes sense (for example, I will always ignore errors created from writing to a *bytes.Buffer), it is actually often the only sensible thing to do. It is fundamentally not only okay, but 99% of times correct to just completely ignore the error returned by fmt.Println. And while it makes sense to check the error returned from json.Marshal in your HTTP handler against *json.MarshalError (to panic/log/complain loudly, because your code is buggy), any other errors should 99% of the time just be ignored. And that's fine.

I believe that to say Go encourages explicit error handling, it would need some mechanism of checked exceptions, Result types, or a requirement to pass an errcheck like analysis in the compiler.

I think it would be closer to say, that Go encourages local error handling. That is, the code that handles an error, is close to the code that produced it. Exceptions encourages the two to be separated: There are usually several or many lines of code in a single try-block, all of which share one catch-block and it is hard to tell which of the lines produced it. And very often, the actual error location is several stack frames deep. You could contrast this with Go, where the error return is immediately obvious from the code and if you have a line of error handling, it is usually immediately attached to the function call that produced it.

However, that still seems to come short, in my view. After all, there is nothing to force you to do that. And in fact, one of the most often cited articles about Go error handling is often interpreted to encourage exactly that. Plus, a lot of people end up writing return err far too often, simply propagating the error to be handled elsewhere. And the proliferation of error-wrapping libraries happens in the same vein: What their proponents phrase as "adding context to the error value", I interpret as "adding back some of the information as a crutch, that you removed when passing the error to non-local handling code". Sadly, far too often, the error then ends up not being handled at all, as everyone just takes advantage of that crutch. This leaves the end-user with an error message that is essentially a poorly formatted, non-contiguous stacktrace.

Personally, I'd characterize Go's approach like this: In Go, error handling is simply first-class code. By forcing you to use exactly the same control-flow mechanisms and treat errors like any other data, Go encourages you to code your error handling. Often that means a bunch of control flow to catch and recover from any errors where they occur. But that's not "clutter", just as it is not "clutter" to write if n < 1 { return 1 } when writing a Fibonacci function (to choose a trivial example). It is just code. And yes, sometimes that code might also store the error away or propagate it out-of-band to reduce repetition where it makes sense - like in above blog post. But focussing on the "happy path" is a bit of a distraction: Your users will definitely be more happy about those parts of the control flow that make the errors disappear or transform them into clean, actionable advise on how to solve the problem.

So, in my reading, the title of the Go blog post puts the emphasis in slightly the wrong place - and often, people take the wrong message from it, in my opinion. Not "errors are values", but "error handling is code".


So, what would be my advise for handling errors? To be honest, I don't know yet - and I'm probably in no place to lecture anyone anyway.

Personally, I've been trying for the last couple of months to take a page out of http.Handlers playbook and try, as much as possible, to completely avoid returning an error. Instead of thinking "I should return an error here, in case I ever do any operation that fails", I instead think "is there any way at all I can get away with not returning an error here?". It doesn't always work and sometimes you do have to pass errors around or wrap them. But I am forcing myself to think very hard about handling my errors and it encourages a programming-style of isolating failing components. The constraint of not being able to return an error tends to make you creative in how to handle it.


[1] You might be tempted to suggest, that you could define an HTTPError, containing the necessary info. Indeed, that's what the official Go blog does, so it can't be bad? And indeed, that is what they do, but note that they do not actually return an error in the end - they return an appError, which contains the necessary information. Exactly because they don't know how to deal with general errors. So they translate any errors into a domain specific type that carries the response. So, that is not the same as returning an error.

I think this particular pattern is fine, though, personally, I don't really see the point. Anything that builds an appError needs to provide the complete response anyway, so you might as well just write it out directly. YMMV.

21. January 2018 23:40:00

15. January 2018

Mero’s Blog

Generating entropy without imports in Go

tl;dr: I come up with a couple of useless, but entertaining ways to generate entropy without relying on any packages.

This post is inspired by a comment on reddit, saying

[…]given the constraints of no imports and the function signature:

func F(map[string]string) map[string]string { ... }

F must use a deterministic algorithm, since it is a deterministic algorithm it can be represented in a finite state machine.

Now, the point of this comment was to talk about how to then compile such a function into a deterministic finite state machine, but it got me thinking about a somewhat different question. If we disallow any imports and assume a standard (gc) Go implementation - how many ways can we find to create a non-deterministic function?

So, the challenge I set to myself was: Write a function func() string that a) can not refer to any qualified identifier (i.e. no imports) and b) is non-deterministic, that is, produces different outputs on each run. To start me off, I did add a couple of helpers, to accumulate entropy, generate random numbers from it and to format strings as hex, without any imports:

type rand uint32

func (r *rand) mix(v uint32) {
    *r = ((*r << 5) + *r) + rand(v)
}

func (r *rand) rand() uint32 {
    mx := rand(int32(*r)>>31) & 0xa8888eef
    *r = *r<<1 ^ mx
    return uint32(*r)
}

func hex(v uint32) string {
    var b []byte
    for v != 0 {
        if x := byte(v & 0xf); x < 10 {
            b = append(b, '0'+x)
        } else {
            b = append(b, 'a'+x-10)
        }
        v >>= 4
    }
    return string(b)
}

Obviously, these could be inlined, but separating them allows us to reuse them for our different functions. Then I set about the actual task at hand.

Method 1: Map iteration

In Go, the iteration order of maps is not specified:

The iteration order over maps is not specified and is not guaranteed to be the same from one iteration to the next.

But gc, the canonical Go implementation, actively randomizes the map iteration order to prevent programs from depending on it. We can use this, to receive some of entropy from the runtime, by creating a map and iterating over it:

func MapIteration() string {
  var r rand

  m := make(map[uint32]bool)
  for i := uint32(0); i < 100; i++ {
    m[i] = true
  }
  for i := 0; i < 1000; i++ {
    for k := range m {
      r.mix(k)
      break // the rest of the loop is deterministic
    }
  }
  return hex(r.rand())
}

We first create a map with a bunch of keys. We then iterate over it a bunch of times; each map iteration gives us a different start index, which we mix into our entropy pool.

Method 2: Select

Go actually defines a way in which the runtime is giving us access to entropy directly:

If one or more of the communications can proceed, a single one that can proceed is chosen via a uniform pseudo-random selection.

So the spec guarantees that if we have multiple possible communications in a select, the case has to be chosen non-deterministically. We can, again, extract that non-determinism:

func Select() string {
    var r rand

    ch := make(chan bool)
    close(ch)
    for i := 0; i < 1000; i++ {
        select {
        case <-ch:
            r.mix(1)
        case <-ch:
            r.mix(2)
        }
    }
    return hex(r.rand())
}

We create a channel and immediately close it. We then create a select-statement with two cases and depending on which was taken, we mix a different value into our entropy pool. The channel is closed, to guarantee that communication can always proceed. This way, we extract one bit of entropy per iteration.

Note, that there is no racing or concurrency involved here: This is simple, single-threaded Go code. The randomness comes directly from the runtime. Thus, this should work in any compliant Go implementation. The playground, however, is not compliant with the spec in this regard, strictly speaking. It is deliberately deterministic.

Method 3: Race condition

This method exploits the fact, that on a multi-core machine at least, the Go scheduler is non-deterministic. So, if we let two goroutines race to write a value to a channel, we can extract some entropy from which one wins this race:

func RaceCondition() string {
    var r rand

    for i := 0; i < 1000; i++ {
        ch := make(chan uint32, 2)
        start := make(chan bool)
        go func() {
            <-start
            ch <- 1
        }()
        go func() {
            <-start
            ch <- 2
        }()
        close(start)
        r.mix(<-ch)
    }

    return hex(r.rand())
}

The start channel is there to make sure that both goroutines become runnable concurrently. Otherwise, the first goroutine would be relatively likely to write the value before the second is even spawned.

Method 4: Allocation/data races

Another thought I had, was to try to extract some entropy from the allocator or GC. The basic idea is, that the address of an allocated value might be non-deterministic - in particular, if we allocate a lot. We can then try use that as entropy.

However, I could not make this work very well, for the simple reason that Go does not allow you to actually do anything with pointers - except dereferencing and comparing them for equality. So while you might get non-deterministic values, those values can't be used to actually generate random numbers.

I thought I might be able to somehow get a string or integer representation of some pointer without any imports. One way I considered was inducing a runtime-panic and recovering that, in the hope that the error string would contain a stacktrace or offending values. However, none of the error strings created by the runtime actually seem to contain any values that could be used here.

I also tried a workaround to interpret the pointer as an integer, by exploiting race conditions to do unsafe operations:

func DataRace() string {
    var r rand

    var data *uintptr
    var addr *uintptr

    var i, j, k interface{}
    i = (*uintptr)(nil)
    j = &data

    done := false
    go func() {
        for !done {
            k = i
            k = j
        }
    }()
    for {
        if p, ok := k.(*uintptr); ok && p != nil {
            addr = p
            done = true
            break
        }
    }

    data = new(uintptr)
    r.mix(uint32(*addr))
    return hex(r.rand())
}

It turns out, however, that at least this particular instance of a data race has been fixed since Russ Cox wrote that blog post. In Go 1.9, this code just loops endlessly. I tried it in Go 1.5, though, and it works there - but we don't get a whole lot of entropy (addresses are not that random). With other methods, we could re-run the code to collect more entropy, but in this case, I believe the escape analysis gets into our way by stack-allocating the pointer, so it will be the same one on each run.

I like this method, because it uses several obscure steps to work, but on the other hand, it's the least reliable and it requires an old Go version.

Your Methods?

These are all the methods I could think of; but I'm sure I missed a couple. If you can think of any, feel free to let me know on Twitter, reddit or hackernews :) I also posted the code in a gist, so you can download and run it yourself, but keep in mind, that the last method busy-loops in newer Go versions.

15. January 2018 01:04:30

13. January 2018

sECuREs Webseite

Off-site backups with an apu2c4

Background

A short summary of my backup strategy is: I run daily backups to my NAS. In order to recover from risks like my apartment burning down or my belongings being stolen, I like to keep one copy of my data off-site, updated less frequently.

I used to store off-site backups with the “unlimited storage” offerings of various cloud providers.

These offers follow a similar pattern: they are announced, people sign up and use a large amount of storage, the provider realizes they cannot make enough money off of this pricing model, and finally the offer is cancelled.

I went through this two times, and my friend Mark’s similar experience and home-grown solution inspired me to also build my own off-site backup.

Introduction

I figured the office would make a good place for an external hard disk: I’m there every workday, it’s not too far away, and there is good internet connectivity for updating the off-site backup.

Initially, I thought just leaving the external hard disk there and updating it over night by bringing my laptop to the office every couple of weeks would be sufficient.

Now I know that strategy doesn’t work for me: the time would never be good (“maybe I’ll unexpectedly need my laptop tonight!”), I would forget, or I would not be in the mood.

Lesson learnt: backups must not require continuous human involvement.

The rest of this article covers the hardware I decided to use and the software setup.

Hardware

The external hard disk enclosure is a T3US41 Sharkoon Swift Case PRO USB 3.0 for 25 €.

The enclosed disk is a HGST 8TB drive for which I paid 290 € in mid 2017.

For providing internet at our yearly retro computing event, I still had a PC Engines apu2c4 lying around, which I repurposed for my off-site backups. For this year’s retro computing event, I’ll either borrow it (setting it up is quick) or buy another one.

The apu2c4 has two USB 3.0 ports, so I can connect my external hard disk to it without USB being a bottle-neck.

Setup: installation

On the apu2c4, I installed Debian “stretch” 9, the latest Debian stable version at the time of writing. I prepared a USB thumb drive with the netinst image:

% wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.2.1-amd64-netinst.iso
% cp debian-9.2.1-amd64-netinst.iso /dev/sdb

Then, I…

  • plugged the USB thumb drive into the apu2c4
  • On the serial console, pressed F10 (boot menu), then 1 (boot from USB)
  • In the Debian installer, selected Help, pressed F6 (special boot parameters), entered install console=ttyS0,115200n8
  • installed Debian as usual.
  • Manually ran update-grub, so that GRUB refers to the boot disk by UUID instead of root=/dev/sda1. Especially once the external hard disk is connected, device nodes are unstable.
  • On the serial console, pressed F10 (boot menu), then 4 (setup), then c to move the mSATA SSD to number 1 in boot order
  • Connected the external hard disk

Setup: persistent reverse SSH tunnel

I’m connecting the apu2c4 to a guest network port in our office, to keep it completely separate from our corporate infrastructure. Since we don’t have permanently assigned publically reachable IP addresses on that guest network, I needed to set up a reverse tunnel.

First, I created an SSH private/public keypair using ssh-keygen(1).

Then, I created a user account for the apu2c4 on my NAS (using cloud-config), where the tunnel will be terminated. This account’s SSH usage is restricted to port forwardings only:

users:
  - name: apu2c4
    system: true
    ssh-authorized-keys:
      - "restrict,command=\"/bin/false\",port-forwarding ssh-rsa AAAA…== root@stapelberg-apu2c4"

On the apu2c4, I installed the autossh Debian package (see the autossh(1) manpage for details) and created the systemd unit file /etc/systemd/system/autossh-nas.service with the following content:

[Unit]
Description=autossh reverse tunnel
After=network.target
Wants=network-online.target

[Service]
Restart=always
StartLimitIntervalSec=0
Environment=AUTOSSH_GATETIME=0
ExecStart=/usr/bin/autossh -M 0 -N -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -o "ExitOnForwardFailure yes" apu2c4@nas.example.net -R 2200:localhost:22

[Install]
WantedBy=multi-user.target

After enabling and starting the unit using systemctl enable --now autossh-nas, the apu2c4 connected to the NAS and set up a reverse port-forwarding.

On the NAS, I configure SSH like so in my /root/.ssh/config:

Host apu2c4
  Hostname localhost
  Port 2200
  User root
  IdentitiesOnly yes

Finally, I authorized the public key of my NAS to connect to the apu2c4.

Note that this concludes the setup of the apu2c4: the device’s only purpose is to make the external hard disk drive available remotely to my NAS, clean and simple.

Setup: full-disk encryption

I decided to not store the encryption key for the external hard disk on the apu2c4, to have piece of mind in case the hard disk gets misplaced or even stolen. Of course I trust my co-workers, but this is a matter of principle.

Hence, I amended my NAS’s cloud-config setup like so (of course with a stronger key):

write_files:
  - path: /root/apu2c4.lukskey
    permissions: 0600
    owner: root:root
    content: |
    ABCDEFGHIJKL0123456789

…and configured the second key slot of the external hard disk to use this key.

Setup: Backup script

I’m using a script roughly like the following to do the actual backups:

#!/bin/bash
# vi:ts=4:sw=4:et
set -e

/bin/ssh apu2c4 cryptsetup luksOpen --key-file - /dev/disk/by-id/ata-HGST_HDN1234 offline_crypt < /root/apu2c4.lukskey

/bin/ssh apu2c4 mount /dev/mapper/offline_crypt /mnt/offsite

# step 1: update everything but /backups
echo "$(date +'%c') syncing NAS data"

(cd /srv && /usr/bin/rsync --filter 'exclude /backup' -e ssh -ax --relative --numeric-ids ./ apu2c4:/mnt/offsite)

# step 2: copy the latest backup
hosts=$(ls /srv/backup/)
for host in $hosts
do
  latestremote=$(ls /srv/backup/${host}/ | tail -1)
  latestlocal=$(/bin/ssh apu2c4 ls /mnt/offsite/backup/${host} | tail -1)
  if [ "$latestlocal" != "$latestremote" ]
  then
    echo "$(date +'%c') syncing $host (offline: ${latestlocal}, NAS: ${latestremote})"
    /bin/ssh apu2c4 mkdir -p /mnt/offsite/backup/${host}
    (cd /srv && /usr/bin/rsync -e ssh -ax --numeric-ids ./backup/${host}/${latestremote}/ apu2c4:/mnt/offsite/backup/${host}/${latestremote} --link-dest=../${latestlocal})

    # step 3: delete all previous backups
    echo "$(date +'%c') deleting everything but ${latestremote} for host ${host}"
    ssh apu2c4 "find /mnt/offsite/backup/${host} \! \( -path \"/mnt/offsite/backup/${host}/${latestremote}/*\" -or -path \"/mnt/offsite/backup/${host}/${latestremote}\" -or -path \"/mnt/offsite/backup/${host}\" \) -delete"
  fi
done

/bin/ssh apu2c4 umount /mnt/offsite
/bin/ssh apu2c4 cryptsetup luksClose offline_crypt
/bin/ssh apu2c4 hdparm -Y /dev/disk/by-id/ata-HGST_HDN1234

Note that this script is not idempotent, lacking in error handling and won’t be updated. It merely serves as an illustration of how things could work, but specifics depend on your backup.

To run this script weekly, I created the following cloud-config on my NAS:

coreos:
  units:
    - name: sync-offsite.timer
      command: start
      content: |
        [Unit]
        Description=sync backups to off-site storage

        [Timer]
        OnCalendar=Sat 03:00

    - name: sync-offsite.service
      content: |
        [Unit]
        Description=sync backups to off-site storage
        After=docker.service srv.mount
        Requires=docker.service srv.mount

        [Service]
        Type=oneshot

        ExecStart=/root/sync-offsite-backup.sh

Improvement: bandwidth throttling

In case your office (or off-site place) doesn’t have a lot of bandwidth available, consider throttling your backups. Thus far, I haven’t had the need.

Improvement: RTC-based wake-up

I couldn’t figure out whether the apu2c4 supports waking up based on a real-time clock (RTC), and if yes, whether that works across power outages.

If so, one could keep it shut down (or suspended) during the week, and only power it up for the actual backup update. The downside of course is that any access (such as for restoring remotely) require physical presence.

If you know the answer, please send me an email.

Conclusion

The presented solution is easier to integrate than most cloud storage solutions.

Of course my setup is less failure-tolerant than decent cloud storage providers, but given the low probability of a catastrophic event (e.g. apartment burning down), it’s fine to just order a new hard disk or apu2c4 when either of the two fails — for this specific class of backups, that’s an okay trade-off to make.

The upside of my setup is that the running costs are very low: the apu2c4’s few watts of electricity usage are lost in the noise, and syncing a few hundred MB every week is cheap enough these days.

by Michael Stapelberg at 13. January 2018 16:30:00

08. January 2018

Mero’s Blog

Monads are just monoids in the category of endofunctors

tl;dr: I explain the mathematical background of a joke-explanation of monads. Contains lots of math and a hasty introduction to category theory.

There is a running gag in the programming community, that newcomers will often be confused by the concept of monads (which is how sequential computations are modeled in purely functional languages) and getting the explanation "it is simple, really: Monads are just monoids in the category of endofunctors". This is not meant as an actual explanation, but rather to poke a bit of fun at the habit of functional programmers to give quite abstract and theoretical explanations at times, that are not all that helpful.

However, given my background in mathematics, I decided that I wanted to actually approach Haskell from this point of view: I am interested in how it uses math to model programming and also to, after several years of doing mostly engineering focused programming work, flex my math muscles again - as there is quite a bit of interesting math behind these concepts.

The quote is from a pretty popular book about category theory and is, in full:

All told, a monad in \(X\) is just a monoid in the category of endofunctors of \(X\), with product \(\times\) replaced by composition of endofunctors and unit set by the identity endofunctor.

This, of course, is an explanation of the mathematical concept of monads, not meant for programmers. Most explanations of the quote that I found either assumed quite a bit of knowledge in Haskell or took a lot of liberties with the mathematical concepts (and relied a lot on "squinting") or both. This write up is my attempt, to walk through all the concepts needed to explain monads as a mathematical concept and how it relates to Haskell - with as little squinting as possible.

Of course, there are a couple of disclaimers, I should start with:

  1. This is not the best way to understand what monads are, if you are actually interested in using them to program. In fact, it is literally the worst way. I would recommend this intro, which takes a much more practical approach.
  2. This is not the best way to understand how category theory works, if you are actually interested in learning mathematics. In fact, it is literally the worst way. I would recommend the book the quote is from, it's quite good (but assumes a math audience).
  3. I haven't done mathematics in years. I also don't know much Haskell either. So I might be getting a bunch of stuff wrong to varying degrees. I'm sure I will hear all about it :)
  4. Even if I would understand everything correctly, there are still a lot of details, mostly of technical nature, I had to omit, to keep this "short". Not that it is short.

Originally, I intended this to be the ultimate explanation, which would teach Haskellers category theory, mathematicians Haskell and people who know neither both. Unsurprisingly, this is not what this is, at all. It ended up mostly a write up to assure myself that I understood the path myself. If anything, you can treat this as a kind of "reading companion": If you want to understand this topic of the intersection between category theory and functional programming, this post can lead you through the correct terms to search for and give you a good idea what to focus on, in the respective Wikipedia articles.

With all that out of the way, let's begin.

Categories

In mathematics, a category is (roughly) a collection of objects and a collection of arrows between them. There is not a lot of meaning behind these, but it will probably help you to think of objects as sets and arrows as mappings. Every arrow goes from an object (the domain) to an object (the codomain) and we write an arrow as \(f:X\to Y\), where \(f\) is the name of the arrow, \(X\) is the domain and \(Y\) is the codomain. Just like with mappings, there can be many arrows between any given pair of objects - or there may be none.

We do need some restrictions: First, we require a specific identity arrow \(\mathrm{id}:X\to X\) attached to every object \(X\), which has \(X\) as both domain and codomain. Secondly, we require (some) arrows to be composable. That is if we have two arrows \(f:X\to Y,g:Y\to Z\) - so, whenever the domain of \(g\) is the codomain of \(f\) - there should also be a composed arrow¹ \(g\circ f: X\to Z\), that shares the domain with \(f\) and the codomain with \(g\).

Furthermore, the identity arrows must act as a unit for composition, that is, for every arrow \(f\) we require \(\mathrm{id}\circ f = f = f \circ\mathrm{id}\). We also require composition to be associative, that is \((f\circ g)\circ h = f\circ(g\circ h)\) (whenever all compositions exist)².

When we talk about a category, we often draw diagrams like this:

\[ \require{AMScd} \begin{CD} X @>{f}>> Y \\ @V{g}VV @VV{p}V \\ Z @>>{q}> W \\ \end{CD} \]

They show some of the objects and arrows from the category in a compact way. This particular diagram indicates that there are four objects and four arrows involved, with obvious domains and codomains. We only draw a subset of the objects and arrows, that is interesting for the point we are trying to make - for example, above diagram could also contain, of course, identity arrows and compositions \(p\circ f\) and \(q\circ g\)), but we didn't draw them. In a square like this, we can take two paths from \(X\) to \(W\). If these paths are identical (that is, \(p\circ f = q\circ g\), we say that the square commutes. A commutative diagram is a diagram, in which any square commutes, that is, it does not matter which path we take from any object to another. Most of the time, when we draw a diagram, we intend it to be commutative.

So, to summarize, to define a mathematical category, we need to:

  1. Specify what our objects are
  2. Specify what our arrows are, where each arrow starts and ends at a certain object
  3. This collection of arrows need to include an arrow \(\mathrm{id}_X\) for every object \(X\), which starts and ends at \(X\)
  4. And we need to be able to glue together arrows \(f:X\to Y\) and \(g:Y\to Z\) to an arrow \(g\circ f: X\to Z\)

In Haskell, we work on the category Hask, which consists of:

  1. The objects are types: Int is an object, String is an object but also Int | String, String -> Int and any other complicated type you can think of.
  2. The arrows are functions: f :: a -> b is a function taking an a as an input and returning a b and is represented by an arrow f, which has a as its domain and b as its codomain. So, for example, length :: String -> Int would start at the type String and end at Int.
  3. Haskell has a function id :: a -> a which gives us the identity arrow for any type a.
  4. We can compose functions with the operator (.) :: (b -> c) -> (a -> b) -> (a -> c). Note, that this follows the swapped notation of \(\circ\), where the input type of the left function is the output type of the right function.

In general, category theory is concerned with the relationship between categories, whereas in functional programming, we usually only deal with this one category. This turns out to be both a blessing and a curse: It means that our object of study is much simpler, but it also means, that it is sometimes hard to see how to apply the general concepts to the limited environment of functional programming.

Monoids

Understanding categories puts us in the position to understand monoids. A monoid is the generalized structure underlying concepts like the natural numbers: We can add two natural numbers, but we can't (in general) subtract them, as there are no negative numbers. We also have the number \(0\), which, when added to any number, does nothing - it acts as a unit for addition. And we also observe, that addition is associative, that is, when doing a bunch of additions, the order we do them in doesn't matter.

The same properties also apply to other constructs. For example, if we take all maps from a given set to itself, they can be composed and that composition is associative and there is a unit element (the identity map).

This provides us with the following elements to define a monoid:

  1. A set \(M\)
  2. An operation \(\star\colon M\times M\to M\), which "adds" together two elements to make a new one
  3. We need a special unit element \(u\in M\), which acts neutrally when added to any other element, that is \(m\star u=m=u\star m\)
  4. The operation needs to be associative, that is we always require \(m\star(n\star k)=(m\star n)\star k\)

There is another way to frame this, which is closer in line with category theory. If we take \(1 := \{0\}\) to be a 1-element set, we can see that the elements of \(M\) are in a one-to-one correspondence to functions \(1\to M\): Every such function chooses an element of \(M\) (the image of \(0\)) and every element \(m\in M\) fixes such a function, by using \(f(0) := m\). Thus, instead of saying "we need a special element of \(M\)", we can also choose a special function \(\eta: 1\to M\). And instead of talking about an "operation", we can talk about a function \(\mu: M\times M\to M\). Which means, we can define a monoid via a commutative diagram like so:

\[ \begin{CD} 1 \\ @V{\eta}VV \\ M \\ \end{CD} \hspace{1em} \begin{CD} M\times M \\ @V{\mu}VV \\ M \\ \end{CD} \hspace{1em} \begin{CD} M\times 1 @>{\mathrm{id}\times\eta}>> M\times M @<{\eta\times\mathrm{id}}<< 1\times M \\ @V{\pi_1}VV @V{\mu}VV @V{\pi_2}VV \\ M @>{\mathrm{id}}>> M @<{\mathrm{id}}<< M \\ \end{CD} \hspace{1em} \begin{CD} M\times M\times M @>{\mu\times\mathrm{id}}>> M\times M \\ @V{\mathrm{id}\times\mu}VV @V{\mu}VV \\ M\times M @>{\mu}>> M \\ \end{CD} \]

\(\pi_1\) and \(\pi_2\) here, are the functions that project to the first or second component of a cross product respectively (that is \(\pi_1(a, b) := a, \pi_2(a, b) := b\)) and e.g. \(\mathrm{id}\times\eta\) is the map that applies \(\mathrm{id}\) to the first component of a cross-product and \(\eta\) to the second: \(\mathrm{id}\times\eta(m, 0) = (m, \eta(0))\).

There are four sub-diagrams here:

  1. The first diagram just says, that we need an arrow \(\eta:1\to M\). This chooses a unit element for us.
  2. Likewise, the second diagram just says, that we need an arrow \(\mu:M\times M\to M\). This is the operation.
  3. The third diagram tells us that the chosen by \(\eta\) should be a unit for \(\mu\). The commutativity of the left square tells us, that it should be right-neutral, that is \[ \forall m\in M: m = \pi_1(m, 0) = \mu(\mathrm{id}\times\eta(m, 0)) = \mu(m, \eta(0)) \] and the commutativity of the right square tells us, that it should be left-neutral, that is \[ \forall m\in M: m = \pi_2(0,m) = \mu(\eta\times\mathrm{id}(0, m)) = \mu(\eta(0), m) \]

Thus, the first diagram is saying that the element chosen by \(\eta\) should act like a unit. For example, the left square says

\[\pi_1(m,0) = \mu((\mathrm{id}\times\eta)(m,0)) = \mu(m,\eta(0))\]

Now, writing \(\mu(m,n) = m\star n\) and \(\eta(0) = u\), this is equivalent to saying \(m = u\star m\).

The second diagram is saying that \(\mu\) should be associative: The top arrow combines the first two elements, the left arrow combines the second two. The right and bottom arrows then combine the result with the remaining element respectively, so commutativity of that square means the familiar \(m\star (n\star k) = (m\star n)\star k\).

Haskell has the concept of a monoid too. While it's not really relevant to the discussion, it might be enlightening to see, how it's modeled. A monoid in Haskell is a type-class with two (required) methods:

class Monoid a where
  mempty :: a
  mappend :: a -> a -> a

Now, this gives us the operation (mappend) and the unit (a), but where are the requirements of associativity and the unit acting neutrally? The Haskell type system is unable to codify these requirements, so they are instead given as a "law", that is, any implementation of a monoid is supposed to have these properties, to be manually checked by the programmer:

  • mappend mempty x = x (the unit is left-neutral)
  • mappend x mempty = x (the unit is right-neutral)
  • mappend x (mappend y z) = mappend (mappend x y) z (the operation is associative)
Functors

I mentioned that category theory investigates the relationship between categories - but so far, everything we've seen only happens inside a single category. Functors are, how we relate categories to each other. Given two categories \(\mathcal{B}\) and \(\mathcal{C}\), a functor \(F:\mathcal{B}\to \mathcal{C}\) assigns to every object \(X\) of \(\mathcal{B}\), an object \(F(X)\) of \(\mathcal{C}\). It also assigns to every arrow \(f:X\to Y\) in \(\mathcal{B}\) a corresponding arrow \(F(f): F(X)\to F(Y)\) in \(\mathcal{C}\)³. So, a functor transfers arrows from one category to another, preserving domain and codomain. To actually preserve the structure, we also need it to preserve the extra requirements of a category, identities and composition. So we need, in total:

  1. An object map, \(F:O_\mathcal{B} \to O_\mathcal{C}\)
  2. An arrow map, \(F:A_\mathcal{B}\to A_\mathcal{C}\), which preserves start and end object, that is the image of an arrow \(X\to Y\) starts at \(F(X)\) and ends at \(F(Y)\)
  3. The arrow map has to preserve identities, that is \(F(\mathrm{id}_X) = \mathrm{id}_{F(X)}\)
  4. The arrow map has to preserve composition, that is \(F(g\circ f) = F(g)\circ F(f)\).

A trivial example of a functor is the identity functor (which we will call \(I\)), which assigns each object to itself and each arrow to itself - that is, it doesn't change the category at all.

A simple example is the construction of the free monoid, which maps from the category of sets to the category of monoids. The Free monoid \(S^*\) on a set \(S\) is the set of all finite length strings of elements of \(S\), with concatenation as the operation and the empty string as the unit. Our object map then assigns to each set \(S\) its free monoid \(S^*\). And our arrow map assigns to each function \(f:S\to T\) the function \(f^*:S^*\to T^*\), that applies \(f\) to each element of the input string.

There is an interesting side note here: Mathematicians love to abstract. Categories arose from the observation, that in many branches of mathematics we are researching some class of objects with some associated structure and those maps between them, that preserve this structure. It turns out that category theory is a branch of mathematics that is researching the objects of categories, with some associated structure (identity arrows and composition) and maps (functors) between them, that preserve that structure. So it seems obvious that we should be able to view categories as objects of a category, with functors as arrows. Functors can be composed (in the obvious way) and every category has an identity functor, that just maps every object and arrow to itself.

Now, in Haskell, Functors are again a type class:

class Functor f where
  fmap :: (a -> b) -> (f a -> f b)

This looks like our arrow map: It assigns to each function g :: a -> b a function fmap g :: f a -> f b. The object map is implicit: When we write f a, we are referring to a new type, that depends on a - so we "map" a to f a .

Again, there are additional requirements the type system of Haskell can not capture. So we provide them as laws the programmer has to check manually:

  • fmap id == id (preserves identities)
  • fmap (f . g) == fmap f . fmap g (preserves composition)

There is one thing to note here: As mentioned, in Haskell we only really deal with one category, the category of types. That means that a functor always maps from the category of types to itself. In mathematics, we call such a functor, that maps a category to itself, an endofunctor. So we can tell, that in Haskell, every functor is automatically an endofunctor.

Natural transformations

We now understand categories and we understand functors. We also understand, that we can look at something like the category of categories. But the definition of a monad given to us talks about the category of endofunctors. So we seem to have to step up yet another level in the abstraction hierarchy and somehow build this category. As objects, we'd like to have endofunctors - and arrows will be natural transformations, which take one functor to another, while preserving its internal structure (the mapping of arrows). If that sounds complicated and abstract, that's because it is.

We need two functors \(F,G:\mathcal{B}\to \mathcal{C}\) of the same "kind" (that is, mapping to and from the same categories). A natural transformation \(\eta:F\dot\to G\) assigns an arrow \(\eta_X: F(X)\to G(X)\) (called a component of \(\eta\)) to every object in \(\mathcal{B}\). So a component \(\eta_X\) describes, how we can translate the action of \(F\) on \(X\) into the action of \(G\) on \(X\) - i.e. how to translate their object maps. We also have to talk about the translation of the arrow maps. For that, we observe that for any arrow \(f:X\to Y\) in \(\mathcal{B}\), we get four new arrows in \(\mathcal{C}\):

\[ \begin{CD} X \\ @V{f}VV \\ Y \\ \end{CD} \hspace{1em} \begin{CD} F(X) @>{\eta_X}>> G(X) \\ @V{F(f)}VV @VV{G(f)}V \\ F(Y) @>>{\eta_Y}> G(Y) \\ \end{CD} \]

For a natural transformation, we require the resulting square to commute.

So, to recap: To create a natural transformation, we need

  1. Two functors \(F,G:\mathcal{B}\to\mathcal{C}\)
  2. For every object \(X\) in \(\mathcal{B}\), an arrow \(\eta_X: F(X)\to G(X)\)
  3. The components need to be compatible with the arrow maps of the functors: \(\eta_Y\circ F(f) = G(f)\circ \eta_X\).

In Haskell, we can define a natural transformation like so:

class (Functor f, Functor g) => Transformation f g where
    eta :: f a -> g a

f and g are functors and a natural transformation from f to g provides a map f a -> g a for every type a. Again, the requirement of compatibility with the actions of the functors is not expressible as a type signature, but we can require it as a law:

  • eta (fmap fn a) = fmap fn (eta a)
Monads

This, finally, puts us in the position to define monads. Let's look at our quote above:

All told, a monad in \(X\) is just a monoid in the category of endofunctors of \(X\), with product \(\times\) replaced by composition of endofunctors and unit set by the identity endofunctor.

It should be clear, how we can compose endofunctors. But it is important, that this is a different view of these things than if we'd look at the category of categories - there, objects are categories and functors are arrows, while here, objects are functors and arrows are natural transformations. That shows, how composition of functors can take the role of the cross-product of sets: In a set-category, the cross product makes a new set out of two other set. In the category of endofunctors, composition makes a new endofunctor out of two other endofunctors.

When we defined monoids diagrammatically, we also needed a cross product of mappings, that is, given a map \(f:X_1\to Y_1\) and a map \(g:X_2\to Y_2\), we needed the map \(f\times g: X_1\times X_2\to Y_1\times Y_2\), which operated on the individual constituents. If we want to replace the cross product with composition of endofunctors, we need an equivalent for natural transformations. That is, given two natural transformations \(\eta:F\to G\) and \(\epsilon:J\to K\), we want to construct a natural transformation \(\eta\epsilon:J\circ F\to K\circ G\). This diagram illustrates how we get there (working on components):

\[ \begin{CD} F(X) @>{\eta_X}>> G(X) @. \\ @V{J}VV @VV{J}V @. \\ J(F(X)) @>{J(\eta_X)}>> J(G(X)) @>{\epsilon_{G(X)}}>> K(G(X)) \\ \end{CD} \]

As we can see, we can build an arrow \(\epsilon_{G(X)}\circ J(\eta_X): J(F(X)) \to K(G(X))\), which we can use as the components of our natural transformation \(\eta\epsilon:J\circ F\to K\circ G\). This construction is called the horizontal composition of natural transformations. We should verify that this is indeed a natural transformation - for now, let's just accept that it follows from the naturality of \(\eta\) and \(\epsilon\).

Lastly, there is an obvious natural transformation taking a functor to itself; each component being just the identity arrow. We call that natural transformation \(\iota\), staying with the convention of using Greek letters for natural transformations.

With this, we can redraw the diagram we used to define monoids above, the replacements indicated by the quotes:

\[ \begin{CD} I \\ @V{\eta}VV \\ M \\ \end{CD} \hspace{1em} \begin{CD} M\circ M \\ @V{\mu}VV \\ M \\ \end{CD} \hspace{1em} \begin{CD} M\circ I @>{\iota\ \eta}>> M\circ M @<{\eta\ \iota}<< I\circ M \\ @VVV @V{\mu}VV @VVV \\ M @>{\iota}>> M @<{\iota}<< M \\ \end{CD} \hspace{1em} \begin{CD} M\circ M\circ M @>{\mu\ \iota}>> M\circ M \\ @V{\iota\ \mu}VV @V{\mu}VV \\ M\circ M @>{\mu}>> M \\ \end{CD} \]

The vertical arrows in the middle diagram now simply apply the composition of functors, using that the identity functor is a unit.

These diagrams encode these conditions on our natural transformations:

  • \(\mu\circ\eta\iota = \mu = \iota\eta\circ\mu\), that is \(\eta\) serves as a unit
  • \(\mu\circ\mu\iota = \mu\circ\iota\mu\), that is \(\mu\) is associative

To recap, a monad, in category theory, is

  • An endofunctor \(M\)
  • A natural transformation \(\eta: I\to M\), which serves as an identity for horizontal composition.
  • A natural transformation \(\mu: M\circ M\to M\), which is associative in respect to horizontal composition.

Now, let's see, how this maps to Haskell monads.

First, what is the identity functor in Haskell? As we pointed out above, the object function of functors is implicit, when we write f a instead of a. As such, the identity functor is simply a - i.e. we map any type to itself. fmap of that functor would thus also just be the identity fmap :: (a -> a) -> (a -> a).

So, what would our natural transformation \(\eta\) look like? As we said, a natural transformation between two functors is just a map f a -> g a. So (if we call our endofunctor m) the identity transformation of our monoid is eta :: a -> m a mapping the identity functor to m. We also need our monoidal operation, which should map m applied twice to m: mu :: m (m a) -> m a.

Now, Haskellers write return instead of eta and write join instead of mu, giving us the type class

class (Functor m) => Monad where
  return :: a -> m a
  join :: m (m a) -> m a

As a last note, it is worth pointing out that you usually won't implement join, but instead a different function, called "monadic bind":

(>>=) :: m a -> (a -> m b) -> m b

The reason is, that this more closely maps to what monads are actually used for in functional programming. But we can move between join and >>= via

(>>=) :: m a -> (a -> m b) -> m b
v >>= f = join ((fmap f) v)

join :: m (m a) -> m a
join v = v >>= id
Conclusion

This certainly was a bit of a long ride. It took me much longer than anticipated both to understand all the steps necessary and to write them down. I hope you found it helpful and I hope I didn't make too many, too glaring mistakes. If so (either), feel free to let me know on Twitter, reddit or Hacker News - but please remember to be kind :)

I want to thank Tim Adler and mxf+ for proof-reading this absurdly long post and for making many helpful suggestions for improvements


[1] It is often confusing to people, that the way the arrows point in the notation and the order they are written seems to contradict each other: When writing \(f:X\to Y\) and \(g:Y\to Z\) you might reasonably expect their composite to work like \(f\circ g: X\to Z\), that is, you glue together the arrows in the order you are writing them.

The fact that we are not doing that is a completely justified criticism, that is due to a historical accident - we write function application from right to left, that is we write \(f(x)\), for applying \(f\) to \(x\). Accordingly, we write \(g(f(x))\), when applying \(g\) to the result of applying \(f\) to \(x\). And we chose to have the composite-notation be consistent with that, instead of the arrow-notation.

I chose to just eat the unfortunate confusion, as it turns out Haskell is doing exactly the same thing, so swapping things around would just increase the confusion.

Sorry.

[2] Keep in mind that this is a different notion from the ones for monoids, which we come to a bit later: While the formulas seem the same and the identities look like a unit, the difference is that only certain arrows can be composed, not all. And that there are many identity arrows, not just one. However, if we would have only one object, it would have to be the domain and codomain of every arrow and there would be exactly one identity arrow. In that case, the notions would be the same and indeed, "a category with exactly one object" is yet another way to define monoids.

[3] It is customary, to use the same name for the object and arrow map, even though that may seem confusing. A slight justification of that would be, that the object map is already given by the arrow map anyway: If \(F\) is the arrow map, we can define the object map as \(X\mapsto \mathrm{dom}(F(\mathrm{id}_X))\). So, given that they are always occurring together and you can make one from the other, we tend to just drop the distinction and save some symbols.

What was that? Oh, you thought Mathematicians where precise? Ha!

[4] It is important to note, that this is not really a function. Functions operate on values of a given type. But here, we are operating on types and Haskell has no concept of a "type of types" built in that a function could operate on. There are constructs operating on types to construct new types, like data, type, newtype or even deriving. But they are special syntactical constructs that exist outside of the realm of functions.

This is one of the things that was tripping me up for a while: I was trying to figure out, how I would map types to other types in Haskell or even talk about the object map. But the most useful answer is "you don't".

[5] An important note here, is that the \(\eta_X\) are arrows. Where the object map of a functor is just a general association which could look anything we like, the components of a natural transformation need to preserve the internal structure of the category we are working in.

[6] You will often see these conditions written differently, namely written e.g. \(\mu M\) instead of \(\mu\iota\). You can treat that as a notational shorthand, it really means the same thing.

[7] There is a technicality here, that Haskell also has an intermediate between functor and monad called "applicative". As I understand it, this does not have a clear category theoretical analogue. I'm not sure why it exits, but I believe it has been added into the hierarchy after the fact.

08. January 2018 00:30:00

02. January 2018

Mero’s Blog

My case for veganism

I'm going to try to make an argument for being vegan, but, to be clear, it is not very likely to convince you to change your eating habits. It is not designed to - it is only supposed to change the way you think about it. I mention all of that, so you are aware that I don't care what your conclusions are here. If you are reading this, you should do so out of a genuine interest of my motives and for the purpose of self-reflection - not to pick a fight with that vegan dude and really show him he's wrong. I will not debate the content of this article with you. So, with that out of the way, here is a thought experiment:

Say, we would live in a Star Trek like post-scarcity society. Energy is all but abundant and we figured out replicator-technology, that can instantly create anything we like out of it. You get offer the choice between two meals, one is a delicious steak dinner (or whatever), made in a traditional manner. The second is the same thing, but from a replicator. Both are indistinguishable, they taste the same, they have the same nutritional and chemical composition, they cost the same. They only differ in how they're made.

You might be trying to rules-lawyer this. You might be trying to make up an argument, for why the replicator-steak would have to be worse. Or that the cow would already be dead, so it wouldn't matter. But that is obviously not the point of this thought experiment (and remember, you don't have to convince anyone of being right, here). The point is, that I strongly believe that the vast majority of people would agree, that all things being equal, choosing the meal that no animal suffered for is the correct choice. And if you truly believe that it isn't, if you can honestly say to yourself that it doesn't matter: You won't gain anything from the rest of this article. You are relieved and might now just as well stop reading.

The point I am trying to make, is that you probably already know all the reasons you should be vegan. It's very likely that you already have an intuitive understanding of all the reasons in the "pro veganism" column of your pro/contra list. And it really shouldn't be necessary to convince you it's a good idea, in general.

Why then, do so few people actually choose to be vegan, if they are fully aware of all the reasons to do so? The obvious answer is: Because not all things are being equal. There is a "contra veganism" column and it's filled with many good reasons not to. What reasons those are, is deeply individual. It might be due to health. Due to an appeal to nature. Convenience. Money. Availability. Taste. Or maybe just priorities: Other things seem more important and deserving of your energy. And that's okay. We all have to make hundreds of decisions every day and weigh these kinds of questions. And sometimes we do things that we shouldn't and we usually have good reasons to. And sometimes we compromise and don't go all the way, but just do the best we feel able to and that's fine too. Nobody has to be perfect all the time.

The reason, I'm writing this article anyway, is that there is a fundamental difference between the two questions "Why are you vegan?" and "Why are you not not vegan?". When you ask me why I am vegan, you are making the conversation inherently about my values and you will usually end up attacking them - not because you disagree with them (you likely are not), but just because that's the natural progression of this question. And to be absolutely clear: I don't owe you a justification for my value system. I'm sorry if that sounds harsh, but the topic is mostly really annoying to me (as hard as that may be to believe at this point).

A more sensible question, though, is to ask how to best mitigate the contra-column. If we agree that, fundamentally, the points in the pro-column are valid and shared reasons, we can proceed into the much more productive conversation about how much weight the downsides really have and how you might be able to reduce at least some of their impact. And, again to be clear: The outcome of that might very well be, that your reasons are completely valid, rational and that, applied to your personal situation, veganism wouldn't be a good choice. (And to be also clear: I might not be in the mood to have this conversation either. But it's much preferable).

So, what I wish people to take away from this is

  1. Stop asking why you should be vegan (or why I am), you more than likely already know. If you are really interested in making an informed choice, bring up your concerns instead, but also accept if I don't want to talk about them at that particular time - it's a lot of emotional labor, to give the same explanations repeatedly. It might not seem like a big deal to me, to ask these questions, but I've repeated most of my answers literally hundreds of times at this point and might prefer to just enjoy my food.
  2. Stop treating veganism as a preference and start treating it as a moral choice. There is a qualitative difference between someone who does not like Italian food and a vegan. This is interesting when choosing what or where to eat as a group: This is hard enough as it is and I at least usually try very hard to accommodate everyone and not be a burden. And I absolutely do not expect to be treated like I'm better for that. But if it actually would come down to a choice between a vegetarian restaurant or a steakhouse, just because you really like meat: Yes, I do absolutely expect us to avoid the steakhouse. (To be clear: In my experience, it rarely actually comes down to only those two choices. And there are good reasons to avoid vegetarian restaurants that are not based on preference which should be given ample weight too - e.g. someone I know has Coeliac disease, fructose intolerance and lactose intolerance and tends to have a very hard time eating non-meat things. In my experience, though, people who have needs and not just preferences tend to ironically be more open to compromise anyway, so it is less often a problem with them).
  3. Maybe think about your reasons for not being vegan and evaluate them seriously. To be clear, this is a stretch-goal and not the actual point of this article.

And if you want to, you can watch someone who does eat meat say essentially the same things here:

Thanks for reading, don't @ me. ;)


Reasons I'm not not vegan

Now, the main point of this post is dedicated to the general question of "how I think about the topic and how I believe you should think about it too". But I also want it to serve as a reference of my personal, specific thoughts driving my decision - so if you don't know me well or are not interested in my personal reasons, this would be a good place to close the tab and do something else.

I'm still writing this, because I hope this can be the last thing I ever have to write about this (yeah, lol). Because again, I don't actually like discussing it, as unbelievable as that may seem. So here is, as a reference, why I am vegan (and I might add to/change this list over time, when my viewpoints evolve). Why, after ten-ish years of thinking "I should be vegan, but…", I decided to make the switch - or at least give it a serious try. So, this is a list of reasons I gave to myself to justify not going vegan and why they stopped being convincing to me. Your mileage may vary.

Meat/Cheese/Eggs/Bailey's is awesome and I can't imagine giving it up.

For most of my life I didn't think about vegetarianism or veganism at all. Eating meat was the default, so that's what I did. When I did start to think about it, I convinced myself that I couldn't give up meat, because most of my favorite foods where meat-based. However, a lot of people in my peer-group around that time (university) where vegetarian or vegan, so I naturally got into contact with a lot of good food that wasn't naturally meat-based. So I started eating less and less meat - and the less meat I ate, the more I realized I didn't really miss it that much, given how much good alternatives there are. Eventually I decided to become a "flexitarian", which very quickly (in ~1-2 months) became "vegetarian", when I realized that it didn't actually bother me to not eat meat at all - and making that commitment meant less decisions, so it made things easier. With Cheese/Eggs/Bailey's, I basically went through exactly the same transition, six or seven years later: "I can't imagine giving them up" - "Actually, there are really good alternatives" - "Let's just try reducing it and see how far I get" - "Meh, might as well just commit completely".

So, to me, giving up animal products just turned out much easier, than expected, when I actually tried. And I'm not saying I don't miss them, every once in a while, I will still look longingly at a steak or think fondly of my scrambled eggs. But for the most part, the alternatives are just as great (or at times better), so it isn't as big a sacrifice as expected.

Being vegetarian/vegan is unhealthy.

There is a bunch of research about this and for a while (especially before actually looking into the details) the health implications of veganism (vegetarianism not so much) did concern me. But, it turns out, this topic is pretty complicated. Nutrition research is very hard - and that manifests in the fact that for most of it, the statistical significance is usually low and the effect sizes usually small. Now, I'm not denying, that there are health downsides to a vegan diet. But even with the general mess that nutritional research is, it doesn't seem very controversial that if you are concerned for your health, there are much more important factors to consider. If weighed against the health benefits of sleeping more, doing more sports, not sitting all day, stop recreational drug use, taking extensive vacations… (neither of which I seem to be willing to do, even though they would be easy enough), the relatively minor health effects of eating animal products (contrasted with a somewhat balanced vegan diet plus supplementation) just did not seem to be a relevant driving force for that decision and more of a rationalization.

That being said, from what I can gather so far, there is a general consensus that if a) you pay attention to having a somewhat balanced diet and b) are willing to supplement the nutrients you can't actually get, the health impact of veganism is pretty low, if any. Personally, I am supplementing Vitamins B12 and D right now, which has very low quality of life impact - so I don't consider that a significant downside. I also pay a little bit more attention to what I'm eating, which I consider a good thing.

If it turns out that I can not sustain a vegan diet, I will reconsider it, but for now, I don't see any realistic danger of that happening.

It is cumbersome to know whether or not something is vegan.

This is mostly true. As a vegetarian, this mostly revolved around looking for gelatin in the ingredients of sweets and asking in a restaurant whether "Lasagna" is made with meat or not. Being a vegan does involve a lot of scanning ingredients-lists of basically every product I buy. Though I'm positively surprised how many vendors are recently starting to choose to get their products certified - and not only brands you would expect to focus on that, but also, increasingly, all kinds of mainstream products.

That being said, there is an availability issue (especially around "may contain traces of…", which is basically saying "if you are allergic, we can't rule out cross-contamination of other things from the same factory"). I tend to be pragmatic about this: If I have the choice, I will buy the certifiably vegan option, otherwise I'm also fine with traces of animal products, personally. If I don't know, I will go with my best guess and how I feel in the moment.

This is definitely the most true and heavy argument still on the contra-side for me, but being kind of pragmatic about it helps alleviate most of the pain.

It's hypocritical to draw the line at X and not at Y.

You can always be more rigorous and there are a lot of line-drawing questions that come up when thinking about vegetarianism/veganism. For the record, a lot of that is just plain nonsense, but there are some legitimate questions to be asked around whether or not insects count, for example, or certain shellfish, whether you would eat meat if it would be thrown away otherwise or would eat an egg, if the Hen laying it was living a happy, free life. In the end, the vast majority of things you can eat will involve some harm to the environment or animals and you won't always know, so where do you draw the line?

Personally, I decided that definite harm is worse than potential harm and more harm is worse than less harm. "It is hypocritical to not eat meat/cheese/eggs but still kill a wasp entering your apartment" is about as convincing an argument to me as "it is hypocritical to eat meat/cheese/eggs but not also eat dogs/jellyfish/human". The world isn't black-and-white and it's fine to choose a gray spot in the middle that makes you comfortable.

Eating out becomes a PITA.

Yes. Going out and eating in a group is a PITA. Honestly, there are no two ways about it. I do have to actively make sure that a chosen food place has options for me and more often than not that does involve making special requests and/or making do with less great meals.

In general, this still works reasonably well. The cafeteria at work has great vegan options most of the time, Zurich has an amazing choice of great restaurants for vegans to offer, most other restaurants can accommodate too and even if not, I'm fine just eating a little thing and then later eat some more at home.

The main problem is working around the social issues associated with it - dealing with people who are unwilling to be accommodating, having to justify/discuss my choice or just exposing something about my person I might prefer to keep private. Basically, I wrote a whole thing about this.

But this is simply one of those downsides I chose to accept. Nobody said going vegan wouldn't come with sacrifices.

Being vegan is expensive

I am not sure this is true in general. I am relatively sure, that being vegetarian at least actually ended up saving me money as a student. But I can't be completely certain, as the change also came with other changes in circumstances. My vegan diet is probably more expensive than my vegetarian one, mainly because it includes a lot more processed substitute products ("faux meat" and various plant milks, which are at least in Switzerland still significantly more expensive than the cow-based variants), but again, I didn't actually run any numbers.

I'm pretty sure it's possible to have an affordable vegan diet, especially if limiting processed substitute products and not eating out so often. Luckily, this isn't really a concern for me, right now, though. Food and Groceries is a relatively small proportion of my monthly expenses and as such, the impact this has on me is pretty limited either way.

I convinced myself, that if I can afford spending money on all kinds of luxury items and electronic gadgets, I can probably afford spending a little more on food.

02. January 2018 00:23:00

11. December 2017

sECuREs Webseite

Dell 8K4K monitor (Dell UP3218K)

Background

Ever since I first used a MacBook Pro with Retina display back in 2013, I’ve been madly in love with hi-DPI displays. I had seen the device before, and marvelled at brilliant font quality with which scientific papers would be rendered. But it wasn’t until I had a chance to use the device for a few hours to make i3 compatible with hi-DPI displays that I realized what a difference it makes in the day-to-day life.

Note that when I say “hi-DPI display”, I mean displays with an integer multiple of 96 dpi, for example displays with 192 dpi or 288 dpi. I explain this because some people use the same term to mean “anything more than 96 dpi”.

In other words, some people are looking for many pixels (e.g. running a 32 inch display with 3840x2160 pixels, i.e. 137 dpi, with 100% scaling), whereas I desire crisp/sharp text (i.e. 200% scaling).

Hence, in 2014, I bought the Dell UP2414Q with 3840x2160 on 24” (185 dpi), which was one of the first non-Apple devices to offer a dpi that Apple would market as “Retina”.

After getting the Dell UP2414Q, I replaced all displays in my life with hi-DPI displays one by one. I upgraded my phone, my personal laptop, my work laptop and my monitor at work.

Dell UP3218K

In January 2017, Dell introduced the Dell Ultrasharp UP3218K monitor at the Consumer Electronics Show (CES). It is the world’s first available 8K monitor, meaning it has a resolution of 7680x4320 pixels at a refresh rate of 60 Hz. The display’s dimensions are 698.1mm by 392.7mm (80cm diagonal, or 31.5 inches), meaning the display shows 280 dpi.

While the display was available in the US for quite some time, it took until October 2017 until it became available in Switzerland.

Compatibility

The UP3218K requires connecting two separate DisplayPort 1.4 cables in order to reach the native resolution and refresh rate. When connecting only one cable, you will be limited to a refresh rate of 30 Hz, which is a very noticeable annoyance on any display: you can literally see your mouse cursor lag behind. Ugh.

Note that this mode of connection does not use Multi-Stream Transport (MST), which was a trick that first-generation 4K displays used. Instead, it uses the regular Single-Stream Transport (SST), but two cables.

As of November 2017, only latest-generation graphics cards support DisplayPort 1.4 at all, with e.g. the nVidia GTX 1060 being marketed as “DisplayPort 1.2 Certified, DisplayPort 1.3/1.4 Ready”.

AMD Radeon Pro WX7100

Hence, I thought I would play it safe and buy a graphics card which is explicitly described as compatible with the UP3218K: I ordered an AMD Radeon Pro WX7100.

Unfortunately, I have to report that the WX7100 is only able to drive the monitor at native resolution when using Windows. On Linux, I was limited to 1920x1080 at 60Hz (!) when using the Open Source amdgpu driver. With the Closed Source amdgpu-pro driver, I reached 3840x2160 at 60Hz, which is still not the native resolution. Also, the amdgpu-pro driver is a hassle to install: it requires an older kernel and isn’t packaged well in Debian.

nVidia GeForce GTX 1060

I returned the WX7100 in exchange for the cheapest and most quiet GeForce 10 series card with 2 DisplayPort outputs I could find. My choice was the MSI GeForce GTX 1060 GAMING X 6G (MSI V328-001R). The card seems like overkill, given that I don’t intend to play games on this machine, but lower-end cards all come with at most one DisplayPort output.

Regardless, I am happy with the card. It indeed is silent, and with the Closed Source driver, it powers the UP3218K without any trouble. Notably, it supports RandR 1.5, which I’ll talk about a bit more later.

Compatibility Matrix

Operating System Graphics Card Driver Resolution
Windows Radeon WX7100 yes 7680x4320 @ 60 Hz
Windows GeForce 1060 yes 7680x4320 @ 60 Hz
Linux Radeon WX7100 amdgpu 1920x1080 @ 60 Hz
Linux Radeon WX7100 pro 3840x2160 @ 60 Hz
Linux GeForce 1060 nVidia 7680x4320 @ 60 Hz

Recommendation

If you want to play it safe, buy an nVidia card of the GeForce 10 series. Verify that it says “DisplayPort 1.4 Ready”, and that it comes with two DisplayPort outputs.

I read about improvements of the amdgpu driver for the upcoming Linux 4.15, but I don’t know whether that will help with the problems at hand.

Impressions

The unboxing experience is well-designed, and all components make a good impression. All cables which you will need (two DisplayPort cables, a power cable, a USB cable) are included and seem to be of high quality.

The display has a thin bezel, much thinner than my other monitors ViewSonic VX2475Smhl-4K or Dell UP2414Q.

The power LED is white and not too bright. The on-screen menu reacts quickly and is reasonably intuitive.

The built-in USB hub works flawlessy, even with devices which don’t work on my standalone USB3 hub (for reasons which I have yet to find out).

Display Quality

The display quality of the screen is stunningly good.

It was only when I configured 300% scaling that I realized why some Chromebooks had a distinctly different look and feel from other computers I had used: I always assumed they differed in font rendering somehow, but the actual difference is just the screen DPI: fonts look distinctly better with 288 dpi than with 192 dpi, which of course looks better than 96 dpi.

Some people might wonder whether an 8K display is any better than a 4K display, and I now can answer that question with a decisive “yes, one can easily see the difference”. I’m not sure if the difference between a 288 dpi and a 384 dpi display would be visible, but we’ll see when we get there :-).

Glossy

What I didn’t expect is that the UP3218K is a glossy display, as opposed to a matte display. Depending on the brightness and colors, you might see reflections. With my preferred brightness of 50%, I can clearly see reflections when displaying darker colors, e.g. on a black terminal emulator background, or even in my grey Emacs theme.

While one can mentally ignore the reflections after a little while, I still consider the glossyness a mild annoyance. I hope as 8K displays become more prevalent, display vendors will offer matte 8K displays as well.

Scaling

I found it interesting that the display works well in both 200% scaling and 300% scaling.

When running the display at 200% scaling, you get 3840x2160 (4K resolution) “logical pixels”, but sharper.

When running the display at 300% scaling, you get 2560x1440 “logical pixels”, but extremely sharp.

I would say it is a subjective preference which of the two settings to use. Most likely, people who prefer many pixels would run the display at 200%, whereas I prefer the 300% scaling mode for the time being.

Linux compatibility / configuration

To use this display without gross hacks, ensure all relevant components in your software stack support RandR 1.5. My known working configuration is:

  • Xorg 1.19.5
  • nVidia driver 375.82
  • libxcb 1.12
  • i3 4.14
  • i3lock 2.10

With the following command, you can create a RandR MONITOR object spanning the DisplayPort outputs DP-4 and DP-2:

xrandr --setmonitor up3218k auto DP-4,DP-2

I place this command in my ~/.xsession before starting i3.

Theoretically, Xorg could create a MONITOR object automatically. I filed a feature request for this.

Scaling compatibility

With regards to scaling issues, the situation is very similar to any other monitor which requires scaling. Applications which were updated to support 200% scaling seem to work with 300% scaling just as well.

Of course, applications which were not yet updated to work with scaling look even smaller than on 200% displays, so it becomes more of a nuisance to use them. As far as I can tell, the most likely offender are Java applications such as JDownloader.

Buzzing noise

Unfortunately, the monitor emits a high-pitched buzzing noise, very similar to Coil Whine. The noise is loud enough to prevent focused work without listening to music.

I verified that this symptom was happening with Windows and Linux, on two different computers, with default monitor settings, and even when no input source was connected at all.

Finally, I contacted Dell support about it. In the call I received on the next business day, they were very forthcoming and offered to replace the monitor.

The replacement monitor still emits some noise, but much less pronounced. I think I can easily ignore the noise.

Wakeup issues

Rarely (less than once a week), when waking up the monitor from DPMS standby mode, only the left half of the screen would appear on my monitor.

This can be fixed by turning the monitor off and on again.

My theory is that one of the scalers does not manage to synchronize with the video card, but I don’t know for sure.

Interestingly enough, I also encountered this issue with the Dell UP2414Q I bought in 2014. My workaround is to power down that display using its power button in the evenings, and power it up in the mornings.

Conclusion

For me, this monitor is worth it: I am okay with paying the hefty Research & Development tax that first-to-market products such as this monitor have. I like to think that I’m voting with my wallet to make sure vendors notice my interest in “Retina” displays.

For most people, I would recommend to wait until the second or third generation of 8K monitors. By then, I expect most issues to be resolved, compatibility to not be a concern, and vendors focusing on extra features. Hopefully, we’ll eventually see matte 8K monitors with higher refresh rates than 60 Hz.

Technical details

In the hope the following is useful (perhaps for debugging?) to anyone:

by Michael Stapelberg at 11. December 2017 09:05:00

02. December 2017

michael-herbst.com

Annual Colloquium 2017: Introduction to Bohrium

Last Thursday the PhDs of my graduate school gathered again for our self-organised mini conference, named Annual Colloquium. Having organised this event myself as well a couple of years back, I had rather mixed feelings this time, since I most likely not be around in Heidelberg for another AC.

For this reason I am very happy that the organisers gave me the chance to once again make a contribution to this great event. This time I was asked to repeat my interactive introductory talk about the Bohrium automatic parallelisation framework. I already presented about Bohrium in one of my c¼h lectures at the Heidelberg Chaostreff earlier this year and in fact this talk turned out to be very similar to the previous one.

Compared to the points mentioned in my earlier blog post, one should probably add a few things, which have changed in Bohrium in the recent months. First of all Bohrium has made quite some progress regarding the interoperability with other efforts like cython, pyopencl and pycuda for improving the performance of python scripts. In fact Bohrium and these projects can now be used side-by-side and will work together flawlessly to accelerate algorithms written in python. Along a similar line, Bohrium started to look into mechanisms, which could be used to speed up places where one would typically require a plain python for-loops. Whilst this destroys the full compatibility with numpy on the one hand, this allows on the other hand to increase performance in settings, which are hard to write only as array operations.

On top of that the recent integration into the Spack package manager makes it comparatively easy to install Bohrium on any machine (including HPC clusters) to give it a try in a production environment. See the Spack section of the Bohrium documentation for more details.

If you want to find out more about Bohrium, I suggest you read my previous post or watch the recording of my previous talk. For completeness I attach below the demonstration script I used for both Bohrium presentations.

Link Licence
Bohrium moments example script Creative Commons License

by Michael F. Herbst at 02. December 2017 23:00:00

29. November 2017

RaumZeitLabor

Hack To The Future – Mentorinnen und Mentoren gesucht

Ihr überlegt noch, was die erste gute Tat des neuen Jahres werden soll? Ihr habt vom 19. bis 21. Januar Zeit?

Das RaumZeitLabor wird die Initiative Kindermedienland am dritten Wochenende im Januar beim “Hack To The Future” in der Stadtbibliothek Mannheim unterstützen.

Technikbegeisterte Jugendliche bilden kleine Teams und entwickeln innerhalb des Wochenendes eigene digitale Projekte. Dabei werden sie von erfahrenen Mentorinnen und Mentoren unterstützt. Ihr habt Lust und Zeit beim Tüfteln, Coden und bei der Projektentwicklung zu helfen? Perfekt! Dann schaut doch mal hier vorbei!

Vergütet wird das Ganze nicht. Für Mentorinnen und Mentoren gibt es ein HTTF-Shirt und kostenlose Verpflegung am ganzen Wochenende sowie ein gemeinsames Essen nach der Veranstaltung als Dankeschön.

httf_logo

by flederrattie at 29. November 2017 00:00:00

13. November 2017

sECuREs Webseite

Network setup for our retro computing event RGB2Rv17

Our computer association NoName e.V. organizes a retro computing event called RGB2R every year, located in Heidelberg, Germany. This year’s version is called RGB2Rv17.

This article describes the network setup I created for this year’s event.

The intention is not so much to provide a fully working setup (even though the setup did work fine for us as-is), but rather inspire to you to create your own network, based vaguely on what’s provided here.

Connectivity

The venue has a DSL connection with speeds reaching 1 Mbit/s if you’re lucky. Needless to say, that is not sufficient for the about 40 participants we had.

Luckily, there is (almost) direct line of sight to my parent’s place, and my dad recently got a 400 Mbit/s cable internet connection, which he’s happy to share with us :-).

WiFi antenna pole

Hardware

For the WiFi links to my parent’s place, we used 2 tp-link CPE510 (CPE stands for Customer Premise Equipment) on each site. The devices only have 100 Mbit/s ethernet ports, which is why we used two of them.

The edge router for the event venue was a PC Engines apu2c4. For the Local Area Network (LAN) within the venue, we provided a few switches and WiFi using Ubiquiti Networks access points.

Software

On the apu2c4, I installed Debian “stretch” 9, the latest Debian stable version at the time of writing. I prepared a USB thumb drive with the netinst image:

% wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.2.1-amd64-netinst.iso
% cp debian-9.2.1-amd64-netinst.iso /dev/sdb

Then, I…

  • plugged the USB thumb drive into the apu2c4
  • On the serial console, pressed F10 (boot menu), then 1 (boot from USB)
  • In the Debian installer, selected Help, pressed F6 (special boot parameters), entered install console=ttyS0,115200n8
  • installed Debian as usual.

Initial setup

Debian stretch comes with systemd by default, but not with systemd-networkd(8) by default, so I changed that:

edge# systemctl enable systemd-networkd
edge# systemctl disable networking

Also, I cleared the MOTD, placed /tmp on tmpfs and configured my usual environment:

edge# echo > /etc/motd
edge# echo 'tmpfs /tmp tmpfs defaults 0 0' >> /etc/fstab
edge# wget -qO- https://d.zekjur.net | bash -s

I also installed a few troubleshooting tools which came in handy later:

edge# apt install tcpdump net-tools strace

Disabling ICMP rate-limiting for debugging

I had to learn the hard way that Linux imposes a rate-limit on outgoing ICMP packets by default. This manifests itself as spurious timeouts in the traceroute output. To ease debugging, I disabled the rate limit entirely:

edge# cat >> /etc/sysctl.conf <<'EOT'
net.ipv4.icmp_ratelimit=0
net.ipv6.icmp.ratelimit=0
EOT
edge# sysctl -p

Renaming network interfaces

Descriptive network interface names are helpful when debugging. I won’t remember whether enp0s3 is the interface for an uplink or the LAN, so I assigned the names uplink0, uplink1 and lan0 to the apu2c4’s interfaces.

To rename network interfaces, I created a corresponding .link file, had the initramfs pick it up, and rebooted:

edge# cat >/etc/systemd/network/10-uplink0.link <<'EOT'
[Match]
MACAddress=00:0d:b9:49:db:18

[Link]
Name=uplink0
EOT
edge# update-initramfs -u
edge# reboot

Network topology

Because our internet provider didn’t offer IPv6, and to keep my dad out of the loop in case any abuse issues should arise, we tunneled all of our traffic.

We decided to set up one tunnel per WiFi link, so that we could easily load-balance over the two links by routing IP flows into one of the two tunnels.

Here’s a screenshot from the topology dashboard which I made using the Diagram Grafana plugin:

Network interface setup

We configured IP addresses statically on the uplink0 and uplink1 interface because we needed to use static addresses in the tunnel setup anyway.

Note that we placed a default route in route table 110. Later on, we used iptables(8) to make traffic use either of these two default routes.

edge# cat > /etc/systemd/network/uplink0.network <<'EOT'
[Match]
Name=uplink0

[Network]
Address=192.168.178.10/24
IPForward=ipv4

[Route]
Gateway=192.168.178.1
Table=110
EOT
edge# cat > /etc/systemd/network/uplink1.network <<'EOT'
[Match]
Name=uplink1

[Network]
Address=192.168.178.11/24
IPForward=ipv4

[Route]
Gateway=192.168.178.1
Table=111
EOT

Tunnel setup

Originally, I configured OpenVPN for our tunnels. However, it turned out the apu2c4 tops out at 130 Mbit/s of traffic through OpenVPN. Notably, using two tunnels didn’t help — I couldn’t reach more than 130 Mbit/s in total. This is with authentication and crypto turned off.

This surprised me, but doesn’t seem too uncommon: on the internet, I could find reports of similar speeds with the same hardware.

Given that our setup didn’t require cryptography (applications are using TLS these days), I looked for light-weight alternatives and found Foo-over-UDP (fou), a UDP encapsulation protocol supporting IPIP, GRE and SIT tunnels.

Each configured Foo-over-UDP tunnel only handles sending packets. For receiving, you need to configure a listening port. If you want two machines to talk to each other, you therefore need a listening port on each, and a tunnel on each.

Note that you need one tunnel per address family: IPIP only supports IPv4, SIT only supports IPv6. In total, we ended up with 4 tunnels (2 WiFi uplinks with 2 address families each).

Also note that Foo-over-UDP provides no authentication: anyone who is able to send packets to your configured listening port can spoof any IP address. If you don’t restrict traffic in some way (e.g. by source IP), you are effectively running an open proxy.

Tunnel configuration

First, load the kernel modules and set the corresponding interfaces to UP:

edge# modprobe fou
edge# modprobe ipip
edge# ip link set dev tunl0 up
edge# modprobe sit
edge# ip link set dev sit0 up

Configure the listening ports for receiving FOU packets:

edge# ip fou add port 1704 ipproto 4
edge# ip fou add port 1706 ipproto 41

edge# ip fou add port 1714 ipproto 4
edge# ip fou add port 1716 ipproto 41

Configure the tunnels for sending FOU packets, using the local interface of the uplink0 interface:

edge# ip link add name fou0v4 type ipip remote 203.0.113.1 local 192.168.178.10 encap fou encap-sport auto encap-dport 1704 dev uplink0
edge# ip link set dev fou0v4 up
edge# ip -4 address add 10.170.0.1/24 dev fou0v4

edge# ip link add name fou0v6 type sit remote 203.0.113.1 local 192.168.178.10 encap fou encap-sport auto encap-dport 1706 dev uplink0
edge# ip link set dev fou0v6 up
edge# ip -6 address add fd00::10:170:0:1/112 dev fou0v6 preferred_lft 0

Repeat for the uplink1 interface:

# (IPv4) Set up the uplink1 transmit tunnel:
edge# ip link add name fou1v4 type ipip remote 203.0.113.1 local 192.168.178.11 encap fou encap-sport auto encap-dport 1714 dev uplink1
edge# ip link set dev fou1v4 up
edge# ip -4 address add 10.171.0.1/24 dev fou1v4

# (IPv6) Set up the uplink1 transmit tunnel:
edge# ip link add name fou1v6 type sit remote 203.0.113.1 local 192.168.178.11 encap fou encap-sport auto encap-dport 1716 dev uplink1
edge# ip link set dev fou1v6 up
edge# ip -6 address add fd00::10:171:0:1/112 dev fou1v6 preferred_lft 0

Load-balancing setup

In previous years, we experimented with setups using MLVPN for load-balancing traffic on layer 2 across multiple uplinks. Unfortunately, we weren’t able to get good results: when aggregating links, bandwidth would be limited to the slowest link. I expect that MLVPN and others would work better this year, if we were to set it up directly before and after the WiFi uplinks, as the two links should be almost identical in terms of latency and throughput.

Regardless, we didn’t want to take any chances and decided to go with IP flow based load-balancing. The downside is that any individual connection can never be faster than the uplink over which it is routed. Given the number of concurrent connections in a typical network, in practice we observed good utilization of both links regardless.

Let’s tell iptables to mark packets coming from the LAN with one of two values based on the hash of their source IP, source port, destination IP and destination port properties:

edge# iptables -t mangle -A PREROUTING -s 10.17.0.0/24 -j HMARK --hmark-tuple src,sport,dst,dport --hmark-mod 2 --hmark-offset 10 --hmark-rnd 0xdeadbeef

Note that the --hmark-offset parameter is required: mark 0 is the default, so you need an offset of at least 1.

For debugging, it is helpful to exempt the IP addresses we use on the tunnels themselves, otherwise we might not be able to ping an endpoint which is actually reachable:

edge# iptables -t mangle -A PREROUTING -s 10.17.0.0/24 -d 10.170.0.0/24 -m comment --comment "for debugging" -j MARK --set-mark 10
edge# iptables -t mangle -A PREROUTING -s 10.17.0.0/24 -d 10.171.0.0/24 -m comment --comment "for debugging" -j MARK --set-mark 11

Now, we need to add a routing policy to select the correct default route based on the firewall mark:

edge# ip -4 rule add fwmark 10 table 10
edge# ip -4 rule add fwmark 11 table 11

The steps for IPv6 are identical.

Note that current OpenWrt (15.05) does not provide the HMARK iptables module. I filed a GitHub issue with OpenWrt.

Connectivity for the edge router

Because our default routes are placed in table 110 and 111, the router does not have upstream connectivity. This is mostly working as intended, as it makes it harder to accidentally route traffic outside of the tunnels.

There is one exception: we need a route to our DNS server:

edge# ip -4 rule add to 8.8.8.8/32 lookup 110

It doesn’t matter which uplink we use for that, since DNS traffic is tiny.

Connectivity to the tunnel endpoint

Of course, the tunnel endpoint itself must also be reachable:

edge# ip rule add fwmark 110 lookup 110
edge# ip rule add fwmark 111 lookup 111

edge# iptables -t mangle -A OUTPUT -d 203.0.113.1/32 -p udp --dport 1704 -j MARK --set-mark 110
edge# iptables -t mangle -A OUTPUT -d 203.0.113.1/32 -p udp --dport 1714 -j MARK --set-mark 111
edge# iptables -t mangle -A OUTPUT -d 203.0.113.1/32 -p udp --dport 1706 -j MARK --set-mark 110
edge# iptables -t mangle -A OUTPUT -d 203.0.113.1/32 -p udp --dport 1716 -j MARK --set-mark 111

Connectivity to the access points

By clearing the firewall mark, we ensure traffic doesn’t get sent through our tunnel:

edge# iptables -t mangle -A PREROUTING -s 10.17.0.0/24 -d 192.168.178.250 -j MARK --set-mark 0 -m comment --comment "for debugging"
edge# iptables -t mangle -A PREROUTING -s 10.17.0.0/24 -d 192.168.178.251 -j MARK --set-mark 0 -m comment --comment "for debugging"
edge# iptables -t mangle -A PREROUTING -s 10.17.0.0/24 -d 192.168.178.252 -j MARK --set-mark 0 -m comment --comment "for debugging"
edge# iptables -t mangle -A PREROUTING -s 10.17.0.0/24 -d 192.168.178.253 -j MARK --set-mark 0 -m comment --comment "for debugging"

Also, since the access points are all in the same subnet, we need to tell Linux on which interface to send the packets, otherwise packets might egress on the wrong link:

edge# ip -4 route add 192.168.178.252 dev uplink0 src 192.168.178.10
edge# ip -4 route add 192.168.178.253 dev uplink1 src 192.168.178.11

MTU configuration

edge# ifconfig uplink0 mtu 1472
edge# ifconfig uplink1 mtu 1472
edge# ifconfig fou0v4 mtu 1416
edge# ifconfig fou0v6 mtu 1416
edge# ifconfig fou1v4 mtu 1416
edge# ifconfig fou1v6 mtu 1416

It might come in handy to quickly be able to disable an uplink, be it for diagnosing issues, performing maintenance on a link, or to work around a broken uplink.

Let’s create a separate iptables chain in which we can place temporary overrides:

edge# iptables -t mangle -N prerouting_override
edge# iptables -t mangle -A PREROUTING -j prerouting_override
edge# ip6tables -t mangle -N prerouting_override
edge# ip6tables -t mangle -A PREROUTING -j prerouting_override

With the following shell script, we can then install such an override:

#!/bin/bash
# vim:ts=4:sw=4
# enforces using a single uplink
# syntax:
#	./uplink.sh 0  # use only uplink0
#	./uplink.sh 1  # use only uplink1
#	./uplink.sh    # use both uplinks again

if [ "$1" = "0" ]; then
	# Use only uplink0
	MARK=10
elif [ "$1" = "1" ]; then
	# Use only uplink1
	MARK=11
else
	# Use both uplinks again
	iptables -t mangle -F prerouting_override
	ip6tables -t mangle -F prerouting_override
	ip -4 rule del to 8.8.8.8/32
	ip -4 rule add to 8.8.8.8/32 lookup "110"
	exit 0
fi

iptables -t mangle -F prerouting_override
iptables -t mangle -A prerouting_override -s 10.17.0.0/24 -j MARK --set-mark "${MARK}"
ip6tables -t mangle -F prerouting_override
ip6tables -t mangle -A prerouting_override -j MARK --set-mark "${MARK}"

ip -4 rule del to 8.8.8.8/32
ip -4 rule add to 8.8.8.8/32 lookup "1${MARK}"

MSS clamping

Because Path MTU discovery is often broken on the internet, it’s best practice to limit the Maximum Segment Size (MSS) of each TCP connection, achieving the same effect (but only for TCP connections).

This technique is called “MSS clamping”, and can be implemented in Linux like so:

edge# iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o fou0v4 -j TCPMSS --clamp-mss-to-pmtu
edge# iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o fou1v4 -j TCPMSS --clamp-mss-to-pmtu
edge# ip6tables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o fou0v6 -j TCPMSS --clamp-mss-to-pmtu
edge# ip6tables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o fou1v6 -j TCPMSS --clamp-mss-to-pmtu

Traffic shaping

Shaping upstream

With asymmetric internet connections, such as the 400/20 cable connection we’re using, it’s necessary to shape traffic such that the upstream is never entirely saturated, otherwise the TCP ACK packets won’t reach their destination in time to saturate the downstream.

While the FritzBox might already provide traffic shaping, we wanted to voluntarily restrict our upstream usage to leave some headroom for my parents.

Hence, we’re shaping each uplink to 8 Mbit/s, which sums up to 16 Mbit/s, well below the available 20 Mbit/s:

edge# tc qdisc replace dev uplink0 root tbf rate 8mbit latency 50ms burst 4000
edge# tc qdisc replace dev uplink1 root tbf rate 8mbit latency 50ms burst 4000

The specified latency value is a best guess, and the burst value is derived from the kernel internal timer frequency (CONFIG_HZ) (!), packet size and rate as per https://unix.stackexchange.com/questions/100785/bucket-size-in-tbf.

Tip: keep in mind to disable shaping temporarily when you’re doing bandwidth tests ;-).

Shaping downstream

It’s somewhat of a mystery to me why this helped, but we achieved noticeably better bandwidth (50 Mbit/s without, 100 Mbit/s with shaping) when we also shaped the downstream traffic (i.e. made the tunnel endpoint shape traffic).

LAN

For DHCP, DNS and IPv6 router advertisments, we set up dnsmasq(8), which worked beautifully and was way quicker to configure than the bigger ISC servers:

edge# apt install dnsmasq
edge# cat > /etc/dnsmasq.d/rgb2r <<'EOT'
interface=lan0
dhcp-range=10.17.0.10,10.17.0.250,30m
dhcp-range=::,constructor:lan0,ra-only
enable-ra
cache-size=10000
EOT

Monitoring

First, install and start Prometheus:

edge# apt install prometheus prometheus-node-exporter prometheus-blackbox-exporter
edge# systemctl enable prometheus
edge# systemctl restart prometheus
edge# systemctl enable prometheus-node-exporter
edge# systemctl restart prometheus-node-exporter
edge# systemctl enable prometheus-blackbox-exporter
edge# systemctl restart prometheus-blackbox-exporter

Then, install and start Grafana:

edge# apt install apt-transport-https
edge# wget -qO- https://packagecloud.io/gpg.key | apt-key add -
edge# echo deb https://packagecloud.io/grafana/stable/debian/ stretch main > /etc/apt/sources.list.d/grafana.list
edge# apt update
edge# apt install grafana
edge# systemctl enable grafana-server
edge# systemctl restart grafana-server

Also, install the excellent Diagram Grafana plugin:

edge# grafana-cli plugins install jdbranham-diagram-panel
edge# systemctl restart grafana-server

Config files

I realize this post contains a lot of configuration excerpts which might be hard to put together. So, you can find all the config files in a git repository. As I mentioned at the beginning of the article, please create your own network and don’t expect the config files to just work out of the box.

Statistics

  • We peaked at about 60 active DHCP leases.

  • The connection tracking table (holding an entry for each IPv4 connection) never exceeded 4000 connections.

  • DNS traffic peaked at about 12 queries/second.

  • dnsmasq’s maximum cache size of 10000 records was sufficient: we did not have a single cache eviction over the entire event.

  • We were able to obtain peaks of over 150 Mbit/s of download traffic.

  • At peak, about 10% of our traffic was IPv6.

WiFi statistics

  • On link 1, our signal to noise ratio hovered between 31 dBm to 33 dBm. When it started raining, it dropped by 2-3 dBm.

  • On link 2, our signal to noise ratio hovered between 34 dBm to 36 dBm. When it started raining, it dropped by 1 dBm.

Despite the relatively bad signal/noise ratios, we could easily obtain about 140 Mbps on the WiFi layer, which results in 100 Mbps on the ethernet layer.

The difference in signal/noise ratio between the two links had no visible impact on bandwidth, but ICMP probes showed measurably more packet loss on link 1.

by Michael Stapelberg at 13. November 2017 21:45:00

10. November 2017

michael-herbst.com

Lazy matrices talk from the IWR school 2017

As mentioned in a previous post on this matter last month, from 2nd to 6th October, the IWR hosted the school Mathematical Methods for Quantum Chemistry, which I co-organised together with my supervisors Andreas Dreuw and Guido Kanschat as well as the head of my graduate school Michael Winckler.

From my personal point of view the school turned out to be a major success, where I had the chance to meet a lot of interesting people and got a bucket of ideas to try out in the future. The feedback we got from the participants was positive as well and so I am very happy that all the effort in the past half a year really turned to be worth the while for all of us.

Even though all relevant documents from the school, including the slides from the lectures and most contributed talks, are finally published at the school's website, I nevertheless want to include a pointer to the slides of my lazy matrices talk from this blog for reference.

The topic and structure of the talk is very similar to the talks of the previous months. I motivate the use of contraction-based methods from the realisation, that storing intermediate computational results in memory can be less optimal than recomputing them in a clever way when needed. Then I present lazy matrices as a solution to the issue that the code needed for performing calculations in the sense of contraction-based methods can become very complicated. Afterwards I hint how we use lazy matrices in the context of the quantum chemistry program molsturm and how molsturm itself really facilitates the implementation of new quantum-chemical methods. For this I show in slide 20 a comparison of parts of a working Coupled-Cluster doubles (CCD) code based on molsturm with the relevant part of the equation for the CCD residual.

Link Licence
Lazy matrices for contraction-based algorithms (IWR school talk) Creative Commons License

by Michael F. Herbst at 10. November 2017 23:00:00

21. October 2017

sECuREs Webseite

Making GitLab authenticate against dex

Because I found it frustratingly hard to make GitLab and dex talk to each other, this article walks you through what I did step-by-step.

Let’s establish some terminology:

  • dex is our OpenID Connect (OIDC) “Provider (OP)”
    in other words: the component which verifies usernames and passwords.

  • GitLab is our OpenID Connect (OIDC) “Relying Party (RP)”
    in other words: the component where the user actually wants to log in.

Step 1: configure dex

First, I followed dex’s Getting started guide until I had dex serving the example config.

Then, I made the following changes to examples/config-dev.yaml:

  1. Change the issuer URL to be fully qualified and use HTTPS.
  2. Configure the HTTPS listener.
  3. Configure GitLab’s redirect URI.

Here is a diff:

--- /proc/self/fd/11	2017-10-21 15:01:49.005587935 +0200
+++ /tmp/config-dev.yaml	2017-10-21 15:01:47.121632025 +0200
@@ -1,7 +1,7 @@
 # The base path of dex and the external name of the OpenID Connect service.
 # This is the canonical URL that all clients MUST use to refer to dex. If a
 # path is provided, dex's HTTP service will listen at a non-root URL.
-issuer: http://127.0.0.1:5556/dex
+issuer: https://dex.example.net:5554/dex
 
 # The storage configuration determines where dex stores its state. Supported
 # options include SQL flavors and Kubernetes third party resources.
@@ -14,11 +14,9 @@
 
 # Configuration for the HTTP endpoints.
 web:
-  http: 0.0.0.0:5556
-  # Uncomment for HTTPS options.
-  # https: 127.0.0.1:5554
-  # tlsCert: /etc/dex/tls.crt
-  # tlsKey: /etc/dex/tls.key
+  https: dex.example.net:5554
+  tlsCert: /etc/letsencrypt/live/dex.example.net/fullchain.pem
+  tlsKey: /etc/letsencrypt/live/dex.example.net/privkey.pem
 
 # Uncomment this block to enable the gRPC API. This values MUST be different
 # from the HTTP endpoints.
@@ -50,7 +48,7 @@
 staticClients:
 - id: example-app
   redirectURIs:
-  - 'http://127.0.0.1:5555/callback'
+  - 'http://gitlab.example.net/users/auth/mydex/callback'
   name: 'Example App'
   secret: ZXhhbXBsZS1hcHAtc2VjcmV0

Step 2: configure GitLab

First, I followed GitLab Docker images to get GitLab running in Docker.

Then, I swapped out the image with computersciencehouse/gitlab-ce-oidc, which is based on the official image, but adds OpenID Connect support.

I added the following config to /srv/gitlab/config/gitlab.rb:

gitlab_rails['omniauth_enabled'] = true

# Must match the args.name (!) of our configured omniauth provider:
gitlab_rails['omniauth_allow_single_sign_on'] = ['mydex']

# By default, third-party authentication results in a newly created
# user which needs to be unblocked by an admin. Disable this
# additional safety mechanism and directly create users:
gitlab_rails['omniauth_block_auto_created_users'] = false

gitlab_rails['omniauth_providers'] = [
  {
    name: 'openid_connect',  # identifies the omniauth gem to use
    label: 'OIDC',
    args: {
      # The name shows up in the GitLab UI in title-case, i.e. “Mydex”,
      # and must match the name in client_options.redirect_uri below
      # and omniauth_allow_single_sign_on above.
      #
      # NOTE that if you change the name after users have already
      # signed up through the provider, you will need to update the
      # “identities” PostgreSQL table accordingly:
      # echo "UPDATE identities SET provider = 'newdex' WHERE \
      #   provider = 'mydex';" | gitlab-psql gitlabhq_production
      'name':          'mydex',

      # Scope must contain “email”.
      'scope':         ['openid', 'profile', 'email'],

      # Discover all endpoints from the issuer, specifically from
      # https://dex.example.net:5554/dex/.well-known/openid-configuration
      'discovery':     true,

      # Must match the issuer configured in dex:
      # Note that http:// URLs did not work in my tests; use https://
      'issuer':        'https://dex.example.net:5554/dex',

      'client_options': {
        # identifier, secret and redirect_uri must match a
	# configured client in dex.
        'identifier':   'example-app',
        'secret':       'ZXhhbXBsZS1hcHAtc2VjcmV0',
        'redirect_uri': 'https://gitlab.example.net/users/auth/mydex/callback'
      }
    }
  }
]

Step 3: patch omniauth-openid-connect

Until dex issue #376 is fixed, the following patch for the omniauth-openid-connect gem is required:

--- /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omniauth-openid-connect-0.2.3/lib/omniauth/strategies/openid_connect.rb.orig	2017-10-21 12:31:50.777602847 +0000
+++ /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omniauth-openid-connect-0.2.3/lib/omniauth/strategies/openid_connect.rb	2017-10-21 12:34:20.063308560 +0000
@@ -42,24 +42,13 @@
       option :send_nonce, true
       option :client_auth_method
 
-      uid { user_info.sub }
-
+      uid { @email }
       info do
-        {
-          name: user_info.name,
-          email: user_info.email,
-          nickname: user_info.preferred_username,
-          first_name: user_info.given_name,
-          last_name: user_info.family_name,
-          gender: user_info.gender,
-          image: user_info.picture,
-          phone: user_info.phone_number,
-          urls: { website: user_info.website }
-        }
+        { email: @email }
       end
 
       extra do
-        {raw_info: user_info.raw_attributes}
+        {raw_info: {}}
       end
 
       credentials do
@@ -165,6 +154,7 @@
               client_id: client_options.identifier,
               nonce: stored_nonce
           )
+          @email = _id_token.raw_attributes['email']
           _access_token
         }.call()
       end

by Michael Stapelberg at 21. October 2017 13:19:00

20. October 2017

Mero’s Blog

A day in the life of an Omnivore

You get an E-Mail. It's an invite to a team dinner. As you have only recently joined this team, it's going to be your first. How exciting! You look forward to get to know your new coworkers better and socialize outside the office. They are nice and friendly people and you are sure it's going to be a great evening. However, you also have a distinct feeling of anxiety and dread in you. Because you know, the first dinner with new people also means you are going to have the conversation again. You know it will come up, whether you want to or not. Because you had it a thousand times - because you are an Omnivore.

You quickly google the place your manager suggested. "Green's Organic Foodery". They don't have a menu on their site, but the name alone makes clear, that meat probably isn't their specialty, exactly. You consider asking for a change of restaurant, but quickly decide that you don't want to get a reputation as a killjoy who forces their habits on everyone just yet. You figure they are going to have something with meat on their menu. And if not, you can always just grab a burger on your way home or throw a steak into a pan when you get back. You copy the event to your calendar and continue working.

At six, everyone gathers to go to dinner together. It's not far, so you decide to just walk. On the way you get to talk to some of your team mates. You talk about Skiing, your home countries, your previous companies. You are having fun - they seem to be easy-going people, you are getting along well. It's going to be an enjoyable night.

You arrive at the restaurant and get seated. The waiter arrives with the menu and picks up your orders for drinks. When they leave, everyone starts flipping through the menu. “You've got to try their Tofu stir-fry. It's amazing”, Kevin tells you. You nod and smile and turn your attention to the booklet in your hand. You quickly take in the symbols decorating some of the items. “G” - safe to assume, these are the gluten-free options. There's also an “O” on a bunch of them. Also familiar, but ambiguous - could be either “Omnivores" or “Ovo-lacto” (containing at least dairy products or eggs), you've seen both usages. There is no legend to help disambiguate and quickly flipping through the rest of the menu, you find no other symbols. Ovo-lacto, then. You are going to have to guess from the names and short descriptions alone, whether they also contain any meat. They have lasagna, marked with an “O”. Of course that's probably just the cheese but they might make it with actual minced beef.

The waiter returns and takes your orders. “The lasagna - what kind is it?”, you ask. You are trying to avoid the O-word as long as you can. “It's Lasagna alla Bolognese, house style”. Uh-oh. House style? “Is it made from cattle, or based on gluten proteins?” (you don't care how awkward you have to phrase your question, you are not saying the magic trigger words!) “Uhm… I'm not sure. I can ask in the kitchen, if you'd like?” “That would be great, thanks”. They leave. Jen, from across the table, smiles at you - are you imagining it, or does it look slightly awkward? You know the next question. “Are you an Omnivore?” “I eat meat, yes”, you say, smiling politely. Frick. Just one meal, is all you wanted. But it had to come up at some point anyway, so fine. “Wow. I'm Ovo-lacto myself. But I couldn't always eat meat, I think. Is it hard?” You notice that your respective seat neighbors have started to listen too. Ovo-lactos aren't a rarity anymore (Onmivores a bit more so, but not that much), but the topic still seems interesting enough to catch attention. You've seen it before. What feels like a hundred thousand times. In fact, you have said exactly the same thing Jen just said, just a year or so ago. Before you just decided to go Omnivore.

“Not really”, you start. “I found it not much harder than when I went Ovo-lacto. You have to get used to it, of course, and pay a little more attention, but usually you find something. Just like when you start eating cheese and eggs.” At that moment, the waiter returns. “I spoke to the chef and we can make the lasagna with beef, if you like”. “Yes please”, you hand the menu back to them with a smile. “I considered going Ovo-lacto”, Mike continues the conversation from the seat next to Jen, “but for now I just try to have some milk every once in a while. Like, in my coffee or cereal. It's not that I don't like it, there are really great dairy products. For example, this place in the city center makes this amazing yogurt. But having it every day just seems very hard“. “Sure”, you simply say. You know they mean well and you don't want to offend them by not being interested; but you also heard these exact literal words at least two dozen times. And always with ample evidence in the room that it's not actually that hard.

“I don't really see the point”, Roy interjects from the end of the table. You make a mental check-mark. “Shouldn't people just eat what they want? I don't get why suddenly we all have to like meat”. It doesn't matter that no one suggested that. “I mean, I do think it's cool if people eat meat”, someone whose name you don't remember adds, “I sometimes think I should eat more eggs myself. But it's just so annoying that you get these Omnivores who try to lecture you about how unhealthy it is to not eat meat or that humans are naturally predisposed to digest meat. I mean, you seem really relaxed about it”, they quickly add as assurance in your direction, “but you can't deny that there are also these Omni-nazis”. You sip on your water, mentally counting backwards from 3. “You know how you find an Omnivore at a party?”, Kevin asks jokingly from your right. “Don't worry, they will tell you”, Rick delivers the punchline for him. How original.

”I totally get Omnivores. If they want to eat meat, that's great. What I don't understand is this weird trend of fake-salad. Like, people get a salad, but then they put french dressing on it, or bacon bits. I mean, if you want a salad, why not just simply have a salad?”. You know the stupidly obvious answer of course and you haven't talked in a while, so you decide to steer the conversation into a more pleasant direction. “It's simple, really. You like salad, right?” “Yeah, of course“ “So, if you like salad, but decide that you also want to eat dairy or meat - doesn't it make sense to get as close to a pure salad as you can? While still staying with your conviction to eat meat? It's a tradeoff, sure, but isn't it better than no salad at all?” There's a brief pause. You can tell that they haven't considered that before. No one has. Which you find baffling. Every single time. “Hm. I guess I haven't thought about it like that before. From that point of view it does kind of make sense. Anyway, I still prefer the real deal”. “That's fine”, you say, smiling “I will continue to eat my salad with french dressing”.

Your food arrives and the conversation continues for a bit, with the usual follow-up questions - do you eat pork too, or just beef? What about dogs? Would you consider eating monkey meat? Or Human? You explain that you don't worry about the exact line, that you are not dogmatic about it and usually just decide based on convenience and what seems obvious (and in luckily, these questions don't usually need an answer in practice anyway). Someone brings up how some of what's labeled as milk actually is made from almonds, because it's cheaper, so you can't even be sure you actually get dairy. But slowly, person by person, the topic shifts back to work, hobbies and family. “How's the lasagna?”, Jen asks. “Great”, you reply with a smile, because it is.

On your way home, you take stock. Overall, the evening went pretty well. You got along great with most of your coworkers and had long, fun conversations. The food ended up delicious, even if you wish they had just properly labeled their menu. You probably are going to have to nudge your team on future occasions, so you go out to Omnivore-friendlier places. But you are also pretty sure they are open to it. Who knows, you might even get them to go to a steak house at some point. You know you are inevitably going to have the conversation again, at some point - whether it will come up at another meal with your team or with a new person, who you eat with for the first time. This time, at least, it went reasonably well.


This post is a work of fiction. ;) Names, characters, places and incidents either are products of the author's imagination or are used fictitiously. Any resemblance to actual events or locales or persons, living or dead, is entirely coincidental.

Also, if we had "the conversation" before, you should know I still love you and don't judge you :) It's just that I had it a thousand times :)

20. October 2017 23:45:00

18. September 2017

michael-herbst.com

Coulomb-Sturmians and molsturm

Yesterday I gave a talk at our annual group retreat at the Darmstädter Haus in Hirschegg, Kleinwalsertal. For this talk I expanded the slides from my lazy matrix talk in Kiel, incorporating a few of the more recent Coulomb-Sturmian results I obtained.

Most importantly I added a few slides to highlight the issues with standard contracted Gaussians and to discuss the fundamental properties of the Coulomb-Sturmians. Furthermore I gave some details why a contraction-based scheme is advantageous if Coulomb-Sturmian basis functions are employed and discussed the design of our molsturm program package and how it allows to perform calculations in a fully contraction-based manor.

Link Licence
Coulomb-Sturmians and molsturm (Slides Kleinwalsertal talk) Creative Commons License

by Michael F. Herbst at 18. September 2017 22:00:00

12. September 2017

Mero’s Blog

Diminishing returns of static typing

I often get into discussions with people, where the matter of strictness and expressiveness of a static type system comes up. The most common one, by far, is Go's lack of generics and the resulting necessity to use interface{} in container types (the container-subpackages are obvious cases, but also context). When I express my view, that the lack of static type-safety for containers isn't a problem, I am treated with condescending reactions ranging from disbelief to patronizing.

I also often take the other side of the argument. This happens commonly, when talking to proponents of dynamically typed languages. In particular I got into debates of whether Python would be suitable for a certain use-case. When the lack of static type-safety is brought up, the proponents of Python defend it by pointing out that it now features optional type hints. Which they say make it possible, to reap the benefits of static typing even in a conventionally dynamically typed language.

This is an attempt to write my thoughts on both of these (though they are not in any way novel or creative) down more thoroughly. Discussions usually don't provide the space for that. They are also often charged and parties are more interested in “winning the argument”, than finding consensus.


I don't think it's particularly controversial, that static typing in general has advantages, even though actual data about those seems to be surprisingly hard to come by. I certainly believe that, it is why I use Go in the first place. There is a difference of opinion though, in how large and important those benefits are and how much of the behavior of a program must be statically checked to reap those benefits.

To understand this, we should first make explicit what the benefits of static type checking are. The most commonly mentioned one is to catch bugs as early in the development process as possible. If a piece of code I write already contains a rigorous proof of correctness in the form of types, just writing it down and compiling it gives me assurance that it will work as intended in all circumstances. At the other end of the spectrum, in a fully dynamic language I will need to write tests exercising all of my code to find bugs. Running tests takes time. Writing good tests that actually cover all intended behavior is hard. And as it's in general impossible to cover all possible execution paths, there will always be the possibility of a rare edge-case that we didn't think of testing to trigger a bug in production.

So, we can think of static typing as increasing the proportion of bug-free lines of code deployed to production. This is of course a simplification. In practice, we would still catch a lot of the bugs via more rigorous testing, QA, canarying and other practices. To a degree we can still subsume these in this simplification though. If we catch a buggy line of code in QA or the canary phase, we are going to roll it back. So in a sense, the proportion of code we wrote that makes it as bug-free into production will still go down. Thus:

This is usually the understanding, that the “more static typing is always better” argument is based on. Checking more behavior at compile time means less bugs in production means more satisfied customers and less being woken up at night by your pager. Everybody's happy.

Why then is it, that we don't all code in Idris, Agda or a similarly strict language? Sure, the graph above is suggestively drawn to taper off, but it's still monotonically increasing. You'd think that this implies more is better. The answer, of course, is that static typing has a cost and that there is no free lunch.

The costs of static typing again come in many forms. It requires more upfront investment in thinking about the correct types. It increases compile times and thus the change-compile-test-repeat cycle. It makes for a steeper learning curve. And more often than we like to admit, the error messages a compiler will give us will decline in usefulness as the power of a type system increases. Again, we can oversimplify and subsume these effects in saying that it reduces our speed:

This is what we mean when we talk about dynamically typed languages being good for rapid prototyping. In the end, however, what we are usually interested in, is what I'd like to call velocity: The speed with which we can deploy new features to our users. We can model that as the speed with which we can roll out bug-free code. Graphically, that is expressed as the product of the previous two graphs:

In practice, the product of these two functions will have a maximum, a sweet spot of maximum velocity. Designing a type system for a programming language is, at least in part, about finding that sweet spot¹.

Now if we are to accept all of this, that opens up a different question: If we are indeed searching for that sweet spot, how do we explain the vast differences in strength of type systems that we use in practice? The answer of course is simple (and I'm sure many of you have already typed it up in an angry response). The curves I drew above are completely made up. Given how hard it is to do empirical research in this space and to actually quantify the measures I used here, it stands to reason that their shape is very much up for interpretation.

A Python developer might very reasonably believe that optional type-annotations are more than enough to achieve most if not all the advantages of static typing. While a Haskell developer might be much better adapted to static typing and not be slowed down by it as much (or even at all). As a result, the perceived sweet spot can vary widely:

What's more, the importance of these factors might vary a lot too. If you are writing avionics code or are programming the control unit for a space craft, you probably want to be pretty darn sure that the code you are deploying is correct. On the other hand, if you are a Silicon Valley startup in your growth-phase, user acquisition will be of a very high priority and you get users by deploying features quicker than your competitors. We can model that, by weighing the factors differently:

Your use case will determine the sweet spot you are looking for and thus the language you will choose. But a language is also designed with a set of use cases in mind and will set its own sweet spot according to that.

I think when we talk about how strict a type system should be, we need to acknowledge these subjective factors. And it is fine to believe that your perception of one of those curves or how they should be weighted is closer to a hypothetical objective reality than another persons. But you should make that belief explicit and provide a justification of why your perception is more realistic. Don't just assume that other people view them the same way and then be confused that they do not come to the same conclusions as you.


Back to Go's type system. In my opinion, Go manages to hit a good sweet spot (that is, its design agrees with my personal preferences on this). To me it seems that Go reaps probably upwards of 90% of the benefits you can get from static typing while still being not too impeding. And while I definitely agree static typing is beneficial, the marginal benefit of making user-defined containers type-safe simply seems pretty low (even if it's positive). In the end, it would probably be less than 1% of Go code that would get this additional type-checking and it is probably pretty obvious code. And meanwhile, I perceive generics as a language feature pretty costly. So I find it hard to justify a large perceived cost with a small perceived benefit.

Now, that is not to say I'm not open to be convinced. Just that simply saying “but more type-safety!” is only looking at one side of the equation and isn't enough. You need to acknowledge that there is no free lunch and that this is a tradeoff. You need to accept that your perceptions of how big the benefit of adding static typing is, how much it costs and how important it is are all subjective. If you want to convince me that my perception of their benefit is wrong, the best way would be to provide specific instances of bugs or production crashes caused by a type-assertion on an interface{} taken out of a container. Or a refactoring you couldn't make because of the lack of type-safety with a specific container. Ideally, this takes the form of an experience report, which I consider an excellent way to talk about engineered tradeoffs.

Of course you can continue to roll your eyes whenever someone questions your perception of the value-curve of static typing. Or pretend that when I say the marginal benefit of type-safe containers is small, I am implying that the total benefit of static typing is small. It's an effective debate-tactic, if your goal is to shut up your opposition. But not if your goal is to convince them and build consensus.


[1] There is a generous and broad exception for research languages here. If the point of your design is to explore the possibility space of type-systems, matters of practicality can of course often be ignored.

12. September 2017 11:05:00

11. September 2017

michael-herbst.com

Advanced bash scripting 2017

I am very happy to announce that my graduate school asked me to repeat the block course about bash scripting, which I first taught in 2015.

In the course we will take a structured look at UNIX shell scripting from the bottom up. We will revise some elements about the UNIX operating system infrastructure, discuss handy features of the shell as well as its syntax elements. Whilst the main focus of the course is the bash shell, we will look at other common utility programs like grep, sed and awk as well as they are very handy in the context of scripting. Last but not least we will discuss common pitfalls and how they can be avoided.

The course runs from 6th till 10th November 2017 at Heidelberg University. You can find further information on the "Advanced bash scripting" course website.

As usual all course material will be published both on the course website as well as the course github repository afterwards.

Update (29/09/2017): Registration is now open.

by Michael F. Herbst at 11. September 2017 08:00:00

05. September 2017

Mero’s Blog

Gendered Marbles

tl;dr: "Some marbles, apparently, have a gender. And they seem to be overwhelmingly male."

A couple of days ago The MarbleLympics 2017 popped into my twitter stream. In case you are unaware (I certainly was): It's a series of videos where a bunch of marbles participate in a made-up Olympics. They are split into teams that then participate in a series of "competitions" in a variety of different events. The whole event is professionally filmed, cut and overlaid both with fake noises from spectators and a well-made, engaging sports commentary. It is really fun to watch. I don't know why, but I find it way more captivating than watching actual sports. I thoroughly recommend it.

Around event 8 (high jump) though, I suddenly noticed that the commentator would occasionally not only personify but actually gender marbles. For most of the commentary he just refers to the teams as a whole with a generic "they". But every once in a while - and especially often during the high-jump event - he would use a singular gendered pronoun. Also, that only really occurred to me when he referred to one of the marbles as "she".

This instantly became one of those things that after noticing it, I couldn't unnotice it. It's not so much that it matters. But from then on, I couldn't stop listening up every time a singular pronoun was used.

Well, you know where this is going. Fully aware of how much of a waste of my time this is, I sat down and counted. More specifically, I downloaded the closed captions of all the videos and grepped through them for pronouns. I did double-check all findings and here is what I found: By my count, 20 distinct marbles are referred to by singular pronouns (yes. I noted their names to filter duplicates. Also I kind of hoped to find a genderfluid marble to be honest). Here is an alphabetized list of gendered marbles:

  • Aqua (Oceanics) - Male
  • Argent (Quicksilvers) - Male
  • Clementin (O'Rangers) - Male
  • Cocoa (Chocolatiers) - Male (in two events)
  • Cosmo (Team Galactic) - Male
  • Imar (Primaries) - Male
  • Jump (Jungle Jumpers) - Male
  • Leap (Jungle Jumpers) - Male
  • Mandarin (O'Rangers) - Male (in two events)
  • Mary (Team Primary) - Female
  • Mercurial (Quicksilvers) - Male
  • Mimo (Team Momo) - Male
  • Momo Momo (Team Momo) - Male (in three events)
  • Pinky Winky (Pinkies) - Male
  • Rapidly (Savage Speeders) - Male
  • Taffy (Jawbreakers) - Male
  • Wespy (Midnight Wisps) - Male
  • Whizzy (Savage Speeders) - Male
  • Yellah (Mellow Yellow) - Male
  • Yellup (Mellow Yellow) - Male

As you can see, the overwhelming majority of gendered marbles are men. There is exactly one exception: Mary. From what I can tell, that's because it's the only name that has clear gender associations. All the other names probably could go either way. And marbles obviously have no gender. They are as non-gendered an object as you could imagine. And yet there seems to be a default assumption that athletic marbles would be men.

Obviously this doesn't matter. Obviously you can't discriminate marbles. You can't misgender them or hurt their feelings. Obviously the commentator didn't sit down and made a list of all the marbles and assigned 95% of them a male gender - it's clearly just an ad-hoc subconscious assignment. And to be absolutely clear: I do not try to fault the makers of these videos at all. They did nothing wrong. It's a ludicrous expectation for them to sit down and make sure that they assign balanced genders to their marbles.

But I do find it an interesting observation. I do think it reflects an implicit, unconscious bias in a striking way. I also think it illustrates nicely that gender bias in language isn't exclusive to languages like German, where all nouns are gendered (take note, German friends). Of course none of this is news. This kind of unconscious gender bias in language is well-researched and documented. It's just that once you know about it, you can't stop noticing the evidence for it popping up everywhere. Even with marbles.

And all of that being said: Yes, I am also aware that all of this is slightly ridiculous.


PS: In case the team behind the MarbleLympics are reading this: Really, thank you for the videos :) They are great.

05. September 2017 23:22:00

19. August 2017

sECuREs Webseite

Why Go is my favorite programming language

I strive to respect everybody’s personal preferences, so I usually steer clear of debates about which is the best programming language, text editor or operating system.

However, recently I was asked a couple of times why I like and use a lot of Go, so here is a coherent article to fill in the blanks of my ad-hoc in-person ramblings :-).

My background

I have used C and Perl for a number of decently sized projects. I have written programs in Python, Ruby, C++, CHICKEN Scheme, Emacs Lisp, Rust and Java (for Android only). I understand a bit of Lua, PHP, Erlang and Haskell. In a previous life, I developed a number of programs using Delphi.

I had a brief look at Go in 2009, when it was first released. I seriously started using the language when Go 1.0 was released in 2012, featuring the Go 1 compatibility guarantee. I still have code running in production which I authored in 2012, largely untouched.

1. Clarity

Formatting

Go code, by convention, is formatted using the gofmt tool. Programmatically formatting code is not a new idea, but contrary to its predecessors, gofmt supports precisely one canonical style.

Having all code formatted the same way makes reading code easier; the code feels familiar. This helps not only when reading the standard library or Go compiler, but also when working with many code bases — think Open Source, or big companies.

Further, auto-formatting is a huge time-saver during code reviews, as it eliminates an entire dimension in which code could be reviewed before: now, you can just let your continuous integration system verify that gofmt produces no diffs.

Interestingly enough, having my editor apply gofmt when saving a file has changed the way I write code. I used to attempt to match what the formatter would enforce, then have it correct my mistakes. Nowadays, I express my thought as quickly as possible and trust gofmt to make it pretty (example of what I would type, click Format).

High-quality code

I use the standard library (docs, source) quite a bit, see below.

All standard library code which I have read so far was of extremely high quality.

One example is the image/jpeg package: I didn’t know how JPEG worked at the time, but it was easy to pick up by switching between the Wikipedia JPEG article and the image/jpeg code. If the package had a few more comments, I would qualify it as a teaching implementation.

Opinions

I have come to agree with many opinions the Go community holds, such as:

Few keywords and abstraction layers

The Go specification lists only 25 keywords, which I can easily keep in my head.

The same is true for builtin functions and types.

In my experience, the small number of abstraction layers and concepts makes the language easy to pick up and quickly feel comfortable in.

While we’re talking about it: I was surprised about how readable the Go specification is. It really seems to target programmers (rather than standards committees?).

2. Speed

Quick feedback / low latency

I love quick feedback: I appreciate websites which load quickly, I prefer fluent User Interfaces which don’t lag, and I will choose a quick tool over a more powerful tool any day. The findings of large web properties confirm that this behavior is shared by many.

The authors of the Go compiler respect my desire for low latency: compilation speed matters to them, and new optimizations are carefully weighed against whether they will slow down compilation.

A friend of mine had not used Go before. After installing the RobustIRC bridge using go get, they concluded that Go must be an interpreted language and I had to correct them: no, the Go compiler just is that fast.

Most Go tools are no exception, e.g. gofmt or goimports are blazingly fast.

Maximum resource usage

For batch applications (as opposed to interactive applications), utilizing the available resources to their fullest is usually more important than low latency.

It is delightfully easy to profile and change a Go program to utilize all available IOPS, network bandwidth or compute. As an example, I wrote about filling a 1 Gbps link, and optimized debiman to utilize all available resources, reducing its runtime by hours.

3. Rich standard library

The Go standard library provides means to effectively use common communications protocols and data storage formats/mechanisms, such as TCP/IP, HTTP, JPEG, SQL, …

Go’s standard library is the best one I have ever seen. I perceive it as well-organized, clean, small, yet comprehensive: I often find it possible to write reasonably sized programs with just the standard library, plus one or two external packages.

Domain-specific data types and algorithms are (in general) not included and live outside the standard library, e.g. golang.org/x/net/html. The golang.org/x namespace also serves as a staging area for new code before it enters the standard library: the Go 1 compatibility guarantee precludes any breaking changes, even if they are clearly worthwhile. A prominent example is golang.org/x/crypto/ssh, which had to break existing code to establish a more secure default.

4. Tooling

To download, compile, install and update Go packages, I use the go get tool.

All Go code bases I have worked with use the built-in testing facilities. This results not only in easy and fast testing, but also in coverage reports being readily available.

Whenever a program uses more resources than expected, I fire up pprof. See this golang.org blog post about pprof for an introduction, or my blog post about optimizing Debian Code Search. After importing the net/http/pprof package, you can profile your server while it’s running, without recompilation or restarting.

Cross-compilation is as easy as setting the GOARCH environment variable, e.g. GOARCH=arm64 for targeting the Raspberry Pi 3. Notably, tools just work cross-platform, too! For example, I can profile gokrazy from my amd64 computer: go tool pprof ~/go/bin/linux_arm64/dhcp http://gokrazy:3112/debug/pprof/heap.

godoc displays documentation as plain text or serves it via HTTP. godoc.org is a public instance, but I run a local one to use while offline or for not yet published packages.

Note that these are standard tools coming with the language. Coming from C, each of the above would be a significant feat to accomplish. In Go, we take them for granted.

Getting started

Hopefully I was able to convey why I’m happy working with Go.

If you’re interested in getting started with Go, check out the beginner’s resources we point people to when they join the Gophers slack channel. See https://golang.org/help/.

Caveats

Of course, no programming tool is entirely free of problems. Given that this article explains why Go is my favorite programming language, it focuses on the positives. I will mention a few issues in passing, though:

  • If you use Go packages which don’t offer a stable API, you might want to use a specific, known-working version. Your best bet is the dep tool, which is not part of the language at the time of writing.
  • Idiomatic Go code does not necessarily translate to the highest performance machine code, and the runtime comes at a (small) cost. In the rare cases where I found performance lacking, I successfully resorted to cgo or assembler. If your domain is hard-realtime applications or otherwise extremely performance-critical code, your mileage may vary, though.
  • I wrote that the Go standard library is the best I have ever seen, but that doesn’t mean it doesn’t have any problems. One example is complicated handling of comments when modifying Go code programmatically via one of the standard library’s oldest packages, go/ast.

by Michael Stapelberg at 19. August 2017 11:00:00

18. August 2017

RaumZeitLabor

Analogspieleabend im RaumZeitLabor

Am 30.09.2017 findet der erste Analogspieleabend im RaumZeitLabor statt. Von Brettspiele über Kartenspiele bis hin zu Rätselspielen und sonstigem hätten wir gerne alles dabei. Deshalb bringt auch eure eigenen Spiele mit! Wir werden dann gemeinsam in netter Runde, zu Mate und Tiefkühlpizza und anderen traditionell nerdigen Speisen, Spiele spielen, wie es im 20. Jahrhundert noch tradition war. Die Veranstaltung beginnt um 17 Uhr. Der Eintritt ist selbstverständlich frei, wir würden uns aber über Spenden freuen. Vorbei ist es, wenn der letzte geht. Ihr wart noch nie im RaumZeitLabor? Schaut euch [Anfahrtbeschreibung](https://raumzeitlabor.de/kontakt/anfahrt/) an und das wichtigste: Schreibt euch die Nummer vom Raum (+49 621 76 23 13 70) auf, damit wir euch helfen können, solltet ihr den Weg nicht finden.

by uwap at 18. August 2017 00:00:00

14. August 2017

Mero’s Blog

Why context.Value matters and how to improve it

tl;dr: I think context.Value solves the important use case of writing stateless - and thus scalable - abstractions. I believe dynamic scoping could provide the same benefits while solving most of the criticism of the current implementation. I thus try to steer the discussion away from the concrete implementation and towards the underlying problem.

This blog post is relatively long. I encourage you to skip sections you find boring


Lately this blogpost has been discussed in several Go forums. It brings up several good arguments against the context-package:

  • It requires every intermediate functions to include a context.Context even if they themselves do not use it. This introduces clutter into APIs and requires extensive plumbing. Additionally, ctx context.Context "stutters".
  • context.Value is not statically type-safe, requiring type-assertions.
  • It does not allow you to express critical dependencies on context-contents statically.
  • It's susceptible to name collisions due to requiring a global namespace.
  • It's a map implemented as a linked list and thus inefficient.

However, I don't think the post is doing a good enough job to discuss the problems context was designed to solve. It explicitly focuses on cancellation. Context.Value is discarded by simply stating that

[…] designing your APIs without ctx.Value in mind at all makes it always possible to come up with alternatives.

I think this is not doing this question justice. To have a reasoned argument about context.Value there need to be consideration for both sides involved. No matter what your opinion on the current API is: The fact that seasoned, intelligent engineers felt the need - after significant thought - for Context.Value should already imply that the question deserves more attention.

I'm going to try to describe my view on what kind of problems the context package tries to address, what alternatives currently exist and why I find them insufficient and I'm trying to describe an alternative design for a future evolution of the language. It would solve the same problems while avoiding some of the learned downsides of the context package. It is not meant as a specific proposal for Go 2 (I consider that way premature at this point) but just to show that a balanced view can show up alternatives in the design space and make it easier to consider all options.


The problem context sets out to solve is one of abstracting a problem into independently executing units handled by different parts of a system. And how to scope data to one of these units in this scenario. It's hard to clearly define the abstraction I am talking about. So I'm instead going to give some examples.

  • When you build a scalable web service you will probably have a stateless frontend server that does things like authentication, verification and parsing for you. This allows you to scale up the external interface effortlessly and thus also gracefully fall back if the load increases past what the backends can handle. By treating requests as independent from each other you can load-balance them freely between your frontends.
  • Microservices split a large application into small individual pieces that each process individual requests, each potentially branching out into more requests to other services. The requests will usually be independent, making it easy to scale individual microservices up and down based on demand, to load-balance between instances and to solve problems in transparent proxies.
  • Functions as a Service goes one step further: You write single stateless functions that transform data and the platform will make them scale and execute efficiently.
  • Even CSP, the concurrency model built into Go, can be viewed through that lens. The programmer expresses her problem as individually executing "processes" and the runtime will execute them efficiently.
  • Functional Programming as a paradigm calls this "purity". The concept that a functions result may only depend on its input parameters means not much more than the absence of shared state and independent execution.
  • The design of a Request Oriented Collector for Go plays exactly into the same assumptions and ideas.

The idea in all these cases is to increase scaling (whether distributed among machines, between threads or just in code) by reducing shared state while maintaining shared usage of resources.

Go takes a measured approach to this. It doesn't go as far as some functional programming languages to forbid or discourage mutable state. It allows sharing memory between threads and synchronizing with mutexes instead of relying purely on channels. But it also definitely tries to be a (if not the) language to write modern, scalable services in. As such, it needs to be a good language to write this kind of stateless services. It needs to be able to make requests the level of isolation instead of the process. At least to a degree.

(Side note: This seems to play into the statement of the author of above article, who claims that context is mainly useful for server authors. I disagree though. The general abstraction happens on many levels. E.g. a click in a GUI counts just as much as a "request" for this abstraction as an HTTP request)

This brings with it the requirement of being able to store some data on a request-level. A simple example for this would be authentication in an RPC framework. Different requests will have different capabilities. If a request originates from an administrator it should have higher privileges than if it originates from an unauthenticated user. This is fundamentally request scoped data. Not process, service or application scoped. And the RPC framework should treat this data as opaque. It is application specific not only how that data looks en détail but also what kinds of data it requires.

Just like an HTTP proxy or framework should not need to know about request parameters or headers it doesn't consume, an RPC framework shouldn't know about request scoped data the application needs.


Let's try to look at specific ways this problem is (or could be) solved without involving context. As an example, let's look at the problem of writing an HTTP middleware. We want to be able to wrap an http.Handler (or a variation thereof) in a way that allows the wrapper to attach data to a request.

To get static type-safety we could try to add some type to our handlers. We could have a type containing all the data we want to keep request scoped and pass that through our handlers:

type Data struct {
    Username string
    Log *log.Logger
    // …
}

func HandleA(d Data, res http.ResponseWriter, req *http.Request) {
    // …
    d.Username = "admin"
    HandleB(d, req, res)
    // …
}

func HandleB(d Data, res http.ResponseWriter, req *http.Request) {
    // …
}

However, this would prevent us from writing reusable Middleware. Any such middleware would need to make it possible to wrap HandleA. But as it's supposed to be reusable, it can't know the type of the Data parameter. You could make the Data parameter an interface{} and require type-assertion. But that wouldn't allow the middleware to inject its own data. You might think that interface type-assertions could solve this, but they have their own set of problems. In the end, this approach won't bring you actual additional type safety.

We could store our state keyed by requests. For example, an authentication middleware could do

type Authenticator struct {
    mu sync.Mutex
    users map[*http.Request]string
    wrapped http.Handler
}

func (a *Authenticator) ServeHTTP(res http.ResponseWriter, req *http.Request) {
    // …
    a.mu.Lock()
    a.users[req] = "admin"
    a.mu.Unlock()
    defer func() {
        a.mu.Lock()
        delete(a.users, req)
        a.mu.Unlock()
    }()
    a.wrapped.ServeHTTP(res, req)
}

func (a *Authenticator) Username(req *http.Request) string {
    a.mu.Lock()
    defer a.mu.Unlock()
    return a.users[req]
}

This has some advantages over context:

  • It is more type-safe.
  • While we still can't express a requirement on an authenticated user statically, we can express a requirement on an Authenticator
  • It's not susceptible to name-collisions anymore.

However, we bought this with shared mutable state and the associated lock contention. It can also break in subtle ways, if one of the intermediate handlers decides to create a new Request - as http.StripPrefix is going to do soon.

Lastly, we might consider to store this data in the *http.Request itself, for example by adding it as a stringified URL parameter. This too has several downsides, though. In fact it checks almost every single item from our list of downsides of context.Context. The exception is being a linked list. But even that advantage we buy with a lack of thread safety. If that request is passed to a handler in a different goroutine we get into trouble.

(Side note: All of this also gives us a good idea of why the context package is implemented as a linked list. It allows all the data stored in it to be read-only and thus inherently thread-safe. There will never be any lock-contention around the shared state saved in a context.Context, because there will never be any need for locks)

So we see that it is really hard (if not impossible) to solve this problem of having data attached to requests in independently executing handlers while also doing significantly better than with context.Value. Whether you believe this a problem worth solving or not is debatable. But if you want to get this kind of scalable abstraction you will have to rely on something like context.Value.


No matter whether you are now convinced of the usefulness of context.Value or still doubtful: The disadvantages can clearly not be ignored in either case. But we can try to find a way to improve on it. To eliminate some of the disadvantages while still keeping its useful attributes.

One way to do that (in Go 2) would be to introduce dynamically scoped variables. Semantically, each dynamically scoped variable represents a separate stack. Every time you change its value the new one is pushed to the stack. It is pop'ed off again after your function returns. For example:

// Let's make up syntax! Only a tiny bit, though.
dyn x = 23

func Foo() {
    fmt.Println("Foo:", x)
}

func Bar() {
    fmt.Println("Bar:", x)
    x = 42
    fmt.Println("Bar:", x)
    Baz()
    fmt.Println("Bar:", x)
}

func Baz() {
    fmt.Println("Baz:", x)
    x = 1337
    fmt.Println("Baz:", x)
}

func main() {
    fmt.Println("main:", x)
    Foo()
    Bar()
    Baz()
    fmt.Println("main:", x)
}

// Output:
main: 23
Foo: 23
Bar: 23
Bar: 42
Baz: 42
Baz: 1337
Bar: 42
Baz: 23
Baz: 1337
main: 23

There are several notes about what I would imagine the semantics to be here.

  • I would only allow dyn-declarations at package scope. Given that there is no way to refer to a local identifier of a different function, that seems logical.
  • A newly spawned goroutine would inherit the dynamic values of its parent function. If we implement them (like context.Context) via linked lists, the shared data will be read-only. The head-pointer would need to be stored in some kind of goroutine-local storage. Thus, writes only ever modify this local storage (and the global heap), so wouldn't need to be synchronized specifically.
  • The dynamic scoping would be independent of the package the variable is declared in. That is, if foo.A modifies a dynamic bar.X, then that modification is visible to all subsequent callees of foo.A, whether they are in bar or not.
  • Dynamically scoped variables would likely not be addressable. Otherwise we'd loose concurrency safety and the clear "down-stack" semantics of dynamic scoping. It would still be possible to declare dyn x *int though and thus get mutable state to pass on.
  • The compiler would allocate the necessary storage for the stacks, initialized to their initializers and emit the necessary instructions to push and pop values on writes and returns. To account for panics and early returns, a mechanism like defer would be needed.
  • There is some confusing overlap with package-scoped variables in this design. Most notably, from seeing foo.X = Y you wouldn't be able to tell whether foo.X is dynamically scoped or not. Personally, I would address that by removing package-scoped variables from the language. They could still be emulated by declaring a dynamically-scoped pointer and never modifying it. Its pointee is then a shared variable. But most usages of package-scoped variables would probably just use dynamically scoped variables.

It is instructive to compare this design against the list of disadvantages identified for context.

  • API clutter would be removed, as request-scoped data would now be part of the language without needing explicit passing.
  • Dynamically scoped variables are statically type-safe. Every dyn declaration has an unambiguous type.
  • It would still not be possible to express critical dependencies on dynamically scoped variables but they also couldn't be absent. At worst they'll have their zero value.
  • Name collision is eliminated. Identifiers are, just like variable names, properly scoped.
  • While a naive implementation would still use linked lists, they wouldn't be inefficient. Every dyn declaration gets its own list and only the head-pointer ever needs to be operated on.
  • The design is still "magic" to a degree. But that "magic" is problem-inherent (at least if I understand the criticism correctly). The magic is exactly the possibility to pass values transparently through API boundaries.

Lastly, I'd like to mention cancellation. While the author of above post dedicates most of it to cancellation, I have so far mostly ignored it. That's because I believe cancellation to be trivially implementable on top of a good context.Value implementation. For example:

// $GOROOT/src/done
package done

// C is closed when the current execution context (e.g. request) should be
// cancelled.
dyn C <-chan struct{}

// CancelFunc returns a channel that gets closed, when C is closed or cancel is
// called.
func CancelFunc() (c <-chan struct, cancel func()) {
    // Note: We can't modify C here, because it is dynamically scoped, which is
    // why we return a new channel that the caller should store.
    ch := make(chan struct)

    var o sync.Once
    cancel = func() { o.Do(close(ch)) }
    if C != nil {
        go func() {
            <-C
            cancel()
        }()
    }
    return ch, cancel
}

// $GOPATH/example.com/foo
package foo

func Foo() {
    var cancel func()
    done.C, cancel = done.CancelFunc()
    defer cancel()
    // Do things
}

This cancellation mechanism would now be usable from any library that wants it without needing any explicit support in its API. This would also make it easy to add cancellation capabilities retroactively.


Whether you like this design or not, it demonstrates that we shouldn't rush to calling for the removal of context. Removing it is only one possible solution to its downsides.

If the removal of context.Context actually comes up, the question we should ask is "do we want a canonical way to manage request-scoped values and at what cost". Only then should we ask what the best implementation of this would be or whether to remove the current one.

14. August 2017 00:17:25

06. August 2017

Mero’s Blog

What I want from a logging API

This is intended as an Experience Report about logging in Go. There are many like it but this one is mine.

I have been trying for a while now to find (or build) a logging API in Go that fills my needs. There are several things that make this hard to get "right" though. This is my attempt to describe them coherently in one place.

When I say "logging", I mean informational text messages for human consumption used to debug a specific problem. There is an idea currently gaining traction in the Go community called "structured logging". logrus is a popular package that implements this idea. If you haven't heard of it, you might want to skim its README. And while I definitely agree that log-messages should contain some structural information that is useful for later filtering (like the current time or a request ID), I believe the idea as often advocated is somewhat misguided and conflates different use cases that are better addressed otherwise. For example, if you are tempted to add a structured field to your log containing an HTTP response code to alert on too many errors, you probably want to use metrics and timeseries instead. If you want to follow a field through a variety of systems, you probably want to annotate a trace. If you want analytics like calculating daily active users or what used user-agents are used how often, you probably want what I like to call request annotations, as these are properties of a request, not of a log-line. If you exclude all these use cases, there isn't a lot left for structured logging to address.

The logs I am talking about is to give a user or the operator of a software more insight into what is going on under the covers. The default assumption is, that they are not looked at until something goes wrong: Be it a test failing, an alerting system notifying of an issue or a bug report being investigated or a CLI not doing what the user expected. As such it is important that they are verbose to a certain degree. As an operator, I don't want to find out that I can't troubleshoot a problem because someone did not log a critical piece of information. An API that requires (or encourages) me to only log structured data will ultimately only discourage me from logging at all. In the end, some form of log.Debugf("Error reading foo: %v", err) is the perfect API for my use case. Any structured information needed to make this call practically useful should be part of the setup phase of whatever log is.

The next somewhat contentious question is whether or not the API should support log levels (and if so, which). My personal short answer is "yes and the log levels should be Error, Info and Debug". I could try and justify these specific choices but I don't think that really helps; chalk it up to personal preference if you like. I believe having some variation on the verbosity of logs is very important. A CLI should be quiet by default but be able to tell the user more specifically where things went wrong on request. A service should be debuggable in depth, but unconditionally logging verbosely would have in unacceptable latency impact in production and too heavy storage costs. There need to be some logs by default though, to get quick insights during an emergency or in retrospect. So, those three levels seem fine to me.

Lastly what I need from a logging API, is the possibility to set up verbosity and log sinks both horizontally and vertically. What I mean by that is that software is usually build in layers. They could be individual microservices, Go packages or types. Requests will then traverse these layers vertically, possibly branching out and interleaved to various degrees.

Request forest

Depending on what and how I am debugging, it makes sense to increase the log verbosity of a particular layer (say I narrowed down the problem to shared state in a particular handler and want to see what happens to that state during multiple requests) or for a particular request (say, I narrowed down a problem to "requests which have header FOO set to BAR" and want to follow one of them to get a detailed view of what it does). Same with logging sinks, for example, a request initiated by a test should get logged to its *testing.T with maximum verbosity, so that I get a detailed context about it if and only if the test fails to immediately start debugging. These settings should be possible during runtime without a restart. If I am debugging a production issue, I don't want to change a command line flag and restart the service.

Let's try to implement such an API.

We can first narrow down the design space a bit, because we want to use testing.T as a logging sink. A T has several methods that would suit our needs well, most notably Logf. This suggest an interface for logging sinks that looks somewhat like this:

type Logger interface {
    Logf(format string, v ...interface{})
}

type simpleLogger struct {
    w io.Writer
}

func (l simpleLogger) Logf(format string, v ...interface{}) {
    fmt.Fprintf(l.w, format, v...)
}

func NewLogger(w io.Writer) Logger {
    return simpleLogger{w}
}

This has the additional advantage, that we can add easily implement a Discard-sink, that has minimal overhead (not even the allocations of formatting the message):

type Discard struct{}

func (Discard) Logf(format string, v ...interface{}) {}

The next step is to get leveled logging. The easiest way to achieve this is probably

type Logs struct {
    Debug Logger
    Info Logger
    Error Logger
}

func DiscardAll() Logs {
    return Logs{
        Debug: Discard{},
        Info: Discard{},
        Error: Discard{},
    }
}

By putting a struct like this (or its constituent fields) as members of a handler, type or package, we can get the horizontal configurability we are interested in.

To get vertical configurability we can use context.Value - as much as it's frowned upon by some, it is the canonical way to get request-scoped behavior/data in Go. So, let's add this to our API:

type ctxLogs struct{}

func WithLogs(ctx context.Context, l Logs) context.Context {
    return context.WithValue(ctx, ctxLogs{}, l)
}

func GetLogs(ctx context.Context, def Logs) Logs {
    // If no Logs are in the context, we default to its zero-value,
    // by using the ,ok version of a type-assertion and throwing away
    // the ok.
    l, _ := ctx.Value(ctxLogs{}).(Logs)
    if l.Debug == nil {
        l.Debug = def.Debug
    }
    if l.Info == nil {
        l.Info = def.Info
    }
    if l.Error == nil {
        l.Error = def.Error
    }
    return l
}

So far, this is a sane, simple and easy to use logging API. For example:

type App struct {
    L log.Logs
}

func (a *App) ServeHTTP(res http.ResponseWriter, req *http.Request) {
    l := log.GetLogs(req.Context(), a.L)
    l.Debug.Logf("%s %s", req.Method, req.URL.Path)
    // ...
}

The issue with this API, however, is that it is completely inflexible, if we want to preserve useful information like the file and line number of the caller. Say, I want to implement the equivalent of io.MultiWriter. For example, I want to write logs both to os.Stderr and to a file and to a network log service.

I might try to implement that via

func MultiLogger(ls ...Logger) Logger {
    return multiLog{ls}
}

type multiLog struct {
    loggers []Logger
}

func (m *multiLog) Logf(format string, v ...interface{}) {
    for _, l := range m.loggers {
        m.Logf(format, v...)
    }
}

However, now the caller of Logf of the individual loggers will be the line in (*multiLog).Logf, not the line of its caller. Thus, caller information will be useless. There are two APIs currently existing in the stdlib to work around this:

  1. (testing.T).Helper (from Go 1.9) lets you mark a frame as a test-helper. When the caller-information is then added to the log-output, all frames marked as a helper is skipped. So, theoretically, we could add a Helper method to our Logger interface and require that to be called in each wrapper. However, Helper itself uses the same caller-information. So all wrappers must call the Helper method of the underlying `testing.T`*, without any wrapping methods. Even embedding doesn't help, as the Go compiler creates an implicit wrapper for that.
  2. (log.Logger).Output lets you specify a number of call-frames to skip. We could add a similar method to our log sink interface. And wrapping loggers would then need to increment the passed in number, when calling a wrapped sink. It's possible to do this, but it wouldn't help with test-logs.

This is a very similar problem to the ones I wrote about last week. For now, I am using the technique I described as Extraction Methods. That is, the modified API is now this:

// Logger is a logging sink.
type Logger interface {
    // Logf logs a text message with the given format and values to the sink.
    Logf(format string, v ...interface{})

    // Helpers returns a list of Helpers to call into from all helper methods,
    // when wrapping this Logger. This is used to skip frames of logging
    // helpers when determining caller information.
    Helpers() []Helper
}

type Helper interface {
    // Helper marks the current frame as a helper method. It is then skipped
    // when determining caller information during logging.
    Helper()
}

// Callers can be used as a Helper for log sinks who want to log caller
// information. An empty Callers is valid and ready for use.
type Callers struct {
    // ...
}

// Helper marks the calling method as a helper. When using Callers in a
// Logger, you should also call this to mark your methods as helpers.
func (*Callers) Helper() {
    // ...
}

type Caller struct {
    Name string
    File string
    Line int
}

// Caller can be used to determine the caller of Logf in a Logger, skipping all
// frames marked via Helper.
func (*Callers) Caller() Caller {
    // ...
}

// TestingT is a subset of the methods of *testing.T, so that this package
// doesn't need to import testing.
type TestingT interface {
    Logf(format string, v ...interface{})
    Helper()
}

// Testing returns a Logger that logs to t. Log lines are discarded, if the
// test succeeds.
func Testing(t TestingT) Logger {
    return testLogger{t}
}

type testLogger struct {
    t TestingT
}

func (l testLogger) Logf(format string, v ...interface{}) {
    l.t.Helper()
    l.t.Logf(format, v...)
}

func (l testLogger) Helpers() []Helper {
    return []Helper{l.t}
}

// New returns a logger writing to w, prepending caller-information.
func New(w io.Writer) Logger {
    return simple{w, new(Callers)}
}

type simple struct {
    w io.Writer
    c *Callers
}

func (l *simple) Logf(format string, v ...interface{}) {
    l.c.Helper()
    c := l.c.Caller()
    fmt.Fprintf(l.w, "%s:%d: " + format, append([]interface{}{c.File, c.Line}, v...)...)
}

func (l *simple) Helpers() []Helper {
    return []Helper{l.c}
}

// Discard discards all logs.
func Discard() Logger {
    return discard{}
}

type discard struct{}

func (Discard) Logf(format string, v ...interface{}) {
}

func (Discard) Helpers() []Helper {
    return nil
}

// MultiLogger duplicates all Logf-calls to a list of loggers.
func MultiLogger(ls ...Logger) Logger {
    var m multiLogger
    for _, l := range ls {
        m.helpers = append(m.helpers, l.Helpers()...)
    }
    m.loggers = ls
    return m
}

type multiLogger struct {
    loggers []Logger
    helpers []Helper
}

func (m multiLogger) Logf(format string, v ...interface{}) {
    for _, h := range m.helpers {
        h.Helper()
    }
    for _, l := range m.loggers {
        l.Logf(format, v...)
    }
}

func (m multiLogger) Helpers() []Helper {
    return m.helpers
}

It's a kind of clunky API and I have no idea about the performance implications of all the Helper-code. But it does work, so it is, what I ended up with for now. Notably, it puts the implementation complexity into the implementers of Logger, in favor of making the actual consumers of them as simple as possible.

06. August 2017 20:08:56

01. August 2017

RaumZeitLabor

Konservieren, Sammeln, Bewahren - Das RZL im ISG

Am vergangenen Donnerstag hatten wir die Gelegenheit, das Stadtarchiv Mannheim etwas genauer unter die Lupe zu nehmen.

Vom großen Mannheim-Luftbild-Teppich im Foyer des Collini-Centers aus, ging es direkt in die heiligen Registerhallen im ersten Stock. Dort durften Ratsprotokollbücher (einige mit mehr als 10 kg Gewicht!) aus den letzten 356 Jahren bestaunt und mehrere Tonnen Rollregal händisch bewegt werden. Akten der städtischen Ämter, Bauunterlagen und historisches Karten- und Plakatmaterial lagern hier in säurefesten Kartons und Magazinen – Insgesamt mehr als 10 Kilometer Archivmaterial, wenn man es aneinanderreihen würde.

Weiter ging’s ins Digitalisierungszentrum des Stadtarchivs: Wertvolle und unersetzliche Dokumente werden dort unter anderem mit Hilfe von Großformat- und Buchscanner für die Zukunft konserviert. Trotz moderner Hilfsmittel werden viele Dokumente noch händisch digitalisiert – OCR funktioniert leider bei vielen Handschriften nicht.

Wer im Onlinearchiv recherchieren oder Einblick in Archivgut nehmen möchte, den führt der Weg in einen der Lesesäle. Nach einem Besuch im Fritz-Cahn-Garnier-Lesesaal endete unsere Tour vor der Informationswand zum bevorstehenden Umzug in den Ochsenpferchbunker.

Unser Dank gilt Herrn Dr. Stockert und Herrn Dr. Schenk für die tolle und informative Tour - wir freuen uns aufs neue „Marchivum“ in der Neckarstadt!

Stadtarchiv Impressionen

by flederrattie at 01. August 2017 00:00:00

30. July 2017

Mero’s Blog

The trouble with optional interfaces

tl;dr: I take a look at the pattern of optional interfaces in Go: what they are used for, why they are bad and what we can do about it.

Note: I wrote most of this article on Wednesday, with the intention to finish and publish it on the weekend. While I was sleeping, Jack Lindamood published this post, which talks about much of the same problems. This was the exact moment I saw that post :) I decided, to publish this anyway; it contains, in my opinion, enough additional content, to be worth it. But I do encourage to (also?) read his post.

What are optional interfaces?

Optional interfaces are interfaces which can optionally be extended by implementing some other interface. A good example is http.Flusher (and similar), which is optionally implemented by an http.ResponseWriter. If a request comes in via HTTP/2, the ResponseWriter will implement this interface to support HTTP/2 Server Push. But as not all requests will be over HTTP/2, this isn't part of the normal ResponseWriter interface and instead provided via an optional interface that needs to be type-asserted at runtime.

In general, whenever some piece of code is doing a type-assertion with an interface type (that is, use an expression v.(T), where T is an interface type), it is very likely offering an optional interface.

A far from exhaustive list of where the optional interface pattern is used (to roughly illustrate the scope of the pattern):

What are people using them for?

There are multiple reasons to use optional interfaces. Let's find examples for them. Note that this list neither claims to be exhaustive (there are probably use cases I don't know about) nor disjunct (in some cases, optional interfaces will carry more than one of these use cases). But I think it's a good rough partition to discuss.

Passing behavior through API boundaries

This is the case of ResponseWriter and its optional interfaces. The API, in this case, is the http.Handler interface that users of the package implement and that the package accepts. As features like HTTP/2 Push or connection hijacking are not available to all connections, this interface needs to use the lowest common denominator between all possible behaviors. So, if more features need to be supported, we must somehow be able to pass this optional behavior through the http.Handler interface.

Enabling optional optimizations/features

io.Copy serves as a good example of this. The required interfaces for it to work are just io.Reader and io.Writer. But it can be made more efficient, if the passed values also implement io.WriterTo or io.ReaderFrom, respectively. For example, a bytes.Reader implements WriteTo. This means, you need less copying if the source of an io.Copy is a bytes.Reader. Compare these two (somewhat naive) implementations:

func Copy(w io.Writer, r io.Reader) (n int64, err error) {
    buf := make([]byte, 4096)
    for {
        rn, rerr := r.Read(buf)
        wn, werr := w.Write(buf[:rn])
        n += int64(wn)
        if rerr == io.EOF {
            return n, nil
        }
        if rerr != nil {
            return n, rerr
        }
        if werr != nil {
            return n, werr
        }
    }
}

func CopyTo(w io.Writer, r io.WriterTo) (n int64, err error) {
    return r.WriteTo(w)
}

type Reader []byte

func (r *Reader) Read(b []byte) (n int, err error) {
    n = copy(b, *r)
    *r = (*r)[n:]
    if n == 0 {
        err = io.EOF
    }
    return n, err
}

func (r *Reader) WriteTo(w io.Writer) (int64, error) {
    n, err := w.Write(*r)
    *r = (*r)[n:]
    return int64(n), err
}

Copy needs to first allocate a buffer, then copy all the data from the *Reader to that buffer, then pass it to the Writer. CopyTo, on the other hand, can directly pass the byte-slice to the Writer, saving an allocation and a copy.

Some of that cost can be amortized, but in general, its existence is a forced consequence of the API. By using optional interfaces, io.Copy can use the more efficient method, if supported, and fall back to the slow method, if not.

Backwards compatible API changes

When database/sql upgraded to use context, it needed help from the drivers to actually implement cancellation and the like. So it needed to add contexts to the methods of driver.Conn. But it can't just do that change; it would be a backwards incompatible API change, violating the Go1 compatibility guarantee. It also can't add a new method to the interface to be used, as there are third-party implementations for drivers, which would be broken as they don't implement the new method.

So it instead resorted to deprecate the old methods and instead encourage driver implementers to add optional methods including the context.

Why are they bad?

There are several problems with using optional interfaces. Some of them have workarounds (see below), but all of them have drawbacks on their own.

They violate static type safety

In a lot of cases, the consumer of an optional interface can't really treat it as optional. For example, http.Hijacker is usually used to support WebSockets. A handler for WebSockets will, in general, not be able to do anything useful, when called with a ResponseWriter that does not implement Hijacker. Even when it correctly does a comma-ok type assertion to check for it, it can't do anything but serve an error in that case.

The http.Hijacker type conveys the necessity of hijacking a connection, but since it is provided as an optional interface, there is no possibility to require this type statically. In that way, optional interfaces hide static type information.

They remove a lot of the power of interfaces

Go's interfaces are really powerful by being very small; in general, the advice is to only add one method, maybe a small handful. This advice enables easy and powerful composition. io.Reader and io.Writer have a myriad of implementations inside and outside of the standard library. This makes it really easy to, say, read uncompressed data from a compressed network connection, while streaming it to a file and hashing it at the same time to write to some content-addressed blob storage.

Now, this composition will, in general, destroy any optional interfaces of those values. Say, we have an HTTP middleware to log requests. It wants to wrap an http.Handler and log the requests method, path, response code and duration (or, equivalently, collect them as metrics to export). This is, in principle, easy to do:

type logResponder struct {
    http.ResponseWriter
    code int
    set bool
}

func (rw *logResponder) WriteHeader(code int) {
    rw.code = code
    rw.set = bool
    rw.ResponseWriter.WriteHeader(code)
}

func LogRequests(h http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        lr := &logResponder{ResponseWriter: w}
        m, p, start := r.Method, r.Path, time.Now()
        defer func() {
            log.Printf("%s %s -> %d (%v)", m, p, lr.code, time.Now().Sub(start))
        }()
        h(lr, r)
    })
}

But *logResponder will now only support the methods declared by http.ResponseWriter, even if the wrapped ResponseWriter also supports some of the optional interfaces. That is because method sets of a type are determined at compile time.

Thus, by using this middleware, the wrapped handler is suddenly unable to use websockets, or HTTP/2 server push or any of the other use cases of optional interfaces. Even worse: this deficiency will only be discovered at runtime.

Optimistically adding the optional interface's methods and type-asserting the underlying ResponseWriter at runtime doesn't work either: handlers would incorrectly conclude the optional interface is always present. If the underlying ResponseWriter does not support adding at the underlying connection there just is no useful way to implement http.Hijacker.

There is one way around this, which is to dynamically check the wrapped interface and create a type with the correct method set, e.g.:

func Wrap(wrap, with http.ResponseWriter) http.ResponseWriter {
    var (
        flusher http.Flusher
        pusher http.Pusher
        // ...
    )
    flusher, _ = wrap.(http.Flusher)
    pusher, _ = wrap.(http.Pusher)
    // ...
    if flusher == nil && pusher == nil {
        return with
    }
    if flusher == nil && pusher != nil {
        return struct{
            http.ResponseWriter
            http.Pusher
        }{with, pusher}
    }
    if flusher != nil && pusher == nil {
        return struct{
            http.ResponseWriter
            http.Flusher
        }{with, flusher}
    }
    return struct{
        http.ResponseWriter
        http.Flusher
        http.Pusher
    }{with, flusher, pusher}
}

This has two major drawbacks:

  • Both code-size and running time of this will increase exponentially with the number of optional interfaces you have to support (even if you generate the code).
  • You need to know every single optional interface that might be used. While supporting everything in net/http is certainly tenable, there might be other optional interfaces, defined by some framework unbeknownst to you. If you don't know about it, you can't wrap it.

What can we use instead?

My general advice is, to avoid optional interfaces as much as possible. There are alternatives, though they also are not entirely satisfying.

Context.Value

context was added after most of the optional interfaces where already defined, but its Value method was meant exactly for this kind of thing: to pass optional behavior past API boundaries. This will still not solve the static type safety issue of optional interfaces, but it does mean you can easily wrap them.

For example, net/http could instead do

var ctxFlusher = ctxKey("flusher")

func GetFlusher(ctx context.Context) (f Flusher, ok bool) {
    f, ok = ctx.Value(ctxFlusher).(Flusher)
    return f, ok
}

This would enable you to do

func ServeHTTP(w http.ResponseWriter, r *http.Request) {
    f, ok := http.GetFlusher(r.Context())
    if ok {
        f.Flush()
    }
}

If now a middleware wants to wrap ResponseWriter, that's not a problem, as it will not touch the Context. If a middleware wants to add some other optional behavior, it can do so easily:

type Frobnicator interface{
    Frobnicate()
}

var ctxFrobnicator = ctxKey("frobnicator")

func GetFrobnicator(ctx context.Context) (f Frobnicator, ok bool) {
    f, ok = ctx.Value(ctxFrobnicator).(Frobnicator)
    return f, ok
}

As contexts form a linked list of key-value-pairs, this will interact nicely with whatever optional behavior is already defined.

There are good reasons to frown upon the usage of Context.Value; but they apply just as much to optional interfaces.

Extraction methods

If you know an interface type that is probable to be wrapped and also has optional interfaces associated it is possible to enforce the possibility of dynamic extension in the optional type. So, e.g.:

package http

type ResponseWriter interface {
    // Methods…
}

type ResponseWriterWrapper interface {
    ResponseWriter

    WrappedResponseWriter() ResponseWriter
}

// GetFlusher returns an http.Flusher, if res wraps one.
// Otherwise, it returns nil.
func GetFlusher(res ResponseWriter) Flusher {
    if f, ok := res.(Flusher); ok {
        return f
    }
    if w, ok := res.(ResponseWriterWrapper); ok {
        return GetFlusher(w.WrappedResponseWriter())
    }
    return nil
}

package main

type logger struct {
    res ResponseWriter
    req *http.Request
    log *log.Logger
    start time.Time
}

func (l *logger) WriteHeader(code int) {
    d := time.Now().Since(l.start)
    l.log.Write("%s %s -> %d (%v)", l.req.Method, l.req.Path, code, d)
    l.res.WriteHeader(code)
}

func (l *logger) WrappedResponseWriter() http.ResponseWriter {
    return l.res
}

func LogRequests(h http.Handler, l *log.Logger) http.Hander {
    return http.HandlerFunc(res http.ResponseWriter, req *http.Request) {
        res = &logger{
            res: res,
            req: req,
            log: l,
            start: time.Now(),
        }
        h.ServeHTTP(res, req)
    }
}

func ServeHTTP(res http.ResponseWriter, req *http.Request) {
    if f := http.GetFlusher(res); f != nil {
        f.Flush()
    }
}

This still doesn't address the static typing issue and explicit dependencies, but at least it enables you to wrap the interface conveniently.

Note, that this is conceptually similar to the errors package, which calls the wrapper-method "Cause". This package also shows an issue with this pattern; it only works if all wrappers use it. That's why I think it's important for the wrapping interface to live in the same package as the wrapped interface; it provides an authoritative way to do that wrapping, preventing fragmentation.

Provide statically typed APIs

net/http could provide alternative APIs for optional interfaces that explicitly include them. For example:

type Hijacker interface {
    ResponseWriter
    Hijack() (net.Conn, *bufio.ReadWriter, error)
}

type HijackHandler interface{
    ServeHijacker(w Hijacker, r *http.Request)
}

func HandleHijacker(pattern string, h HijackHandler) {
    // ...
}

For some use cases, this provides a good way to side-step the issue of unsafe types. Especially if you can come up with a limited set of scenarios that would rely on the optional behavior, putting them into their own type would be viable.

The net/http package could, for example, provide separate ResponseWriter types for different connection types (for example HTTP2Response). It could then provide a func(HTTP2Handler) http.Handler, that serves an error if it is asked to serve an unsuitable connection and otherwise delegates to the passed Handler. Now, the programmer needs to explicitly wire a handler that requires HTTP/2 up accordingly. They can rely on the additional features, while also making clear what paths must be used over HTTP/2.

Gradual repair

I think the use of optional interfaces as in database/sql/driver is perfectly fine - if you plan to eventually remove the original interface. Otherwise, users will have to continue to implement both interfaces to be usable with your API, which is especially painful when wrapping interfaces. For example, I recently wanted to wrap importer.Default to add behavior and logging. I also needed ImporterFrom, which required separate implementations, depending on whether the importer returned by Default implements it or not. Most modern code, however, shouldn't need that.

So, for third party packages (the stdlib can't do that, because of compatibility guarantees), you should consider using the methodology described in Russ Cox' excellent Codebase Refactoring article and actually deprecate and eventually remove the old interface. Use optional interfaces as a transition mechanism, not a fix.

How could Go improve the situation?

Make it possible for reflect to create methods

There are currently at least two GitHub issues which would make it possible to do extend interfaces dynamically: reflect: NamedOf, reflect: MakeInterface. I believe this would be the easiest solution - it is backwards compatible and doesn't require any language changes.

Provide a language mechanism for extension

The language could provide a native mechanism to express extension, either by adding a keyword for it or, for Go2, by considering to make extension the default behavior for interface->struct embedding. I'm not sure either is a good idea, though. I would probably prefer the latter, because of my distaste for keywords. Note, that it would still be possible to then compose an interface into a struct, just not via embedding but by adding a field and delegation-methods. Personally, I'm not a huge fan of embedding interfaces in structs anyway except when I'm explicitly trying to extend them with additional behavior. Their zero-value is not usable, so it requires additional hoops to jump through.

Conclusion

I recommend:

  • If at all possible, avoid optional interfaces in APIs you provide. They are just too inconvenient and un-Go-ish.
  • Be careful when wrapping interfaces, in particular when there are known optional interfaces for them.

Using optional interfaces correctly is inconvenient and cumbersome. That should signal that you are fighting the language. The workarounds needed all try to circumvent one or more design decision of Go: to value composition over inheritance, to prefer static typing and to make computation and behavior obvious from code. To me, that signifies that optional interfaces are fundamentally not a good fit for the language.

30. July 2017 18:39:00

24. July 2017

michael-herbst.com

[c¼h] Parallelised numerics in Python: An introduction to Bohrium

Thursday a week ago I gave a brief introductory talk in our Heidelberg Chaostreff about the Bohrium project. Especially after the HPC day at the Niels Bohr Institute during my recent visit to Copenhagen, I became rather enthusiastic about Bohrium and wanted to pass on some of my experiences.

The main idea of Bohrium is to build a fully numpy-compatible framework for high-performance computing, which can automatically parallelise numpy array operations and/or execute them on a general-purpose graphics cards. The hope is that this eradicates the step of rewriting a prototypical python implementation of a scientific model in more low-level languages like C++ or CUDA before dealing with the actual real-world problems in mind.

In practice Bohrium achieves this by translating the python code (via some intermediate steps) into small pieces of C or CUDA code. These are then automatically compiled at runtime of the script, taking into account the current hardware setup, and afterwards executed. The results of such a just-in-time compiled kernel are again available in numpy-like arrays and can be passed to other scripts for post-processing, e.g. plotting in matplotlib.

It is important to note, that the effect of Bohrium is limited to array operations. So for example the usual Python for loops are not touched. This is, however, hardly a problem if the practice of so-called array programming is followed. In array programming one avoids plain for-loops and similar traditional python language elements in preference for special syntax which works on blocks (numpy arrays) of data at once. Examples of such operations is pretty much the typical numpy workflow:

  • views and slices: array[1:3]
  • broadcasting: array[:, np.newaxis]
  • elementwise operations: array1 * array2
  • reduction: np.sum(array1, axis=1)

A slightly bigger drawback of Bohrium is, that the just-in-time compilation takes time, where no results are produced. In other words Bohrium does only start to pay of at larger problem sizes or if exactly the same sequence of instructions is to be executed many times.

In my c¼h I demonstrate Bohrium by the means of this example script

#!/usr/bin/env python3

import numpy as np
import sys
import time


def moment(n, a):
    avg = np.sum(a) / a.size
    return np.sum((a - avg)**n) / a.size


def compute(a):
    start = time.time()

    mean = np.sum(a) / a.size
    var  = moment(2, a)
    m3   = moment(3, a)
    m4   = moment(4, a)

    end = time.time()

    fmt = "After {0:8.3f}s: {1:8.3f} {2:8.3f} {3:8.3f} {4:8.3f}"
    print(fmt.format(end - start, float(mean), float(var),
                     float(m3), float(m4)))


def main(size, repeat=6):
    for i in range(repeat):
        compute(np.random.rand(size))


if __name__ == "__main__":
    size = 30
    if len(sys.argv) >= 2:
        size = int(sys.argv[1])
    main(size)

which is also available for download. The script performs a very simple analysis of the (randomly generated) input data: It computes some statistical moments and displays them to the user. For bigger arrays the single-threaded numpy starts to get very slow, whereas the multi-threaded Bohrium version wins even thought it needs to compile first. Running the script with Bohrium does not require one to change even a single line of code! Just

python3 -m bohrium ./2017.07.13_moments.py

does kick off the automatic parallelisation.

The talk has been recorded and is available on youtube. Note, that the title of the talk and the description are German, but the talk by itself is in English.

by Michael F. Herbst at 24. July 2017 22:00:00

15. July 2017

Insantity Industries

Embedded IPython for script development

Every once and a while, when working with Python-code, there comes the point where I think “well, if I could just have a Python-shell at this very point in the code…”, be it for debugging or for inspection of objects because I’m to lazy to look up the documentation.

The solution is quite simple. Make sure you have the ipython-package installed and just place

import IPython
IPython.embed()

at the point where you want to have your Python-shell. And there you go, an ipython-shell with the full state of the code at the position of usage.

by Jonas Große Sundrup at 15. July 2017 00:00:00

06. July 2017

michael-herbst.com

IWR School: Mathematical Methods for Quantum Chemistry

From 2nd to 6th October 2017 the IWR hosts the school Mathematical Methods for Quantum Chemistry, which aims at shining some light into the simulation of quantum chemical problems and their mathematical properties. I am very happy to be part of the organising committee for this school, especially since I have been missing an opportunity like this myself in the recent years.

Starting from an introduction into the basics of quantum chemistry as well as the required concepts about optimisation and non-linear eigenvalue problems, we will look at ways to tackle the relevant differential equations from a numerical and algorithmic viewpoint. Hands-on practical sessions allow to apply the treatment introduced in the lectures to simple practical problems.

After the school participants will have an idea of the required interplay between the physical modelling, the mathematical representation and the computational treatment needed in this highly interdisciplinary field of research.

Speakers of the school include

The school mainly targets postgraduate students, postdocs and young researchers from mathematics and computational chemistry. Everyone is encouraged to participate actively by giving a short presentation of his or her work during the available sessions of contributed talks.

For further information as well as the program check out our website. The deadline for the application is 8th August 2017 (direct registration link).

We intend to publish the material of the school as we go along at this location, so feel free to check back regularly for updates.

by Michael F. Herbst at 06. July 2017 22:00:00

05. July 2017

RaumZeitLabor

Exkursion ins Stadtarchiv Mannheim

Verehrte Freunde alter Aktenschränke!

Am Donnerstag, den 20. Juli 2017 werden wir eine Tour durch das alte Stadtarchiv Mannheim machen. Ab 15.30 Uhr werden wir die Räumlichkeiten im Collini-Center unter die Lupe nehmen können, etwas zur Geschichte des Stadtarchivs hören und alles Wissenswerte zum bevorstehenden Umzug in den Ochsenpferchbunker erfahren. Außerdem werden wir uns natürlich das Digitalisierungslabor des Stadtarchivs ganz genau anschauen und über die Vor- und Nachteile der digitalen Archivierung sprechen.

Wenn ihr also wissen wollt, was man im Umgang mit alten Archivalien beachten muss, ob immernoch alles auf Mikrofilm gespeichert wird und was genau jetzt eigentlich das Marchivum ist, solltet ihr euch das Ganze nicht entgehen lassen.

Falls ihr dabei sein sollt, stanzt mir bis zum 16. Juli 2017 einfach eine Lochkarte mit eurem Namen und den Namen der Personen, die ihr mitnehmen wollt.

by flederrattie at 05. July 2017 00:00:00

24. June 2017

Insantity Industries

Persistent IRC

Hanging around in IRC is fun. However, when you don’t stay connected all the time (why do you shut down your computer anyways…) you are missing out a lot of that fun! This is obviously an unbearable state, this needs immediate fixing!

Persistent IRC Setup

There are several ways of building a persistent IRC-setup. One is a so called bouncer that works like a proxy that mediates between you and the IRC network and buffers the messages. Another, simpler method is just running a IRC-client on a machine that is always powered on, like a server, that we can reconnect to to read our messages (and of course write new ones).

Requirements

  • A computer that is powered and connected to the internet 247
  • an account on said machine that is accessible via SSH
  • Software installed on said machine: tmux, weechat

Alternatively to tmux, screen can be used if you cannot get tmux installed on the machine you want to run this setup on, it works very similar. This post, however, will focus on the tmux-setup.

SSH

We can access our account via

ssh username@server.name

We should end up in a plain terminal prompt where we can type.

weechat

In this shell we can then start weechat, a fully keyboard-controlled terminal-IRC-client.

Adding a server

To add a server, type

/server add <servername> <serveraddress> -ssl -autoconnect

The options -ssl and -autoconnect are both optional. -ssl will enable encryption to the IRC network by default, -autoconnect will enable autoconnect in the server config, so that our weechat will automatically connect to that server when we start it.

<servername> will be the weechat-internal name for this server, <serveraddress> can also include a port by appending it via <serveraddress>/<port>.

Adding the freenode-network could therefore read

/server add freenode chat.freenode.net/6697 -ssl -autoconnect

Afterwards, we can connect to the freenode-network by issuing

/connect freenode

as the autoconnect only works upon starting of weechat. Alternatively, /quit weechat and start it again we should get autoconnected. And now, we are connected to the freenode-network!

Setting your name

By default, our nick will be set according to our username on the system. It can be changed via

/nick <newnickname>

To change it persistently, we can set the corresponding option in weechat via

/set irc.server.<servername>.nicks "demodude"

to a custom nickname. Generally, options in weechat can be set by /set <optionname> <value> or read by /set <optionname>. Weechat also supports globbing in the optionname, so getting all options for our added server can be done by

/set irc.server.<servername>.*

Joinging channels

Communication on IRC happens in channels. Each channel usually has a certain focus of stuff that happens there. We can then join channels via

/join #mytestchannel

which is a very boring channel as no one except for ChanServ and us is here, which you can see on the user list for this channel on the right. But it technically worked and we can now just type to post messages in this channel.

In the bar right above where we type we see a 2:#mytestchannel. Weechat has the concept of buffers. Each buffer represends one server or channel. The channel we are in is now buffer 2. To get back to our server-buffer, we can type /buffer 1 or hit F5, both will bring us back to buffer 1, which is our server-buffer. To get back to our channel-buffer, /buffer <buffernumber> or F6 will bring you there.

To enter channels on a server upon starting weechat, we can set the option irc.server.<servername>.autojoin, to a comma-separated list of channels "#channel1,#channel2,#channel3". To find all the channels on a IRC-server, we can issue a /list (for freenode, be aware, the list is HUGE).

We can save our changes via /save and exit weechat via /quit.

Scrolling

We can scroll backward and forward by using the PgUp- and PgDown-Keys. If we are at the very bottom of a buffer, weechat will automatically scroll down with incoming messages. If you have scrolled up, weechat will not follow as it assumes you want to read the part you scrolled to.

command recap

  • adding server: /server add <servername> <serveraddress>[/<port>] [-ssl] [-autoconnect]
  • connecting to a server: /connect <servername>
  • joining channels: /join <channelname>
  • switching buffers: /buffer <targetbuffer> or F5/F6
  • leaving channels: /close in channelbuffer
  • scrolling: PgUp, PgDown

We are up and running for IRC now! However, once we exit our weechat, we are no longer connected and missing out all the fun! So our weechat needs to run continuously.

Introducing…

tmux

Usually, upon SSH-disconnect, all of our processes will be killed (including our weechat). This is different with tmux. Tmux allows to reattach to what we have done when last SSH-ing into our server.

So we exit our weechat and are back on our shell. There, we start

tmux

We now see a green bar at the bottom ouf our screen.

This is a tmux and a shell running inside of it.

We can now hit Ctrl+b to tell tmux to await a tmux-command and not forward our typing to the shell but instead interpret it as a command to tmux itself. We can then type d to detach and our shell is gone.

Afterwards, we can reattach to our tmux by running tmux attach and our shell is back! This also works when we detach and then log out of our machine, log in again and then reattach our tmux.

Now the only thing left is running a weechat inside of our tmux and we are done. We can detach (or just close the terminal, also works perfectly fine) and then reattach later to read what we have been missing out on. Our persistent IRC-setup is ready to rumble.

Improving our setup

Up to now we have a running setup for persistent IRC. However, the user-experience of this setup can be significantly improved.

Tab completion

Weechat is capable of tab completion, e.g. when using /set. However, by default, weechat autocompletes the first option it finds fully instead of to the first ., which is the configsection-delimiter in weechat.

To change this, we search for completion options via

/set *completion*

and afterwards we

/set weechat.completion.partial_completion_command on
/set weechat.completion.partial_completion_command_arg on
/set weechat.completion.partial_completion_completion_other on

Weechat plugins

Weechat is a highly extendable software. A full list of extensions can be found here, some of the most useful ones are listed in the following.

You can install all scripts directly from within weechat via

/script install <scriptname>
  • buffers.pl provides a visual list of open buffers on the left side of weechat (from weechat 1.8 onwards this can be replaced by weechat’s built-in buflist, which provides the same feature)
  • beep.pl triggers our terminal bell when we are highlighted, mentioned or queried
  • autojoin_on_invite.py does basically what the name says
  • screen_away.py will detect when we detach from our IRC-setup and report “I’m not here” to the other person if we are queried
  • autosort.py keeps our buffers sorted alphabetically, no matter how many you have open
  • autojoin.py writes our currently joined channels into irc.server.<servername>.autojoin by issuing /autojoin --run

For autosort.py you most likely also want to

/set buffers.look.indenting on
/set irc.look.server_buffer independent

autoconnecting tmux

To automate reattaching to tmux every time you SSH into the machine your persistent IRC-setup is running on, we can put

if [[ $TERM != 'screen' ]]
then
    tmux attach || tmux
fi

at the end of the file ~/.profile.

getting a shell besides your weechat

You can also open more shells in tmux. To do so, hit Ctrl+b and then ‘c’ for create. You will find another buffer (not the weechat-buffer, but the concept in tmux is equivalent) down in your tmux-bar.

The buffers read

<buffernumber>:terminaltitle

You can then switch between buffers via Crtl+b, <buffernumber>.

FUO (frequently used options)

A lot of IRC-network allow registration of usernames to ensure that we can reuse our nick and no one else grabs it. If we have done that, Weechat can automatically identify us upon connecting. To do so, we just need to set the password we chose when registering in the option

irc.server.<servername>.password

However, we just need to be aware that weechat saves that password in plaintext in the configuration.

Weechat can also trigger arbitrary commands when connecting to a server. This is useful for things like self-invites into invite-only-channels or other things that you want to trigger. To use this, we just need to set

irc.server.<servername>.command

to a semicolon-separated list of commands as you would issue them manually in weechat.

by Jonas Große Sundrup at 24. June 2017 00:00:00

22. June 2017

Insantity Industries

Hardwaretriggered Backups

No one wants to do backups, but everyone wants restore once something went wrong. The solution we look for is fully automated backups that trigger once you plug-in your backupdrive (or it appears $somehow).

So our objective is to detect the plugging-in of a USB-stick or harddisk and trigger a backupscript.

udev

The solution is: udev. udev is the solution in most Linux-Distributions responsible for handling hotplug-events.

Those events can be monitored via udevadm monitor. When then plugging in a device, we see events like

KERNEL[1108.237335] add      /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2 (usb)
KERNEL [1108.778873] add    /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.2/1-1.2:1.0/host6 (scsi)

for different subsystems (USB-subsystem and SCSI-subsystem in this case) which we can hook into and trigger arbitrary commands.

writing udev-rules

To identify a device, we need to take a look at its properties. For a harddisk this can be done by

udevadm info -a -p $(udevadm info -q path -n $devicename)

where $devicename can be sdb or /dev/sdb, for example. udevadm info -q path -n $devicename gives us the indentifier in the /sys-subsystem we already have seen in the output of udevadm monitor, udevadm info -a -p $devicepath uses this identifier to walk through the /sys-subsystem and gives us all the properties that are associated with the device or its parent nodes, for example my thumbdrive’s occurrence in the scsi-subsystem:

KERNELS=="6:0:0:0"
SUBSYSTEMS=="scsi"
DRIVERS=="sd"
ATTRS{device_blocked}=="0"
ATTRS{device_busy}=="0"
ATTRS{dh_state}=="detached"
ATTRS{eh_timeout}=="10"
ATTRS{evt_capacity_change_reported}=="0"
ATTRS{evt_inquiry_change_reported}=="0"
ATTRS{evt_lun_change_reported}=="0"
ATTRS{evt_media_change}=="0"
ATTRS{evt_mode_parameter_change_reported}=="0"
ATTRS{evt_soft_threshold_reached}=="0"
ATTRS{inquiry}==""
ATTRS{iocounterbits}=="32"
ATTRS{iodone_cnt}=="0x117"
ATTRS{ioerr_cnt}=="0x2"
ATTRS{iorequest_cnt}=="0x117"
ATTRS{max_sectors}=="240"
ATTRS{model}=="DataTraveler 2.0"
ATTRS{queue_depth}=="1"
ATTRS{queue_type}=="none"
ATTRS{rev}=="PMAP"
ATTRS{scsi_level}=="7"
ATTRS{state}=="running"
ATTRS{timeout}=="30"
ATTRS{type}=="0"
ATTRS{vendor}=="Kingston"

We can use those attributes to identify our device amongst all the devices that we have.

A udev-rule now contains two major component types:

  • matching statements that identify the event and device we want to match
  • action statements that take action on the matched device

which is of course a simplification, but it will suffice for the purpose of this blogpost and most standard applications.

To do so we first create a file /etc/udev/rules.d/myfirstudevrule.rules. The filename doesn’t matter as long as it ends in .rules as only those files will be read by udev.

In this udev-rule, we first need to match our device. To do so, I will pick three of the properties above that sound like they are sufficient to uniquely match my thumbdrive.

SUBSYSTEMS=="scsi", ACTION=="add" ATTRS{model}=="DataTraveler 2.0", ATTRS{vendor}=="Kingston"

I have added a statement matching the ACTION, as we of course only want to trigger a backup when the device appears. You can also match entire device classes by choosing the matcher-properties accordingly.

To trigger a command, we can add it to the list of commands RUN that shall run when the device is inserted, for example creating the file /tmp/itsalive:

RUN+="/usr/bin/touch /tmp/itsalive"

So our entire (rather lengthy) udev-rule in /etc/udev/rules.d/myfirstudevrule.rules reads

SUBSYSTEMS=="scsi", ACTION=="add" ATTRS{model}=="DataTraveler 2.0", ATTRS{vendor}=="Kingston", RUN+="/usr/bin/touch /tmp/itsalive"

and we can trigger arbitrary commands with it

trigger long runnig jobs

However, commands in the RUN-list have a time constraint of 180 seconds to run. For a backup, this is likely to be be insufficient. So we need a way to create long-running jobs from udev.

The solution for this is to outsource the command into a systemd-unit. Besides being able to run for longer than 180s, that way we also get proper logging of our backup-command in the journal, which is always good to have.

So we create a file /etc/systemd/system/mybackup.service containing

[Unit]
Description=backup system

[Service]
User=backupuser  # optional, might be yourself, root, …
Type=simple
ExecStart=/path/to/backupscript.sh

We then need to modify the action part of our rule from appending to RUN to read

TAG+="systemd", ENV{SYSTEMD_WANTS}="mybackup.service"

Our entire udev-rule then reads

SUBSYSTEMS=="scsi", ACTION=="add" ATTRS{model}=="DataTraveler 2.0", ATTRS{vendor}=="Kingston", TAG+="systemd", ENV{SYSTEMD_WANTS}="mybackup.service"

improve usability

To further improve usability, we can additionally append.

SYMLINK+="backupdisk"

This way, an additional symlink /dev/backupdisk will be created upon the device appearing that can be used for scripting.

using a disk in a dockingstation

At home I have a dockingstation for my laptop and the disk I use for local backups is built into the bay of the dockingstation. From my computer’s point of view, this disk is connected over an internal port. When docking onto the station it is not recognized, as internal ports are not monitored for hotplugging. Upon docking, it is therefore necessary to rescan the internal scsi-bus the disk is connected to. This can be done issuing an entire rescan of the scsi-host host1 by

echo '- - -' > /sys/class/scsi_host/host1/scan

In my case the disk is connected to the port at scsi-host 1. You can find out the scsi-host of your disk by running

ls -l /sys/block/sdb

where sda is the device mapper of the block device whose scsi-host you are interested in. This returns a string like

../devices/pci0000:00/0000:00:1f.2/ata1/host1/target0:0:0/0:0:0:0/block/sda

where you can see that for this disk, the corresponding scsi-host is host1. This can then be used to issue the rescan of the correct scsi-bus. The according udev-rule in my case reads

SUBSYSTEM=="platform", ACTION=="change", ATTR{type}=="ata_bay", RUN+="/usr/bin/echo '- - -' > /sys/class/scsi_host/host1/scan"

Afterwards, the disk should show up and can be matched like any other disk as described above.

by Jonas Große Sundrup at 22. June 2017 00:00:00

18. June 2017

michael-herbst.com

Lazy matrices in quantum chemistry

On my way back from Copenhagen last Thursday I stopped in Kiel to visit Henrik Larsson and some other people from last year's Deutsche Schülerakademie in Grovesmühle.

Henrik is doing his PhD at the group of Prof. Bernd Hartke and so I took the opportunity and gave a short talk at their group seminar on my progress in Copenhagen (slides are attached below).

Most people in the Hartke group work on global optimisation of structures or reaction pathways or on fitting reactive force fields. Hence my talk turned out to be a little off-topic. On the other hand I had an afternoon of enjoyful and rather unusual discussions, where I learned quite a lot about their work.

Link Licence
Lazy-matrices in quantum chemistry (Slides group seminar talk) Creative Commons License

by Michael F. Herbst at 18. June 2017 22:00:00

25. May 2017

sECuREs Webseite

Auto-opening portal pages with NetworkManager

Modern desktop environments like GNOME offer UI for this, but if you’re using a more bare-bones window manager, you’re on your own. This article outlines how to get a login page opened in your browser when you’re behind a portal.

If your distribution does not automatically enable it (Fedora does, Debian doesn’t), you’ll first need to enable connectivity checking in NetworkManager:

# cat >> /etc/NetworkManager/NetworkManager.conf <<'EOT'
[connectivity]
uri=http://network-test.debian.org/nm
EOT

Then, add a dispatcher hook which will open a browser when NetworkManager detects you’re behind a portal. Note that the username must be hard-coded because the hook runs as root, so this hook will not work as-is on multi-user systems. The URL I’m using is an always-http URL, also used by Android (I expect it to be somewhat stable). Portals will redirect you to their login page when you open this URL.

# cat > /etc/NetworkManager/dispatcher.d/99portal <<EOT
#!/bin/bash

[ "$CONNECTIVITY_STATE" = "PORTAL" ] || exit 0

USER=michael
USERHOME=$(eval echo "~$USER")
export XAUTHORITY="$USERHOME/.Xauthority"
export DISPLAY=":0"
su $USER -c "x-www-browser http://www.gstatic.com/generate_204"
EOT

by Michael Stapelberg at 25. May 2017 09:37:17