Planet NoName e.V.

07. October 2014

RaumZeitLabor

Wir laden zur Kryptoparty!

Cryptoparty
Encrypt everything! Das RaumZeitLabor lädt ein zur Kryptoparty: Am 17.10. ab 19h erzählen euch Nivatius und Cheatha, warum ihr eure Daten und eure Kommunikation verschlüsseln wollt. Wir erklären euch verständlich, was in diesem Internet passiert und welche Zusammenhänge ihr kennen müsst um euch dort angemessen zu verhalten. Statt euch mit solch kryptische Begriffe wie OTR, PGP, Tor und VPN zu langweilen zeigen wir praxisbezogen und individuell auf, für was ihr diese Dinge braucht und wie man sie benutzt. Unabhängig von eurem Kenntnisstand nehmen wir euch an die Hand und erleichtern euch den Einstieg in die Verschlüsselung von E-Mails und Festplatten mit freier Software.

Nach einer allgemeinen Einführung ist Zeit das Erlernte an eurem eigenen Laptop in entspannter Atmosphäre direkt umzusetzen! Fleißige Laboranten werden euch dabei bei Fragen und Problemen tatkräftig unterstützen.

Die Teilnahme ist kostenlos. Wir bitten um eine Voranmeldung über die Kommentarfunktion, über das Facebookeevent oder per Mail an anmeldung@raumzeitlabor.de

by Cheatha at 07. October 2014 19:59:18

03. October 2014

Raphael Michel

Deploying Django without downtime

One of my active projects is abiapp.net, a SaaS content management tool for the collaborative creation of yearbook contents, primarily targeted to German high scool students. While we currently have few enough users to run all of abiapp.net on a single machine, we have enough users that at any time of the day one of us is working on the project, a user is online. We have a philosophy of pushing new code in very small intervals, days with multiple updates on the same day are nothing uncommon.

However, I do not enjoy pushing new code if I know that all users currently using the website will be interrupted – pretty much everything in our application happens in real-time, so users will surely recognize a downtime of up to a minute. There would have been several possibilities to shorten this time frame but I have decided to work on a solution without any measurable downtime at all.

The general setup

Our application is implemented in Python using Django and runs on a Debian stable virtual machine. The Django application lives inside a gunicorn instance, which is a lightweight HTTP server on top of the WSGI protocol you should use if you do web stuff in Python. On top of all, there is a nginx webserver which servers static content and acts as a proxy for gunicorn. We use MySQL as our database backand and we have a background task queue for long-running tasks powered by Celery and RabbitMQ as a message broker. The gunicorn instance is being controlled by supervisord.

The old deployment setup

We currently deploy using git. We have a git remote on our production server which has a post-receive hook which executed the following tasks:

  • Load the new source code into the working direcotry
  • Make a database backup (we learned this the hard way)
  • Perform database migrations
  • Compile LESS to CSS
  • Compress static files
  • Execute unit tests, just to be sure
  • Reload gunicorn

However, this setup has some very huge problems. The biggest one is that in the moment we load our new code into the working directory, Django will use our new templates and static files even though we are still running on the old python code. This is already bad, but it gets way worse in the unlikely event that the unit tests fail and the new python code is not loaded – then we're stuck in this intermediate state of broken things.

The new deployment setup

We now have two completely independent instances of the application. We have have our git repository three times on the production server:

$ ls app.*
app.src/
app.run.A/
app.run.B/

app.src is the bare git repository we push our changes to and app.run.A and app.run.B are two copies of it used for running the application. The application always runs twice:

$ supervisorctl status
abiapp.A       RUNNING    pid 6184, uptime 0:00:02
abiapp.B       RUNNING    pid 6185, uptime 0:00:02

One of those processes runs with the code and templates from app.run.A, one with the other. They listen on different sockets and supervisord knows them as distinct services.

We also have two copies of our nginx webserver config, one of them pointing to the socket of process A and one to the socket of process B. Only one of them is enabled at the same time:

$ ls /etc/nginx/sites-available/
abiapp.net-A
abiapp.net-B
$ ls /etc/nginx/sites-enabled/
abiapp.net-A

The nginx config

The nginx configuration looks a bit like this:

upstream abiapp_app_server_A {
    server unix:/home/abiapp/run/gunicorn.A.sock fail_timeout=0;
}

server {
    listen 443;
    server_name .abiapp.net;

    # … SSL foo …

    location /static/ { # The static files directory
        alias /var/www/users/abiapp/www.abiapp.net/static.A/;
        access_log off;
        expires 7d;
        add_header Cache-Control public;
        add_header Pragma public;
        access_log off;
    }

    location /media/ {
        # …
    }

    location / { # The application proxy
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        if (!-f $request_filename) {
            proxy_pass http://abiapp_app_server_A;
            break;
        }
        root /var/www/users/abiapp/www.abiapp.net/;
    }

    # error pages …
}

The git hook

So our post-receive hook has to find out which of those two processes currently serves users and replace the other process with the new code. We'll now walk through this git hook pice by piece.

The first part determines which instance is running by looking into a text file containing the string A or B. We'll write the current instance name into this file later in our hook.

#!/bin/bash
if [[ "$(cat /home/abiapp/run/selected)" == "B" ]]; then
    RUNNING="B"
    STARTING="A"
else
    RUNNING="A"
    STARTING="B"
fi

We now activate the virtual python environment our application lives in and move to the application's source code.

unset GIT_DIR
source /home/abiapp/appenv/bin/activate
cd /home/abiapp/app.run.$RUNNING/src

First of all, we'll do a backup, just to be sure. You could also use mysqldump here.

echo "* Backup"
python manage.py dumpdata > ~/backup/dump_$(date +"%Y-%m-%d-%H-%M")_push.json

We now pull the new source code into our current directory.

echo "* Reset server repository"
git reset --hard
git pull /home/abiapp/app.src master || exit 1
cd src

Then, we perform database migrations and deploy our static files.

echo "* Perform database migrations"
python manage.py migrate || exit 1
echo "* Deploy static files"
python manage.py collectstatic --noinput || exit 1
echo "* Compress static files"
python manage.py compress || exit 1

Note: The two source code directories have slightly different Django configuration files: Their STATIC_ROOT settings point to different directories.

We now perform our unit tests on the production server:

echo "* Unit Tests"
python manage.py test app || exit 1;

And finally restart the gunicorn process.

echo "* Restart app"
sudo supervisorctl restart abiapp.$STARTING

Remember, we just restarted a process which was not visible to the Internet, we replaced the idling one of our instances. Now the time has come to reload our webserver:

echo "* Reload webserver"
sudo /usr/local/bin/abiapp-switch $STARTING

The abiapp-switch script does no more than replacing the link in /etc/nginx/sites-enabled by the other configuration file and then calling service nginx reload.

This is the moment our new code goes live. On the reload call, nginx spawns up new workers using the new configuration1. All old workers will finish their current requests and then shut down, so that there really is no measurable downtime. To have the hook complete, we restart celery (which waits for all running workers to finish their tasks, then restarts with the new code):

echo "* Restart task queue"
sudo service celeryd restart 

And finally we report success and store the name of the newly running instance.

echo "Done :-)"
echo "Instance $STARTING is running now"
echo $STARTING > /home/abiapp/run/selected

So we're done here. As you may have noticed, all early steps of the hook included an || exit 1. In case of a failed migration, unit test or compression, the whole process would just abort and leave us virtually unharmed, as the working instance keeps running.

A word on database migrations

As you may have noticed, we still have one flaw in our workflow: The database migrations are applied some time before the new code is running. The only really clean solution is to split each of your 'destructive' database migrations into multiple deployment iterations: If you for example remove a field from a model (a column from a table), you'd first push the new code with all usage of the field being removed and then, in a second push, you'd deploy the database migration which removes the column.

03. October 2014 22:00:00

29. September 2014

RaumZeitLabor

RGB2R v11 – Das Retro-Gaming-Event in Heidelberg

Wie jeden Herbst richtet auch dieses Jahr der befreundete NoName e.V. vom Freitag, den 31.10., bis zum Sonntag, den 2.11., die roots go back to the roots aus, eine gemütliche Retrogame-Veranstaltung in Heidelberg.

Von der Veranstaltungsseite:

Die RGB2R ist das richtige Event für alle, die schon lange mal wieder die gute, alte Zeit der Videospiele erleben möchten, als Spiele noch von ihrer Grundidee, ihrem (Midi)Soundtrack oder ihrer genialen Story lebten, und selbiges nicht einfach nur Lückenfüller für aufwändige 3D-Grafik waren.

Es gibt wenige Regeln und genauso wenig ein festes Programm. Es ist ein lockeres Event unter Gleichgesinnten, jeder bringt ein Stück Videospiel-Historie mit. Den guten alten C64, Atari, Super Nintendo, GameBoy, 486er… eben die Ursprünge von dem, was heute den Computer(Spiele)-Markt beherrscht.

Doch es soll nicht nur gespielt werden! Es darf natürlich auch gebastelt, gecoded, gelötet und gehackt werden…

Wenn dich das anspricht und du schon immer mal wieder Monkey Island, Street Fighter 2, Elite oder Turrican spielen und mal wieder ein paar C64-Grafikdemos sehen wolltest, bist du hier richtig!

Das Team R.E.T.R.O. des RaumZeitLabors sowie einige andere Laborant*innen werden natürlich vor Ort sein – wenn du das jetzt auch vor hast: zur Anmeldung geht es hier.

Wir freuen uns auf spassiges Retro-Gedaddel!

by slowpoke at 29. September 2014 16:02:45

22. September 2014

RaumZeitLabor

Sticken leicht gemacht – Workshop für Kinder

Das RaumZeitLabor bietet am 22. und 23. November Stickworkshops an. Von 15:00 Uhr bis 19:00 Uhr könnt ihr Kinder zwischen 8 und 15 Jahren Textilien mit der Stickmaschine gestalten.
Was du lernst? Wir zeigen dir, wie eine Stickmaschine aufgebaut ist und mit welchen Materialien sie arbeitet. Danach helfen wir dir dabei, ein T-Shirt aus unserem großen Vorrat oder deinen mitgebrachten Lieblingspulli zu besticken. Du kannst dein eigenes Motiv vor Ort entwerfen oder dich beim Konvertieren eines Superhelden, eines Ponys oder was immer du magst unterstützen lassen.

Anmelden kannst du dich ganz einfach per E-Mail an info@raumzeitlabor.de
 

Damit jedes Kind mitmachen kann, ist die Teilnahme kostenlos. Als gemeinnütziger Verein freut sich das RaumZeitLabor aber über freiwillige Spenden, um die Materialkosten decken zu können.

 

by Alexander Brock at 22. September 2014 22:19:03

Maus Türöffnertag

Kaum ist das Agenda-Diplom zu Ende geht es auch schon weiter mit den Angeboten für unseren Nachwuchs. Das RZL macht beim Maus Türöffnertag am 03. Oktober mit.

Bei uns könnt ihr eigene Ansteckbuttons gestalten und selbst pressen. Mitgebrachte Textilien können mit Hilfe einer Textilpresse und einer Stickmaschine verschönert werden, außerdem können kleine Tierfiguren aus Holz angemalt werden.

Die Veranstaltung richtet sich an Kinder zwischen 5 und 12 Jahren, anmelden könnt ihr euch per E-Mail an info@raumzeitlabor.de.

Es gibt zwei Termine, einmal von 12 bis 14:30 und einmal von 15 bis 17:30.

by Alexander Brock at 22. September 2014 11:38:32

Trollcon 2014 am 29. November 2014 im RaumZeitLabor Mannheim

Das Phänomen des Trolling ist seit Jahrzehnten verbreitet aber auch 2014 noch erschreckend wenig erforscht.

Auch dieses Jahr treffen sich daher im RaumZeitLabor Mannheim wieder die führenden Trollspezialisten Deutschlands und des Internets um an einem intensiven Tag gemeinsam über neueste Ergebnisse der Troll- und Sozialforschung zu diskutieren.

Die reflexsive und extensive Beschäftigungsmethodik welche eine traditionell exponierte Stellung im Kontext der Veranstaltung Trollcon einnimmt, erlaubt es, dass Thema Trolling invasiv zu sensualisieren und führt unausweichlich zu einem ergebnisorientierten Dialog.

Tagsüber finden im Rahmen der Konferenz zahlreiche Vorträge und Diskussion rund um aktuelle Entwicklungen im Thema “Trolling” statt, am Abend wird der oder die Preisträger des Positivpreis “Troll des Jahres 2014″ im Rahmen einer Preisverleihung verkündet.

Der Preis “Troll des Jahres” wird zum mittlerweile dritten Mal vergeben. Nachdem er 2012 an den Piratenpolitiker Christopher Lauer für seinen außergewöhnlichen Politikstil vergeben wurde, konnte der Preis 2013 vom Neu-Ulmer Oberbürgermeister Gerold Noerenberg (CSU) für seine kreative Zivilcourage gegen rechte Gesellschaftsfeinde erobert werden. Vorschläge für 2014 werden noch durch die Jury angenommen.

Das Anmeldeformular für die Trollcon 2014 ist zusammen mit weiteren Information auf der Veranstaltungswebseite unter trollcon.de zu finden. Die Anmeldung funktioniert schnell & einfach per Fax. Vortragseinreichungen sind noch möglich.

by Alexander Brock at 22. September 2014 11:37:30

12. September 2014

Mero’s Blog

The four things I miss about go

As people who know me know, my current favourite language is go. One of the best features of go is the lack of features. This is actually the reason I preferred C over most scripting languages for a long time – it does not overburden you with language-features that you first have to wrap your head around. You don't have to think for a while about what classes or modules or whatever you want to have, you just write your code down and the (more or less) entire language can easily fit inside your head. One of the best writeups of this (contrasting it with python) was done by Gustavo Niemeyer in a blogpost a few years back.

So when I say, there are a few things popping up I miss about go, this does not mean I wish them to be included. I subjectively miss them and it would definitely make me happy, if they existed. But I still very much like the go devs for prioritizing simplicity over making me happy.

So let's dig in.

  1. Generics
  2. Weak references
  3. Dynamic loading of go code
  4. Garbage-collected goroutines

Generics

So let's get this elephant out of the room first. I think this is the most named feature lacking from go. They are asked so often, they have their own entry in the go FAQ. The usual answers are anything from "maybe they will get in" to "I don't understand why people want generics, go has generic programming using interfaces". To illustrate one shortcoming of the (current) interface approach, consider writing a (simple) graph-algorithm:

type Graph [][]int

func DFS(g Graph, start int, visitor func(int)) {
    visited := make([]bool, len(g))

    var dfs func(int)
    dfs = func(i int) {
        if visited[i] {
            return
        }
        visitor(i)
        visited[i] = true
        for _, j := range g[i] {
            dfs(j)
        }
    }

    dfs(start)
}

This uses an adjacency list to represent the graph and does a recursive depth-first-search on it. Now imagine, you want to implement this algorithm generically (given, a DFS is not really hard enough to justify this, but you could just as easily have a more complex algorithm). This could be done like this:

type Node interface{}

type Graph interface {
    Neighbors(Node) []Node
}

func DFS(g Graph, start Node, visitor func(Node)) {
    visited := make(map[Node]bool)

    var dfs func(Node)
    dfs = func(n Node) {
        if visited[n] {
            return
        }
        visitor(n)
        visited[n] = true
        for _, n2 := range g.Neighbors(n) {
            dfs(n2)
        }
    }

    dfs(start)
}

This seems simple enough, but it has a lot of problems. For example, we loose type-safety: Even if we write Neighbors(Node) []Node there is no way to tell the compiler, that these instances of Node will actually always be the same. So an implementation of the graph interface would have to do type-assertions all over the place. Another problem is:

type AdjacencyList [][]int

func (l AdjacencyList) Neighbors(n Node) []Node {
    i := n.(int)
    var result []Node
    for _, j := range l[i] {
        result = append(result, j)
    }
    return result
}

An implementation of this interface as an adjacency-list actually performs pretty badly, because it can not return an []int, but must return a []Node, and even though int satisfies Node, []int is not assignable to []Node (for good reasons that lie in the implementation of interfaces, but still).

The way to solve this, is to always map your nodes to integers. This is what the standard library does in the sort-package. It is exactly the same problem. But it might not always be possible, let alone straightforward, to do this for Graphs, for example if they do not fit into memory (e.g. a web-crawler). The answer is to have the caller maintain this mapping via a map[Node]int or something similar, but… meh.

Weak references

I have to admit, that I am not sure, my use case here is really an important or even very nice one, but let's assume I want to have a database abstraction that transparently handles pointer-indirection. So let's say I have two tables T1 and T2 and T2 has a foreign key referencing T1. I think it would be pretty neat, if a database abstraction could automatically deserialize this into a pointer to a T1-value A. But to do this. we would a) need to be able to recognize A a later Put (so if the user changes A and later stores it, the database knows what row in T1 to update) and b) hand out the same pointer, if another row in T2 references the same id.

The only way I can think how to do this is to maintain a map[Id]*T1 (or similar), but this would prevent the handed out values to ever be garbage-collected. Even though there a hacks that would allow some use cases for weak references to be emulated, I don't see how they would work here.

So, as in the case of generics, this mainly means that some elegant APIs are not possible in go for library authors (and as I said, in this specific case it probably isn't a very good idea. For example you would have to think about what happens, if the user gets the same value in two different goroutines from the database).

Dynamic loading of go code

It would be useful to be able to dynamically load go code at runtime, to build plugins for go software. Specifically I want a good go replacement for jekyll because I went through some ruby-version-hell with it lately (for example jekyll serve -w still does not work for me with the version packaged in debian) and I think a statically linked go-binary would take a lot of possible pain-points out here. But plugins are a really important feature of jekyll for me, so I still want to be able to customize a page with plugins (how to avoid introducing the same version hell with this is another topic).

The currently recommended ways to do plugins are a) as go-packages and recompiling the whole binary for every change of a plugin and b) using sub-processes and net/rpc.

I don't feel a) being a good fit here, because it means maintaining a separate binary for every jekyll-site you have which just sounds like a smallish nightmare for binary distributions (plus I have use cases for plugins where even the relatively small compilation times of go would result in an intolerable increase in startup-time).

b) on the other hand results in a lot of runtime-penalty: For example I can not really pass interfaces between plugins, let alone use channels or something and every function call has to have its parameters and results serialized and deserialized. Where in the same process I can just define a transformation between different formats as a func(r io.Reader) io.Reader or something, in the RPC-context I first have to transmit the entire file over a socket, or have the plugin-author implement a net/rpc server himself and somehow pass a reference to it over the wire. This increases the burden on the plugin-authors too much, I think.

Luckily, it seems there seems to be some thought put forward recently on how to implement this, so maybe we see this in the nearish future.

Garbage-collected goroutines

Now, this is the only thing I really don't understand why it is not part of the language. Concurrency in go is a first-class citizen and garbage-collection is a feature emphasized all the time by the go-authors as an advantage. Yet, they both seem to not play entirely well together, making concurrency worse than it has to be.

Something like the standard example of how goroutines and channels work goes a little bit like this:

func Foo() {
    ch := make(chan int)
    go func() {
        i := 0
        for {
            ch <- i
            i++
        }
    }()

    for {
        fmt.Println(<-ch)
    }
}

Now, this is all well, but what if we want to exit the loop prematurely? We have to do something like this:

func Foo() {
    ch := make(chan int)
    done := make(chan bool)
    go func() {
        i := 0
        for {
            select {
                case ch <- i:
                    i++
                case <-done:
                    return
            }
        }
    }()
    for {
        i := <-ch
        if i > 1000 {
            break
        }
        fmt.Println(i)
    }
}

Because otherwise the goroutine would just stay around for all eternity, effectively being leaked memory. There are entire talks build around this and similar problems, where I don't really understand why. If we add a break to our first version, Foo returns and suddenly, all other references to ch, except the one the goroutine is blocking on writing to are gone and can be garbage-collected. The runtime can already detect if all goroutines are sleeping and we have a deadlock, the garbage-collector can accurately see what references there are to a given channel, why can we not combine the two to just see "there is absolutely no way, this channel-write can ever succeed, so let's just kill it and gc all it's memory"? This would have zero impact on existing programs (because as you can not get any references to goroutines, a deadlocked one can have no side-effect on the rest of the program), but it would make channels so much more fun to work with. It would make channels as iterators a truly elegant pattern, it would simplify pipelines and it would possibly allow a myriad other use cases for channels I can not think of right now. Heck, you could even think about (not sure if this is possible or desirable) running any deferred statements, when a goroutine is garbage-collected, so all other resources held by it will be correctly released.

This is the one thing I really wish to be added to the language. Really diving into channels and concurrency right now is very much spoiled for me because I always have to think about draining every channel, always think about what goroutine closes what channels, passing cancellation-channels…

12. September 2014 17:10:28

05. September 2014

sECuREs Webseite

Fiber7 performance

Ever since I moved to Zürich, I wanted to get a fiber internet connection. I’ve lived with a 6 Mbps DSL line at my parent’s place for about 10 years, so I was looking forward to a lot more Megabits and a lot less latency. For reasons that I won’t go into in this article, it took me about a year to get a fiber connection, and in the end I had to go with Swisscom (instead of init7 on top of EWZ).

But then fiber7 launched. They provide a 1 Gbps symmetrical connection (Swisscom provided a 1 Gbps/100 Mbps down/up connection) for a lot less money than Swisscom, and with native, static IPv6.

A couple of people are interested in how fiber7 performs, and after being connected for about 2 months, I think I can answer this question by now :-).

Latency

I started running smokeping to see how my internet connection performs back when I was with Swisscom, because they had some routing issues to certain networks. This would manifest itself with getting 50 KB/s transfer rates, which is unacceptable for image boards or any other demanding application.

So, here is the smokeping output for google.ch during the time period that covers both my Swisscom line, the temporary cablecom connection and finally fiber7:

smokeping latency to google.ch (annotated)

What you can see is that with Swisscom, I had a 6 ms ping time to google.ch. Interestingly, once I used the MikroTik RB2011 instead of the Swisscom-provided internet box, the latency improved to 5 ms.

Afterwards, latency changed twice. For the first change, I’m not sure what happened. It could be that Swisscom turned up a new, less loaded port to peer with Google. Or perhaps they configured their systems in a different way, or exchanged some hardware. The second change is relatively obvious: Swisscom enabled GGC, the Google Global Cache. GGC is a caching server provided by Google that is placed within the ISP’s own network, typically providing much better latencies (due to being placed very close to the customer) and reducing the traffic between the ISP and Google. I’m confident that Swisscom uses that because of the reverse pointer record of the IP address to which google.ch resolves to. So with that, latency is between 1 ms and 3 ms.

Because switching to Fiber7 involves recabling the physical fiber connection in the POP, there is a 2-day downtime involved. During that time I used UPC cablecom’s free offering, which is a 2 Mbps cable connection that you can use for free (as long as you pay for the cable connection itself, and after paying 50 CHF for the modem itself).

As you can see on the graph, the cable connection has a surprisingly good latency of around 8 ms to google.ch — until you start using it. Then it’s clear that 2 Mbps is not enough and the latency shoots through the roof.

The pleasant surprise is that fiber7’s ping time to google.ch is about 0.6 ms (!). They achieve such low latency with a dedicated 10 gig interconnect to Google at the Interxion in Glattbrugg.

Longer-term performance

smokeping latency measurements to google.ch over more than a week

Let me say that I’m very happy with the performance of my internet connection. Some of the measurements where packet loss is registered may be outside of fiber7’s control, or even caused by me, when recabling my equipment for example. Overall, the latency is fine and consistent, much more so than with Swisscom. I have never experienced an internet outage during the two months I’ve been with fiber7 now.

Also, while I am not continuously monitoring my bandwidth, rest assured that whenever I download something, I am able to utilize the full Gigabit, meaning I get an aggregate speed of 118 MB/s from servers that support it. Such servers are for example one-click hosters like uploaded, but also Debian mirrors (as soon as you download from multiple ones in parallel).

Conclusion

tl;dr: fiber7 delivers. Incredible latency, no outages (yet), full download speed.

by Michael Stapelberg at 05. September 2014 12:00:00

31. August 2014

sECuREs Webseite

Replicated PostgreSQL with pgpool2

I run multiple web services, mostly related to i3wm.org. All of them use PostgreSQL as their database, so the data that is stored in that PostgreSQL database is pretty important to me and the users of these services.

Since a while now, I have been thinking about storing that data in a more reliable way. Currently, it is stored on a single server, and is backed up to two different locations (one on-site, one off-site) every day. The server in question has a RAID-1 of course, but still: the setup implies that if that one server dies, the last backup may be about a day old in the worst case, and also it could take me significant time to get the services back up.

The areas in which I’d like to improve my setup are thus:

  1. Durability: In case the entire server dies, I want to have an up-to-date copy of all data.
  2. Fault tolerance: In case the entire server dies, I want to be able to quickly switch to a different server. A secondary machine should be ready to take over, albeit not fully automatically because fully automatic solutions typically are either fragile or require a higher number of servers than I’m willing to afford.

For PostgreSQL, there are various settings and additional programs that you can use which will provide you with some sort of clustering/replication. There is an overview in the PostgreSQL wiki (“Replication, Clustering, and Connection Pooling”). My solution of choice is pgpool2 because it seems robust and mature (yet under active development) to me, it is reasonably well documented and I think I roughly understand what it does under the covers.

The plan

I have two servers, located in different data centers, that I will use for this setup. The number of servers does not really matter, meaning you can easily add a third or fourth server (increasing latency with every server of course). However, the low number of servers places some restrictions on what we can do. As an example, solutions that involve global consistency based on paxos/raft quorums will not work with only two servers. As a consequence, master election is out of the question and a human will need to do the actual failover/recovery.

Each of the two servers will run PostgreSQL, but only one of them will run pgpool2 at a time. The DNS records for e.g. faq.i3wm.org will point to the server on which pgpool2 is running, so that server handles 100% of the traffic. Let’s call the server running pgpool2 the primary, and the other server the secondary. All queries that modify the database will still be sent to the secondary, but the secondary does not handle any user traffic. This could be accomplished by either not running the applications in question, or by having them connect to the pgpool2 on the primary.

When a catastrophe happens, the DNS records will be switched to point to the old-secondary server, and pgpool2 will be started there. Once the old-primary server is available again, it will become the secondary server, so that in case of another catastrophe, the same procedure can be executed again.

With a solution that involves only two servers, an often encountered problem are split-brain situations. This means both servers think they are primary, typically because there is a network partition, meaning the servers cannot talk to each other. In our case, it is important that user traffic is not handled by the secondary server. This could happen after failing over because DNS heavily relies on caching, so switching the record does not mean that suddenly all queries will go to the other server — this will only happen over time. A solution for that is to either kill pgpool2 manually if possible, or have a tool that kills pgpool2 when it cannot verify that the DNS record points to the server.

Configuration

I apologize for the overly long lines in some places, but there does not seem to be a way to use line continuations in the PostgreSQL configuration file.

Installing and configuring PostgreSQL

The following steps need to be done on each database server, whereas pgpool2 will only be installed on precisely one server.

Also note that a prerequisite for the configuration described below is that hostnames are configured properly on every involved server, i.e. hostname -f should return the fully qualified hostname of the server in question, and other servers must be able to connect to that hostname.

apt-get install postgresql postgresql-9.4-pgpool2 rsync ssh
cat >>/etc/postgresql/9.4/main/postgresql.conf <<'EOT'
listen_addresses = '*'

max_wal_senders = 1
wal_level = hot_standby
archive_mode = on
archive_command = 'test ! -f /var/lib/postgresql/9.4/main/archive_log/backup_in_progress || (test -f /var/lib/postgresql/9.4/main/archive_log/%f || cp %p /var/lib/postgresql/9.4/main/archive_log/%f)'
EOT
install -o postgres -g postgres -m 700 -d \
  /var/lib/postgresql/9.4/main/archive_log
systemctl restart postgresql.service

pgpool comes with an extension (implemented in C) that provides a couple of functions which are necessary for recovery. We need to “create” the extension in order to be able to use these functions. After running the following command, you can double-check with \dx that the extension was installed properly.

echo 'CREATE EXTENSION "pgpool_recovery"' | \
  su - postgres -c 'psql template1'

During recovery, pgpool needs to synchronize data between the PostgreSQL servers. This is done partly by running pg_basebackup on the recovery target via SSH and using rsync (which connects using SSH). Therefore, we need to create a passwordless SSH key for the postgres user. For simplicity, I am implying that you’ll copy the same id_rsa and authorized_keys files onto every database node. You’ll also need to connect to every other database server once in order to get the SSH host fingerprints into the known_hosts file.

su - postgres
ssh-keygen -f /var/lib/postgresql/.ssh/id_rsa -N ''
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
exit

We’ll also need to access remote databases with pg_basebackup non-interactively, so we need a password file:

su - postgres
echo '*:*:*:postgres:wQgvBEusf1NWDRKVXS15Fc8' > .pgpass
chmod 0600 .pgpass
exit

When pgpool recovers a node, it first makes sure the data directory is up to date, then it starts PostgreSQL and tries to connect repeatedly. Once the connection succeeded, the node is considered healthy. Therefore, we need to give the postgres user permission to control postgresql.service:

apt-get install sudo
cat >/etc/sudoers.d/pgpool-postgres <<'EOT'
postgres ALL=(ALL:ALL) NOPASSWD:/bin/systemctl start postgresql.service
postgres ALL=(ALL:ALL) NOPASSWD:/bin/systemctl stop postgresql.service
EOT

Now enable password-based authentication for all databases and replication traffic. In case your database nodes/clients don’t share a common hostname suffix, you may need to use multiple entries or replace the hostname suffix by “all”.

cat >>/etc/postgresql/9.4/main/pg_hba.conf <<'EOT'
host    all             all             .zekjur.net             md5     
host    replication     postgres        .zekjur.net             md5     
EOT

After enabling password-based authentication, we need to set a password for the postgres user which we’ll use for making the base backup:

echo "ALTER USER postgres WITH PASSWORD 'wQgvBEusf1NWDRKVXS15Fc8';" | \
  su postgres -c psql

Installing pgpool2

apt-get install pgpool2
cd /etc/pgpool2
gunzip -c /usr/share/doc/pgpool2/examples/\
pgpool.conf.sample-replication.gz > pgpool.conf

To interact with pgpool2, there are a few command-line utilities whose name starts with pcp_. In order for these to work, we must configure a username and password. For simplicity, I’ll re-use the password we set earlier for the postgres user, but you could chose to use an entirely different username/password:

echo "postgres:$(pg_md5 wQgvBEusf1NWDRKVXS15Fc8)" >> pcp.conf

In replication mode, when the client should authenticate towards the PostgreSQL database, we also need to tell pgpool2 that we are using password-based authentication:

sed -i 's/trust$/md5/g' pool_hba.conf
sed -i 's/\(enable_pool_hba =\) off/\1 on/g' pgpool.conf

Furthermore, we need to provide all the usernames and passwords that we are going to use to pgpool2:

touch pool_passwd
chown postgres.postgres pool_passwd
pg_md5 -m -u faq_i3wm_org secretpassword

For the use-case I am describing here, it is advisable to turn off load_balance_mode, otherwise queries will be sent to all healthy backends, which is slow because they are not in the same network. In addition, we’ll assign a higher weight to the backend which runs on the same machine as pgpool2, so read-only queries are sent to the local backend only.

sed -i 's/^load_balance_mode = on/load_balance_mode = off/g' \
    pgpool.conf

Now, we need to configure the backends.

sed -i 's/^\(backend_\)/# \1/g' pgpool.conf

cat >>pgpool.conf <<'EOT'
backend_hostname0 = 'midna.zekjur.net'
backend_port0 = 5432
backend_weight0 = 2
backend_data_directory0 = '/var/lib/postgresql/9.4/main'

backend_hostname1 = 'alp.zekjur.net'
backend_port1 = 5432
backend_weight1 = 1
backend_data_directory1 = '/var/lib/postgresql/9.4/main'
EOT

Overview: How recovery works

Let’s assume that pgpool is running on midna.zekjur.net (so midna is handling all the traffic), and alp.zekjur.net crashed. pgpool will automatically degrade alp and continue operation. When you tell it to recover alp because the machine is available again, it will do three things:

  1. (“1st stage”) SSH into alp and run pg_basebackup to get a copy of midna’s database.
  2. (“2nd stage”) Disconnect all clients so that the database on midna will not be modified anymore. Flush all data to disk on midna, then rsync the data to alp. pg_basebackup from 1st stage will have copied almost all of it, so this is a small amount of data — typically on the order of 16 MB, because that’s how big one WAL file is.
  3. Try to start PostgreSQL on alp again. pgpool will wait for 90 seconds by default, and within that time PostgreSQL must start up in such a state that pgpool can connect to it.

So, during the 1st stage, which copies the entire database, traffic will still be handled normally, only during 2nd stage and until PostgreSQL started up no queries are served.

Configuring recovery

For recovery, we need to provide pgpool2 with a couple of shell scripts that handle the details of how the recovery is performed.

sed -i 's/^\(recovery_\|client_idle_limit_in_recovery\)/# \1/g' \
    pgpool.conf

cat >>pgpool.conf <<'EOT'
recovery_user = 'postgres'
recovery_password = 'wQgvBEusf1NWDRKVXS15Fc8'

# This script is being run by invoking the pgpool_recovery() function on
# the current master(primary) postgresql server. pgpool_recovery() is
# essentially a wrapper around system(), so it runs under your database
# UNIX user (typically "postgres").
# Both scripts are located in /var/lib/postgresql/9.4/main/
recovery_1st_stage_command = '1st_stage.sh'
recovery_2nd_stage_command = '2nd_stage.sh'

# Immediately disconnect all clients when entering the 2nd stage recovery
# instead of waiting for the clients to disconnect.
client_idle_limit_in_recovery = -1
EOT

The 1st_stage.sh script logs into the backend that should be recovered and uses pg_basebackup to copy a full backup from the master(primary) backend. It also sets up the recovery.conf which will be used by PostgreSQL when starting up.

cat >/var/lib/postgresql/9.4/main/1st_stage.sh <<'EOF'
#!/bin/sh
TS=$(date +%Y-%m-%d_%H-%M-%S)
MASTER_HOST=$(hostname -f)
MASTER_DATA=$1
RECOVERY_TARGET=$2
RECOVERY_DATA=$3

# Move the PostgreSQL data directory out of our way.
ssh -T $RECOVERY_TARGET \
    "[ -d $RECOVERY_DATA ] && mv $RECOVERY_DATA $RECOVERY_DATA.$TS"

# We only use archived WAL logs during recoveries, so delete all
# logs from the last recovery to limit the growth.
rm $MASTER_DATA/archive_log/*

# With this file present, our archive_command will actually
# archive WAL files.
touch $MASTER_DATA/archive_log/backup_in_progress

# Perform a backup of the database.
ssh -T $RECOVERY_TARGET \
    "pg_basebackup -h $MASTER_HOST -D $RECOVERY_DATA --xlog"

# Configure the restore_command to use the archive_log WALs we’ll copy
# over in 2nd_stage.sh.
echo "restore_command = 'cp $RECOVERY_DATA/archive_log/%f %p'" | \
    ssh -T $RECOVERY_TARGET "cat > $RECOVERY_DATA/recovery.conf"
EOF
cat >/var/lib/postgresql/9.4/main/2nd_stage.sh <<'EOF'
#! /bin/sh
MASTER_DATA=$1
RECOVERY_TARGET=$2
RECOVERY_DATA=$3
port=5432

# Force to flush current value of sequences to xlog
psql -p $port -t -c 'SELECT datname FROM pg_database WHERE NOT datistemplate AND datallowconn' template1|
while read i
do
  if [ "$i" != "" ];then
    psql -p $port -c "SELECT setval(oid, nextval(oid)) FROM pg_class WHERE relkind = 'S'" $i
  fi
done

# Flush all transactions to disk. Since pgpool stopped all connections,
# there cannot be any data that does not reside on disk until the
# to-be-recovered host is back on line.
psql -p $port -c "SELECT pgpool_switch_xlog('$MASTER_DATA/archive_log')" template1

# Copy over all archive logs at once.
rsync -avx --delete $MASTER_DATA/archive_log/ \
    $RECOVERY_TARGET:$RECOVERY_DATA/archive_log/

# Delete the flag file to disable WAL archiving again.
rm $MASTER_DATA/archive_log/backup_in_progress
EOF
cat >/var/lib/postgresql/9.4/main/pgpool_remote_start <<'EOF'
#!/bin/sh
ssh $1 sudo systemctl start postgresql.service
EOF

chmod +x /var/lib/postgresql/9.4/main/1st_stage.sh
chmod +x /var/lib/postgresql/9.4/main/2nd_stage.sh
chmod +x /var/lib/postgresql/9.4/main/pgpool_remote_start

Now, let’s start pgpool2 and verify that it works and that we can access our first node. The pcp_node_count command should return an integer number like “2”. The psql command should be able to connect and you should see your database tables when using \d.

systemctl restart pgpool2.service
pcp_node_count 10 localhost 9898 postgres wQgvBEusf1NWDRKVXS15Fc8
psql -p 5433 -U faq_i3wm_org faq_i3wm_org

Monitoring

pgpool2 intercepts a couple of SHOW statements, so you can use the SQL command SHOW pool_nodes to see how many nodes are there:

> SHOW pool_nodes;
 node_id |     hostname     | port | status | lb_weight |  role  
---------+------------------+------+--------+-----------+--------
 0       | midna.zekjur.net | 5432 | 2      | 0.666667  | master
 1       | alp.zekjur.net   | 5432 | 2      | 0.333333  | slave
(2 rows)

You could export a cgi-script over HTTP, which just always runs this command, and then configure your monitoring software to watch for certain strings in the output. Note that you’ll also need to configure a ~/.pgpass file for the www-data user. As an example, to monitor whether alp is still a healthy backend, match for “alp.zekjur.net,5432,2” in the output of this script:

#!/bin/sh
cat <<'EOT'
Content-type: text/plain

EOT
exec echo 'SHOW pool_nodes;' | psql -t -A -F, --host localhost \
  -U faq_i3wm_org faq_i3wm_org

Performing/Debugging a recovery

In order to recover node 1 (alp in this case), use:

pcp_recovery_node 300 localhost 9898 postgres wQgvBEusf1NWDRKVXS15Fc8 1

The “300” used to be a timeout, but these days it’s only supported for backwards compatibility and has no effect.

In case the recovery fails, the only thing you’ll get back from pcp_recovery_node is the text “BackendError”, which is not very helpful. The logfile of pgpool2 contains a bit more information, but to debug recovery problems, I typically strace all PostgreSQL processes and see what the scripts are doing/where they are failing.

pgpool2 behavior during recovery

In order to see how pgpool2 performs during recovery/degradation, you can use this little Go program that tries to do three things every 0.25 seconds: check that the database is healthy (SELECT 1;), run a meaningful SELECT, run an UPDATE.

When a database node goes down, a single query may fail until pgpool2 realizes that the node needs to be degraded. If your database load is light, chances are that pgpool2 will realize the database is down without even failing a single query, though.

2014-08-13 23:15:27.638 health: ✓  select: ✓  update: ✓
2014-08-13 23:15:28.700 insert failed: driver: bad connection
2014-08-13 23:15:28.707 health: ✓  select: ✓  update: x

During recovery, there is a time when pgpool2 will just disconnect all clients and not answer any queries any more (2nd stage). In this case, the state lasted for about 20 seconds:

…
2014-08-13 23:16:01.900 health: ✓  select: ✓  update: ✓
2014-08-13 23:16:02.161 health: ✓  select: ✓  update: ✓
# no queries answered here
2014-08-13 23:16:23.625 health: ✓  select: ✓  update: ✓
2014-08-13 23:16:24.308 health: ✓  select: ✓  update: ✓
…

Conclusion

Setting up a PostgreSQL setup that involves pgpool2 is definitely a lot of work. It could be a bit easier if the documentation was more specific on the details of how recovery is supposed to work and would include the configuration that I came up with above. Ideally, something like pgpool2 would be part of PostgreSQL itself.

I am not yet sure how much software I’ll need to touch in order to make it gracefully deal with the PostgreSQL connection dying and coming back up. I know of at least one program I use (buildbot) which does not handle this situation well at all — it needs a complete restart to work again.

Time will tell if the setup is stable and easy to maintain. In case I make negative experiences, I’ll update this article :).

by Michael Stapelberg at 31. August 2014 16:00:00

26. August 2014

Raphael Michel

SSL-enabled web servers on Android devices

Scenario

I am currently working on a software for a client, in which there are multiple Android devices involved. One of them works as a server while the other devices work as clients, fetching and pushing data from and to the server to keep all the data in sync. The software is supposed to be deployed on trusted devices in a trusted wireless network specifically set up for this application. Nevertheless, experience tells that in reality, it probably will be used in wireless networks which are not isolated.

The software is to be used at event locations and may not require internet connection at any time. It should work with any set of Android (4.0+) devices with having the app installed being the only prerequisite, wheras the same app should work as both client and server. The software should assume that the connection between the devices might be broken at any time and configuration should be simple for non-technical people.

As I'm pretty sure I'm not the only one having this requirements for an Android app project, I'm gonna talk a bit about how I've done it.

Implementation idea

The protocol used between the Android devices is HTTP. As I must assume that the app is being used in unisolated WiFi networks, the communication has to be SSL encrypted.

It is quite easy to run a Jetty server inside an Android app1, and it is also possible to use SSL encryption with Jetty. However, all documentation and examples on this I managed to find, were suggesting creating a SSL cert with keytool on your computer, storing it in a BKS keystore and shipping it with your application, or, having your users do this and letting them specify a path on the SD card to the keystore.

Neither of those is a real option for me: I cannot assume any of my users ever will create his own certificate with keytool and I also cannot ship a hard-coded private key with my application, as the app itself might not be considered a secret and using SSL with known keys is not even slightly better than not using encryption at all. Therefore, I must generate the SSL key and certificate on each device seperately on the first use of the app. I will present my code for this in the next section.

After the certificate is generated, the clients need to know the certificate's fingerprint: If the client would just accept any certificate, I could have avoided all the key generation work above, because a client accepting all certificates is nearly as bad as shipping the same keys on every device. As the system has to work offline and ad-hoc, there is no way to use something like a CA infrastructure.

Luckily, there is an elegant way to solve both the certificate and the configuration problem at once: The server device shows a QR code containing ip address, port and SSL fingerprint of the server as well as an authentication token (being in a public network, we want both encryption and authentication). The client just has to scan this QR code and gets all informaction necessary for etablishing a secure connection.

Implementation details

Dependencies

This effort has introduced a bunch of new dependencies to my application

  • My webserver component is built on Jetty 8.1.15, of which I created my own jar bundle (using jar xf and jar cf) containing:
    • jetty-continuation-8.1.15.v20140411.jar
    • jetty-http-8.1.15.v20140411.jar
    • jetty-io-8.1.15.v20140411.jar
    • jetty-security-8.1.15.v20140411.jar
    • jetty-server-8.1.15.v20140411.jar
    • jetty-servlet-8.1.15.v20140411.jar
    • jetty-util-8.1.15.v20140411.jar
    • jetty-webapp-8.1.15.v20140411.jar
    • jetty-xml-8.1.15.v20140411.jar
    • servlet-api-3.0.jar
  • Bouncycastle's bcprov-jdk15on-146.jar for keystore handling
  • Apache Commons Codec for fingerprinting keys
  • ZXing's core.jar for QR code generation or David Lazaro's wonderful QRCodeReaderView for easy QR code scanning (already includes core.jar)

Key generation

The hardest part was generating the keypair and certificate. I've got2 some3 inspiration4 from the web, but as I did not find an example ready to work on Android, here's mine:

/**
 * Creates a new SSL key and certificate and stores them in the app's
 * internal data directory.
 * 
 * @param ctx
 *            An Android application context
 * @param keystorePassword
 *            The password to be used for the keystore
 * @return boolean indicating success or failure
 */
public static boolean genSSLKey(Context ctx, String keystorePassword) {
    try {
        // Create a new pair of RSA keys using BouncyCastle classes
        RSAKeyPairGenerator gen = new RSAKeyPairGenerator();
        gen.init(new RSAKeyGenerationParameters(BigInteger.valueOf(3),
                new SecureRandom(), 1024, 80));
        AsymmetricCipherKeyPair keypair = gen.generateKeyPair();
        RSAKeyParameters publicKey = (RSAKeyParameters) keypair.getPublic();
        RSAPrivateCrtKeyParameters privateKey = (RSAPrivateCrtKeyParameters) keypair
                .getPrivate();

        // We also need our pair of keys in another format, so we'll convert
        // them using java.security classes
        PublicKey pubKey = KeyFactory.getInstance("RSA").generatePublic(
                new RSAPublicKeySpec(publicKey.getModulus(), publicKey
                        .getExponent()));
        PrivateKey privKey = KeyFactory.getInstance("RSA").generatePrivate(
                new RSAPrivateCrtKeySpec(publicKey.getModulus(), publicKey
                        .getExponent(), privateKey.getExponent(),
                        privateKey.getP(), privateKey.getQ(), privateKey
                                .getDP(), privateKey.getDQ(), privateKey
                                .getQInv()));

        // CName or other certificate details do not really matter here
        X509Name x509Name = new X509Name("CN=" + CNAME);

        // We have to sign our public key now. As we do not need or have
        // some kind of CA infrastructure, we are using our new keys
        // to sign themselves

        // Set certificate meta information
        V3TBSCertificateGenerator certGen = new V3TBSCertificateGenerator();
        certGen.setSerialNumber(new DERInteger(BigInteger.valueOf(System
                .currentTimeMillis())));
        certGen.setIssuer(new X509Name("CN=" + CNAME));
        certGen.setSubject(x509Name);
        DERObjectIdentifier sigOID = PKCSObjectIdentifiers.sha1WithRSAEncryption;
        AlgorithmIdentifier sigAlgId = new AlgorithmIdentifier(sigOID,
                new DERNull());
        certGen.setSignature(sigAlgId);
        ByteArrayInputStream bai = new ByteArrayInputStream(
                pubKey.getEncoded());
        ASN1InputStream ais = new ASN1InputStream(bai);
        certGen.setSubjectPublicKeyInfo(new SubjectPublicKeyInfo(
                (ASN1Sequence) ais.readObject()));
        bai.close();
        ais.close();

        // We want our keys to live long
        Calendar expiry = Calendar.getInstance();
        expiry.add(Calendar.DAY_OF_YEAR, 365 * 30);

        certGen.setStartDate(new Time(new Date(System.currentTimeMillis())));
        certGen.setEndDate(new Time(expiry.getTime()));
        TBSCertificateStructure tbsCert = certGen.generateTBSCertificate();

        // The signing: We first build a hash of our certificate, than sign
        // it with our private key
        SHA1Digest digester = new SHA1Digest();
        AsymmetricBlockCipher rsa = new PKCS1Encoding(new RSAEngine());
        ByteArrayOutputStream bOut = new ByteArrayOutputStream();
        DEROutputStream dOut = new DEROutputStream(bOut);
        dOut.writeObject(tbsCert);
        byte[] signature;
        byte[] certBlock = bOut.toByteArray();
        // first create digest
        digester.update(certBlock, 0, certBlock.length);
        byte[] hash = new byte[digester.getDigestSize()];
        digester.doFinal(hash, 0);
        // and sign that
        rsa.init(true, privateKey);
        DigestInfo dInfo = new DigestInfo(new AlgorithmIdentifier(
                X509ObjectIdentifiers.id_SHA1, null), hash);
        byte[] digest = dInfo.getEncoded(ASN1Encodable.DER);
        signature = rsa.processBlock(digest, 0, digest.length);
        dOut.close();
        
        // We build a certificate chain containing only one certificate
        ASN1EncodableVector v = new ASN1EncodableVector();
        v.add(tbsCert);
        v.add(sigAlgId);
        v.add(new DERBitString(signature));
        X509CertificateObject clientCert = new X509CertificateObject(
                new X509CertificateStructure(new DERSequence(v)));
        X509Certificate[] chain = new X509Certificate[1];
        chain[0] = clientCert;

        // We add our certificate to a new keystore
        KeyStore keyStore = KeyStore.getInstance("BKS");
        keyStore.load(null);
        keyStore.setKeyEntry(KEY_ALIAS, (Key) privKey,
                keystorePassword.toCharArray(), chain);
        
        // We write this keystore to a file
        OutputStream out = ctx.openFileOutput(FILE_NAME,
                Context.MODE_PRIVATE);
        keyStore.store(out, keystorePassword.toCharArray());
        out.close();
        return true;
    } catch (Exception e) {
        // Do your exception handling here
        // There is a lot which might go wrong
        e.printStackTrace();
    }
    return false;
}

The key generation takes roughly fifteen seconds on my Motorola Moto G, so it is strongly discouraged to do this in the UI thread – do it in your Service (you should have one for your server!) or in an AsyncTask.

FILE_NAME is the name of your key store (I use keystore.bks) and KEY_ALIAS the alias of the new key inside the keystore (I use ssl).

Jetty initialization

In the initialization code of our jetty servlet, we have to load our newly created keystore into a SslContextFactory, which is quite easy:

SslContextFactory sslContextFactory = new SslContextFactory();
InputStream in = openFileInput(SSLUtils.FILE_NAME);
KeyStore keyStore = KeyStore.getInstance("BKS");
try {
    keyStore.load(in, KEYSTORE_PASSWORD.toCharArray());
} finally {
    in.close();
}
sslContextFactory.setKeyStore(keyStore);
sslContextFactory.setKeyStorePassword(KEYSTORE_PASSWORD);
sslContextFactory.setKeyManagerPassword(KEYSTORE_PASSWORD);
sslContextFactory.setCertAlias(SSLUtils.KEY_ALIAS);
sslContextFactory.setKeyStoreType("bks");
// We do not want to speak old SSL and we only want to use strong ciphers
sslContextFactory.setIncludeProtocols("TLS");
sslContextFactory.setIncludeCipherSuites("TLS_DHE_RSA_WITH_AES_128_CBC_SHA");

Server server = new Server();
SslSelectChannelConnector sslConnector = new SslSelectChannelConnector(
        sslContextFactory);
sslConnector.setPort(PORT);
server.addConnector(sslConnector);

// As before:
server.setHandler(handler); // where handler is an ``AbstractHandler`` instance
server.start();

QR Code generation

In order to display the QR code, we first need to create a SHA1 hash of our certificate:

public static String getSHA1Hash(Context ctx, String keystorePassword) {
    InputStream in = null;
    KeyStore keyStore;
    try {
        in = ctx.openFileInput(FILE_NAME);
        keyStore = KeyStore.getInstance("BKS");
        keyStore.load(in, keystorePassword.toCharArray());
        return new String(Hex.encodeHex(DigestUtils.sha1(keyStore
                .getCertificate(KEY_ALIAS).getEncoded())));
    } catch (Exception e) {
        // Should not go wrong on standard Android devices
        // except possible IO errors on reading the keystore file
        e.printStackTrace();
    } finally {
        try {
            if (in != null)
                in.close();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
    return null;
}

Using this method, we can generate a QR code containing ip, port and certificate information and draw it onto an ImageView:

protected void genQrCode(ImageView view) {
    QRCodeWriter writer = new QRCodeWriter();
    try {
        WifiManager wifiManager = (WifiManager) getActivity()
                .getSystemService(WIFI_SERVICE);
        int ipAddress = wifiManager.getConnectionInfo().getIpAddress();
        final String formatedIpAddress = String.format(Locale.GERMAN,
                "%d.%d.%d.%d", (ipAddress & 0xff),
                (ipAddress >> 8 & 0xff), (ipAddress >> 16 & 0xff),
                (ipAddress >> 24 & 0xff));

        JSONObject qrdata = new JSONObject();
        qrdata.put("server", formatedIpAddress);
        qrdata.put("port", ServerService.PORT);
        qrdata.put("cert", getSHA1Hash(getActivity(), KEYSTORE_PASSWORD));
        qrdata.put("secret", SECRET); // for authentitication. Generate yourself ;)

        BitMatrix bitMatrix = writer.encode(qrdata.toString(),
                BarcodeFormat.QR_CODE, 500, 500);
        int width = bitMatrix.getWidth();
        int height = bitMatrix.getHeight();
        Bitmap bmp = Bitmap.createBitmap(width, height,
                Bitmap.Config.RGB_565);
        for (int x = 0; x < width; x++) {
            for (int y = 0; y < height; y++) {
                bmp.setPixel(x, y, bitMatrix.get(x, y) ? Color.BLACK
                        : Color.WHITE);
            }
        }
        view.setImageBitmap(bmp);

    } catch (WriterException e) {
        e.printStackTrace();
    } catch (JSONException e) {
        e.printStackTrace();
    }
}

Client side

On the client side, HttpsURLConnection is being used for the connection. This works roughly like this:

// My application saves server address, port and certificate
// in a SharedPreferences store
SharedPreferences sp = getSharedPreferences("server",
        Context.MODE_PRIVATE);

X509PinningTrustManager trustManager = new X509PinningTrustManager(
        sp.getString("cert", ""));

SSLContext sc = null;
DataSource data = getDataSource();
HttpsURLConnection urlConnection = null;
try {
    data.open();

    URL url = new URL("https://" + sp.getString("server", "") + ":"
            + sp.getInt("port", ServerService.PORT) + "/path");
    urlConnection = (HttpsURLConnection) url.openConnection();

    // Set our SSL settings settings
    urlConnection
            .setHostnameVerifier(trustManager.new HostnameVerifier());
    try {
        sc = SSLContext.getInstance("TLS");
        sc.init(null, new TrustManager[] { trustManager },
                new java.security.SecureRandom());
        urlConnection.setSSLSocketFactory(sc.getSocketFactory());
    } catch (NoSuchAlgorithmException e1) {
        // Should not happen...
        e1.printStackTrace();
    } catch (KeyManagementException e1) {
        // Should not happen...
        e1.printStackTrace();
    }

    // do HTTP POST or authentication stuff here...
    InputStream in = new BufferedInputStream(
            urlConnection.getInputStream());
    // process the response...
} catch (javax.net.ssl.SSLHandshakeException e) {
    // We got the wrong certificate
    // (or handshake was interrupted, we can't tell here)
    e.printStackTrace();
} catch (IOException e) {
    // other IO errors
    e.printStackTrace();
} finally {
    if (urlConnection != null)
        urlConnection.disconnect();
    try {
        data.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

If you /only/ connect to this server in your whole application, you can use the HttpsUrlConnection.setDefault* methods instead of specifying it with every request. With writing a clever HostnameVerifier and TrustManager you could also achieve that the pinning is only enforced for your server but the system defaults are used for other servers.

The above code example makes use of a TrustManager class I implemented myself. It only accepts exactly one certificate:

/**
 * This class provides an X509 Trust Manager trusting only one certificate.
 */
public class X509PinningTrustManager implements X509TrustManager {

    String pinned = null;

    /**
    * Creates the Trust Manager.
    * 
    * @param pinnedFingerprint
    *            The certificate to be pinned. Expecting a SHA1 fingerprint in
    *            lowercase without colons.
    */
    public X509PinningTrustManager(String pinnedFingerprint) {
        pinned = pinnedFingerprint;
    }

    public java.security.cert.X509Certificate[] getAcceptedIssuers() {
        return new X509Certificate[] {};
    }

    public void checkClientTrusted(X509Certificate[] certs, String authType)
            throws CertificateException {
        checkServerTrusted(certs, authType);
    }

    public void checkServerTrusted(X509Certificate[] certs, String authType)
            throws CertificateException {
        for (X509Certificate cert : certs) {
            try {
                String fingerprint = new String(Hex.encodeHex(DigestUtils
                        .sha1(cert.getEncoded())));
                if (pinned.equals(fingerprint))
                    return;
            } catch (CertificateEncodingException e) {
                e.printStackTrace();
            }
        }
        throw new CertificateException("Certificate did not match, pinned to "
                + pinned);
    }

    /**
    * This hostname verifier does not verify hostnames. This is not necessary,
    * though, as we only accept one single certificate.
    *
    */
    public class HostnameVerifier implements javax.net.ssl.HostnameVerifier {
        public boolean verify(String hostname, SSLSession session) {
            return true;
        }
    }
}

Security considerations

This approach should be very secure against most attackers, as all network traffic is encrypted and traditional man-in-the-middle is not possible, as the client exactly know which certificate it expects and does not accept others. We even have Perfect Forward Secrecy! The fact that this information is not being transferred via network but via a QR code adds additional security.

However, there are some security problems left:

  • Most important: We only have TLS 1.0 available. I did not find a possibility to enable TLS 1.2. This is sad. I suspect the reason is that Android is still based on Java 6 and TLS 1.2 was introduced with Java 7. There is a possibility of running Java 7 code starting with Android KitKat, but this did not help in my quick test.
  • The keystore password is hardcoded. The only other option would be to prompt the user for the password on every application startup, which is not desirable. This, however, is only important if someone gains access to the keystore file, which is only readable for the app itself and root. And if our potential attacker is root on your phone, I guess you've got bigger problems than this keystore… Remember, my application is supposed to run on phones dedicated to run this application (with not many 3rd-party applications introducing vulnerabilities being installed).
  • SecureRandom is not cryptographically secure in Android 4.3 or older, as Android Lint points out. This official blog post5 has some more details and a workaround, but if you care about security, you should not run an old operating system anyway ;)

26. August 2014 22:00:00

24. August 2014

Moredreads Blog

Atsutane's blog

FrOSCon 2014 Summary

This year’s FrOSCon’s over, we had an unused booth (later used by someone for presenting emacs) and our own room, including some talks and a workshop. The talks were surprisingly well visited, we did not expect so many people to come. The original expectation was that we might be 15 people, but we got a larger room, probably a good decision by the organization team.

There are some things we can and should do better with future events, this years organization was – as always – really chaotic.

There are few things we did well and others we learned that we handled them… well bad is the proper term.

  • We need to give talks on each day we get a room. This year we only gave these on the first day and the second day the room was quiet. (Positive effect is the reogarnization of the SSL certificates.)

  • We have to give the talks in a better prepared way and not come up with the topics the weekend before the event and create the slides last-minute.

  • We used an Etherpad for organization of a todo list this was from my point of view well handled.

  • When we want to get a room skip a booth. We did not use the booth as we had no hardware to present something down there.

  • Merchandising, it’s every year the same, users want to get some merchandising. If I saw it correctly Debian sells T-Shirts, with german law we probably can’t do something like that to get a bit profit invested into our servers. To give away free stickers to actual users (not arrogant Fedora folks) might be an option, yet we would have to get them on our own costs. Another option would be to talk with the people from the merchandising booth at the entry.

  • When we give workshops we need a better organization, especially with installation stuff. I haven’t done an installation with the pacstrap script yet, leaving me in the same situation as someone new: How the heck do I initialize an installation.

24. August 2014 17:34:38

19. August 2014

Atsutane's blog

FrOSCon 2014

For those who may miss to see us people downstairs next to some other distro/BSD: This year Arch Linux has its own room at FrOSCon, you’ll find us people in C125.

19. August 2014 05:42:49

12. August 2014

Mero’s Blog

Applying permutation in constant space (and linear time)

I stumbled upon a mildly interesting problem yesterday: Given an Array a and a permutation p, apply the permutation (in place) to the Array, using only O(1) extra space. So, if b is the array after the algorithm, we want that a[i] == b[p[i]].

Naively, we would solve our problem by doing something like this (I'm using go here):

func Naive(vals, perm []int) {
    n := len(vals)
    res := make([]int, n)
    for i := range vals {
        res[perm[i]] = vals[i]
    }
    copy(vals, res)
}

This solves the problem in O(n) time, but it uses of course O(n) extra space for the result array. Note also, that it does not really work in place, we have to copy the result back.

The simplest iteration of this, would be to simply use a sorting-algorithm of our choice, but use as a sorting key not the value of the elements, but the position of the corresponding field in the permutation array:

import "sort"

type PermSorter struct {
    vals []int
    perm []int
}

func (p PermSorter) Len() int {
    return len(p.vals)
}

func (p PermSorter) Less(i, j int) bool {
    return p.perm[i] < p.perm[j]
}

func (p PermSorter) Swap(i, j int) {
    p.vals[i], p.vals[j] = p.vals[j], p.vals[i]
    p.perm[i], p.perm[j] = p.perm[j], p.perm[i]
}

func Sort(vals, perm []int) {
    sort.Sort(PermSorter{vals, perm})
}

This appears a promising idea at first, but as it turns out, this doesn't really use constant space after all (at least not generally). The go sort package uses introsort internally, which is a combination of quick- and heapsort, the latter being chosen if the recursion-depth of quicksort exceeds a limit in O(log(n)). Thus it uses actually O(log(n)) auxiliary space. Also, the running time of sorting is O(n log(n)) and while time complexity wasn't part of the initially posed problem, it would actually nice to have linear running time, if possible.

Note also another point: The above implementation sorts perm, thus destroying the permutation array. Also not part of the original problem, this might pose problems if we want to apply the same permutation to multiple arrays. We can rectify that in this case by doing the following:

type NDPermSorter struct {
    vals []int
    perm []int
}

func (p NDPermSorter) Len() int {
    return len(p.vals)
}

func (p NDPermSorter) Less(i, j int) bool {
    return p.perm[p.vals[i]] < p.perm[p.vals[j]]
}

func (p NDPermSorter) Swap(i, j int) {
    p.vals[i], p.vals[j] = p.vals[j], p.vals[i]
}

func NDSort(vals, perm []int) {
    sort.Sort(NDPermSorter{vals, perm})
}

But note, that this only works, because we want to sort an array of consecutive integers. In general, we don't want to do that. And I am unaware of a solution that doesn't have this problem (though I also didn't think about it a lot).

The solution of solving this problem in linear time lies in a simple observation: If we start at any index and iteratively jump to the target index of the current one, we will trace out a cycle. If any index is not in the cycle, it will create another cycle and both cycles will be disjoint. For example the permutation

i    0  1  2  3  4  5  6  7  8  9  10 11 12 13 14 15 16 17 18 19
p[i] 2  13 1  5  3  15 14 12 8  10 4  19 16 11 9  7  18 6  17 0

will create the following set of cycles:

So the idea is to resolve every cycle separately, by iterating over the indices and moving every element to the place it belongs:

func Cycles(vals, perm []int) {
    for i := 0; i < len(vals); i++ {
        v, j := vals[i], perm[i]
        for j != i {
            vals[j], v = v, vals[j]
            perm[j], j = j, perm[j]
        }
        vals[i], perm[i] = v, i
    }
}

This obviously only needs O(1) space. The secret, why it also only uses O(n) time lies in the fact, that the inner loop will not be entered for elements, that are already at the correct position. Thus this is (from a complexity standpoint at least) the optimal solution to the problem, as it is impossible to use less than linear time for applying a permutation.

There is still one small problem with this solution: It also sorts the permutation array. We need this, to know when a position is already occupied by it's final element. In our algorithm this is represented by the fact, that the permutation is equal to it's index at that point. But really, it would be nice if we could mark the index without losing the order of the permutation. But that is not hard either - because every index is non-negative, we can simply negate every index we are done with. This will make a negative index out of it and we can check for that if we encounter it later and skip it in this case. After we are done, we only need to take care to flip everything back and all should be fine:

func NDCycles(vals, perm []int) {
    for i := 0; i < len(vals); i++ {
        if perm[i] < 0 {
            // already correct - unmark and go on
            // (note that ^a is the bitwise negation
            perm[i] = ^perm[i]
            continue
        }

        v, j := vals[i], perm[i]
        for j != i {
            vals[j], v = v, vals[j]
            // When we find this element in the future, we must not swap it any
            // further, so we mark it here
            perm[j], j = ^perm[j], perm[j]
        }
        vals[i] = v
    }
}

Here we only mark the elements we will again encounter in the future. The current index will always be unmarked, once we are done with the outer loop.

I am aware, that this is technically cheating; This solution relies on the fact, that the upper-most bit of the permutation elements won't ever be set. Thus, we actually do have O(n) auxiliary space (as in n bit), because these bits are not necessary for the algorithm. However, since it is pretty unlikely, that we will find an architecture where this is not possible (and go guarantees us that it actually is, because len(vals) is always signed, so we cant have arrays that are big enough for the msb being set anyway), I think I am okay with it ;)

I ran sum Benchmarks on this an these are the figures I came up with:

n 10 100 1000 10000
Naive 332 ns 883 ns 15046 ns 81800 ns
NDCycle 130 ns 1019 ns 17978 ns 242121 ns
NDSort 1499 ns 27187 ns 473078 ns 4659433 ns

I did not measure space-use. The time of NDCycle for 10000 elements seems suspicious - while it is not surprising, that in general it takes more time than the naive approach, due to it's complexity, this jump is unexpected. Maybe if I have the time I will investigate this and also measure memory use. In the meantime, I uploaded all the code used here, so you can try it out yourself. You can run it with go run perm.go and run the benchmarks with go test -bench Benchmark.*.

12. August 2014 11:10:21

11. August 2014

Moredreads Blog

Werbungskritik

Normalerweise kommentiere ich wenig im Netz, da die Diskussionskultur durch Trolle und anderes ziemlich schlecht ist. Und so sicher bin ich mir nicht mit dem was ich denke, dass ich das anderen aufbürden wollte. :p

Aber trotzdem habe ich mal zu “Das Experiment”, einem Gastbeitrag auf zu der Stimmungsmanipulationsstudie von Facebook, einen Kommentar (sogar mit Klarnamen oO) verfasst, und damit meine lange Abstinenz von öffentlichen Diskussionen abseits Twitter etc. gebrochen. Vielleicht liest’s ja jemand und findet es nachdenkenswert. :p

Um diesen Beitrag ein wenig mit Inhalt zu füllen, vielleicht noch eine Ergänzung.

Ich finde es sehr bemerkenswert wie normal Werbung wahrgenommen wird, als sei es ein Gott-gegebenes Recht von Unternehmen zu “Käufern” degradierte Bürger zum Kauf ihrer Waren zu bewegen, wobei nur in Grenzfällen gesetzliche Hürden gesetzt werden (i. B. bei an Kinder gerichtete Werbung und ironischer Weise, wenn Sie andere Unternehmen zu stark betrifft (Unlauterer Wettbewerb)).

Im Gegensatz zu Kindern, von denen angenommen wird, dass sie nicht fähig sind, den “Wahrheitsgehalt” von Werbung zu durchschauen, wird dem/der Erwachsenen eine “Mündigkeit” als Verbraucher zugeschrieben, bewusst und mit Bedacht das Warenangebot zu betrachten, bevor er/sie eine Entscheidung trifft. Von einem logischen und rationalem Marktteilnehmer/-Innen gehen auch “der” Neoliberalismus aus, dessen Ideologie seit Jahrzehnten immer mehr die Wirtschaftspolitik geprägt hat.

Alleine die Existenz von Werbung müsste dieses Weltbild ins Wanken bringen, denn es ist vielerlei in Studien (sorry, müsst selber suchen :p) gezeigt worden, dass auch darauf achtende Menschen leicht von ihr beeinflussbar sind, und insbesondere in der Überflutung von verschiedensten Informationen nicht alle in den Entscheidungsprozess einfließen lassen können. Aber Marketing, public relations, Lobbying etc. wird auch gerade von Anhängern des Marktliberalismus betrieben, wobei die “Initiative Neue Soziale Marktwirtschaft” und die Bertelsmann-Stiftung nur zwei prominente Akteure sind. Kritik an nicht rein informatinsbasierter “Öffentlichkeitsarbeit”, sei es durch Unternehmen, Lobbyisten, oder anderen “Marktteilnehmern”, kenne ich von Markt-liberalen nicht. Das das keine signifikante kognitive Dissonanz auslöst…

Auch ist Werbung in meinen Augen ein signifikantes Volkswirtschaftliches Problem. Alleine die direkte Investition in Werbung beträgt ca. 1% des Bruttoinlandprodukts in Deutschland, oder ca. 30 Milliarden Euro; vermutlich sind darin indirekte Investitionen, wie interne Produktoptimierungen nicht enthalten. Dies ist Produktivkraft, die verschwendet wird, um alleine das Konsumverhalten zu lenken.

Gerade sozial wichtige Bereiche sind hiervon Betroffen, z.B. fließen bei Pharma-Konzernen große Mengen in Werbung, teils mehr als in Forschung und Entwicklung (siehe z.B. diesen Spiegel Artikel). Dabei ist noch nicht berücksichtigt, dass die Grundlagenforschung die Innovationen erst ermöglicht, größtenteils aus öffentlichen Mitteln finanziert wird. Des weiteren wird durch Lobbying versucht besonderer Schutz fuer Pharma-Unternehmen durch internationale Abkommen zu erreichen, um z.B. den großen Generika Markt in Indien, von dem u. A. Staaten in Afrika stark profitieren, zu sabotieren. (Eine humorige Entgleisung eines Pharmavertreters ist hier ein Interview geführt von Martin Sonneborn)

Dies nur so als ein Paar Beispiele warum ich es, wie Eingangs erwähnt, bemerkenswert finde, dass Werbung eine so prominente Stellung in unserer Gesellschaft einnehmen kann. Andere kritische Beispiele finden sich z. B. in den Werken von Noam Chomsky, aber auch Systemtragende sollten Grund haben da sich zu wundern.

11. August 2014 19:15:00

sECuREs Webseite

Configuring a Ubiquiti EdgeRouter Lite (Erlite-3) for Fiber7

I immediately ordered a fiber7 internet connection once it became available, and I’ve been connected since a few weeks. They offer a 1 Gbps symmetrical fiber connection, with native (static) IPv6 and no traffic limit — for 65 CHF per month (about 54 €).

In the order form, they let you choose whether you want to order a pre-configured MikroTik RB2011UiAS-2HnD including fiber optic and fiber patch cable. I assumed this would be an excellent choice, so I ordered it.

I really like the MikroTik device. Its CLI and web interface are well thought-out and easy to use once you understand their way of thinking. It’s small, absolutely silent and just works. However, there’s one shortcoming: it doesn’t do IPv4 hardware acceleration (they call it “fast path”) when you enable NAT, which you need for a fiber7 connection. Thus, the top bandwidth maxes out at 500 to 600 Mbps, so effectively you only use half of your available bandwidth.

Therefore, I looked around for other routers which can do a full Gigabit WAN-to-LAN, i.e. with IPv4-NAT enabled. The selection of routers that can do that is very small, see for example the smallnetbuilder WAN-to-LAN router charts.

In my first try, I went with the Netgear R7000 (“Nighthawk”) which is the highest-performing router with regards to WAN-to-LAN bandwidth on smallnetbuilder. It indeed does hardware acceleration for IPv4-NAT, so you can reach the full 118 MB/s TCP bandwidth that a Gigabit line offers. However, the firmware does not do DHCPv6-PD (Prefix Delegation), even though it’s certified as IPv6-ready. There are alternative firmwares, e.g. Tomato and DD-WRT. Tomato (v121 as of writing) comes with the kernel module that enables IPv4-NAT hardware acceleration, but has a nasty bug: the latency jumps up to 500ms for most of your packets, which is clearly not acceptable. DD-WRT does not come with such a kernel module because they use a newer kernel, so the speed maxes out at 400 Mbps (that’s what they claim, I didn’t even bother testing it).

Ubiquiti EdgeRouter Lite (Erlite-3)

So, as a second try, I ordered what everyone recommended me in the first place: the Ubiquiti EdgeRouter Lite (Erlite-3).

The EdgeRouter Lite (with firmware v1.5.0) offers IPv4 and IPv6 offloading, and in fact reaches Gigabit line rate (118 MB/s measured TCP performance). An unwelcome surprise is that hardware acceleration only works when not using bridging at all, so if you want to connect two devices to your router in the same subnet, like a computer and a switch, you cannot do that. Effectively, the EdgeRouter needs to sit between the internet connection and a switch.

With regards to the web interface of EdgeOS: the web interface feels very polished and modern, but it seems to lack a number of features that are only accessible in the CLI interface. The MikroTik web interface had a much higher coverage of features. In general, I like how Ubiquiti does many things right, though: firmware updates are quick and painless, the model selection and download on their website is very simple to find and use, and you even get a link to the relevant GPL tarball without asking :).

Configuring the EdgeRouter Lite for fiber7

First of all, you should disconnect the MikroTik (or your current router) from the network. I recommend doing that by explicitly disabling both DHCP clients, so that the fiber7 router knows you are not using the old device any more. This is important because fiber7 uses a Cisco feature called “IP source guard”, which will disable any MAC address on your port that does not have a DHCP lease. Therefore, if you just switch routers, you need to wait for the old lease to expire before you get a new lease. In my first tests, this worked relatively well, but then a lease got stuck for some reason and I had to escalate the problem to their NOC. So, better disable the DHCP:

/ip dhcp-client set disabled=yes numbers=0
/ipv6 dhcp-client set disabled=yes numbers=0

In my configuration, I connect a switch to eth0 and a media converter (the TP-LINK MC220L) to eth1. As a general tip: if you mess up your configuration, you can always use the link-local address of the EdgeRouter and SSH into that. Find the link-local address using ping6 ff02::1%eth0.

After logging into the web interface, set the eth1 address to DHCP and it should get a public IPv4 address from fiber7. Afterwards, enable NAT by clicking on NAT → Add Source NAT Rule. Set the outbound interface to eth1 and select the “masquerade” radio button. You’ll also need to switch to the “services” tab and enable a DHCP and DNS server. This should give you IPv4 connectivity to the internet.

IPv6 is a bit harder, since EdgeOS in its current version (1.5.0) does not support DHCPv6-PD via its Web or CLI interface. The necessary software (wide-dhcpv6) is included, though, so we can configure it manually.

Use ssh ubnt@192.168.1.1 to log into the CLI. In order to set the proper IPv6 address on the transfer network, run ip -6 address show dev eth1 and look for a line that says inet6 fe80::de9f:dbff:fe81:a906/64 scope link. Copy everything after the :: and prefix it with 2a02:168:2000:5: (the fiber7 transfer network range), then configure that as static IPv6 address on eth1 and set the default route (and enable IPv6 offloading):

configure
set system offload ipv6 forwarding enable
set interfaces ethernet eth1 address 2a02:168:2000:5:de9f:dbff:fe81:a906/64
set protocols static route6 ::/0 next-hop 2a02:168:2000:5::1 interface eth1
commit
save
exit

Now you should be able to run ping6 google.ch and get a reply. We still need to enable DHCPv6 though so that the router gets a prefix and hands that out to its clients. Run sudo -s to get a root shell and configure DHCPv6:

cat >/etc/wide-dhcpv6/dhcp6c-script-zkj <<'EOT'
#!/bin/sh
# wide-dhcpv6-client 20080615-12 does not properly close
# file descriptors when starting the script.
# https://bugs.debian.org/757848
exec 4>&- 5>&- 6>&- 7>&-
# To prevent radvd from sending the final router advertisment
# that unconfigures the prefixes.
killall -KILL radvd
/etc/init.d/radvd restart
exit 0
EOT
chmod +x /etc/wide-dhcpv6/dhcp6c-script-zkj

cat >/etc/wide-dhcpv6/dhcp6c.conf <<'EOT'
interface eth1 {
        send ia-pd 0;
        request domain-name-servers;
        script "/etc/wide-dhcpv6/dhcp6c-script-zkj";
};

id-assoc pd 0 {
        prefix-interface eth0 {
                sla-id 1;
                sla-len 0;
        };
};
EOT

sed -i 's/eth0/eth1/g' /etc/default/wide-dhcpv6-client

cat >/config/scripts/post-config.d/dhcpv6.sh <<'EOT'
#!/bin/sh
/etc/init.d/wide-dhcpv6-client start
EOT
chmod +x /config/scripts/post-config.d/dhcpv6.sh

/config/scripts/post-config.d/dhcpv6.sh

Now, when running ip -6 address show dev eth0 you should see that the router added an IPv6 address like 2a02:168:4a09:0:de9f:dbff:fe81:a905/48 to eth0. Let’s enable router advertisments so that clients get an IPv6 address, route and DNS server:

configure
set interfaces ethernet eth0 ipv6 router-advert prefix ::/64
set interfaces ethernet eth0 ipv6 router-advert radvd-options
  "RDNSS 2001:4860:4860::8888 {};"
commit
save
exit

That’s it! On clients you should be able to ping6 google.ch now and get replies.

Bonus: Configuring a DHCPv6-DUID

fiber7 wants to hand out static IPv6 prefixes based on the DHCPv6 option 37, but that’s not ready yet. Until then, they offer you to set a static prefix based on your DUID (a device identifier based on the MAC address of your router). Since I switched from the MikroTik, I needed to port its DUID to the EdgeRouter to keep my static prefix.

Luckily, wide-dhcpv6 reads a file called dhcp6c_duid that you can create with the proper DUID. The file starts with a 16-bit integer containing the length of the DUID, followed by the raw DUID:

echo -en '\x00\x0a\x00\x03\x00\x01\x4c\x5e\x0c\x43\xbf\x39' > /var/lib/dhcpv6/dhcp6c_duid

Conclusion

I can see why fiber7 went with the MikroTik as their offer for customers: it combines a media converter (for fiber to ethernet), a router, a switch and a wifi router. In my configuration, those are now all separate devices: the TP-LINK MC220L (27 CHF), the Ubiquiti EdgeRouter Lite Erlite-3 (170 CHF) and the TP-LINK WDR4300 (57 CHF). The ping latency to google.ch has gone up from 0.6ms to 0.7ms due to the additional device, but the download rate is about twice as high, so I think this is the setup that I’ll keep for a while — until somebody comes up with an all-in-one device that provides the same features and achieves the same rates :-).

by Michael Stapelberg at 11. August 2014 09:55:00

04. August 2014

Raphael Michel

Monitoring email

I most recently moved my mails from the great guys at Uberspace to an own server, because I like owning things and running them by myself. This is no problem at all with hosting my own sync server for contacts, calendar and photos, but it is a problem with mail, because mail is critical infrastructure to me. So, although I like the thought of having my mail on a machine I own, I do not like the thought of my mailserver going broken without me noticing it.

As I run a bunch of online services for myself and for others, I already had an Icinga instance running to notify me if one of my services happens to go down. This Icinga instance already was configured to check whether my mailserver correctly responds on the SMTP and IMAP ports but this is not enough. This does not fire alarm, if there is a configuration mistake anywhere in the local mail transport (postfix), the spam filter (SpamAssassin), other filters (sieve), or the authentication system.

The only solution to this problem is to set up End to End monitoring for email. Most surprisingly, I failed to find a ready-to-use Icinga plugin for doing this. So this is what I came up with by myself.

For testing all of the steps in the flow of an email, you need a second mailserver you can regularly exchange mails with. You can use any mail account at one of the big mail hosters, but there is a chance they'll consider your testing mails as spam, so I just paired up with Ben, a friend of mine, to monitor each other's mailserver.

The basic idea is very simple: You have your mailserver (A) and your friend's mailserver (B) and on every run, the Icinga check script connects of the SMTP servers and sends one email each from A to B and from B to A, containing a timestamp. On the next run, it connects to both servers via IMAP, checks whether both mails arrived, deletes them and sends out new ones once again.

The check script is written in Python and requires at least Python 3.2 and the Python nagiosplugin. On Debian stable, just do a quick

apt-get install python3 python3-pip
pip-3.2 install nagiosplugin

Then, grab my script from Github:

wget https://raw.githubusercontent.com/raphaelm/monitoring/master/mail/check_mail_twoway
chmod +x check_mail_twoway
mv check_mail_twoway /usr/lib/nagios/plugins/

You can execute ./check_mail_twoway -h to get an overview of the command-line options and play around with them.

My Icinga configuration looks as follows:

define command {
    command_name    check_mail_twoway
    command_line    /usr/lib/nagios/plugins/check_mail_twoway $ARG1$ 
}
define service {
    use                 generic-service
    host_name           dannyl
    service_description Mail E2E
    check_command       check_mail_twoway!-s1 serverA \
        -a1 userA@serverA -i1 serverA -u1 userA \
        -p1 passwordA -s2 serverB -i2 serverB \
        -a2 userB@serverB -u2 userB -p2 passwordB
}

On the first run, the script will always report a failure, as it can not find any emails (as none have been sent out yet). I do not think there is a simple and perfect way to prevent this, please correct me if I'm wrong.

I'd love to hear if this works for you or to see any pull requests!

(And of course, my Icinga does not try to notify me via email if my email fails, but via SMS.)

04. August 2014 22:35:00

31. July 2014

RaumZeitLabor

Mario Kart Turnier #7

Ladys, Gentlemen, Koopa, Pilzköpfe und sonstige Entitäten!
Es ist wieder soweit, das RaumZeitLabor lädt ein zum 7. Mario Kart Turnier! Egal ob ihr alle Strecken im Schlaf beherrscht oder noch nie mit einem Kart eure Runden gedreht habt: Fahrerinnen und Fahrer aller Erfahrungsstufen sind willkommen!

Wir werden das Turnier in den Disziplinen Super Mario Kart (Super Nintendo) und Mario Kart 64 (Nintendo 64) im Doppel-K.O.-System austragen. Je nach Anzahl der Teilnehmer wird es auch eine Gruppenphase geben. Das Turnier findet am Samstag, den 23. August 2014 statt. Die Teilnahme ist kostenfrei, ihr müsst euch aber bis spätestens 19h im RaumZeitLabor eingefunden haben. Zur weiteren Planung wäre es toll, wenn ihr euch mit einem Kommentar unter diesem Blogpost oder auf der Facebookseite anmeldet.

Als kleines Warm-Up hier das Finale des letzten Turniers zwischen Jannik (Bowser, oben) und Cheatha (Wario, unten):

Update: Wenn die Technik mitspielt, dann werden wir sowohl eine Aufzeichnung der Rennen haben als auch einen Livestream. So könnt ihr später auf Youtube nochmal das Turnier Revue passieren lassen oder sogar am Livestream mitfiebern. Für weitere Updates folgt am besten @RZL_TeamRetro auf Twitter!

by Cheatha at 31. July 2014 19:33:50

29. July 2014

RaumZeitLabor

Ludum Dare 30 im RaumZeitLabor

Am Wochenende vom 22-25 August 2014 findet zum 30. Mal der internationale Gamejam “ludum dare” statt. Um 3 Uhr in der Nacht von Freitag auf Samstag fällt der Startschuss, alleine in 48 oder als Team in 72 Stunden ein Spiel zu einem vorgegebenen Thema zu programmieren.

Wer keine Lust hat, in Einsamkeit zu programmieren oder schon immer mal jemandem beim Spiele programmieren zuschauen wollte, ist eingeladen, im RaumZeitLabor vorbeizukommen.

Bei gutem Wetter wird auch gegrillt!

Anfahrt

by xAndy at 29. July 2014 17:32:58

28. July 2014

sECuREs Webseite

Don’t type WiFi passwords again: use a yubikey

In my flat, I have a printed QR code which contains the WiFi credentials. You can scan it with your Smartphone (provided you have a barcode scanner installed) and then connect to the WiFi network.

For notebook computers, this doesn’t work so well. Sometimes they don’t have a usable camera, and even if they have one, people don’t typically have a barcode scanner installed.

Therefore, you typically end up typing in the (long) password, which is annoying. I thought there should be a better way, perhaps by faking a USB Human Interface Device (like a keyboard). Then I saw a YubiKey lying around and a couple minutes later I had my solution working :-).

YubiKey

YubiKey

The YubiKey is a small USB device (about as big as a USB memory stick) that is typically used for two-factor authentication. You may be familiar with this concept from Gmail, where you log in with a password and then provide a number generated by the Google Authenticator on your phone as a second factor. The YubiKey does this by simulating you typing in the number after you touch it.

Luckily, the device’s firmware also supports a mode where it essentially just types a static password when being touched. Unfortunately, this feature is a bit hard to use, so I wrote pw-to-yubi, a little script which translates a (WiFi) password into a command line to program your YubiKey.

$ sudo apt-get install yubikey-personalization
$ git clone https://github.com/stapelberg/pw-to-yubi.git
$ cd pw-to-yubi
$ ./pw-to-yubi.pl mywifipassword
ykpersonalize -1 -o-append-cr -o-static-ticket -oshort-ticket \
  -o-strong-pw1 -o-strong-pw2 -oman-update \
  -ofixed=h:101c1a0c090c130416161a1215070000 -ouid=000000000000 \
  -a00000000000000000000000000000000

Run the command and any further touches of the YubiKey that is currently plugged into your computer will result in the key typing your WiFi password.

pw-to-yubi in a terminal emulator

by Michael Stapelberg at 28. July 2014 19:15:00

26. July 2014

Moredreads Blog

19. July 2014

Moredreads Blog

14. July 2014

RaumZeitLabor

Zweiter Agenda Diplom 2014 Workshop

Nach dem erfolgreichen Start im Mai fand am vergangenen Wochenende der zweite Termin des „Agenda Diplom für Kinder“ der Stadt Mannheim im RaumZeitLabor statt. Unter dem Motto „Wir erwecken Roboter zum Leben“ haben wir erneut mit drei Gruppen von acht- bis zwölfjähriger Kinder Lego-Mindstorms-Roboter programmier, mit Aufgaben steigenden Schwierigkeitsgrades.

IMG_6393b

IMG_6402b

IMG_6408b

IMG_6417b

IMG_6426b

IMG_6456b

IMG_6464b

IMG_6473b

IMG_6450b

IMG_6465b

IMG_6483b

IMG_6489b

by rami at 14. July 2014 18:13:12

04. July 2014

Moredreads Blog

22. June 2014

Raphael Michel

GPN14 CTF Writeup

Auf der GPN14 haben die squareroots ein Capture the Flag veranstaltet. In diesem Artikel schildere ich die Lösungen zu den Challenges, die ich gelöst habe und zu denen ich im Nachhinein noch genug Spuren auf meinem Rechner habe, um etwas über sie zu erzählen. Nachts und schnell gelöst, handelt es sich hier natürlich nicht um vorbildlichen Code…

Time is up

In einer der Challenges unter dem Namen „Time is up“ war die Datei mit dem Flag über knapp 800 .rar-Teildateien verteilt, die man alle einzeln über ein Downloadportal herunterladen musste. In diesem Downloadportal musste man vor dem Download jeder einzelnen Datei ein CAPTCHA eingeben, das oft, aber nicht immer, sehr schwer war und in unter 5 Sekunden gelöst werden musste.

Die Lösung hierzu stammt nicht nur von mir, sondern zum größten Teil auch von NoMoKeTo.

Ausgangspunkt der Challenge war eine Textdatei mit den Links zu den einzelnen Dateien in willkürlicher Reihenfolge, die man ebenfalls über besagtes Download-Portal herunterladen musste. Dies haben wir von Hand erledigt und in ein Python-Skript gefüttert, das ich im folgenden beschreibe.

Das erste Python-Skript, download.py, lädt in einer Schleife für jede Datei die Downloadseite und ein CAPTCHA. Die Downloadseite wird auf den Dateinamen der herunterzuladenden Datei durchsucht, damit wir keine Mühe auf .rar-Teildateien verschwenden, die wir bereits bei einem vorherigen Ausführen heruntergeladen haben.

Anschließend wird das CAPTCHA versucht zu lösen (siehe unten) und zurück an den Server gesendet. War das CAPTCHA falsch, laden wir uns einfach ein neues und versuchen es erneut, bis es klappt. War das CAPTCHA richtig, wird die betreffende Datei gespeichert.

Nun zum interessanten Teil, unserem CAPTCHA-OCR. Die CAPTCHAs bestehen immer aus vier bunten und gedrehten Großbuchstaben oder Zahlen verschiedener Schriftarten auf insgesamt zehn verschiedenen Hintergründen.

CAPTCHA CAPTCHA CAPTCHA CAPTCHA

In einem ersten Trainingsschritt (collectbgs.py) haben wir so lange CAPTCHAs heruntergeladen, bis wir jeden Hintergrund mindestens 50 mal gesehen haben, wobei wir zwei CAPTCHAs als mit dem gleichen Hintergrund erkennen, wenn mindestens die Hälfte der Pixel farblich übereinstimmt. Anhand dieser Bilder extrahieren wir nun den puren Hintergrund, indem wir für jeden Pixel die Farbe setzen, die in den mindestens 50 Bilder an dieser Stelle am häufigsten vorkommt.

CAPTCHA CAPTCHA CAPTCHA

Um nun ein einzelnes CAPTCHA zu lösen (captchasolver.py), finden wir zuerst mit der bereits geschilderten Methode heraus, welchen Hintergrund das CAPTCHA verwendet und subtrahieren diesen (bzw. färben alle Pixel schwarz, die sich vom Hintergrund unterscheiden), womit wir den reinen Text erhalten.

Wir suchen nun nach leeren Spalten in unserem so behandelten CAPTCHA um die Zwischenräume zwischen den vier Buchstaben zu finden und so die einzelnen Buchstaben zu isolieren. Nun müssen wir noch die Buchstaben erkennen.

Wir zählen dazu drei Werte des Buchstabens: Die Anzahl Pixel, die der Buchstabe ausfüllt, die Anzahl der schwarzen Pixel, die am Rand des Buchstabens liegen und die Anzahl der weißen Pixel, die komplett vom Buchstaben eingeschlossen sind, also z.B. in den Bäuchen eines B oder in einem O.

Nach einer halbstündigen manuellen Trainingsrunde hat sich dieses Zahlentripel als charakteristisch genug herausgestellt, um zumindest bei einigen der in den CAPTCHAs verwendeten Schriftarten eine hohe Trefferquote zu erzielen. Dadurch, dass vier Buchstaben korrekt erkannt werden mussten und das Skript bei manchen Schriftarten keine Chance hatte, war die Trefferquote insgesamt eher gering, aber ausreichend um in einer halben Stunde alle 796 Dateien herunterzuladen und das Flag zu finden.

Auszug aus einer Trainigsdatei für einen der Hintergründe:

{
    "186-145-0": "Y",
    "198-164-13": "N",
    "141-110-62": "O",
    "110-88-1": "1"
}

Was hätten die CAPTCHAs tun müssen, um unserem Skript zu entgehen? Nun, Linien quer durch die Buchstaben hätten die Trennung der Buchstaben erschwert, exotische Schriftarten hätten die Erkennung über das Zahlentripel nicht erlaubt und bei Einsatz von Fail-to-ban wären wir verloren gewesen. Andererseits: Manche der CAPTCHAs waren so ekelhaft, dass ich sie auch als Mensch nicht geschafft habe zu lösen und so betrachtet war unser Skript wohl ähnlich gut wie ein Mensch, was mal wieder zeigt: CAPTCHAs sind keine effektive Methode, um Menschen von Skripten zu unterscheiden und stattdessen verursachen Sie Frustration beim User. Do not use them. Danke.

Der Source Code unserer Skripte zum nachlesen findet sich in diesem GitHub Gist, aber ist wahrscheinlich alles andere als portabel und auf andere Dinge übertragbar (was auch ein wenig so gedacht ist).

Colorful life

In einer „Programming“-Challenge gab es eine Website, auf der man ein Bild präsentiert bekommen hat, das mit einer zufälligen Farbe ausgefüllt war, und musste in extrem kurzer Zeit in ein Formular die Rot-, Gelb- und Grünwerte dieser Farbe eingeben. Folgendes JavaScript, auf der Entwicklerkonsole ausgeführt, löst das Problem.

var img = document.getElementsByTagName('img')[0]; 
var canvas = document.createElement('canvas');
canvas.width = img.width;
canvas.height = img.height;
canvas.getContext('2d').drawImage(img, 0, 0, img.width, img.height);
var pixelData = canvas.getContext('2d').getImageData(1, 1, 1, 1).data;
document.getElementsByName('r')[0].value = pixelData[0];
document.getElementsByName('g')[0].value = pixelData[1];
document.getElementsByName('b')[0].value = pixelData[2];
document.forms[0].submit()

Weird Garbage

Bei der Challenge „Weird Garbage“ war ein Server gegeben, der -- über Telnet angesprochen -- jeweils eine Zeile Brainfuck zurückgegeben hat.

Ausgeführt, gibt dieses Brainfuck jeweils drei Zeichen aus. Der Trick war, sich alle diese Zeichentripel zu besorgen und dann in die richtige Reihenfolge zu bringen, wobei man eigentlich nur den ersten Teil („sqr“) identifizieren musste, um dann den nächsten Teil zu finden, der mit den letzten beiden Zeichen begann („qrt“).

Where is Kurt Krabbel?

Eine „Misc“-Challenge bestand in einem Screenshot des Foursquare-Profils von „Kurt Krabbel“, der verschiedene Orte in Mannheim aufgesucht hat. Das besondere an Mannheim ist, dass die Innenstadt nicht nur in Quadraten aufgebaut, sondern sogar benannt ist. Die Adressen, die Kurt aufgesucht hat, sind in chronologischer Reihenfolge in den Quadraten H1, A1, C5, K1, L14, A5, B3 zu finden und das Flag setzte sich aus den Buchstaben davon zusammen: MD5(hacklab).

22. June 2014 22:00:00

Mero’s Blog

GPN14 GameJam - Crazy cat lady

tl:dr: We made a gamejam-game

At the GPN14 we (meaning me and Lea, with a bit of help by sECuRE) participated in the gamejam. It was the first time for us both, I did all the coding and Lea provided awesome graphics.

The result is a small minigame “crazy cat lady”, where you throw cats at peoples faces and - if you hit - scratch the shit out of them (by hammering the spacebar). The game mechanics are kind of limited, but the graphics are just epic, in my opinion:

Screenshot

Because sounds make every game 342% more awesome, we added a creative commons licensed background-music. We also wanted some cat-meowing and very angry pissed of hissing, which was more of a problem to come by. Our solution was to just wander about the GPN and asking random nerds to make cat sounds and recording them. That gave a pretty cool result, if you ask me.

On the technical side we used LÖVE, an open 2D game engine for lua, widely used in gamejams. I am pretty happy with this engine, it took about 3 hours to get most of the game-mechanics going, the rest was pretty much doing the detailed animations. It is definitely not the nicest or most efficient code, but for a gamejam it is a well suited language and engine.

If you want to try it (I don't think it is interesting for more than a few minutes, but definitely worth checking out), you should install LÖVE (most linux-distributions should have a package for that) and just download it, or check out the sourcecode.

We did not make first place, but that is okay, the game that won is a nice game and a deserved winner. We had a lot of fun and we are all pretty happy with the result as first-timers.

22. June 2014 13:45:28

13. June 2014

Moredreads Blog

06. June 2014

Moredreads Blog

15. May 2014

sECuREs Webseite

Horrible WiFi in hotels

I’ve been travelling a bit to foreign countries lately and noticed that all of the places I’ve stayed at (low to medium price range hotels) have one thing in common: their WiFi is absolutely horrible.

Even worse, it seems like the more money you pay, the more horrible the WiFi gets. As an example, I’ve stayed at the NH city centre in Amsterdam recently. Curiously, swisscom runs the WiFi there — I didn’t even know they do more than access and mobile network in Switzerland. We booked a double room (so the hotel knew there are two people staying in the room), but swisscom only allows one login. I’ll repeat: one (1) login. In 2014. Where people travel with smartphones, tablets and notebooks. Per person. Even better: every time you lose connection, you need to re-acknowledge the terms of service. Of course this breaks automated updates of Android smartphones because they try to save battery and connect to the WiFi only sporadically.

Not only with swisscom it frequently happens that WiFi providers offer you laughably low volume limits like 100 MiB, which is less than what I have with my tiny international roaming package. Or 256 kbit/s (!) of bandwidth, which of course is no comparison to mobile data connections these days. Also, I’ve seen latencies as high as half a minute, effectively preventing any internet usage from working.

The horrible WiFi PDF

In order to not suffer in silence any more, but quickly express my frustration, I’ve created a form that I’ll carry with me when visiting a hotel. In case (hah!) their WiFi sucks, I can just leave them that note and hope they’ll eventually take action.

Feel free to do the same: store horrible-wifi.pdf (19 KiB) (Public Domain, please share) and leave it at every hotel with inacceptable WiFi.

by Michael Stapelberg at 15. May 2014 23:30:00

13. May 2014

RaumZeitLabor

Roboter erfolgreich zum Leben erweckt – Agenda Diplom 2014 im RaumZeitLabor startet mit großem Interesse

Nach dem großen Zuspruch beim Agenda Diplom für Kinder der Stadt Mannheim im letzten Jahr nimmt das RaumZeitLabor auch in diesem Jahr wieder mit vier Veranstaltungen unter dem Motto “Wir erwecken Lego-Roboter zum Leben – Deine ersten Schritte mit Lego Mindstorms EV3 Robotern” teil.

agenda19 agenda18
agenda20

Der erste Termin am 10. Mai war im Nu ausgebucht, bei den restlichen Terminen sind augenblicklich nur noch wenige Plätze frei.

Pünktlich um 15 Uhr waren sowohl 12 Mädchen und Jungs, als auch eine stattliche Anzahl interessierter Eltern im RaumZeitLabor, um zu erfahren wie man Lego Roboter mit dem System Mindstorms EV3 zum Leben erwecken kann. Dabei ging es primär darum den programmierbaren Elementen beizubringen Abläufe in der richtigen Reihenfolge zu vollführen, damit eine selbst gestellte Aufgabe bewältigt werden konnte.

agenda17

Es brauchte nur einen ganz kurze Einführung und schon sah man die Kinder, die sich in drei Teams zusammengefunden hatten, zusammen erste Schritte zur Aufgabenstellung zu diskutieren und diese dann auch sofort im Programmiersystem von Lego Mindstorms EV3 umzusetzen.

agenda16 agenda15 agenda14 agenda07 agenda06 agenda05

Aber nicht nur die Kids waren fasziniert von der Materie, nein, auch die mitgekommennen Eltern erhielten ihrerseits Einblicke in das was ihre Kinder, draußen, in einem anderen Raum, gerade machten und waren begeistert. Für Sie gab es nämlich einen kleinen Extra Workshop im angrenzenden Workshopraum des RaumZeitLabors. Sie wurden sowohl über die Möglichkeiten des Lego Systems informiert, als auch über das was man sonst noch so alles im RaumZeitLabor machen kann und der eine oder die andere fand sogar Angebote für sich selbst so ansprechend. So wird es wohl auch mit einigen der Eltern zu einem Wiedersehen kommen.

agenda04 agenda03

Nach etwa der Hälfte der Zeit konnten die Robotik-Neulinge schon erste fast komplette Bewegungsabläufe präsentieren. Weil das ganze Programmieren, Diskutieren und Ausprobieren anstrengend ist, hatten sich die zwölf  und natürlich auch die Eltern eine gemeinsame Pause verdient. Dazu gab es ein kleines gesundes Buffet mit Obst, Gemüse und Getränken an dem sich alle stärken konnten.

agenda10 agenda09 agenda08

Nach der Pause wurden dann die Teams gemischt, so dass auch wirklich dem Nachhaltigkeitsgedanken des Agenda Diploms Rechnung getragen wurde, indem die Acht- bis Zwölfjährigen so nicht nur den Umgang mit zukunftsweisenden Themen wie Robotik, sondern auch wichtige sogenannte Softskills wie Teamfähigkeit lernen konnten.

agenda13 agenda12 agenda02

Am Ende der Veranstaltung erhielten die Teilnehmer noch eine eigens im Kartendrucker angefertigte Teilnehmerkarte mit Chip und Magnetstreifen zum Andenken an das Agenda Diplom.

agenda11

Wir sind sicher, dass wir einige der Kinder, aber auch manche Eltern in nächster Zeit wieder im RaumZeitLabor begrüßen dürfen. Darauf freuen wir uns ebenso wie auf die nächsten Teilnehmer der Veranstaltungen am 12. Juli (augebucht), 15. August und 20. September.

by zwetschgo at 13. May 2014 12:49:14