Welcome to my web log. See the first post for an introduction. See the archive page for all posts, and comments for a feed of comments only. (There is an english language feed if you don't want to see Finnish.)

Archives Tags Recent Comments Moderation policy Main site

All content outside of comments is copyrighted by Lars Wirzenius, and licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License. Comments are copyrighted by their authors.

Recently, Daniel visited us in Helsinki. In addition to enjoying local food and scenerey, we spent some time together in front of a whiteboard to sketch out designs for Ick2. Ick is my continuous integration system, and it's all Daniel's fault for suggesting the name. Ahem.

I am currently using the first generation of Ick and it is a rigid, cumbersome, and fragile thing. It works well enough that I don't miss Jenkins, but I would like something better. That's the second generation of Ick, or Ick2, and that's what we discussed with Daniel.

Where pretty much everything in Ick1 is hardcoded, everything in Ick2 will be user-configurable. It's my last, best chance to go completely overboard in the second system syndrome manner.

Where Ick1 was written in a feverish two-week hacking session, rushed because my Jenkins install at the time had broken one time too many, we're taking our time with Ick2. Slow and careful is the tune this time around.

Our "minimum viable product" or MVP for Ick2 is defined like this:

Ick2 builds static websites from source in a git repository, using ikiwiki, and published to a web server using rsync. A change to the git repository triggers a new build. It can handle many separate websites, and if given enough worker machines, can build many of them concurrently.

This is a real task, and something we already do with Ick1 at work. It's a reasonable first step for the new program.

Some decisions we made:

  • The Ick2 controller, which decides which projects to build, and what's the next build step at any one time, will be reactive only. It will do nothing except in response to an HTTP API request. This includes things like timed events. An external service will need to poke the controller at the right time.

  • The controller will be accompanied by worker manager processes, which fetch instructions of what to do next, and control actual worker over ssh.

  • Provisioning of the workers is out of scope for the MVP. For the MVP we are OK with a static list of workers. In the future we might make worker registration be a dynamic things, but not for the MVP. (Parts or all of this decision may be changed in the future, but we need to start somewhere.)

  • The MVP publishing will happen by running rsync to a web server. Providing credentials for the workers to do that is the sysadmin's problem, not something the MVP will handle itself.

  • The MVP needs to handle more than one worker, and more than one pipelines, and needs to build things concurrently when there's call for it.

  • The MVP will need to read the pipelines (and their steps and any other info) from YAML config files, and can't have that stuff hardcoded.

  • The MVP API will have no authentication or authorization stuff yet.

The initial pipelines will be basically like this, but expressed in some way by the user:

  1. Clone the source repoistory.
  2. Run ikiwiki --build to build the website.
  3. Run rsync to publish the website on a server.


  • Every worker can clone from the git server.
  • Every worker has all the build tools.
  • Every worker has rsync and access to every web server.
  • Every pipeline run is clean.

Actions the Ick2 controller API needs to support:

  • List all existing projects.
  • Trigger a project to build.
  • Query what project builds are running.
  • Get build logs for a project: current log (from the running build), and the most recent finished build.

A sketch API:

  • POST /projects/foo/+trigger

    Trigger build of project foo. If the git hasn't changed, the build runs anyway.

  • GET /projects

    List names of all projects.

  • GET /projects/foo

    On second thought, I can't think of anything useful for this to return for the MVP. Scratch.

  • GET /projects/foo/logs/current

    Return entire known build log captured so far for the currently running build.

  • GET /projects/foo/logs/previous

    Return entire build log for latest finished build.

  • GET /work/bar

    Used by worker bar: return next not-yet-finished step to run as a JSON object containing fields "project" (name of project for which to run the step) and "shell" (a shell command to run). The call will return the same JSON object until the worker reports it as having finished.

  • POST /work/bar/snippet

    Used by worker bar to report progress on the currently running step: a JSON object containing fields "stdout" (string with output from the shell command's stdout), "stderr" (ditto but stderr), and "exit_code" (the shell command's exit code, if it's finished, or null).


  • Git server has a hook that calls "GET /projects/foo/+trigger" (or else this is simulated by user).

  • Controller add a build of project foo to queue.

  • Worker manager calls "GET /work/bar", gets a shell command to run, and starts running it on its worker.

  • While worker runs shell command, every second or so, worker manager calls "POST /work/bar/snippet" to report progress including collected output, if any.

  • Controller responds with OK or KILL, and if the latter, worker kills the command it is running. Worker manager continues reporting progress via snippet until shell command is finished (on its own or by having been killed).

  • Controller appends any output reported via .../snippet. When it learns a shell command has finished, it updates its idea of the next step to run.

  • When controller learns a project has finished building, it rotates the current build log to be the previous one.

The next step will probably be to sketch a yarn test suite of the API and implement a rudimentary one.

Posted Sun May 7 19:00:00 2017 Tags:

It is a new month, and time to publish the next chapter in Hacker Noir. This is chapter 2, titled "Development setup phase". I hope you enjoy it. Feedback via email, irc, identi.ca, twitter are welcome. Or come talk to me at FOSDEM if you're there.

Posted Wed Feb 1 13:26:00 2017 Tags:

Half a year ago I wrote a blog post about debugging over email. This is a follow-up.

The blog post summarised:

  • Have an automated way to collect all usual informaion needed for debugging: versions, config and log files, etc.

  • Improve error messages so the users can solve their issues themselves.

  • Give users better automated diagnostics tools.

Based on further thinking and feedback, I add:

  • When a program notices a problem that may indicate a bug in it, it should collect the necessary information itself, automatically, in a way that the user just needs to send to the developers / support.

  • The primary goal should be to help people solve their own problems.

  • A secondary goal is to make the problem reproducible by the developers, or otherwise make it easy to fix bugs without access to the original system where the problem was manifested.

I've not written any code to help with this remote debugging, but it's something I will start experimenting with in the near future.

Further ideas welcome.

Posted Sun Jan 22 17:35:00 2017 Tags:

I participated in Nanowrimo in November, but I failed to actually finish the required 50,000 words during the month. Oh well. I plan on finishing the book eventually, anyway.

Furthermore, as an open source exhibitionist I thought I'd publish a chapter each month. This will put a bit of pressure on me to keep writing, and hopefully I'll get some nice feedback too.

The working title is "Hacker Noir". I've put the first chapter up on http://noir.liw.fi/.

Posted Sat Jan 7 22:46:00 2017 Tags:

I spent a few days in Cambridge for a minidebconf. This is a tiny version of the full annual Debconf. We had a couple of days for hacking, and another two days for talks.

I spent my hacking time on thinking about vmdebootstrap (my tool for generating disk images with an installed Debian), and came to the conclusion I need to atone my sins for writing such crappy code by rewriting it from scratch to be nicer to use. I gave a talk about this, too. The mailing list post has the important parts, and meetings-archive has a video.

I haven't started the rewrite, and it's not going to make it for stretch.

I also gave two other talks, on the early days of Linux, and Qvarn, the latter being what I do at work.

Thank you to ARM, for sponsoring the location, and the other sponsors for sponsoring food. These in-real-life meetings between developers are important for the productivity and social cohesion of Debian.

Posted Tue Nov 22 19:08:00 2016 Tags:

I have just released version 1.20 of Obnam, my backup program. It's been nine months since the previous release, and that's a long time: I've had an exciting year, and not entirely in a good way. Unfortuntely that's eaten up a lot of my free time and enthusiasm for my hobby projects.

See below for a snippet of NEWS, with a summary of the user-visible changes. A lot of the effort has gone into improving FORMAT GREEN ALBATROSS, but that isn't documented in the NEWS file.

I've received patches and actionable bug reports from a number of people, and I'm grateful for those. I try to credit them by name in the NEWS file.

Obnam NEWS

This file summarizes changes between releases of Obnam.

NOTE: Obnam has an EXPERIMENTAL repository format under development, called green-albatross-20160813. It is NOT meant for real use. It is likely to change in incompatible ways without warning. DO NOT USE it unless you're willing to lose your backup.

Version 1.20, released 2016-10-29

  • The format name for green-albatross is renamed to green-albatross-20160813 and will henceforth be renamed every time there's a change, to avoid confusing Lars because of backwards incompatibilities. When it reaches stability and the on-disk format is frozen, it'll be renamed back to a date-less version.

  • Those using the experimental green-albatross repository format will have to start over with fresh repositories. This release contains backwards incompatible changes that mean existing repositories no longer work. Sorry, but that's what experimental means.

  • A green-albatross change is that the "chunk index" data structure is no longer a single blob, and instead it's broken down into smaller objects. This avoids keeping all of the chunk indexes in memory at once, which should reduce memory use.

  • Remi Rampin started updating and continuing the French translation of the Obnam manual.

  • Lars Wirzenius changed the default so that Obnam reads random data when creating encryption key from /dev/urandom instead of /dev/random. The goal is to make it less likely that Obnam stops at the key generation stage on machines with little entropy. Set weak-random = no in your configuration to override this.

Minor changes:

  • Lars Wirzenius changed obnam forget so that if there is nothing to do, it doesn't even try to connect to the repository.

  • Lars Wirzenius added a chapter on participating in the Obnam project to the manual.

  • Lars Wirzenius changed --one-file-system to work for bind mounts. It only works for bind mounts that exist at the time when Obnam starts, however. Also, /proc/mounts must be an accurate list of mount points.

  • Lars Wirzenius added the gpg command line to the error message about gpg failing.

Bug fixes:

  • The manual and manual page used to claim you could break only the locks for one client. This was not true. The manuals has been fixed.

  • A whole bunch of typo fixes, from Andrea Gelmini.

  • Michel Alexandre Salim fixed a bug in the FUSE (obnam mount) plugin, which was a typo in a method name (get_clientgeneration_ids).

  • Lars Wirzenius fixed obnam restore to require a target set with --to. Jonathan Dowland reported the problem.

  • Lars Wirzenius fixed obnam list-errors so that it doesn't crash on error classes that only exist to make the exception hierarchy neater, such as EncryptionError. Bug reported by Rik Theys.

  • Ian Cambell fixed a bug in obnam kdirstat and its handling of FIFO sockets.

Posted Sat Oct 29 12:30:00 2016 Tags:

We needed a router and wifi access point in the office, and simultaneously both I and my co-worker Ivan needed such a thing at our respective homes. After some discussion, and after reading articles in Ars Technica about building PCs to act as routers, we decided to do just that.

  • The PC solution seem to offer better performance, but this is actually not a major reason for us.

  • We want to have systems we understand and can hack. A standard x86 PC running Debian sounds ideal to use.

  • Why not a cheap commercial router? They tend to be opaque and mysterious, and can't be managed with standard tooling such as Ansible. They may or may not have good security support. Also, they may or may not have sufficient functionality to be nice things, such as DNS for local machines, or the full power if iptables for firewalling.

  • Why not OpenWRT? Some models of commercial routers are supported by OpenWRT. Finding good hardware that is also supported by OpenWRT is a task in itself, and not the kind of task especially I like to do. Even if one goes this route, the environment isn't quite a standard Linux system, because of various hardware limitations. (OpenWRT is a worthy project, just not our preference.)

We got some hardware:

Component Model Cost
Barebone Qotom Q190G4, VGA, 2x USB 2.0, 134x126x36mm, fanless 130€
CPU Intel J1900, 2-2.4GHz quad-core -
NIC Intel WG82583, 4x 10/100/1000 -
Memory Crucial CT102464BF160B, 8GB DDR3L-1600 SODIMM 1.35V CL11 40€
SSD Kingston SSDNow mS200, 60GB mSATA 42€
WLAN AzureWave AW-NU706H, Ralink RT3070L, 300M 802.11b/g/n, half mPCIe 17€
mPCIe adapter Half to full mPCIe adapter 3€
Antennas 2x 2.4/5GHz 6dBi, RP-SMA, U.FL Cables 7€

These were bought at various online shops, including AliExpress and verkkokauppa.com.

After assembling the hardware, we installed Debian on them:

  • Connect the PC to a monitor (VGA) and keyboard (USB), as well as power.

  • I built a "factory image" to be put on the SSD, and a USB stick installer image, which includes the factory one. Write the installer image on a USB stick, boot off that, then copy the factory image to the SSD and reboot off the SSD.

  • The router now runs a very bare-bones, stripped-down Debian system, which runs a DHCP server on eth3 (marked LAN4 on the box). You can log as root on the console (no password), or via ssh, but for ssh you need to replace the /home/ansible/.ssh/authorized_keys file with one that contains only your public ssh key.

  • Connect a laptop to the Ethernet port marked LAN4, and get an IP address with DHCP.

  • Log in with ssh to ansible@, and verify that sudo id works without password. Except you can't do this, unless you put in your ssh key in the authorized keys file above.

  • Git clone the ansible playbooks, adjust their parameters in minipc-router.yml as wanted, and run the playbook. Then reboot the router again.

  • You should now have wifi, routing (with NAT), and be generally speaking able to do networking.

There's a lot of limitations and problems:

  • There's no web UI for managing anything. If you're not comfortable doing sysadmin via ssh (with or without ansible), this isn't for you.

  • No IPv6. We didn't want to enable it yet, until we understand it better. You can, if you want to.

  • No real firewalling, but adjust roles/router/files/ferm.conf as you wish.

  • The router factory image is 4 GB in size, and our SSD is 60 GB. That's a lot of wasted space.

  • The router factory image embeds our public keys in the ansible user's authorized keys file for ssh. This is because we built this for ourselves first. If there's interest by others in using the images, we'll solve this.

  • Probably a lot of stupid things. Feel free to tell us what it is (bugs@liw.fi would be a good address for that).

If you'd like to use the images and Ansible playbooks, please do. We'd be happy to get feedback, bug reports, and patches. Send them to me (liw@liw.fi) or my ticketing system (bugs@liw.fi).

Posted Tue Oct 4 09:56:23 2016 Tags:

A year ago I got tired of Jenkins and wrote a CI system for myself, Ick. It's served me well since, but it's a bit clunky and awkward and I have to hope nobody else wants to use it.

I've been thinking about re-architecting Ick from scratch, and so I wrote down some of my thinking about this. It's very raw, but just in case someone else might be interested, I put it online at ick2.

At this point I'm still thinking about very high level concepts. I've not written any code, and probably won't in the next couple of months. But I had to get this out of my brain.

Posted Fri Sep 9 18:02:41 2016 Tags:

I gave a talk about the early days of Linux at the jubilee symposium arranged by the University of Helsinki CS department. Below is an outline of what I meant to speak about, but the actual talk didn't follow it exactly. You can compare these to the video once it comes online.

  • Linus and I met at uni, the only 2 Swedish speaking new students that year, so we naturally migrated towards each other.
  • After a year away for military service, got back in touch, summer of
    1. .
  • C & Unix course fall of 1990; Minix.
  • Linus didn't think atime updates in real time were plausible, but I showed him; funnily enough, atime updates have been an issue in Linux until fairly recently, since they slow things down (without being particularly useful)
  • Jan 5, 1991 bought his first PC (i386 + i387 + 4 MiB RAM and a small hard disk); he had a Sinclair QL before that.
  • Played Prince of Persia for a couple of months.
  • Then wanted to learn i386 assembly and multitasking.
  • A/B threading demo.
  • Terminal emulation, Usenet access from home.
  • Hard disk driver, mistaking hard disk for a modem.
  • More ambition, announced Linux to the world for the first time
  • first ever Linux installation.
  • Upload to ftp.funet.fi, directory name by Ari Lemmke.
  • Originally not free software, licence changed early 1992.
  • First mailing list was created and introduced me to a flood of email (managed with VAX/VMS MAIL and later mush on Unix).
  • I talked a lot with Linus about design at this time, but never really participated in the kernel work (partly because disagreeing with Linus is a high-stress thing).
  • However, I did write the first sprintf for the kernel, since Linus hadn't learnt about varargs functions in C; he then ruined it and added the comment "Wirzenius wrote this portably..." (add google hit count for wirzenius+fucked).
  • During 1992 Linux grew fast, and distros happened, and a lot of packaging and porting of software; porting was easier because Linus was happy to add/change things in the kernel to accomodate software
  • A lot of new users during 1992 as well.
  • End of 1992 I and a few others founded the Linux Documentation Project to help all the new users, some of who didn't come from a Unix background.
  • In fact, things progressed so fast in 1992 that Linus thought he'd release 1.0 very soon, resulting in a silly sequence of version numbers: 0.12, 0.95, 0.96, 0.96b, 0.96c, 0.96c++2.
  • X server ported to Linux; almost immediate prediction of the year of the Linux desktop never happening unless ALL the graphics cards were supported immediately.
  • Linus was of the opinion that you needed one process (not thread) per window in X; I taught him event driven programming.
  • Bug in network code, resulting in ban on uni network.
  • Pranks in the shared office room.
  • We released 1.0 in an event at the CS dept in March, 1994; this included some talks and a ritual compilation of the release version during the event.
Posted Mon Aug 22 15:00:43 2016 Tags:

Today it is 23 years ago since Ian Murdock published his intention to develop a new Linux distribution, Debian. It also about 20 years since I became a Debian developer and made my first package upload.

In the time since:

  • I've retired a couple of times, to pursue other interests, and then un-retired.

  • I've maintained a bunch of different packages, most importantly the PGP2 software in the 90s. (I now only maintain software for which I'm also upstream, in order to make jokes about my upstream being an unco-operative jerk, and my packager being unhelpful in the extreme.)

  • Got kicked out from the Debian mailing lists for insulting another developer. Not my proudest moment. I was allowed back later, and I've tried to be polite ever since. (See also rules 6.)

  • I've been to a few Debconfs (3, 5, 6, 9, 10, 15). I'm looking forward to going to many more in the future. It's clear that seeing many project members at least every now and then has a very big impact on project cohesion.

  • I had a gig where I was paid to improve the technical quality of Debian. After a few months of bug fixing (which isn't my favourite pastime), I wrote piuparts in order to find new bugs. (I gave that project away many years ago, but it seems to still be going strong.)

  • I've almost ran for DPL twice, but I'm glad I didn't actually. I've carefully avoided any positions of power or responsibility in the project. (I live in fear that someone decides to nominate me for something where I'd actually have make important decisions.)

    Not being responsible means I can just ignore the project for a while when something annoying happens. (Or retire again.) With such a large project, eventually something really annoying does happen.

  • Came up with the DEP process with Zack and Dato. I also ran the second half of the DEP5 process to get the debian/copyright machine readable format accepted. (I'm no longer involved, though, and I don't think DEP is much now.)

  • I've taught several workshops about Debian packaging, including online for Debian-Women. It's always fun when others "get" how easy packaging really is, despite all the efforts of the larger variety in tooling and random web pages go to to obscure the fundamental simplicity.

  • Over the years Í've enjoyed many of the things developed within Debian (without claiming any credit for myself):

    • the policy manual, perhaps the most important technical achievement of the project

    • the social contract and Debian free software guidelines, unarguably the most important non-technical achievements of the project

    • the whole package management system, but especially apt

    • debhelper's dh, which made the work of packaging simple cases so easy it's nearly a no-brainer

    • d-i made me not hate installing Debian (although I think time is getting ripe to replace d-i with something new; catch me in a talkative mood at party to hear more)

    • Debian-Women made an almost immediate improvement to the culture of the larger project (even if there's still much too few women developers)

    • the diversity statement made me a lot happier about being a project member.

    I'd like to thank everyone who's worked on these and made them happen. These are important milestones in Debian.

  • I've opened my mount in a lot of places over the years, which means a lot of people know of me, but nobody can actually point at anything useful I've actually done. Which is why when I've given talks at, say, FOSDEM, I get introduced as "the guy who shared an office with Linus Torvalds a long time ago".

  • I've made a number of friends via participation in Debian. I've found jobs via contacts in Debian, and have even started a side business with someone.

It's been a good twenty years. And the fun ain't over yet.

Posted Tue Aug 16 15:42:44 2016 Tags:

For more, see the archive.