Feed for Planet Debian.

tl;dr: Get vmdebootstrap replacement from http://git.liw.fi/vmdb2 and run it from the source tree. Tell me if something doesn't work. Send patches.

Many years ago I wrote vmdebootstrap, a tool for installing Debian on a disk image for virtual machines. I had a clear personal need: I was setting up a CI system and it needed six workers: one each for Debian oldstable, stable, and unstable, on two architectures (i386, amd64). Installing Debian six times in the same way is a lot of work, so I figured how difficult can it be to automate it. Turns out that not difficult at all, except to install a bootloader.

(Don't ask me why I didn't use any of the other tools for this. It was long ago, and while some of the tools that now exist probably did exist then, I like writing code and learning things while doing it.)

After a while I was happy with what the program did, but didn't want to upload it to Debian, and didn't want to add the kinds of things other people wanted, so I turned vmdebootstrap over to Neil Williams, who added a ton of new features. Unfortunately, it turned out that my initial architecture was not scaleable, and also the code I wrote wasn't very good, and there weren't any tests.

Neil did heroic work forcing my crappy software into doing things I never envisioned. Last year he needed a break and asked me to take vmdebootstrap back. I did, and have been hiding from the public eye ever since, since I was so ashamed of the code. (I created a new identity and pretended to be an international assassin and backup specialist, travelling the world forcing people to have at least one tested backup of their system. If you've noticed reports in the press about people reporting near-death experiences while holding a shiny new USB drive, that would've been my fault.)

Pop quiz: if you have a program with ten boolean options ("do this, except if that option is given, do the other thing"), how many black box tests do you need to test all the functionality? If one run of the program takes half an hour, how long will a full test suite run?

I did some hard thinking about vmdebootstrap, and came to the sad conclusion that it had reached the end of its useful life as a living software project. There was no reasonable way to add most of the additional functionality people were asking for, and even maintaining the current code was too tedious a task to consider seriously. It was time to make a clean break of the past and start over, without caring about backwards compatibility. After all, the old code wasn't going anywhere so anyone who needed it could still use it. There was no need to burden a new program with my past mistakes. All new mistakes were called for.

At the Cambridge mini-Debconf of November, 2016, I gave a short presentation of what I was going to do. I also posted about my plans to the debian-cloud list. In short, I would write a new, more flexible and cleaner replacement to be called vmdb2. For various personal reasons, I've not been able to spend as much time on vmdb2 as I'd like to, but I've now reached the point where I'd like to announce the first alpha version publically.

The source code is hosted here: http://git.liw.fi/vmdb2 . There are .deb packages at my personal public APT repo (http://liw.fi/code/), but vmdb2 is easy enough to run directly from a git checkout:

sudo ./vmdb2 foo.vmdb --output foo.img

There's no need to install it to try it.

What works:

  • vmdb2 can build a disk image with Debian installed, for amd64 only at this time
  • the boot loader is GRUB, either for UEFI or BIOS
  • the image boots under Qemu / KVM and also on actual hardware

What doesn't work:

  • other architecures, including building for a foreign archiecture
  • live CD building (no squashfs support)

I'm not opposed to adding support for those, but they're not directly interesting to me. For example, I only have amd64 machines. The best way to get support for additional features is to tell me how, preferably in the form of patches. (If I have to read tons of docs, or other people's code, and then write code and iterate while other people tell me it doesn't work, it's probably not happening.)

Why would you be interested in vmdb2? There's a lot of other tools to do what it does, so perhaps you shouldn't care. That's fine. I like writing tools for myself. But if this kind of tool is of interest to you, please do have a look.

A short tutorial: vmdb2 wants you to give it a "specification file" (conventionally suffixed .vmdb, because someone stole the .spec suffix, but vmdb2 doesn't care about the name). Below is an example. vmdb2 image specification files are in YAML, since I like YAML, and specify a sequence of steps to take to build the image. Each step is a tiny piece of self-contained functionality provided by a plugin.

steps:
  - mkimg: "{{ output }}"
    size: 4G

  - mklabel: msdos
    device: "{{ output }}"

  - mkpart: primary
    device: "{{ output }}"
    start: 0%
    end: 100%
    part-tag: root-part

The above create an image (name is specified with the --output option), which is four gigabytes in size, and create a partitition table and a single partition that fills the whole disk. The "tag" is given so that later steps can easily refer to the partition.

If you prefer another way to partition the disk, you can achieve that by adding more "mkpart" steps. For example, for UEFI you'll want to have an EFI partition.

  - mkfs: ext4
    partition: root-part

  - mount: root-part
    fs-tag: root-fs

The above formats the partition with the ext4 filesystem, and then mounts it. The mount point will be a temporary directory created by vmdb2, and a tag is again given to the mount point so it can be referred to.

  - unpack-rootfs: root-fs

The above unpacks a tar archive to put content into the filesystem, if the tar archive exists. The tar archive is specified with the --rootfs-tarball command line option.

  - debootstrap: stretch
    mirror: http://http.debian.net/debian
    target: root-fs
    unless: rootfs_unpacked

  - apt: linux-image-amd64
    fs-tag: root-fs
    unless: rootfs_unpacked

  - cache-rootfs: root-fs
    unless: rootfs_unpacked

The above will run debootstrap and install a kernel into the filesystem, but skip doing that if the rootfs tarball was used. Also, the tarball is created if it didn't exist. This way the tarball is used by all but the first run, which saves a bit of time. On my laptop and with a local mirror, debootstrap and kernel installation takes on the order of nine minutes (500 to 600 seconds), whereas unpacking the tar archive is a bit faster (takes around 30 seconds). When iterating over things other than debootstrap, this speeds things up something wonderful, and seems worth the complexity.

The "unless:" mechanism is generic. All the steps share some state, and the unpack-rootfs step sets the "rootfs_unpacked" flag in the shared state. The "unless:" field tells vmdb2 to check for the flag and if it is not set, or if it is set to false ("unless it is set to true"), vmdb2 will execute the step. vmdb2 may get more such flags in the future, if there's need.

  - chroot: root-fs
    shell: |
      sed -i '/^root:[^:]*:/s//root::/' /etc/passwd
      echo pc-vmdb2 > /etc/hostname

The above executes a couple of shell commands in a chroot of the root filesystem we've just created. In this case they remove a login password from root, and set the hostname. This is a replacement of the vmdebootstrap "customize" script, but it can be inserted anywhere into the sequence of steps. There's boot chroot and non-chroot variants of the step.

This is a good point to mention that writing customize scripts gets quite repetitive and tedious after a while, so vmdb2 has a plugin to run Ansible instead. You can customize your image with that instead, while the image is being built and not have to wait until you boot the image and running Ansible over ssh.

  - grub: bios
    root-fs: root-fs
    root-part: root-part
    device: "{{ output }}"
    console: serial

Finally, install a boot loader, grub. This shows the BIOS variant, UEFI is also supported. This also configures grub and the kernel to use a serial console. There's a "yarn" (test suite) to build and smoke test an image with vmdb2 to make sure at least the basic functionality works. The smoke test boots the image under Qemu, logs in as root, and tells the VM to power off. Very, very basic, but has already found actual bugs in vmdb2. The smoke test needs the serial console to work.

As with vmdebootstrap originally, I don't particularly want to maintain the package in Debian. I've added Debian packaging (so that I can install it on my own machines), but I already have enough packages to maintain, so I'm hoping someone else will volunteer to take on the Debian maintainership and bug handling duties.

If you would like vmdb2 to do more things to suit you better, I'm happy to explain how to write plugins to provide more types of steps.

If you are currently using vmdebootstrap, either directly or as part of another tool, I encourage you to have a look at vmdb2. In the long term, I would like to retire vmdebootstrap entirely, once vmdb2 can do everything vmdebootstrap can do, and few people use vmdebootstrap. This may take a while.

In any case, whether you want a new image building tool or not, happy hacking.

Posted Sun Jun 4 15:08:00 2017 Tags:

Introduction

I've written before about using a U2F key with PAM. This post continues the theme and explains how to use a smartcard with GnuPG for storing OpenPGP private keys. Specifically, a Yubikey 4 card, because that's what I have, but any good GnuPG compatible card should work. The Yubikey is both a GnuPG compatible smart card, and a U2F card. The Yubikey 4 can handle keys up to 4096 bits. Older Yubikeys can only handle keys up to 2095 bits.

The reason to do this is to make it harder for an attacker to steal your encryption keys.

I will assume you don't already have an OpenPGP key, or are willing to generate a new one. I will also assume you run Debian stretch; some of the desktop environment setup details may differ between Debian versions or between Linux distributions. You will need:

  • A Yubikey 4 (or other GnuPG compatible smartcard).
  • Two USB memory sticks to store master copies of the key you create.
  • Either a lot of patience, or a ChaosKey or something else to generate a lot of entropy for the kernel. Entropy is used by GnuPG to create encryption keys.

Terminology

Some terminology:

  • OpenPGP is a standard of encryption keys.

  • GnuPG (also known as GPG or gpg) is an encryption program that supports OpenPGP.

  • Encryption key or key pair is a secret and a public key for encryption. Key and key pair are often used as synonyms.

  • The secret keys is yours, and only you will have it or be able to you use. The public key is public, anyone can have a copy. The two are linked and you can't create a secret key linked to a public key. (If you can, you will become a famous cryptographer. Like Isaac Newton famous among apple growers.)

  • Master key is the important thing to keep track of. In the encryption world, the master key is what identifies you.

  • A subkey is a secondary key or key pair for encryption, derived from the master key. Subkeys can be created and revoked almost at will, and there can be any number of them. Subkeys are always associated with a single master key.

  • Subkeys can be dedicated for encryption, signing, and authentication. You can have one of each.

  • Encryption is the process of taking a file and making it illegible to everyone except the owner of a secret key. When the public key is used to encrypt, only the secret key can decrypt.

  • Signing is using a secret key to make a separate file that others can verify using the corresponding public key. This means the signer can "prove" they have the file. For example, you might sign emails to prove they came from you.

  • Authentication is the process of proving you're you. Typically, for example, a server will know your public key, and will give a large random number encrypted with your public key. Only you have the corresponding secret key, so only you can send the random number back to the server, and thereby the server knows you're you.

Outline

The process outline is:

  1. Create a new, signing-only master key with GnuPG.

  2. Create three "subkeys", one each for encryption, signing, and authentication. These subkeys are what everyone else uses.

  3. Export copies of the master key pair and the subkey pairs and put them in a safe place.

  4. Put the subkeys on the Yubikey.

  5. GnuPG will automatically use the keys from the card. You have to have the card plugged into a USB port for things to work. If someone steals your laptop, they won't get the private subkeys. Even if they steal your Yubikey, they won't get them (the smartcard is physically designed to prevent that), and can't even use them (because there's PIN codes or passphrases and getting them wrong several times locks up the smartcard).

  6. Use gpg-agent as your SSH agent, and the authentication-only subkey on the Yubikey is used as your ssh key.

Configure GnuPG

The process in more detail:

  • Configure GnuPG with regards to checksum and encryption algorithms. You can use the defaults, but depending on the version of GnuPG you have, they may be weaker than is recommended. These values are from Riseup.net OpenPGP guide, see link at the end.

    Add the following lines to ~/.gnupg/gpg.conf:

      personal-digest-preferences SHA512
      cert-digest-algo SHA512
      default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed
      personal-cipher-preferences TWOFISH CAMELLIA256 AES 3DES
      keyserver pool.sks-keyservers.net
    

Create new keys

  • Create new sign-only master key.
$ gpg --full-generate-key
gpg (GnuPG) 2.1.18; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 4
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
       = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 1y
Key expires at Tue 29 May 2018 06:43:54 PM EEST
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name: Lars Wirzenius
Email address: liw@liw.fi
Comment: test key
You selected this USER-ID:
    "Lars Wirzenius (test key) <liw@liw.fi>>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: key 25FB738D6EE435F7 marked as ultimately trusted
gpg: directory '/home/liw/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/liw/.gnupg/openpgp-revocs.d/A734C10BF2DF39D19DC0F6C025FB738D6EE435F7.rev'
public and secret key created and signed.

Note that this key cannot be used for encryption. You may want to use
the command "--edit-key" to generate a subkey for this purpose.
pub rsa4096 2017-05-29 [SC] [expires: 2018-05-29]
      A734C10BF2DF39D19DC0F6C025FB738D6EE435F7
      A734C10BF2DF39D19DC0F6C025FB738D6EE435F7
uid Lars Wirzenius (test key) <liw@liw.fi>
  • Note that I set a 1-year expiration for they key. The expiration can be extended at any time (if you have the master secret key), but unless you do, the key won't accidentally live longer than the chosen time.

  • Review the key:

$ gpg --list-secret-keys
/home/liw/.gnupg/pubring.kbx
----------------------------
sec rsa4096 2017-05-29 [SC] [expires: 2018-05-29]
      A734C10BF2DF39D19DC0F6C025FB738D6EE435F7
uid [ultimate] Lars Wirzenius (test key) <liw@liw.fi>
  • You now have the signing-only master key. You should now create three subkeys (keyid is the key identifier shown in the key listing, A734C10BF2DF39D19DC0F6C025FB738D6EE435F7 above). Use the --expert option to be able to add an authentication-only subkey.
$ gpg --edit-key --expert A734C10BF2DF39D19DC0F6C025FB738D6EE435F7z
gpg (GnuPG) 2.1.18; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

sec rsa4096/25FB738D6EE435F7
     created: 2017-05-29 expires: 2018-05-29 usage: SC
     trust: ultimate validity: ultimate
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> addkey
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (12) ECC (encrypt only)
  (13) Existing key
Your selection? 4
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
       = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 1y
Key expires at Tue 29 May 2018 06:44:52 PM EEST
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

sec rsa4096/25FB738D6EE435F7
     created: 2017-05-29 expires: 2018-05-29 usage: SC
     trust: ultimate validity: ultimate
ssb rsa4096/05F88308DFB71774
      created: 2017-05-29 expires: 2018-05-29 usage: S
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> addkey
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (12) ECC (encrypt only)
  (13) Existing key
Your selection? 6
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 1y
Key expires at Tue 29 May 2018 06:45:22 PM EEST
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

sec rsa4096/25FB738D6EE435F7
     created: 2017-05-29 expires: 2018-05-29 usage: SC
     trust: ultimate validity: ultimate
ssb rsa4096/05F88308DFB71774
      created: 2017-05-29 expires: 2018-05-29 usage: S
ssb rsa4096/2929E8A96CBA57C7
      created: 2017-05-29 expires: 2018-05-29 usage: E
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> addkey
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (12) ECC (encrypt only)
  (13) Existing key
Your selection? 8

Possible actions for a RSA key: Sign Encrypt Authenticate
Current allowed actions: Sign Encrypt

   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? a

Possible actions for a RSA key: Sign Encrypt Authenticate
Current allowed actions: Sign Encrypt Authenticate

   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? s

Possible actions for a RSA key: Sign Encrypt Authenticate
Current allowed actions: Encrypt Authenticate

   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? e

Possible actions for a RSA key: Sign Encrypt Authenticate
Current allowed actions: Authenticate

   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? q
RSA keys may be btween 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
       = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 1y
Key expires at Tue 29 May 2018 06:45:56 PM EEST
Is this correct? (y/N) y
Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

sec rsa4096/25FB738D6EE435F7
     created: 2017-05-29 expires: 2018-05-29 usage: SC
     trust: ultimate validity: ultimate
ssb rsa4096/05F88308DFB71774
     created: 2017-05-29 expires: 2018-05-29 usage: S
ssb rsa4096/2929E8A96CBA57C7
     created: 2017-05-29 expires: 2018-05-29 usage: E
ssb rsa4096/4477EB0AEF1C440A
     created: 2017-05-29 expires: 2018-05-29 usage: A
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> save

Export secret keys to files, make a backup

  • You now have a master key and three subkeys. They are hidden in the ~/.gnupg directory. It is time to "export" the secret keys out from there.
$ gpg --export-secret-key --armor keyid > master.key
$ gpg --export-secret-subkeys --armor keyid > subkeys.key
  • You should keep these files safe. You don't want to lose them, and you don't want anyone else to get access to them. I recommend you format two USB memory sticks, format them using full-disk encryption, and copy the exported files to both of them. Then keep them somewhere safe.

    There's ways of making this part more sophisticated, but that's for another time.

  • The next step involves some hoop-jumping. What we want is to have the master secret key NOT on you machine, so we tell GnuPG to remove it. We exported it above, so we won't lose it. However, deleting the master secret key also removes the secret subkeys. But we can import those without importing the master secret key.

$ gpg --delete-secret-key keyid
$ gpg --import subkeys.key
  • Now verify that you have the secret subkeys, but not the master key. There should be one line starting with sec# (note the hash mark, which indicates the key isn't available), and three lines starting with ssb (no hash mark).
$ gpg -K
/home/liw/.gnupg/pubring.kbx
----------------------------
sec# rsa4096 2017-05-29 [SC] [expires: 2018-05-29]
      A734C10BF2DF39D19DC0F6C025FB738D6EE435F7
uid [ultimate] Lars Wirzenius (test key) <liw@liw.fi>
ssb rsa4096 2017-05-29 [S] [expires: 2018-05-29]
ssb rsa4096 2017-05-29 [E] [expires: 2018-05-29]
ssb rsa4096 2017-05-29 [A] [expires: 2018-05-29]

Install subkeys on a Yubikey

  • Now insert the Yubikey in a USB slot. We can start transferring the secret subkeys to the Yubikey. If you want, you can set your name and other information, and change PIN codes. There's several types of PIN codes: normal use, unblocking a locked card, and a third PIN code for admin operations. Changing the PIN codes is a good idea, otherwise everyone will just try the default of 123456 (admin 12345678). However, I'm skipping that in the interest of brevity.
$ gpg -card-edit
...
  • Actually move the subkeys to the card. Note that this does a move, not a copy, and the subkeys will be removed from your ~/.gnupg (check with gpg -K).
$ gpg --edit-key liw
gpg (GnuPG) 2.1.18; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

pub rsa4096/25FB738D6EE435F7
created: 2017-05-29 expires: 2018-05-29 usage: SC
trust: ultimate validity: ultimate
ssb rsa4096/05F88308DFB71774
created: 2017-05-29 expires: 2018-05-29 usage: S
ssb rsa4096/2929E8A96CBA57C7
created: 2017-05-29 expires: 2018-05-29 usage: E
ssb rsa4096/4477EB0AEF1C440A
created: 2017-05-29 expires: 2018-05-29 usage: A
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> key 1

pub rsa4096/25FB738D6EE435F7
created: 2017-05-29 expires: 2018-05-29 usage: SC
trust: ultimate validity: ultimate
ssb* rsa4096/05F88308DFB71774
created: 2017-05-29 expires: 2018-05-29 usage: S
ssb rsa4096/2929E8A96CBA57C7
created: 2017-05-29 expires: 2018-05-29 usage: E
ssb rsa4096/4477EB0AEF1C440A
created: 2017-05-29 expires: 2018-05-29 usage: A
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> keytocard
Please select where to store the key:
(1) Signature key
(3) Authentication key
Your selection? 1

pub rsa4096/25FB738D6EE435F7
created: 2017-05-29 expires: 2018-05-29 usage: SC
trust: ultimate validity: ultimate
ssb* rsa4096/05F88308DFB71774
created: 2017-05-29 expires: 2018-05-29 usage: S
ssb rsa4096/2929E8A96CBA57C7
created: 2017-05-29 expires: 2018-05-29 usage: E
ssb rsa4096/4477EB0AEF1C440A
created: 2017-05-29 expires: 2018-05-29 usage: A
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> key 1

pub rsa4096/25FB738D6EE435F7
created: 2017-05-29 expires: 2018-05-29 usage: SC
trust: ultimate validity: ultimate
ssb rsa4096/05F88308DFB71774
created: 2017-05-29 expires: 2018-05-29 usage: S
ssb rsa4096/2929E8A96CBA57C7
created: 2017-05-29 expires: 2018-05-29 usage: E
ssb rsa4096/4477EB0AEF1C440A
created: 2017-05-29 expires: 2018-05-29 usage: A
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> key 2

pub rsa4096/25FB738D6EE435F7
created: 2017-05-29 expires: 2018-05-29 usage: SC
trust: ultimate validity: ultimate
ssb rsa4096/05F88308DFB71774
created: 2017-05-29 expires: 2018-05-29 usage: S
ssb* rsa4096/2929E8A96CBA57C7
created: 2017-05-29 expires: 2018-05-29 usage: E
ssb rsa4096/4477EB0AEF1C440A
created: 2017-05-29 expires: 2018-05-29 usage: A
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> keytocard
Please select where to store the key:
(2) Encryption key
Your selection? 2

pub rsa4096/25FB738D6EE435F7
created: 2017-05-29 expires: 2018-05-29 usage: SC
trust: ultimate validity: ultimate
ssb rsa4096/05F88308DFB71774
created: 2017-05-29 expires: 2018-05-29 usage: S
ssb* rsa4096/2929E8A96CBA57C7
created: 2017-05-29 expires: 2018-05-29 usage: E
ssb rsa4096/4477EB0AEF1C440A
created: 2017-05-29 expires: 2018-05-29 usage: A
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> key 2

pub rsa4096/25FB738D6EE435F7
created: 2017-05-29 expires: 2018-05-29 usage: SC
trust: ultimate validity: ultimate
ssb rsa4096/05F88308DFB71774
created: 2017-05-29 expires: 2018-05-29 usage: S
ssb rsa4096/2929E8A96CBA57C7
created: 2017-05-29 expires: 2018-05-29 usage: E
ssb rsa4096/4477EB0AEF1C440A
created: 2017-05-29 expires: 2018-05-29 usage: A
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> key 3

pub rsa4096/25FB738D6EE435F7
created: 2017-05-29 expires: 2018-05-29 usage: SC
trust: ultimate validity: ultimate
ssb rsa4096/05F88308DFB71774
created: 2017-05-29 expires: 2018-05-29 usage: S
ssb rsa4096/2929E8A96CBA57C7
created: 2017-05-29 expires: 2018-05-29 usage: E
ssb* rsa4096/4477EB0AEF1C440A
created: 2017-05-29 expires: 2018-05-29 usage: A
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> keytocard
Please select where to store the key:
(3) Authentication key
Your selection? 3

pub rsa4096/25FB738D6EE435F7
created: 2017-05-29 expires: 2018-05-29 usage: SC
trust: ultimate validity: ultimate
ssb rsa4096/05F88308DFB71774
created: 2017-05-29 expires: 2018-05-29 usage: S
ssb rsa4096/2929E8A96CBA57C7
created: 2017-05-29 expires: 2018-05-29 usage: E
ssb* rsa4096/4477EB0AEF1C440A
created: 2017-05-29 expires: 2018-05-29 usage: A
[ultimate] (1). Lars Wirzenius (test key) <liw@liw.fi>

gpg> save
  • If you want to use several Yubikeys, or have a spare one just in case, repeat the previous four steps (starting from importing subkeys back into ~/.gnupg).

  • You're now done, as far GnuPG use is concerned. Any time you need to sign, encrypt, or decrypt something, GnuPG will look for your subkeys on the Yubikey, and will tell you to insert it in a USB port if it can't find the key.

Use subkey on Yubikey as your SSH key

  • To actually use the authentication-only subkey on the Yubikey for ssh, you need to configure your system to use gpg-agent as the SSH agent. Add the following line to .gnupg/gpg-agent.conf:

     enable-ssh-support
    
  • On a Debian stretch system with GNOME, edit /etc/xdg/autostart/gnome-keyring-ssh.desktop to have the following line, to prevent the GNOME ssh agent from starting up:

     Hidden=true
    
  • Edit /etc/X11/Xsession.options and remove or comment out the line that says use-ssh-agent. This stops a system-started ssh-agent from being started when the desktop start.

  • Create the file ~/.config/autostart/gpg-agent.desktop with the following content:

     [Desktop Entry]
     Type=Application
     Name=gpg-agent
     Comment=gpg-agent
     Exec=/usr/bin/gpg-agent --daemon
     OnlyShowIn=GNOME;Unity;MATE;
     X-GNOME-Autostart-Phase=PreDisplayServer
     X-GNOME-AutoRestart=false
     X-GNOME-Autostart-Notify=true
     X-GNOME-Bugzilla-Bugzilla=GNOME
     X-GNOME-Bugzilla-Product=gnome-keyring
     X-GNOME-Bugzilla-Component=general
     X-GNOME-Bugzilla-Version=3.20.0
    
  • To test, log out, and back in again, run the following in a terminal:

$ ssh-add -l

The output should contain a line that looks like this:

    4096 SHA256:PDCzyQPpd9tiWsELM8LwaLBsMDMm42J8/eEfezNgnVc cardno:000604626953 (RSA)
  • You need to export the authentication-only subkey in the SSH key format. You need this for adding to .ssh/authorized_keys, if nothing else.
$ gpg --export-ssh-key keyid > ssh.pub
  • Happy hacking.

See also

See also the following links. I've used them to learn enough to write the above.

Edited to fix:

  • Output of gpg -K after removing secret master key.
Posted Mon May 29 18:03:00 2017 Tags:

Distix is my distributed ticketing system. I initially wrote the core of it as a bit of programming performance art, to celebrate my 30 years as a programmer. Distix is built on top of git and emails in Maildirs. It is a silent listener to your issue and bug discussions: as long as you ensure it gets a copy of each mail, it takes care of automatically arranging things into separate tickets based on email threading. Users and customers do not need to even know Distix is being used. Only the "support staff" need ever interact with Distix, and they mostly only need to close tickets that have been dealt with.

I've been using Distix for my own stuff for some time now, and recently we've started using it at work. I slowly improve it as we find problems.

It's not a sleek, smooth, finished tool. It's clunky, weird, and probably not what you want. But it's what I want.

Changes in recent months:

  • There is a new website: http://distix.eu/. No particular good reason for a new website, but I won the domain for free a couple of years ago, so I might as well use it.

  • In addition, a ticketing system for Distix itself: http://tickets.distix.eu/. Possibly I should've called the subdomain dogfood, but I'm a serious person, not prone to trying to be funny.

  • Mails can now be imported using IMAP.

  • Importing has been optimized for speed and memory use, making my own production use more practical.

I've discussed with a friend the possibility of writing a web UI, and some day maybe that will happen. For now, distix is a command line applicaton that can generate a static HTML site.

Posted Sat May 27 11:07:00 2017 Tags:

Recently, Daniel visited us in Helsinki. In addition to enjoying local food and scenerey, we spent some time together in front of a whiteboard to sketch out designs for Ick2. Ick is my continuous integration system, and it's all Daniel's fault for suggesting the name. Ahem.

I am currently using the first generation of Ick and it is a rigid, cumbersome, and fragile thing. It works well enough that I don't miss Jenkins, but I would like something better. That's the second generation of Ick, or Ick2, and that's what we discussed with Daniel.

Where pretty much everything in Ick1 is hardcoded, everything in Ick2 will be user-configurable. It's my last, best chance to go completely overboard in the second system syndrome manner.

Where Ick1 was written in a feverish two-week hacking session, rushed because my Jenkins install at the time had broken one time too many, we're taking our time with Ick2. Slow and careful is the tune this time around.

Our "minimum viable product" or MVP for Ick2 is defined like this:

Ick2 builds static websites from source in a git repository, using ikiwiki, and published to a web server using rsync. A change to the git repository triggers a new build. It can handle many separate websites, and if given enough worker machines, can build many of them concurrently.

This is a real task, and something we already do with Ick1 at work. It's a reasonable first step for the new program.

Some decisions we made:

  • The Ick2 controller, which decides which projects to build, and what's the next build step at any one time, will be reactive only. It will do nothing except in response to an HTTP API request. This includes things like timed events. An external service will need to poke the controller at the right time.

  • The controller will be accompanied by worker manager processes, which fetch instructions of what to do next, and control actual worker over ssh.

  • Provisioning of the workers is out of scope for the MVP. For the MVP we are OK with a static list of workers. In the future we might make worker registration be a dynamic things, but not for the MVP. (Parts or all of this decision may be changed in the future, but we need to start somewhere.)

  • The MVP publishing will happen by running rsync to a web server. Providing credentials for the workers to do that is the sysadmin's problem, not something the MVP will handle itself.

  • The MVP needs to handle more than one worker, and more than one pipelines, and needs to build things concurrently when there's call for it.

  • The MVP will need to read the pipelines (and their steps and any other info) from YAML config files, and can't have that stuff hardcoded.

  • The MVP API will have no authentication or authorization stuff yet.

The initial pipelines will be basically like this, but expressed in some way by the user:

  1. Clone the source repoistory.
  2. Run ikiwiki --build to build the website.
  3. Run rsync to publish the website on a server.

Assumptions:

  • Every worker can clone from the git server.
  • Every worker has all the build tools.
  • Every worker has rsync and access to every web server.
  • Every pipeline run is clean.

Actions the Ick2 controller API needs to support:

  • List all existing projects.
  • Trigger a project to build.
  • Query what project builds are running.
  • Get build logs for a project: current log (from the running build), and the most recent finished build.

A sketch API:

  • POST /projects/foo/+trigger

    Trigger build of project foo. If the git hasn't changed, the build runs anyway.

  • GET /projects

    List names of all projects.

  • GET /projects/foo

    On second thought, I can't think of anything useful for this to return for the MVP. Scratch.

  • GET /projects/foo/logs/current

    Return entire known build log captured so far for the currently running build.

  • GET /projects/foo/logs/previous

    Return entire build log for latest finished build.

  • GET /work/bar

    Used by worker bar: return next not-yet-finished step to run as a JSON object containing fields "project" (name of project for which to run the step) and "shell" (a shell command to run). The call will return the same JSON object until the worker reports it as having finished.

  • POST /work/bar/snippet

    Used by worker bar to report progress on the currently running step: a JSON object containing fields "stdout" (string with output from the shell command's stdout), "stderr" (ditto but stderr), and "exit_code" (the shell command's exit code, if it's finished, or null).

Sequence:

  • Git server has a hook that calls "GET /projects/foo/+trigger" (or else this is simulated by user).

  • Controller add a build of project foo to queue.

  • Worker manager calls "GET /work/bar", gets a shell command to run, and starts running it on its worker.

  • While worker runs shell command, every second or so, worker manager calls "POST /work/bar/snippet" to report progress including collected output, if any.

  • Controller responds with OK or KILL, and if the latter, worker kills the command it is running. Worker manager continues reporting progress via snippet until shell command is finished (on its own or by having been killed).

  • Controller appends any output reported via .../snippet. When it learns a shell command has finished, it updates its idea of the next step to run.

  • When controller learns a project has finished building, it rotates the current build log to be the previous one.

The next step will probably be to sketch a yarn test suite of the API and implement a rudimentary one.

Posted Sun May 7 19:00:00 2017 Tags:

It is a new month, and time to publish the next chapter in Hacker Noir. This is chapter 2, titled "Development setup phase". I hope you enjoy it. Feedback via email, irc, identi.ca, twitter are welcome. Or come talk to me at FOSDEM if you're there.

Posted Wed Feb 1 13:26:00 2017 Tags:

Half a year ago I wrote a blog post about debugging over email. This is a follow-up.

The blog post summarised:

  • Have an automated way to collect all usual informaion needed for debugging: versions, config and log files, etc.

  • Improve error messages so the users can solve their issues themselves.

  • Give users better automated diagnostics tools.

Based on further thinking and feedback, I add:

  • When a program notices a problem that may indicate a bug in it, it should collect the necessary information itself, automatically, in a way that the user just needs to send to the developers / support.

  • The primary goal should be to help people solve their own problems.

  • A secondary goal is to make the problem reproducible by the developers, or otherwise make it easy to fix bugs without access to the original system where the problem was manifested.

I've not written any code to help with this remote debugging, but it's something I will start experimenting with in the near future.

Further ideas welcome.

Posted Sun Jan 22 17:35:00 2017 Tags:

I participated in Nanowrimo in November, but I failed to actually finish the required 50,000 words during the month. Oh well. I plan on finishing the book eventually, anyway.

Furthermore, as an open source exhibitionist I thought I'd publish a chapter each month. This will put a bit of pressure on me to keep writing, and hopefully I'll get some nice feedback too.

The working title is "Hacker Noir". I've put the first chapter up on http://noir.liw.fi/.

Posted Sat Jan 7 22:46:00 2017 Tags:

I spent a few days in Cambridge for a minidebconf. This is a tiny version of the full annual Debconf. We had a couple of days for hacking, and another two days for talks.

I spent my hacking time on thinking about vmdebootstrap (my tool for generating disk images with an installed Debian), and came to the conclusion I need to atone my sins for writing such crappy code by rewriting it from scratch to be nicer to use. I gave a talk about this, too. The mailing list post has the important parts, and meetings-archive has a video.

I haven't started the rewrite, and it's not going to make it for stretch.

I also gave two other talks, on the early days of Linux, and Qvarn, the latter being what I do at work.

Thank you to ARM, for sponsoring the location, and the other sponsors for sponsoring food. These in-real-life meetings between developers are important for the productivity and social cohesion of Debian.

Posted Tue Nov 22 19:08:00 2016 Tags:

I have just released version 1.20 of Obnam, my backup program. It's been nine months since the previous release, and that's a long time: I've had an exciting year, and not entirely in a good way. Unfortuntely that's eaten up a lot of my free time and enthusiasm for my hobby projects.

See below for a snippet of NEWS, with a summary of the user-visible changes. A lot of the effort has gone into improving FORMAT GREEN ALBATROSS, but that isn't documented in the NEWS file.

I've received patches and actionable bug reports from a number of people, and I'm grateful for those. I try to credit them by name in the NEWS file.

Obnam NEWS

This file summarizes changes between releases of Obnam.

NOTE: Obnam has an EXPERIMENTAL repository format under development, called green-albatross-20160813. It is NOT meant for real use. It is likely to change in incompatible ways without warning. DO NOT USE it unless you're willing to lose your backup.

Version 1.20, released 2016-10-29

  • The format name for green-albatross is renamed to green-albatross-20160813 and will henceforth be renamed every time there's a change, to avoid confusing Lars because of backwards incompatibilities. When it reaches stability and the on-disk format is frozen, it'll be renamed back to a date-less version.

  • Those using the experimental green-albatross repository format will have to start over with fresh repositories. This release contains backwards incompatible changes that mean existing repositories no longer work. Sorry, but that's what experimental means.

  • A green-albatross change is that the "chunk index" data structure is no longer a single blob, and instead it's broken down into smaller objects. This avoids keeping all of the chunk indexes in memory at once, which should reduce memory use.

  • Remi Rampin started updating and continuing the French translation of the Obnam manual.

  • Lars Wirzenius changed the default so that Obnam reads random data when creating encryption key from /dev/urandom instead of /dev/random. The goal is to make it less likely that Obnam stops at the key generation stage on machines with little entropy. Set weak-random = no in your configuration to override this.

Minor changes:

  • Lars Wirzenius changed obnam forget so that if there is nothing to do, it doesn't even try to connect to the repository.

  • Lars Wirzenius added a chapter on participating in the Obnam project to the manual.

  • Lars Wirzenius changed --one-file-system to work for bind mounts. It only works for bind mounts that exist at the time when Obnam starts, however. Also, /proc/mounts must be an accurate list of mount points.

  • Lars Wirzenius added the gpg command line to the error message about gpg failing.

Bug fixes:

  • The manual and manual page used to claim you could break only the locks for one client. This was not true. The manuals has been fixed.

  • A whole bunch of typo fixes, from Andrea Gelmini.

  • Michel Alexandre Salim fixed a bug in the FUSE (obnam mount) plugin, which was a typo in a method name (get_clientgeneration_ids).

  • Lars Wirzenius fixed obnam restore to require a target set with --to. Jonathan Dowland reported the problem.

  • Lars Wirzenius fixed obnam list-errors so that it doesn't crash on error classes that only exist to make the exception hierarchy neater, such as EncryptionError. Bug reported by Rik Theys.

  • Ian Cambell fixed a bug in obnam kdirstat and its handling of FIFO sockets.

Posted Sat Oct 29 12:30:00 2016 Tags:

We needed a router and wifi access point in the office, and simultaneously both I and my co-worker Ivan needed such a thing at our respective homes. After some discussion, and after reading articles in Ars Technica about building PCs to act as routers, we decided to do just that.

  • The PC solution seem to offer better performance, but this is actually not a major reason for us.

  • We want to have systems we understand and can hack. A standard x86 PC running Debian sounds ideal to use.

  • Why not a cheap commercial router? They tend to be opaque and mysterious, and can't be managed with standard tooling such as Ansible. They may or may not have good security support. Also, they may or may not have sufficient functionality to be nice things, such as DNS for local machines, or the full power if iptables for firewalling.

  • Why not OpenWRT? Some models of commercial routers are supported by OpenWRT. Finding good hardware that is also supported by OpenWRT is a task in itself, and not the kind of task especially I like to do. Even if one goes this route, the environment isn't quite a standard Linux system, because of various hardware limitations. (OpenWRT is a worthy project, just not our preference.)

We got some hardware:

Component Model Cost
Barebone Qotom Q190G4, VGA, 2x USB 2.0, 134x126x36mm, fanless 130€
CPU Intel J1900, 2-2.4GHz quad-core -
NIC Intel WG82583, 4x 10/100/1000 -
Memory Crucial CT102464BF160B, 8GB DDR3L-1600 SODIMM 1.35V CL11 40€
SSD Kingston SSDNow mS200, 60GB mSATA 42€
WLAN AzureWave AW-NU706H, Ralink RT3070L, 300M 802.11b/g/n, half mPCIe 17€
mPCIe adapter Half to full mPCIe adapter 3€
Antennas 2x 2.4/5GHz 6dBi, RP-SMA, U.FL Cables 7€

These were bought at various online shops, including AliExpress and verkkokauppa.com.

After assembling the hardware, we installed Debian on them:

  • Connect the PC to a monitor (VGA) and keyboard (USB), as well as power.

  • I built a "factory image" to be put on the SSD, and a USB stick installer image, which includes the factory one. Write the installer image on a USB stick, boot off that, then copy the factory image to the SSD and reboot off the SSD.

  • The router now runs a very bare-bones, stripped-down Debian system, which runs a DHCP server on eth3 (marked LAN4 on the box). You can log as root on the console (no password), or via ssh, but for ssh you need to replace the /home/ansible/.ssh/authorized_keys file with one that contains only your public ssh key.

  • Connect a laptop to the Ethernet port marked LAN4, and get an IP address with DHCP.

  • Log in with ssh to ansible@10.0.0.4, and verify that sudo id works without password. Except you can't do this, unless you put in your ssh key in the authorized keys file above.

  • Git clone the ansible playbooks, adjust their parameters in minipc-router.yml as wanted, and run the playbook. Then reboot the router again.

  • You should now have wifi, routing (with NAT), and be generally speaking able to do networking.

There's a lot of limitations and problems:

  • There's no web UI for managing anything. If you're not comfortable doing sysadmin via ssh (with or without ansible), this isn't for you.

  • No IPv6. We didn't want to enable it yet, until we understand it better. You can, if you want to.

  • No real firewalling, but adjust roles/router/files/ferm.conf as you wish.

  • The router factory image is 4 GB in size, and our SSD is 60 GB. That's a lot of wasted space.

  • The router factory image embeds our public keys in the ansible user's authorized keys file for ssh. This is because we built this for ourselves first. If there's interest by others in using the images, we'll solve this.

  • Probably a lot of stupid things. Feel free to tell us what it is (bugs@liw.fi would be a good address for that).

If you'd like to use the images and Ansible playbooks, please do. We'd be happy to get feedback, bug reports, and patches. Send them to me (liw@liw.fi) or my ticketing system (bugs@liw.fi).

Posted Tue Oct 4 09:56:23 2016 Tags: