TL;DR

Obnam, a backup program, first ALPHA release, called version 0.10, at http://braawi.org/obnam/. Please try it out and tell me what you think.

Obnam 0.10: A New Hope

A long time ago, at a computer far, far away, I realized that I hate making backups.

When the galaxy was still young, I made backups on 5.25 inch floppies. I did not have much data, only the programs I had written myself, so they all fit on one floppy. I had another floppy for backups. I would make a backup every time I felt I had done something I wanted to keep.

Later, as suns grew red and then white, I generated more data, and got a lot of data from elsewhere, and then got a hard disk. I made backups of that on 3.5 inch floppies, and needed a whole pile of them. It took an entire evening to make a full backup. I tried many programs for making incremental backups, but they all got confused, or made me confused, or couldn't handle the inevitable failure of a floppy.

For a while I had a tape drive. It failed more often than it worked. When it worked, it still took an evening to make a backup.

Then CDs happened, and later DVDs. They were much more reliable than floppies, and you needed many fewer of them. As long as everything fit on one disk, I was happy, but that didn't last too long.

Eventually I ended up backing up to a USB disk. Then two of them. It ended up as a pile of hard disks. After a couple of years I upgraded the disks to double the size, and halved the pile, but it was still too many.

I would still occasionally spend an evening doing backups. That's an evening I could spend on something interesting. Since it isn't fun, it also happens too rarely, and when I needed my backups, they're too old. Screw that.

There had to be an easier way. In 2004 I figured out what that was: a combination of fast networks (1 Mbit/s!), online storage (gigabytes of it!), and a sophisticated backup application would make backups easy, fast, automatic. I would never, ever have to pay any attention to them, they would just be there, ready for when I needed them.

I started thinking about ways to implement this, and sketched out some code, and experimented a bit. However, I was distracted by stupid things like having to make a living, and making several false starts in development. While I was doing my thing, on my own, the world moved on, and now everyone's dog are doing backups online. There are, I think, about 12765 applications for making online backups.

That's a good thing. At least one of them will be good.

However, I am now finally getting ready to start using my own application for real. In case anyone else is interested, I am making a release. This is the first alpha release of the fourth complete rewrite from scratch.

I have not made detailed comparisons between Obnam and all the other programs for online backups. I don't intend to start: there's too many of them. Here's the core of my vision (not all of it yet implemented):

  • The backup server can be anything I can access via ssh. No special software on the server end.
  • Entirely free software. I do not want to rely on a server that uses proprietary software to provide me disk space.
  • Everything is encrypted locally, before sent to the server.
  • Fast enough to saturate the network link.
  • Non-intrusive enough to run all the time in the backgroun on my laptop.
  • Handles all corner cases (empty files, device files, etc).
  • Handles duplicate files, and moving files, efficiently.

Interested? Please try it out. I would very much welcome any feedback you may have, preferably via e-mail. Please be gentle.

Highlights

  • Version: 0.10
  • Development status: ALPHA
  • License: GNU GPL 3 (or later)
  • Nota Bene: do not assume this works, and do not assume the on-disk data structures won't change in incompatible ways.
  • On-disk data structure uses the same kind of copy-on-write B-tree as btrfs, designed by Odeh Rodah. It's an independent implementation in pure Python.
  • Test suite has almost 100% coverage (12 excluded statements).

Links

Dependencies

Installation on Debian (squeeze or later)

Add the following line to your sources.list file:

deb http://code.liw.fi/debian squeeze main

Then:

# apt-get update
# apt-get install obnam

Installation on other systems

See above for where to download tarballs for Obnam and its dependencies. Unpack them, and install using the setup.py script in each. (I hope that works; I have only tested this on Debian.)

What you get

  • Backups to local disk.
    • Each backup generation looks like a full snapshot, but only new files, and modified files are backed up.
    • Data de-duplication: if a chunk of data is already backed up, it won't be backed up again, even if it is in a different file, or an older generation. Renamed files, log files, etc, benefit greatly from this.
  • List generations.
  • List contents of generations.
  • Restore a full generation, or specific files.
  • Remove generations.

Major missing features

This is an alpha version, so obviously there are things still missing. I have so

  • Encryption. All data will be encrypted before it is put into the backup store. This includes all metadata, such as filenames.
  • Online backups: the code for sftp is there, but not currently working.
    • If someone wants to write a plugin to support another protocol, I would very much welcome that. However, sftp is what I'll be using myself.
  • Extreme speed. (It's slow, for now.)
  • ACLs and xattr support. Plus possibly other metadata (is there any?).
  • Stable on-disk data structures.
    • I'm nowhere near finished optimizing, and many optimizations will require changing the way things are stored on disk. I am not yet ready to commit supporting the on-disk data structures indefinitely.

Help!

Help! I am trapped inside a ... no, wait, that's the wrong announcement.

I would very much welcome help developing Obnam. If you'd like to help, here is a start of a list of things that need doing:

  • Use it. Does it work for you? If you see anything that could be improved, tell me, and let's see if it can be fixed.
  • Review the code. I've had my head buried in this code for so many years now that it is likely I do not see its glaring problems. Have a look and tell me!
  • Make improvements. Send me fixes for bugs, typos, missing documentation, anything. Or update the Obnam home page (it's a wiki).
  • Take charge. Would you like to be in charge of running benchmarks, making releases, managing bugs, documentation, or anything else? Step up and say so!

If you'd rather help with something else, I would welcome that, too. The list above is just for inspiration.

Contact info

If you have any feedback to me, please e-mail (liw@liw.fi), or see my contact page for other ways. Commenting on this blog post is also fine.