Feed for Planet Debian.

At Debconf15 I gave a talk on topic of having backups be a default service on Debian machines. In that talk, I proposed that we create infrastructure to be included in a default Debian install to manage backups.

I still think this is a good idea, but over the past several months, I've had nearly no time at all to actually do this.

I'm afraid I have to say now that I won't be able to work on this in time for the stretch release. I would be very happy for others to do that, however.

The Debian wiki page https://wiki.debian.org/Backup acts as a central point of information for this. If you're interested in working on this, you can just do it.

Posted Sun Jan 31 10:50:17 2016 Tags:

I have just released version 1.19.1 of Obnam, the backup program. See the website at http://obnam.org for details on what the program does. The new version is available from git (see http://git.liw.fi) and as Debian packages from http://code.liw.fi/debian, and uploaded to Debian, and soon in unstable.

The NEWS file extract below gives the highlights of what's new in this version. Basically, it fixes a bug.

NOTE: Obnam has an EXPERIMENTAL repository format under development, called green-albatross. It is NOT meant for real use. It is likely to change in incompatible ways without warning. Do not use it unless you're willing to lose your backup.

Version 1.19.1, released 2016-01-30

Bug fix:

  • The check for paramiko version turned out not to work with versions 1.7.8 through 1.10.4, due to the paramiko.__version_info__ variable being missing. It's there in earlier and later versions. Lars Wirzenius added code to make the check work if the paramiko.__version__ variable is there. Jan Niggemann provided research and testing.
Posted Sat Jan 30 10:33:40 2016 Tags:

Survey URL: http://goo.gl/forms/hdoQZKjs80

I am doing an Obnam survey. The goal of this survey is to collect feedback from those who use Obnam, or have tried it, to guide the project in the future.

The survey will run until February 29, 2016.

Goals:

  • Get a feel for the number of people using Obnam, and how they are using it.
  • Find out why those who've tried Obnam have chosen to not use it.
  • Get input on roadmap planning: what things are wanted most, or least. What is important for Obnam users?
  • Get feedback on what's good or bad about Obnam in general.
  • Get feedback about the project in addition to the software.
  • Get a feel for whether it's worth pursuing business opportunities around Obnam.

All questions in this survey are optional. I do not collect personal information at all. The survey is implemented using Google Forms, and so Google probably collects some information; sorry. You don't need to log in to Google to fill in the survey, though, and I encourage you to use all the privacy protection tools you have.

I hope as many Obnam users as possible fill in the survey.

Posted Sun Jan 17 11:05:32 2016 Tags:

I have just released version 1.19 of Obnam, the backup program. See the website at http://obnam.org for details on what the program does. The new version is available from git (see http://git.liw.fi) and as Debian packages from http://code.liw.fi/debian, and uploaded to Debian, and soon in unstable.

The NEWS file extract below gives the highlights of what's new in this version.

NOTE: Obnam has an EXPERIMENTAL repository format under development, called green-albatross. It is NOT meant for real use. It is likely to change in incompatible ways without warning. Do not use it unless you're willing to lose your backup.

Version 1.19, released 2016-01-15

Bug fixes:

  • Backup no longer ignores a closed SSH connection. This means it won't keep trying to use it, forever. Instead, it crashes and terminates the backup.

  • The Paramiko SSH implementation, which Obnam uses, changed the interface to the prefetch method in its 1.16 version. Obnam can now deal with either variant of the method. Found and reported by Kyle Manna, who provided a patch that Lars Wirzenius rewrote to be backwards compatible to older versions of Paramiko.

Improvements to the manual:

  • The manual now has an appendix listing all Obnam errors, with codes and explanations. This will need to be updated manually from time to time.

  • The manual now has sections on turning on full debug logging and reporting problems.

Improvements to functionality:

  • The output of obnam generations now show time zone. Lars Wirzenius implemented based on suggestion by Limdi.
Posted Fri Jan 15 19:18:42 2016 Tags:

It was a shock to be back in Finland after five years. The country had gone bad.

I was born in Finland, and lived here all my life, until January, 2010. I then moved abroad for five years, with my then-girlfriend, now wife. We lived in New Zealand, Scotland, and England. In October 2014 we came back, because my next job happened to be in Finland.

When we came back, I found that the country had changed, or I had lost my rose-coloured glasses. The Finland I left was a place where equality and solidarity were assumed, though not universal. The country I came back to turned out to be racist, scared, and quickly abandoning everything that used to be examples of why the country is a good place to live. Some of the change has, possibly, only happened during 2015 and the refugee crisis in Europe, but the change started earlier.

Examples:

  • Racist behaviour and violence is commonplace. Those who do not look like the natives are insulted, threatened with physical violence, and sometimes experience it.
  • Political violence is becoming commonplace. Refugee centers are burnt down, sometimes with people inside. Government ministers are attacked, albeit by throwing a drink on them.
  • Politicians get away with saying and doing clearly racist things, up to and including celebrating neo-nazism.
  • The government spends a lot of effort trying to break any power the labour unions have, so that the employers get to do what they want. The whole stabilising system of negotating between government, employer, and labour organisations is being torn down, and labour is being stripped of any power.
  • Prominent cabinet ministers talk condescendingly about science, universities, and cut down educational funding. Universities should concentrate on being R&D factories for business, forget about advancing civilisation, or the humanities, or the arts. Unless it's the type of arts that becomes best-selling games.

Now, it is clear that Finland was never the kind of idyllic utopia that I thought it was, back when I was young. The older I get, the clearer it is that I was naive.

The change over five years is still quite remarkable. I can't ever go back home. Now, I am sometimes ashamed of being Finnish.

Posted Sun Dec 27 12:11:30 2015 Tags:

I just donated to Software Freedom Conservancy as a supporter. They do great, important work in GPL enforcement, and they need some money to continue it.

You can help, too: https://sfconservancy.org/supporter/.

See also, https://bits.debian.org/2015/12/conservancy-fundraising-campaign.html.

Posted Sun Dec 20 18:25:07 2015 Tags:

In September, The FUUG Foundation gave me a grant to buy some hardware for Obnam development. I used this money to buy a new desktop-ish machine, see below for details. It's sat in a corner, and I use it as a server: it's not normally connected to a monitor or keyboard. It runs Obnam benchmarks. Before this, I ran Obnam benchmarks and experiments on my laptop, or on BigV virtual servers donated by Bytemark.

The hardware

The parts:

  • CPU: Intel Core i7 4790K 4.0 GHz (4 cores, total of 8 hyperthreads)
  • RAM: Kingston HyperX Beast 32 GB
  • Mainboard: 46400 Asus H97M-Plus Intel H97 LGA 1150 micro-ATX
  • SSD: Samsung 850 EVO SSD 120 GB
  • HDD: 4 x WD Red 4 TB
  • PSU: Corsair CX750M ATX
  • Case: BitFenix Phenom micro-ATX

Not to brag, but it's a nice machine. Much more power than my 2012 era laptop.

The SSD is the system drive, the HDDs are for running Obnam benchmarks on. The HDDs are not RAIDed. Each drive is a PV for LVM2. All the data on those drives is scratch data: it's not valuable, and I do not care if it is lost. In fact, most of the data gets created and deleted during a benchmark run, and usually the disks are empty. The SSD contains the host operating system, and the virtual disks for all the virtual machines.

I assembled the machine myself, with the help of a friend, and installed Debian jessie on it. The Debian installation is pretty bare bones, just enough to run and manage a bunch of virtual machines using libvirt and ansible. All the actual work, including benchmarks, are run in virtual machines.

The benchmarks

The benchmarks are run by a kludge, called obbench, and the results are published at http://benchmark.obnam.org/. Here's a snapshot of the results so far:

date many files one big file
2015-09-28 2165.0 1381.9
2015-12-06 1461.6 384.0

In a bit over two months, I've made some significant progress, I think.

The two benchmarks that I currently run are:

  • A million tiny files, containing a single, random byte.
  • A single, 10 GB file.

These are two extreme cases of what a backup needs to deal with: either the file metadata, or its content. They incur different costs for a backup program. Thus, two benchmarks.

In both cases, the benchmark consists of an initial backup, a restore, and a second backup, without changes to the live data. The second backup is an extreme case of what backups usually do: most data usually doesn't change, so keeping that in mind for optimisation is important.

The above benchmarks are synthetic: they use data that's generated by a program (genbackupdata), so that they can be reproduced. Synthetic benchmarks are useful, especially for looking at particular aspects of a program's operation for optimisation. However, they do not necessarily reflect how a program behaves in actual use.

I also run, by hand, experiments with real data. I have a snapshot of our home file server and my laptop on the benchmark machine. The snapshots are static, and do not get updated. I experiment by running Obnam backups manually, the initial full backup and a no-change incremental one. In early October, I couldn't finish the initial full backups. They took too long, more than a week. Now I can finish them in about a day. This remarkable change is not evident from the synthetic benchmarks.

In numbers: 572986 files in the live data, containing 4.5 TiB. Initial backup, about 18.5 hours. Incremental backup, 4m13s. This is from a local disk to a local disk.

In addition to these, I've run numerous experiments on the new machine. These would have been much less easy to run on my laptop, and so I probably wouldn't have. Running benchmarks was always painful on my laptop, since it does not have the necessary disk space, and I'd really rather like to use it for other things.

The results

Thanks to the benchmarks and experiments I've been able to take the in-development version of Obnam from being quite impractical for real use to being in experimental use for real data. I now use the new version as my primary backup of my laptop, with two secondary backups (with the old Obnam version, and rsync) in parallel. This would not have happened this year without the extra hardware.

In addition to the Obnam work, I've used the new machine to develop a test suite for vmdebootstrap.

My actual development still happens on my laptop, except for things that are heavy enough to be slow on the laptop. I've made sure I can do most development purely on my laptop, while offline, including running a CI system, and testing things on two architecture and three releases of Debian. I do not want my development to be dependent on incidental things such as network access, unless I'm doing things that by their nature depend on the network, such as publishing changes or releases.

Posted Sun Dec 20 16:57:04 2015 Tags:

I have just released version 1.18 of Obnam, my backup program. See the website at http://obnam.org for details on what the program does. The new version is available from git (see http://git.liw.fi) and as Debian packages from http://code.liw.fi/debian, and uploaded to Debian, and soon in unstable.

The NEWS file extract below gives the highlights of what's new in this version.

Version 1.18, released 2015-11-04

Bug fixes:

  • William Boughton fixed parsing for sftp URLs with IPv6 addresses. Previously, sftp://[::1] would be interpreted by Obnam as an address [ followed by the port :1], but now it is correctly interpreted as the adddress ::1 and no explicit port.

  • Ian Campbell fixed a bug in the kdirstat plugin, improving the handling of unknown file types.

  • Lars Wirzenius changed the scan_tree code to not be recursive, to avoid problems with directory trees that are deeper than Python's call stack limit allows.

Minor changes:

  • Lars Wirzenius added support for a multiline progress message during backup. Version 0.24 or newer of ttystatus is needed for this, but Obnam will work with an older version by displaying the same single-line progress message as before.

  • Ben Boeckel added the --gnupghome setting so that Obnam can be configured to use a separate GnuPG (gpg) configuration directory.

  • Henri Sivonen improved the compression code to not compress if the result would be larger.

Posted Thu Nov 5 11:01:59 2015 Tags:

I disagree with pretty much everything Norbert Preining says in his blog post about Sarah Sharp in response to Sharp's blog post Closing a door about leaving Linux kernel development. I think Preining mis-characterises what happens on the Linux kernel mailing list, and in Debian, in free software development in general, and what Sarah Sharp has said and done.

I do not wish to participate in yet another discussion on the Debian Code of Conduct, respect, appropriate behaviour, or whether these things result in worse software. I won't spend any further of my time on debating, point-by-point, the arguments and straw-men Preining makes. Those discussions have happened often enough, without any effect on Preining and those who think like him. Likewise, despite the many times he makes the arguments, I continue to think he is wrong about everything on this topic.

I do feel it is important to make it clear to the people reading Planet Debian, where both Preining's and my blogs are published, that his opinions are not mainstream in the Debian project, and that despite what he says, Debian development continues to be fun.

Posted Thu Oct 8 16:56:16 2015 Tags:

I'm very pleased to say that the FUUG foundation in Finland has awarded me a grant to buy some hardware to help development of Obnam, by backup program. The announcement has more details in Finnish.

Posted Thu Sep 24 19:23:17 2015 Tags: