Feed for Planet Debian.
I don't really like any of the ticketing systems I've ever needed to use, whether they've been used as bug tracking systems, user support issue management systems, or something else. Some are not too bad. I currently rely most on debbugs and ikiwiki.
debbugs is the Debian bug tracking system. See https://www.debian.org/Bugs/ for an entry point. It's mostly mail based, with a read-only web interface. You report a bug by sending an email to submission address, and (preferably) include a few magic "pseudo-headers" at the top of your message body ot identify the package and version. There's tools to make this easier, but mostly it's just about sending an e-mail. All replies are via e-mails as well. Effectively, each bug becomes is own little dedicated mailing list.
This is important. A ticket, whether it is a bug report or a support request, is all about the discussion. "Hey I have this problem..." followed by "Have you tried..." and so forth. Anything that makes that discussion easier and faster to have is better.
It is my very strong opinion, and long experience, that the best way to have such a discussion is over e-mail. A lot of modern ticketing systems are web based. They might have an e-mail mode, perhaps read-only, but that's mostly an afterthought. It's a thing bolted onto the side of the system because people like me whinge otherwise.
I like e-mail for this for several reasons.
E-mail is push, not pull. I don't need to go look at a web page to be notified that something's happened.
E-mail requires no extra usernames and passwords to manage. I don't need to create a new account every time I encounter a new ticketing system instance.
E-mail makes it very easy to respond. I can just reply to a message. I don't need to go to a web site, log in, and find a reply button.
I already have archives of my e-mail, so referring to old messages (or finding them) is easy and quick. (Mutt, offlineimap, and notmuch is my particular set of choices. But I'm not locked to them, and you can use whatever you like, too.)
E-mail is a very rich format. Discussions are inherently threaded, and various character sets, languages, attachments, and other such things just work.
For these reasons, I strongly prefer ticketing systems in which e-mails are the primary form of discussions, and e-mail is a first class citizen. I don't mind if there's other ways to participate in the discussion, but if I have to use something else than e-mail, I tend not to be happy.
I use ikiwiki to provide a distributed, shared notebook on bugs. It's a bit cumbersome, and doesn't work well for discussions.
I think we can improve on the way debbugs works, however. I've been thinking about ticketing systems for Obnam (my backup program), since it gaining enough users that it's getting hard to keep track of discussions with just an e-mail client.
Here's what I want:
Obnam users do not need to care about there being a ticketing system. They report a problem by e-mailing the support mailing list, and they keep the list in cc when conducting the discussion. This is very similar to debbugs, with the distinction that there's no ticket numbers that must be kept in the replies.
The support staff (that's me, but hopefully others as well) have access to the ticketing system, which automatically sorts incoming messages into tickets. Tickets have sufficient metadata that it's possible to track which ones have been dealt with, or still need work, and perhaps other things. Each ticket contain a Maildir with all the e-mails belonging to that ticket.
The ticketing system is distributed. I need to be able to work on tickets offline, and to synchronise instances between different computers. Just like git. It's not enough to have an offline mode (e.g., queuing e-mails on my laptop for sending to debbugs when I'm back online).
There is a reasonably powerful search engine that can quickly find the relevant tickets, and messages, based on various criteria.
I will eventually have this. I'm not saying I'm working on this, since I don't have enough free time to do that, but there's a git repository, and some code, and it imports e-mails automatically now.
Some day there may even be a web interface.
(This has been a teaser.)
I have just tagged Obnam (my backup program) 1.8 in git, and built and uploaded Debian packages to code.liw.fi and Debian unstable. NEWS snippet below.
Version 1.8, released 2014-05-13
The error message has been improved for when setting metadata (owner, permission, and similar) of a restored file fails.
obnam force-locknow works even when the client running it is not in the client list.
- Joey Hess found a problem in
obnam restore: restored files would be created with quite liberal default permissions, which would be set to the backed-up permissions later. This could allow a snooper to read files they shouldn't be. This has been fixed now by using restrictive default permissions. A workaround for older versions is to create a directory, set its permissions to 0700, and restore to a subdirectory of that directory.
--helpoutput no longer shows the default value of any options. It was shown only for a few options anyway. The proper way to see the current settings is with the
--dump-configoption. The bug that was fixed that the generated manual page no longer contains values that are specific to the machine doing the generation, such as the hostname as the default value for
--client-name. Reported by SanskritFritz.
When a file was backed up, and later excluded with
--exclude, Obnam wouldn't remove it from the new backups. Now it does. Bug fixed by Anssi Hannula, though his patch got changed because it no longer applied.
When restoring extended attributes not in the user namespace (named like
user.foo) Obnam now ignores them, instead of trying to set them and crashing.
When restoring from a directory that is not a repository, the error message is now clearer.
Obnam would previously allow the backup root to be a symbolic link pointing at a directory. However, this only worked for backups. No other operations would work and would only see the symbolic link, not the directory it pointed at. Obnam now gives an error message even for the backup.
Obnam no longer excludes files named
none, if the setting
Thirty years ago I started to learn programming. To celebrate this, I'm doing a bit of programming as a sort of performance art. I will write a new program, from scratch, until it is ready for me to start using it for real. The program won't be finished, but it will be ready for my own production use. It'll be something I have wanted to have for a while, but I'm not saying beforehand what it will be. For me, the end result is interesting; for you, the interesting part is watching me be stupid and make funny mistakes.
The performance starts Friday, 18 April 2014, at 09:00 UTC. I apologise if this is an awkward time for you. No time is good for everyone, so I picked a time that is good for me.
Run the following command to see what the local time will be for you.
date --date '2014-04-18 09:00:00 UTC'
While I write this program, I will broadcast my terminal to the Internet for anyone to see. For instructions, see the http://liw.fi/distix/performance-art/ page.
There will be an IRC channel as well:
#distix on the OFTC network
irc.oftc.net). Feel free to join there if you want to provide real
time feedback (the laugh track).
When you have a big goal, do at least a little of it every day. Cory Doctorow writes books and stuff, and writes for at least twenty minutes every day. I write computer software, primarily Obnam, my backup program, and recently wrote the first rough draft of a manual for it, by writing at least a little every day. In about two months I got from nothng to something that is already useful to people.
I am now applying this to coding as well. Software development is famously an occupation that happens mostly in one's brain and where being in hack mode is crucial. Getting into hack mode takes time and a suitable, distraction-free environment.
I have found, however, that there are a lot of small, quick tasks that do not require a lot of concentration. Fixing wordings of error messages, making small, mechanical refactorings, confirming bugs by reproducing them and writing test cases to reproduce them, etc. I have foubd that if I've prepared for and planned such tasks properly, in the GTD planning phase, I can do such tasks even on trains and traun stations.
This is important. I commute to work and if I can spend the time I wait for a train, or on the train, productively, I can significant, real progress. But to achieve this I really do have to do the preparation beforehand. Th 9:46 train to work is much too noisy to do any real thinking in.
I have just released version 1.7.4 of Obnam, my backup program. Actually, the release date was yesterday, but I had trouble building the binaries.
Version 1.7.4, released 2014-03-31
The manual is now dual-licensed under GNU GPL v3 or later, and Creative Commons CC-BY-SA 4.0.
The 1.7.3 release never went out. Let's pretend it wasn't even tagged in git, and everyone will be happy.
Obnam FUSE got another bug fix from Valery Yundin, to fix a bug I introduced in 1.7. Reading big files via
obnam mountshould now work better.
Fix count of backed up files. It used to always count directories. Reported by Alberto Fuentes as Debian bug 742384.
obnam diff latestwould fail due to a programming error. Reported by Junyx.
I have just released version 1.7.2 of Obnam, my backup program. While I was releasing 1.7.1, I found a new problem, so I fixed that. Due to sillinesses in my CI/release system, I had to bump the version number to 1.7.2.
Version 1.7.2, released 2014-03-22
- Fix another bug in the FUSE plugin's file reading code, found during the release process of 1.7.2.
Version 1.7.1, released 2014-03-22
dump-repocommand now outputs JSON instead of YAML. The dependency on PyYAML is no longer.
Nemo Inis found a bug in the FUSE plugin (
obnam mount), where Obnam would return the wrong data when the program reading the file didn't read the whole file from the beginning in one read(2) system call.
The test suite now skips tests that require use of extended attributes in the
usernamespace. This should allow the test suite to be run on more build servers run by various distributions.
I have just released version 1.7 of Obnam, my backup program.
Version 1.7, released 2014-03-15
WARNING: This release has had fairly large parts of the internals re-written. There shouldn't be any externally visible changes due to that, but there is a chance of bugs. Be careful. Make a copy of your backup repository before upgrading, if you can.
convert5to6subcommand has been removed. If you need to convert from a pre-1.0 backup repository, and haven't done so yet, please use Obnam version 1.6.1 or earlier to do so.
backup-finishedhook is provided by the backup plugin, so that other plugins may do processing at the end of a backup, such as report the successful backup to a monitoring system. Patch by Enrico Tröger.
The FUSE plugin can now refresh its view, by having the user read the
.pidfile. Patch by Valery Yundin.
--always-restore-setuidto always restore setuid/setgid flags in permissions, even if the restore is not being run by
rootor the owner of the files (as recorded in the backup).
--exclude-fromallows exclusion patterns to be given in a separate file (one per line), instead of in a configuration file or on the command line. Patch by Enrico Tröger.
A start of a manual for Obnam. This will gain more content with new releases. The current versions is mainly an edited version of Lars's blog posts about backups, plus the Obnam tutorial from the Obnam homepage. See http://code.liw.fi/obnam/manual/ for rendered versions (PDF, HTML).
Most of the error messages Obnam produces now have a unique error code:
ERROR: R0B15DX: Cannot find requested generation for client havelockfor example. More error messages will gain error codes in future releases. The error codes are meant to be easy to search for, and will allow error messages to be translated in the future.
obnam-benchmarkprogram got rewritten so that it'll do something useful, but at the same time, it is no longer useful as a general tool. It is now expected to be run from the Obnam source tree (a cloned git repository), and isn't installed anymore.
The log file now includes information about the transfer overhead to the repository. Overhead is all the bytes that are not file content data: filenames, permission bits, extended attributes, etc, plus Obnam internal bookkeeping.
obnam verifynow shows progress both based on number of files and amount of data.
Obnam now doesn't remove chunks that are shared between clients. Previously, this would sometimes happen, because only the first client would correctly record itself as using a chunk. Now all clients do that.
Obnam now creates a
trustdb.gpgin the temporary GNUPGHOME it uses during encryption operations. From version 2.0.22 (or thereabouts),
gpginsists on having a
trustdb.gpgin the GNUPGHOME it uses.
When backing up a large file, and making a checkpoint generation in the middle of it, Obnam would say "continuing backup" after the checkpoint was finished, instead of saying the name of the file. This is now fixed.
obnamlib.Errorexception class has been replaced by the
obnamlib.ObnamErrorclass, which derives from the new
obnamlib.StructuredErrorclass. All new exceptions will need to be derived from
obnamlib.Errorin the future. Also, due to the way
StructuredErrorworks, it is now necessary to create a new exception class for each kind of error. This gives us unique the error codes mentioned above.
obnamlib.Repositoryclass is gone, and replaced with the
obnamlib.RepositoryInterfaceclass, which gets implemented for each repository format (there is only one, for now, but there will be more).
Russ Allbery has written up summaries of the research he's done in preparation of a Debian Technical Committee vote on default init system in Debian's next release. The mails are long, and well worth reading completely if the topic interests you at all, but I'd like to quote separately a few paragraphs that I found thoughtful, and inspiring, in the greater Debian context and not just for the init system discussion.
Here again, I think we have an opportunity for Debian to be more innovative and forward-looking in what we attempt to accomplish in the archive by adopting frameworks that let us incorporate the principles of least privilege and defense in depth into our standard daemon configurations.
And later, from the same mail:
I think we should make wise decisions about which areas we want to invest project effort. I dislike investing significant project effort in catch-up efforts that, when complete, merely get us back to where we would have been if we'd chosen a different solution. I don't think that's wise stewardship of project resources. I want to see Debian focus its efforts on places where we can make a real difference, where we can be leaders. That means adopting the best-of-breed existing solutions and building on top of them, not reinventing wheels and thereby starting from a trailing position.
It is not, in general, necessary to justify what you want to do in Debian. It doesn't matter if it's going to be used by thousands of people or two people. If you can do your work to the standards that we expect to be maintained across the archive, and without negative impact on Debian's other contributors, you can work on whatever you love. And, furthermore, we all support each other in our passions. Debian is built on a culture of deference to other people's work, and reasonable accomodation to the projects that other people want to work on.
Now, there is a fine balance here. Deference to other people's work does not mean a requirement to join their work. Reasonable accomodation does not mean that every Debian developer is required to test their software on non-Linux ports. The goal is that all of us should be empowered to work on the things we are passionate about, which implicitly includes being empowered to not work on the things that are not of interest to us. Therefore, for some efforts, and specifically for the non-Linux port efforts, the work is mostly born by the porters. They're expected to test, diagnose problems, and submit patches. The deference and reasonable accomodation that we expect of Debian contributors is to merge those patches when they are reasonable, to not undermine the porting effort when there is a reasonable approach that preserves it, and to be aware of the implications of their packaging for those ports in the broad sense (such as qualifying build-dependencies with [linux-any] where appropriate).
We do not expect Debian contributors to reject significant new upstream versions or significant new integrations because they will affect the non-Linux ports, or, for that matter, other projects in Debian. We do expect those changes to follow the standards that we expect of Debian as a whole, and that porting efforts will be treated with deference and reasonable accomodation.
I won't offer a comment on these quotes—I prefer to let them speak for themselves—but I will say I find these to be among the wisest things said within Debian all year.
Backups are fun and exciting! Restores are exciting, in a terrifying and stressful way. Fixing serious bugs is gratifying, in a depressing way.
I am glad to announce two software releases: Obnam version 1.6.1 and larch version 1.20131130. Obnam is my backup application, larch is a Python copy-on-write B-tree implementation that Obnam uses. I've uploaded the new versions to Debian unstable and to my own apt repository at [code.liw.fi] (also for wheezy). (I don't have the energy to upload to Debian backports: help is welcome there.)
These are mainly bug fix releases, and minor improvements. The larch release fixes serious problems, and everyone should be upgrading.
NEWS for larch:
Serious bug fixed: the "KeyError" crash for reference counts. This was false memory use optimisation, which triggered a rare bug in related code. Repeatable test case by Rob Kendrick, and helpful analysis by Itamar Turing-Trauring.
Serious bug fixed: another "node missing" bug. This crash was caused by a bug that overwrote on-disk reference count groups with zeroes. Repeatable test case by Rob Kendrick.
Fixes to fsck from Antoine Brenner.
NEWS for Obnam (combining versions 1.6 and 1.6.1, both released today):
Fix Debian package dependencies correctly.
Stop logging paramiko exceptions that get converted into another type of exception by the SFTP plugin in Obnam.
obnam-benchmarkcan now use an installed version of larch. Patch by Lars Kruse.
Obnam has been ported to FreeBSD by Itamar Turner-Trauring of HybridCluster.
Backup progress reporting now reports scanned file data, not just backed up file data. This will hopefully be less confusing to people.
list-toplevelscommands now obey a new option,
--key-details, to show the usernames attached to each public key. Patch by Lars Kruse.
--ssh-commandto set the command Obnam runs when invoking ssh. patch by Lars Kruse.
obnam clientscan now be used without being an existing client. Patch by Itamar Turner-Trauring.
--ssh-host-keys-checkto better specify how SSH host keys should be checked. Patch by Itamar Turner-Trauring.
"obnam list-toplevelsso it doesn't give an error when it's unable to read the per-client directory of another client, when encryption is used. Fix by Lars Kruse.
Fix the encryption plugin to give a better error message when it looks for client directories but fails to find them. Fix by Lars Kruse.
obnam list-toplevelsgot confused when the repository contained extra files, such as "lock" (left there by a previous, crashed Obnam run). It no longer does. Fix by Lars Kruse.
The SFTP plugin now handles another error code (EACCESS) when writing a file and the directory it should go into not existing. Patch by Armin Größlinger.
Obnam's manual page now explains about breaking long logical lines into multiple physical ones.
/~/path prefix in SFTP URLs works again, at least with sufficiently new versions of Paramiko (22.214.171.124 in Debian wheezy is OK). Reported by Lars Kruse.
The Nagios plugin to report errors in a way Nagios expects. Patch by Martijn Grendelman.
The Nagios plugin for Obnam now correctly handles the case where a backup repository for a client exists, but does not have a backup yet. Patch by Lars Kruse.
obnam lsnow handles trailing slashes in filename arguments. Reported by Biltong.
When restoring a backup, Obnam will now continue past errors, instead of aborting with the first one. Patch by Itamar Turner-Trauring.