Recent changes to this wiki:
fix markup
diff --git a/posts/mix-half-albanian.mdwn b/posts/mix-half-albanian.mdwn index e5f5b12..d838ce5 100644 --- a/posts/mix-half-albanian.mdwn +++ b/posts/mix-half-albanian.mdwn @@ -1,4 +1,4 @@ -[[!meta title='"MIX 1/2 Albanian" Kickstater']] +[[!meta title="MIX 1/2 Albanian Kickstater"]] [[!tag personal]] Soile, my partner and love, is a film-maker. She is funding
Post about Soile's Kickstarter
diff --git a/posts/mix-half-albanian.mdwn b/posts/mix-half-albanian.mdwn new file mode 100644 index 0000000..e5f5b12 --- /dev/null +++ b/posts/mix-half-albanian.mdwn @@ -0,0 +1,9 @@ +[[!meta title='"MIX 1/2 Albanian" Kickstater']] +[[!tag personal]] + +Soile, my partner and love, is a film-maker. She is funding +her first feature length documentary via crowd-funding, and has just opened +the [Kickstarter +page](http://www.kickstarter.com/projects/docstory/mix-1-2-albanian). +Have a look and see if you like the project and if you do, any donation +is most welcome.
Added a comment: Microsoft snprintf
diff --git a/posts/strncpy/comment_11_2330e83c82d1c6d07d30fbe2909a8d1d._comment b/posts/strncpy/comment_11_2330e83c82d1c6d07d30fbe2909a8d1d._comment new file mode 100644 index 0000000..42fc2eb --- /dev/null +++ b/posts/strncpy/comment_11_2330e83c82d1c6d07d30fbe2909a8d1d._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Kenny" + ip="216.68.89.80" + subject="Microsoft snprintf" + date="2012-02-01T19:40:37Z" + content=""" +Except you forgot about that fact that Windows's (MS CRT) version of snprintf **DOES NOT** Null terminate if the string will be equal or greater in length than n - http://msdn.microsoft.com/en-us/library/2ts7cx93(v=vs.80).aspx +"""]]
Added a comment
diff --git a/posts/luxury/comment_2_2fd49588c43101ed7aa1afc9d0a55318._comment b/posts/luxury/comment_2_2fd49588c43101ed7aa1afc9d0a55318._comment new file mode 100644 index 0000000..066a84d --- /dev/null +++ b/posts/luxury/comment_2_2fd49588c43101ed7aa1afc9d0a55318._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawn7PWsVDDNfAZte8rEUXUwSbKzlufOi9OQ" + nickname="Jeff" + subject="comment 2" + date="2012-01-20T22:06:15Z" + content=""" +For gender balance, have a Gertrude Stein (or Sophie Tucker or Mae West) quote: + \"I've been rich and I've been poor. Believe me, honey, rich is better.\" +Good luck at FOSDEM 2012! +"""]]
remove now-useless local.css
diff --git a/local.css b/local.css deleted file mode 100644 index 79a8bf6..0000000 --- a/local.css +++ /dev/null @@ -1 +0,0 @@ -@import "http://files.liw.fi/ikiwiki-theme/local.css";
add my name to the front page
diff --git a/index.mdwn b/index.mdwn index fc3c8d2..992023b 100644 --- a/index.mdwn +++ b/index.mdwn @@ -3,4 +3,6 @@ introduction. See the [[archive|posts]] page for all posts, and [[comments]] for a feed of comments only. (There is an [[english language feed|englishfeed]] if you don't want to see Finnish.) +Lars Wirzenius + [[!inline pages="posts/* and !posts/*/* and !tagged(draft)"]]
Comment moderation
diff --git a/posts/cafe-coding/comment_1_566e54c858b5922884fcddb4c34f6ff8._comment b/posts/cafe-coding/comment_1_566e54c858b5922884fcddb4c34f6ff8._comment new file mode 100644 index 0000000..09cd7c8 --- /dev/null +++ b/posts/cafe-coding/comment_1_566e54c858b5922884fcddb4c34f6ff8._comment @@ -0,0 +1,18 @@ +[[!comment format=mdwn + ip="212.21.251.92" + claimedauthor="Iñigo" + url="http://poisonbit.wordpress.com" + subject="Courious" + date="2011-12-23T09:58:17Z" + content=""" +It's courious, I think such scenario does not work well for my concentration. + +I prefer when I can work from home, sometimes I realize I've been many hours in absolute silence working without distraction, in the morning, with not more noise than birds and my keyboard. + +Other thing about sound Vs concentration I've realized, it's about sound origin. + +I work better if \"sound\" (i.e. a playlist of music) comes from the front of my head, than if it comes from behind me. When sound (music, radio, tv, anything) comes behind my head and my eyes are working in front, it distracts me more often than when it comes from the front. + +I hope I could concentrate in a cafe... maybe just spain is different :) + +"""]]
blog about cafe coding
diff --git a/posts/cafe-coding.mdwn b/posts/cafe-coding.mdwn new file mode 100644 index 0000000..eb3e3b1 --- /dev/null +++ b/posts/cafe-coding.mdwn @@ -0,0 +1,22 @@ +[[!meta title="Cafe coding and other productivity hacks"]] +[[!tag productivity programming]] + +I've come to realize that I quite like coding in cafes, from time to time, +and it can be very productive for me. +This is strange, since I'm usually very sensitive to noise when I work. +Almost any unpredictable sounds at the office distract me, +which is why I often listen to music. +However, in a cafe, I don't need that, I just automatically block out +everything but my laptop. + +At the office, everything that happens around +me might concern or interest me. +In a cafe, nobody says anything that I'm interested in. + +Another distraction that cafes lack is the Internet. +Even when there's Internet access, +I tend to turn my laptop's wifi off so that I have a longer battery life. +I might occasionally check mail or IRC, but only for a few minutes at a time. +The rest of the time I crank the widget and write code or take care of +other things from my next actions list. +
Added a comment: This is not about Debian
diff --git a/posts/like-buttons-in-about/comment_5_3b250141f468eba61d948b243468344c._comment b/posts/like-buttons-in-about/comment_5_3b250141f468eba61d948b243468344c._comment new file mode 100644 index 0000000..0eddd7f --- /dev/null +++ b/posts/like-buttons-in-about/comment_5_3b250141f468eba61d948b243468344c._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="This is not about Debian" + date="2011-12-20T19:55:26Z" + content=""" +This is about the people who make the software getting feedback. Debian already has popcon, which is an inadequate feedback mechanism **for Debian**. Not upstream. + +popcon also doesn't provide an answer to the question \"are there people who like this software\" as opposed to \"how many people happen to have this software installed\". +"""]]
Comment moderation
diff --git a/posts/like-buttons-in-about/comment_4_d0040d4a8ff0c21fc72c38233ff2b059._comment b/posts/like-buttons-in-about/comment_4_d0040d4a8ff0c21fc72c38233ff2b059._comment new file mode 100644 index 0000000..b5759f2 --- /dev/null +++ b/posts/like-buttons-in-about/comment_4_d0040d4a8ff0c21fc72c38233ff2b059._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + ip="128.178.247.237" + claimedauthor="DHardy" + subject="comment 4" + date="2011-12-20T16:45:52Z" + content=""" +Lars, how many users really use shared systems so much they're going to start rating packages? The host/user distinction seems mostly irrelevant to me. + +I don't see much use of this in debian; popcon already says how much packages are used, project websites allow some feedback, and bug reports allow some feedback (though I don't know how many users report wishlist items — maybe only consistent users). + +What I'd like to see is more are library rating systems (how many projects, including private closed-source ones, use a library?). +"""]]
Added a comment: On feedback comments and popcon
diff --git a/posts/like-buttons-in-about/comment_3_695e24a1a1c56b0964a570f5019bfd79._comment b/posts/like-buttons-in-about/comment_3_695e24a1a1c56b0964a570f5019bfd79._comment new file mode 100644 index 0000000..dc4bdfb --- /dev/null +++ b/posts/like-buttons-in-about/comment_3_695e24a1a1c56b0964a570f5019bfd79._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="On feedback comments and popcon" + date="2011-12-19T18:36:58Z" + content=""" +Elessar, a simple like/dislike is, well, simple. Having to read a lot of textual feed back would eat up a fair bit of development time. Anyone who wants to provide thoughtful feedback already has the addresses to do so in the About dialog. + +Bob, the Debian popcon is per-host, not per-user, which is an important distinction. It also gathers information to Debian rather than sending it directly to the developers of the software. This is also an important distinction. There's no need for Debian to act as a middleman here. In my opinion, of course. +"""]]
Comment moderation
diff --git a/posts/like-buttons-in-about/comment_2_558829a44b9b13a94e5e7771bc4cc68d._comment b/posts/like-buttons-in-about/comment_2_558829a44b9b13a94e5e7771bc4cc68d._comment new file mode 100644 index 0000000..c0a8309 --- /dev/null +++ b/posts/like-buttons-in-about/comment_2_558829a44b9b13a94e5e7771bc4cc68d._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + ip="151.33.1.152" + claimedauthor="Bob" + url="http://madbob.org/" + subject="Popularity Contest" + date="2011-12-19T16:45:50Z" + content=""" +It already exists Debian Popularity Contest ( http://popcon.debian.org/ ) to measure diffusion of a given package. Probably that same tool can be extended with an interface to the user, a \"Like\" button, an area for comments and so on. It would be better to add that kind of informations in the graphical package manager, too. +"""]]
Comment moderation
diff --git a/posts/like-buttons-in-about/comment_1_39580217c2b793815c86696a98de205c._comment b/posts/like-buttons-in-about/comment_1_39580217c2b793815c86696a98de205c._comment new file mode 100644 index 0000000..ba0b9ea --- /dev/null +++ b/posts/like-buttons-in-about/comment_1_39580217c2b793815c86696a98de205c._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + ip="62.210.161.232" + claimedauthor="Elessar" + url="http://tanguy.ortolo.eu/" + subject="With comment" + date="2011-12-19T14:44:27Z" + content=""" +Well, with a comment area, then. I doubt that the simple information that a user likes or does not like a piece of software would be useful without a little more detail. :-) +"""]]
post about idea
diff --git a/posts/like-buttons-in-about.mdwn b/posts/like-buttons-in-about.mdwn new file mode 100644 index 0000000..14cc393 --- /dev/null +++ b/posts/like-buttons-in-about.mdwn @@ -0,0 +1,12 @@ +[[!meta title="Like/dislike buttons in free software"]] +[[!tag idea]] + +It's a perennial problem that authors of free software get little +feedback from users. Here's an idea: add "I like this program" and +"I don't like this program" buttons to Help/About dialogs. +They would send some kind of anonymous signal to the author, +probably by making an HTTP call to some server somewhere. + +This is not a "call home" functionality, which is questionable in +many ways. It's completely opt-in, and it's pretty clear to the +user what's going on.
fix link
diff --git a/posts/obnam-0.24.mdwn b/posts/obnam-0.24.mdwn index 190d087..0c743cd 100644 --- a/posts/obnam-0.24.mdwn +++ b/posts/obnam-0.24.mdwn @@ -1,7 +1,7 @@ [[!meta title="Obnam 0.24 released (backup software)"]] [[!tag obnam]] -I've released version 0.24 of [Obnam](http://braawi.org/obnam/), +I've released version 0.24 of [Obnam](http://liw.fi/obnam/), my backup application. USER VISIBLE CHANGES
post about obnam 0.24
diff --git a/posts/obnam-0.24.mdwn b/posts/obnam-0.24.mdwn new file mode 100644 index 0000000..190d087 --- /dev/null +++ b/posts/obnam-0.24.mdwn @@ -0,0 +1,47 @@ +[[!meta title="Obnam 0.24 released (backup software)"]] +[[!tag obnam]] + +I've released version 0.24 of [Obnam](http://braawi.org/obnam/), +my backup application. + +USER VISIBLE CHANGES + +* The way file timestamps (modification and access times) have changed, + to fix inaccuracies introduced by the old way. Times are now stored + as two integers giving full seconds and nanoseconds past the full + second, instead of the weird earlier system that was imposed by Python's + use of floating point for the timestamps. This causes the repository + format version to be bumped, resulting in a need to start over with an + empty repository. +* Extended file attributes are now backed up from and restored to local + filesystems. They are neither backed up, nor restored for live data + accessed over SFTP. +* If the `--exclude` regular expression is wrong, Obnam now gives an + error message and then ignores the regexp, rather than crashing. +* There is now a compression plugin, enabled with `--compress-with=gzip`. +* De-duplication mode can now be chosen by the user: the new + `--deduplicate` setting can be one of `never` (fast, but uses more space); + `verify` (slow, but handles hash collisions gracefully); and + `fatalist` (fast, but lossy, if there is a hash collision). `fatalist` + is the default mode. +* Restores now obey the `--dry-run` option. Thanks to Peter Palfreder for + the bug report. +* New option `--verify-randomly` allows you to check only a part of the + backup, instead of everything. +* Verify now has some progress reporting. +* Forget is now much faster. +* Forget now has progress reporting. It is not fast enough to do without, + sorry. +* Backup now removes any checkpoint generations it created during a backup + run, if it succeeds without errors. + +BUG FIXES: + +* Now works with a repository on sshfs. Thanks to Dafydd Harries for + reporting the problem. +* Now depends on a newer version of the larch library, fixing a problem + when the Obnam default node size changes and an existing repository + has a different size. +* User and group names for sftp live data are no longer queried from the + local system. Instead, they're marked as unknown. +
creating tag page tag/effi
diff --git a/tag/effi.mdwn b/tag/effi.mdwn new file mode 100644 index 0000000..4e07918 --- /dev/null +++ b/tag/effi.mdwn @@ -0,0 +1,4 @@ +[[!meta title="pages tagged effi"]] + +[[!inline pages="tagged(effi)" actions="no" archive="yes" +feedshow=10]]
new post
diff --git a/posts/effi-lakanakampanja.mdwn b/posts/effi-lakanakampanja.mdwn new file mode 100644 index 0000000..726ba64 --- /dev/null +++ b/posts/effi-lakanakampanja.mdwn @@ -0,0 +1,21 @@ +[[!meta title="Effin lakanakampanja"]] +[[!tag in-finnish finnish-politics effi]] + +[Effi](http://effi.org/) järjesti [kampanjan kasettimaksua +vastaan](http://riisto.effi.org/). Tästä nousi poru. + +On se vaan niin, että aina kun tekee jotain, niin kritisoidaan ja +jos tekee jotain radikaalia kritisoidaan paljon. Jos on hiljaa +nurkassa niin ei kritisoida, harvoin edes tekemättömyydestä. + +Minusta kampanja on onnistunut, mutta oleellisinta on se, että +Effi tekee jotain näkyvää. Hiljaisia asiantuntijalausuntoja +virkamiehille kirjoittamalla ei saa aikaan keskustelua ja ilman +julkista keskustelua asioista päätetään lobbaajien pään mukaan. + +Effin ajamat asiat ovat tärkeitä: sananvapaus, yksityisyys, muutkin +perusoikeudet, digitaalisessa maailmassa. On se nyt kumma, jos +niitä ei saa puolustaa äänekkäästi. + +(Kiisto: olen entinen hallituksen jäsen ja nykyinen epäaktiivinen +rivijäsen. En ole edes aktivistilistalla.)
Comment moderation
diff --git a/posts/backuphackers/comment_4_017d16e9a274ce9d867897776976c1af._comment b/posts/backuphackers/comment_4_017d16e9a274ce9d867897776976c1af._comment new file mode 100644 index 0000000..1f3c51a --- /dev/null +++ b/posts/backuphackers/comment_4_017d16e9a274ce9d867897776976c1af._comment @@ -0,0 +1,13 @@ +[[!comment format=mdwn + ip="69.168.48.47" + claimedauthor="STDOUBT" + subject="comment 4" + date="2011-12-05T17:32:31Z" + content=""" +Firstly, I wasn't trying to be offensive, just realistic. Further, there are already tons of reliable backup solutions out there which are mature and stable and Free. Creating a new one only seems useful as an excersize for a student programmer else it smacks of the 'not invented here syndrome'. + +As to the distinction between a file synchronizer and a backup tool... I think you're arguing semantics there. If \"synchronizing\" files between computers or between hard drives isn't a form of backup creation, well, I guess I'm just out to lunch huh? Unison is mature and very stable. rsync is one of its components. It's perfectly suitable for my own needs and actually does a perfect job in most use cases. + +There is nothing at all wrong with creating tools for oneself. All good developers \"scratch their own itch\". But expecting to create something which would replace time-honored tools seems like a stretch unless you're talking about inventing a new data transfer protocol ala bittorrent. +In any case, good luck to you. +"""]]
Added a comment: *sigh*
diff --git a/posts/backuphackers/comment_3_5e805812a9eb52eab18f6d7de79447f5._comment b/posts/backuphackers/comment_3_5e805812a9eb52eab18f6d7de79447f5._comment new file mode 100644 index 0000000..03c773c --- /dev/null +++ b/posts/backuphackers/comment_3_5e805812a9eb52eab18f6d7de79447f5._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="*sigh*" + date="2011-12-05T05:19:30Z" + content=""" +STDOUBT, Unison is a file synchronization tool. It's not a backup tool, though I'm sure you can build a backup system on top of it, just like you can on any tool that make a copy of a file. + +I'm sorry your chosen tool isn't as good as you'd like. That's not a reason for you to be offensive towards other people and the tools they're making for themselves. +"""]]
Comment moderation
diff --git a/posts/backuphackers/comment_1_af016e407229b684ccf39aeeb61f7581._comment b/posts/backuphackers/comment_1_af016e407229b684ccf39aeeb61f7581._comment new file mode 100644 index 0000000..1cfe038 --- /dev/null +++ b/posts/backuphackers/comment_1_af016e407229b684ccf39aeeb61f7581._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + ip="86.30.246.241" + claimedauthor="rjc" + subject="tester" + date="2011-12-04T22:25:16Z" + content=""" +Hi Lars, + +I'm more than happy to run tests - will email you directly. + +rjc +"""]] diff --git a/posts/backuphackers/comment_2_ccf377a74f810b43aa26772f0001592e._comment b/posts/backuphackers/comment_2_ccf377a74f810b43aa26772f0001592e._comment new file mode 100644 index 0000000..c661083 --- /dev/null +++ b/posts/backuphackers/comment_2_ccf377a74f810b43aa26772f0001592e._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + ip="69.168.48.47" + claimedauthor="STDOUBT" + subject="How about help to improve what's out there?" + date="2011-12-05T04:08:33Z" + content=""" +Backups are important, right? + +There is no way I'm going to trust my bups to some kiddie project. I have always relied upon Unison. +Recently, I heard it doesn't officially, technically support UTF-8/Unicode. Why not band together to fix that? +That way your group would get some real recognition without re-inventing the wheel... a wheel that would in all likelyhood go nowhere. Think about it. + +https://alliance.seas.upenn.edu/~bcpierce/wiki/index.php?n=Main.UnisonFAQCharacterEncoding + +Anyone who pulls this off would get some serious recognition. + +"""]]
new blog post
diff --git a/posts/backuphackers.mdwn b/posts/backuphackers.mdwn new file mode 100644 index 0000000..3f09e35 --- /dev/null +++ b/posts/backuphackers.mdwn @@ -0,0 +1,11 @@ +[[!meta title="#backuphackers and plea for help"]] +[[!tag obnam backups]] + +The other day I was talking with someone else who is making backup +software (`bup`, to be precise). That was nice, so I thought I'd +create an IRC channel for our kind of weirdos. If you like making +backup software, please join `#backuphackers` on irc.oftc.net. + +Also, I think there's a need for neutral third parties to run +validation tests and benchmarks on backup software. Anyone interested +in doing that?
Added a comment: obnam-benchmark to the rescue
diff --git a/posts/obnam-localfs-speed/comment_2_f8a4bb1e880d0099aaf308b5b2e35de7._comment b/posts/obnam-localfs-speed/comment_2_f8a4bb1e880d0099aaf308b5b2e35de7._comment new file mode 100644 index 0000000..91e03b4 --- /dev/null +++ b/posts/obnam-localfs-speed/comment_2_f8a4bb1e880d0099aaf308b5b2e35de7._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="obnam-benchmark to the rescue" + date="2011-12-01T08:01:54Z" + content=""" +I haven't studied the benchmark profiling data for the latest release yet, so I don't know what the current time-wasters are. It's easy to run a benchmark, though: see the `obnam-benchmark` script in the source directory. It runs everything using the Python profiler, and you can examine the resulting files to see for yourself. Or you can run `obnam` with the environment variable `OBNAM_PROFILE=foo.prof` and it will run the code under profiling and write the profile to `foo.prof`. + +It's certainly possible to get more speed by rewriting things in C, and PyPy might be useful too, but for now, I'm interested in algorithmic improvements rather than rewrites. I'm not going to do any big rewrites for 1.0. PyPy isn't packaged for Debian, so it's not really a current option for me. +"""]]
Comment moderation
diff --git a/posts/write-intent-bitmaps/comment_1_a959d4cc78d9b873ac3a29e2af732580._comment b/posts/write-intent-bitmaps/comment_1_a959d4cc78d9b873ac3a29e2af732580._comment new file mode 100644 index 0000000..86889db --- /dev/null +++ b/posts/write-intent-bitmaps/comment_1_a959d4cc78d9b873ac3a29e2af732580._comment @@ -0,0 +1,7 @@ +[[!comment format=mdwn + ip="204.15.85.60" + subject="Did you adjust the resolution of the bitmap?" + date="2011-12-01T06:42:59Z" + content=""" +Note that you can adjust the granularity of the bitmap, which can help performance substantially. Basically you'd trade a little less rebuild speed for a lower resolution bitmap and hence less performance impact on an internal bitmap. If you're running arrays in the terabytes, it may be worthwhile to only keep track of changes in 256M chunks, for example, which might mean a rebuild takes 5-10 minutes since you're replaying larger chunks, but much better than a full rebuild. +"""]]
Comment moderation
diff --git a/posts/obnam-localfs-speed/comment_1_9b98f43ed08cef5fa68d1f0f1946c3aa._comment b/posts/obnam-localfs-speed/comment_1_9b98f43ed08cef5fa68d1f0f1946c3aa._comment new file mode 100644 index 0000000..cd51dfb --- /dev/null +++ b/posts/obnam-localfs-speed/comment_1_9b98f43ed08cef5fa68d1f0f1946c3aa._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + ip="50.43.15.19" + claimedauthor="Josh" + url="https://joshtriplett.org/" + subject="comment 1" + date="2011-11-30T20:05:36Z" + content=""" +Apart from the sequential nature of obnam, what currently slows it down to 15MB/s, rather than running at the full speed of the disk? Does the various data processing, such as checksumming, take that much time in Python? Would PyPy or a C module speed that up significantly? +"""]]
Comment moderation
diff --git a/posts/sd-to-ikiwiki/comment_1_a0932f7192665724d8431d49a0d13377._comment b/posts/sd-to-ikiwiki/comment_1_a0932f7192665724d8431d49a0d13377._comment new file mode 100644 index 0000000..88d4cef --- /dev/null +++ b/posts/sd-to-ikiwiki/comment_1_a0932f7192665724d8431d49a0d13377._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + ip="78.181.102.36" + claimedauthor="Mithat" + subject="DokuMicroBugTracker" + date="2011-11-27T22:19:15Z" + content=""" +I sorta like the [DokuMicroBugTracker](http://www.dokuwiki.org/plugin:dokumicrobugtracker) plugin for Dokuwiki. +"""]] diff --git a/posts/sd-to-ikiwiki/comment_2_b88b903fbfb20d18e799db1c97ff008e._comment b/posts/sd-to-ikiwiki/comment_2_b88b903fbfb20d18e799db1c97ff008e._comment new file mode 100644 index 0000000..ab54298 --- /dev/null +++ b/posts/sd-to-ikiwiki/comment_2_b88b903fbfb20d18e799db1c97ff008e._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + claimedauthor="Alexandre Franke" + url="http://www.alexandrefranke.com/" + subject="SD interoperability" + date="2011-11-28T16:31:31Z" + content=""" +I'd love to use SD to complement git when working on GNOME stuff, unfortunately it doesn't seem to work with Bugzilla. +"""]]
foo
diff --git a/mc.mdwn b/mc.mdwn new file mode 100644 index 0000000..5e60084 --- /dev/null +++ b/mc.mdwn @@ -0,0 +1 @@ +[[!inline pages="comment_pending(*)" template=comment]]
New blog post
diff --git a/posts/obnam-localfs-speed.mdwn b/posts/obnam-localfs-speed.mdwn new file mode 100644 index 0000000..d9dacf9 --- /dev/null +++ b/posts/obnam-localfs-speed.mdwn @@ -0,0 +1,16 @@ +[[!meta title="Obnam backup speed to local disk"]] +[[!tag obnam]] + +I ran a backup of my backup home directory, to a USB drive with +full-disk encryption. Here's the result: + + 04h00m13s 172739 files; 211.24 GiB up (15.01 MiB/s) /home/liw + +Not a speed daemon, but adequate. + +Backing up over the network, or using encryption in Obnam (rather +than the device mapper) makes things slower, but I'll optimze +those next. The slowness is because Obnam is currently entirely +sequential, so anything that adds a frequent delay (e.g., network round +trip times, or running gpg) has a pretty big impact on the overall +speed. But that's fixable.
Post about SD to ikiwiki for bug tracking
diff --git a/posts/sd-to-ikiwiki.mdwn b/posts/sd-to-ikiwiki.mdwn new file mode 100644 index 0000000..b040425 --- /dev/null +++ b/posts/sd-to-ikiwiki.mdwn @@ -0,0 +1,17 @@ +[[!meta title="Moving from SD to Ikiwiki for bug tracking"]] +[[!tag sd ikiwiki bug-tracking]] + +I've moved from using [SD](http://syncwith.us/sd/) for bug tracking +to using [Ikiwiki](http://ikiwiki.info/). SD is a distributed bug +tracker, and I quite like it. However, it seems to be hard to get +anyone else to use it. + +Ikiwiki is a wiki engine, and makes it easier for others to deal with +it, since they can just use a web browser. Since I don't really use any +metadata for bugs, apart from open/closed, Ikiwiki is a simple solution +that suits my needs. It's probably not suitable for a large project with +lots of open bugs. + +I've only just switched, we'll see how this goes. I really hope distributed +bug tracking takes off in a big way, however, since it's oh so nice to +be able to take full advantage of one's bug tracker even when offline.
Add link to moderation policy to sidebar
diff --git a/sidebar.mdwn b/sidebar.mdwn index 3936fec..ccae437 100644 --- a/sidebar.mdwn +++ b/sidebar.mdwn @@ -9,6 +9,8 @@ Lars Wirzenius [[Recent Comments|comments]] +[Moderation policy](http://liw.fi/moderation/) + #### **All tags** [[!pagestats pages="tag/*"]]
Add links to moderation policy
diff --git a/index.mdwn b/index.mdwn index a4dde45..fc3c8d2 100644 --- a/index.mdwn +++ b/index.mdwn @@ -2,6 +2,5 @@ Welcome to my web log. See the [[first_post|posts/welcome]] for an introduction. See the [[archive|posts]] page for all posts, and [[comments]] for a feed of comments only. (There is an [[english language feed|englishfeed]] if you don't want to see Finnish.) -See also [identi.ca](http://identi.ca/liw). [[!inline pages="posts/* and !posts/*/* and !tagged(draft)"]] diff --git a/posts/welcome.mdwn b/posts/welcome.mdwn index e4ebf37..b632217 100644 --- a/posts/welcome.mdwn +++ b/posts/welcome.mdwn @@ -1,15 +1,24 @@ [[!meta title="Welcome to my newest blog"]] [[!tag meta]] +[[!meta date="2010-11-29 09:18:04 +0000"]] -Once upon a time, I had a [diary on Advogato](http://advogato.org/person/liw/), but rarely used that. Then I wrote my own web log engine, and [used that for many years](http://liw.iki.fi/liw/log/). Then I switched my entire site to [Ikiwiki](http://ikiwiki.info), but other things made [the next blog](http://liw.iki.fi/liw/log2/) a pain to use. This is my newest approach: a dedicated web site just for my blog. +Once upon a time, I had a [diary on Advogato](http://advogato.org/person/liw/), +but rarely used that. Then I wrote my own web log engine, and [used that for +many years](http://liw.iki.fi/liw/log/). Then I switched my entire site to +[Ikiwiki](http://ikiwiki.info), but other things made +[the next blog](http://liw.iki.fi/liw/log2/) a pain to use. +This is my newest approach: a dedicated web site just for my blog. Welcome. -I will be writing here about all the things that interest me and that I want to discuss in public. This blog does not have a topic as such, so it's as narrow-minded or ecletic as I am, or as I want to portray myself in public. +I will be writing here about all the things that interest me and that +I want to discuss in public. This blog does not have a topic as such, +so it's as narrow-minded or ecletic as I am, or as I want to portray +myself in public. -For the first time, readers can comment directly: this whole site is a wiki, every blog post is a page, and you can comment on a post by editing the corresponding discussion page, using the link at the bottom of the page. You have to log in to do that, and you can either use OpenID (preferred) or register on this site separately. I will delete spam and other abuse, as usual on discussion forums. +For the first time, readers can comment directly: this whole site is a wiki, +every blog post is a page, and you can comment on a post by editing the +corresponding discussion page, using the link at the bottom of the page. +See [moderation policy](http://liw.fi/moderation/). -The web design of this site is based on -Nonzero1.0 by -<a href="http://www.nodethirtythree.com/">NodeThirtyThree Design</a>, -with some modification by me. +**EDIT:** Updated in 2011 to point at a more explicit moderation policy.
Remove edit template since I always edit via git
diff --git a/index.mdwn b/index.mdwn index 50b46f7..a4dde45 100644 --- a/index.mdwn +++ b/index.mdwn @@ -5,7 +5,3 @@ introduction. See the [[archive|posts]] page for all posts, and See also [identi.ca](http://identi.ca/liw). [[!inline pages="posts/* and !posts/*/* and !tagged(draft)"]] - - -[[!edittemplate template="posttemplate" match="posts/* and !posts/*/*" - silent=yes]]
Added a comment
diff --git a/posts/future-of-linux-distros/comment_4_c541b1661a79b8e1ddfa9c2b3d6a0a1d._comment b/posts/future-of-linux-distros/comment_4_c541b1661a79b8e1ddfa9c2b3d6a0a1d._comment new file mode 100644 index 0000000..f13fbc3 --- /dev/null +++ b/posts/future-of-linux-distros/comment_4_c541b1661a79b8e1ddfa9c2b3d6a0a1d._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawk7vThnE-3yh9TKV9iQ0ZVkrH7-RyHhQIo" + nickname="Adam" + subject="comment 4" + date="2011-10-29T19:21:11Z" + content=""" +> \"less fine-grained software packaging for overall simplification of the system and its development\" + +People who don't learn the lessons of RedHat are doomed to repeat them, it seems. +"""]]
Added a comment
diff --git a/posts/future-of-linux-distros/comment_3_36520355b518eb624676d24f6cc602eb._comment b/posts/future-of-linux-distros/comment_3_36520355b518eb624676d24f6cc602eb._comment new file mode 100644 index 0000000..2b640b4 --- /dev/null +++ b/posts/future-of-linux-distros/comment_3_36520355b518eb624676d24f6cc602eb._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="Anonymous" + ip="84.155.176.252" + subject="comment 3" + date="2011-10-29T16:39:41Z" + content=""" +\"Sometimes this is non-choice is a good thing.\" - That sentence doesn't parse for me. +"""]]
Added a comment
diff --git a/posts/future-of-linux-distros/comment_2_05f625c357a7282fb799f2e98377a84b._comment b/posts/future-of-linux-distros/comment_2_05f625c357a7282fb799f2e98377a84b._comment new file mode 100644 index 0000000..b2b1946 --- /dev/null +++ b/posts/future-of-linux-distros/comment_2_05f625c357a7282fb799f2e98377a84b._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkN91jAhoesnVI9TtWANaBPaYjd1V9Pag8" + nickname="Benjamin" + subject="comment 2" + date="2011-10-28T21:03:21Z" + content=""" +Interesting post, but I couldn't help but think of how distributions like Gentoo seems to solve many of the issues you raise. + +@Roger, wrt to how recompiling after an update help I had to think of [Ken Thompson ](http://cm.bell-labs.com/who/ken/trust.html). +"""]]
Added a comment: Build time versions
diff --git a/posts/future-of-linux-distros/comment_1_457ad5b0c60dadcf7ff2ecf5f96d9fcc._comment b/posts/future-of-linux-distros/comment_1_457ad5b0c60dadcf7ff2ecf5f96d9fcc._comment new file mode 100644 index 0000000..d3464dd --- /dev/null +++ b/posts/future-of-linux-distros/comment_1_457ad5b0c60dadcf7ff2ecf5f96d9fcc._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://rogerbinns.com/" + nickname="Roger Binns" + subject="Build time versions" + date="2011-10-28T20:52:30Z" + content=""" +One thing that scares me is how the build time is not controlled. For example if package A uses package B at buildtime (eg B is a compiler or header files) and then later A and B get updated separately you can end up with A being compiled against different versions of B on different build systems. A concrete example would be a gnome package getting updated and then the next day one of the gnome libraries it builds against getting updated. Depending on timing some build systems would use an earlier version of the library and some later. If a compiler gets updated then in theory everything that was compiled with it should be recompiled in order to be 100% reproducible. + +In practise Debian et al rely on the packages to have no applicable bugs on upgrading and discovery of issues is rare, but I would be shocked if we somehow had perfect software. In theory the only way to be totally reproducible is to recompile the entire system from the ground up several times over to ensure any version change in any component fully percolates through the system (somewhat like bootstrapping gcc). +"""]]
Remove draft tag.
diff --git a/posts/future-of-linux-distros.mdwn b/posts/future-of-linux-distros.mdwn index bd0777b..023a3a0 100644 --- a/posts/future-of-linux-distros.mdwn +++ b/posts/future-of-linux-distros.mdwn @@ -1,5 +1,5 @@ [[!meta title="On the future of Linux distributions"]] -[[!tag draft debian linux baserock]] +[[!tag debian linux baserock]] Executive summary -----------------
creating tag page tag/baserock
diff --git a/tag/baserock.mdwn b/tag/baserock.mdwn new file mode 100644 index 0000000..bf0f83d --- /dev/null +++ b/tag/baserock.mdwn @@ -0,0 +1,4 @@ +[[!meta title="pages tagged baserock"]] + +[[!inline pages="tagged(baserock)" actions="no" archive="yes" +feedshow=10]]
Post about the future of Linux distros
diff --git a/posts/future-of-linux-distros.mdwn b/posts/future-of-linux-distros.mdwn new file mode 100644 index 0000000..bd0777b --- /dev/null +++ b/posts/future-of-linux-distros.mdwn @@ -0,0 +1,403 @@ +[[!meta title="On the future of Linux distributions"]] +[[!tag draft debian linux baserock]] + +Executive summary +----------------- + +Debian should consider: + +* less fine-grained software packaging for overall simplification of + the system and its development +* less flexibility at the lower levels of the software stack, again + for overall simplicity +* reducing friction in the development workflow, especially by making + it possible to branch and merge at the whole system level + +Introduction +------------ + +I've taken a job with [Codethink](http://www.codethink.co.uk/), +as part of a team to develop +a new embedded Linux system, called +[Baserock](http://wiki.baserock.org/). Baserock is not a +distribution as such, but deals with many of the same issues. +I thought it might be interesting to write out my new thoughts +related to this, since they might be useful for proper +distributions to think about too. + +I'll hasten to add that many of these thoughts are not originally +mine. I shamelessly adopt any idea that I think is good. +The core ideas for Baserock +come from Rob Taylor, Daniel Silverstone, and Paul Sherwood, my +colleagues and bosses at Codethink. At the recent GNOME Summit +in Montreal, I was greatly influenced by Colin Walters. + +This is also not an advertisment for Baserock, but since my of +my current thinking come from that project, I'll discuss things +in the context of Baserock. + +Finally, I'm writing this to express my personal opinion and +thoughts. I'm not speaking for anyone else, least of all Codethink. + + +On the package abstraction +-------------------------- + +Baserock abandons the idea of packages for all individual programs. +In the early 1990s, when Linux was new, and the number of packages in +a distribution could be counted on two hands in binary, packages were +a great idea. It was feasible to know at least something about every +piece of software, and pick the exact set of software to install on +a machine. + +It is still feasible to do so, but only in quite restricted circumstances. +For example, picking the packages to install a DNS server, or an NFS +server, or a mail server, by hand, without using meta packages or +tasks (in Debian terminology), is still quite possible. On embedded +devices, there's also usually only a handful of programs installed, +and the people doing the install can be expected to understand all +of them and decide which ones to install on each specific device. + +However, those are the exceptions, and they're getting rarer. For +most people, manually picking software to install is much too tedious. +In Debian, we've realised this a great many years ago, and developed +meta packages (whose only purpose is to depend on other packages) and +tasks (solves the same problem, but differently). These make it possible +for a user to say "I want to have the GNOME desktop environment", and +not have to worry about finding every piece that belongs in GNOME, +and install that separately. + +For much of the past decade, computers have had sufficient hard disk +space that it is no longer necessary to be quite so picky about what +to install. A new cheap laptop will now typically come with at least 250 +gigabytes of disk space. An expensive one, with an SSD drive, will have +at least 128 gigabytes. A fairly complete desktop install uses less than +ten gigabytes, so there's rarely a need to pick and choose between +the various components. + +From a usability point of view, choosing from a list of a dozen or +two options is much easier than from a list of thirty five thousand +(the number of Debian packages as I write this). + +This is one reason why Baserock won't be using the traditional +package abstraction. Instead, we'll collect programs into larger +collections, called strata, which form some kind of logical or +functional whole. So there'll be one stratum for the core userland +software: init, shell, libc, perhaps a bit more. There'll be another +for a development environment, one for a desktop environment, etc. + +Another, equally important reason, to move beyond packages is the +problems caused by the combinatorial explosion of packages and versions. +Colin Walters talks about this very well. When every system has a +fairly unique set of packages, and versions of them, it becomes much +harder to ensure that software works well for everyone, that upgrades +work well, and that any problems get solved. When the number of +possible package combinations is small, getting the interactions between +various packages right is easier, QA has a much easier time to +test all upgrade paths, and manual test coverage improves a lot when +everyone is testing the same versions. +Even debugging gets easier, when everyone can easily run the same +versions. + +Grouping software into bigger collections does reduce flexibility +of what gets installed. In some cases this is important: very +constrainted embedded devices, for example, still need to be very +picky about what software gets installed. However, even for them, +the price of flash storage is low enough that it might not matter +too much, anymore. The benefit of a simpler overall system may well +outweigh the benefit of fine-grained software choice. + + +Everything in git +----------------- + +In Baserock, we'll be building everything from source in git. +It will not be possible to build anything, unless the source is +committed. This will allow us to track, for each binary blob we +produce, the precise sources that were used to bulid it. + +We will also try +to achieve something a bit more ambitious: anything that affects +any bit in the final system image can be traced to files committed +to git. This means tracking also all configuration settings for the +build, and the whole build environment, in git. + +This is important for us so that we can reproduce an image used in +the field. When a customer is deploying a specific image, and needs +it to be changed, we want to be able to make the change with the +minimal changes compared to the previous version of the image. This +requires that we can re-create the original image, from source, bit for bit, +so that when we make the actual change, only the changes we need to +make affect the image. + +We will make it easy to branch and merge not just individual +projects, but the whole system. This will make it easy to do +large changes to the system, such as transitioning to a new +version of GNOME, or the toolchain. Currently, in Debian, such +large changes need to be serialised, so that they do not affect +each other. It is easy, for example, for a GNOME transition to +be broken by a toolchain transition. + +Branching and merging has long been considered the best available +solution for concurrent development within a project. With Baserock, +we want to have that for the whole system. Our build servers will +be able to build images for each branch, without +requiring massive hardware investment: any software that is shared +between branches only gets built once. + +Launchpad PPAs and similar solutions provide many of the benefits +of branching and merging on the system level. However, they're much +more work than "git checkout -b gnome-3.4-transition". I believe +that the git based approach will make concurrent development much +more efficient. Ask me in a year if I was right. + + +Git, git, and only git +---------------------- + +There are a lot of version control systems in active use. For the +sake of simplicity, we'll use only git. When an upstream project +uses something else, we'll import their code into git. Luckily, there +are tools for that. The import and updates to it will be +fully automatic, of course. + +Git is not my favorite version control system, but it's clearly +the winner. Everything else will eventually fade away into +obscurity. Or that's what we think. If it turns out that we're wrong +about that, we'll switch to something else. However, we do not +intend to have to deal with more than one at a time. Life's too short +to use all possible tools at the same time. + + +Tracking upstreams closely +-------------------------- + +We will track upstream version control repositories, and we will have +an automatic mechanism of building our binaries directly from git. This +will, we hope, make it easy to follow closely upstream development, so +that when, say, GNOME developers make commits, we want to be able to +generate a new system image which includes those changes the same day, +if not within minutes, rather than waiting days or weeks or months. + +This kind of closeness is greatly enhanced by having everything in +version control. When upstream commits changes to their version control +system, we'll mirror them automatically, and this then triggers a +new system image build. When upstream makes changes that do not work, +we can easily create a branch from any earlier commit, and build images +off that branch. + +This will, we hope, also make it simpler to make changes, and give them +back to upstream. Whenever we change anything, it'll be done in a branch, +and we'll have system images available to test the change. So not only +will upstream be able to easily get the change from our git repository, +they'll also be easily verify, on a running system, that the change +fixes the problem. (Diff truncated)
Added a comment: Not sure if that is luxery.
diff --git a/posts/luxury/comment_1_d79c92c4e648e4ab5bb82685ef7994df._comment b/posts/luxury/comment_1_d79c92c4e648e4ab5bb82685ef7994df._comment new file mode 100644 index 0000000..a4d5d74 --- /dev/null +++ b/posts/luxury/comment_1_d79c92c4e648e4ab5bb82685ef7994df._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://www.joachim-breitner.de/" + nickname="nomeata" + subject="Not sure if that is luxery." + date="2011-10-24T13:54:00Z" + content=""" +Luxery might be to buy a golden faucet, even if that means that one has to cut short on space and hot water. Or, spoken with Oscal Wilde’s words: “Let me be surrounded by luxury.I can do without the necessities!“ + +(But I’d prefer the hot water, too.) +"""]]
Post about luxury.
diff --git a/posts/luxury.mdwn b/posts/luxury.mdwn new file mode 100644 index 0000000..81d2a12 --- /dev/null +++ b/posts/luxury.mdwn @@ -0,0 +1,6 @@ +[[!meta title="Luxury"]] +[[!tag opinion]] + +Luxury is a roomy shower with plenty of hot water and a non-slippery +floor. A gold-plated faucet really doesn't make it luxurious, just ugly. +
Post about Codethink, Baserock, MCR, engagement.
diff --git a/posts/codethink-etc.mdwn b/posts/codethink-etc.mdwn new file mode 100644 index 0000000..da9e814 --- /dev/null +++ b/posts/codethink-etc.mdwn @@ -0,0 +1,18 @@ +[[!meta title="In search of boredom"]] +[[!tag personal]] + +A month ago I started a new job, at [Codethink](http://www.codethink.co.uk/), +a British software development company. We work mainly on and with free +software. I'm working on a new project called +[Baserock](http://wiki.baserock.org/), a way to +build embedded Linux systems that we think will avoid many of the +pain points that people usually have. Baserock is still in its formative +stages, not something you can really play with. + +Codethink has its main office in Manchester, so the past weekend I moved +there with my fiancé. + +Oh yeah, Soile and I got engaged. Rings of Palladium. + +While I've been having a lot of fun, I'd like to be bored for a while now. +Instead, I'm going to Prague for ELC Europe.
Obnam 0.23. Remove draft tag.
diff --git a/posts/obnam-0.23.mdwn b/posts/obnam-0.23.mdwn index 3be2c87..f389a33 100644 --- a/posts/obnam-0.23.mdwn +++ b/posts/obnam-0.23.mdwn @@ -1,5 +1,5 @@ [[!meta title="Obnam 0.23 released (backup software)"]] -[[!tag obnam draft]] +[[!tag obnam]] I've released version 0.23 of [Obnam](http://braawi.org/obnam/), my backup application.
Obnam 0.23.
diff --git a/posts/obnam-0.23.mdwn b/posts/obnam-0.23.mdwn new file mode 100644 index 0000000..3be2c87 --- /dev/null +++ b/posts/obnam-0.23.mdwn @@ -0,0 +1,46 @@ +[[!meta title="Obnam 0.23 released (backup software)"]] +[[!tag obnam draft]] + +I've released version 0.23 of [Obnam](http://braawi.org/obnam/), +my backup application. + +USER VISIBLE CHANGES: + +* `restore` now shows a progress bar. +* `fsck` now has more useful progress reporting, and does more checking, + including the integrity of the contents of file content. +* `fsck` now also checks the integrity of the B-trees in the repository, + so that it is not necessary to run `fsck-larch` manually anymore. This + works remotely as well, whereas `fsck-larch` only worked on B-trees + on the local filesystem. +* `force-lock` now gives a warning if the client does not exist in the + repository. +* Subcommands for encryption now give a warning if encryption key is not + given. +* The `--fsck-fix` option will now instruct `obnam fsck` to try to fix + problems found. For this release, it only means fixing B-tree missing + node problems, but more will follow. +* The default sizes have been changed for B-tree nodes (256 KiB) + and file contents chunks (1 MiB), based on benchmarking. +* SFTP protocol use has been optimized, which should result in some + more speed. This also highlights the need to change obnam so it can + do uploads in the background. +* If a client does not exist in the repository, `force-lock` now gives + a warning to the user, rather than ignoring it silently. + +DEVELOPER CHANGES: + +* New `--sftp-delay=100` option can be used to simulate SFTP backups over + networks with long round trip times. +* `obnam-benchmark` can now use `--sftp-delay` and other changes to make + it more useful. + +INTERNAL CHANGES: + +* Got rid of terminal status plugin. Now, the `Application` class provides + a `ttystatus.TerminalStatus` instance instead, in the `ts` attribute. + Other plugings are supposed to use that for progress reporting and + messaging to the user. +* The `posix_fadvise` system call is used only if available. This should + improve Obnam's portability a bit. +
Added a comment: randomsound
diff --git a/posts/ekey/comment_4_4bfb145c2615c7fa45734139585a8853._comment b/posts/ekey/comment_4_4bfb145c2615c7fa45734139585a8853._comment new file mode 100644 index 0000000..2197832 --- /dev/null +++ b/posts/ekey/comment_4_4bfb145c2615c7fa45734139585a8853._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://svend.myopenid.com/" + ip="69.91.227.154" + subject="randomsound" + date="2011-09-06T23:36:21Z" + content=""" +I've been using the [randomsound](http://packages.debian.org/squeeze/randomsound) package to generate entropy on laptops running live CDs. Without randomsound, generating GPG keys takes ages on these systems. I'll give dieharder a try to check the quality of the entropy randomsound produces. +"""]]
Added a comment: units, users and pool sizes
diff --git a/posts/ekey/comment_3_a414a3997732356d02bb426b0542aabd._comment b/posts/ekey/comment_3_a414a3997732356d02bb426b0542aabd._comment new file mode 100644 index 0000000..282b91f --- /dev/null +++ b/posts/ekey/comment_3_a414a3997732356d02bb426b0542aabd._comment @@ -0,0 +1,17 @@ +[[!comment format=mdwn + username="mirabilos" + subject="units, users and pool sizes" + date="2011-09-06T21:07:33Z" + content=""" +Hi, + +the unit is bits. The size of the pool has been constant +for quite a while. + +In recent kernels (not 2.6.18 / etch, but newer ones) +spawning new processes also eats up entropy, for example +for ASLR (Address Space Layout Randomisation). + +And yes, I’ve been happy with my eKey for quite a loooong +while now. +"""]]
Added a comment: kernel.random.poolsize is readonly on my system
diff --git a/posts/ekey/comment_2_1b7b63c618a594261f3f664edeeac1c6._comment b/posts/ekey/comment_2_1b7b63c618a594261f3f664edeeac1c6._comment new file mode 100644 index 0000000..4043f83 --- /dev/null +++ b/posts/ekey/comment_2_1b7b63c618a594261f3f664edeeac1c6._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlgR1ZliRZvRLvuNG0ifo8Kw9MA6yzYKE0" + nickname="Jeff" + subject="kernel.random.poolsize is readonly on my system" + date="2011-09-06T21:05:25Z" + content=""" +.. and looking at the source, I think it's 0444 for everybody using the mainline kernel. <https://github.com/torvalds/linux/blob/master/drivers>/char/random.c#L1255 + +I think the idea of having a personal entropy source is a neat one. I've built one from scratch which can produce more than 100kB/s of bits; happily, it passes the DIEHARDER random number testing suite, and can also serve as an entropy source for linux. (hint: write the data to /dev/random and then ioctl RNDADDTOENTCT to say how many bits of entropy you estimate were in the data you wrote) Design files and python source: <http://emergent.unpy.net/01257868826> + +The idea of using a shared secret to secure communication between the PC and the RNG dongle is one that hadn't crossed my mind. I could potentially add this to my design, but it would negatively impact the throughput (the AVR is pretty taxed already with the rudimentary whitening algorithm that I implemented, let alone doing robust secret-key crypto!) +"""]]
Added a comment: entropy poolsize
diff --git a/posts/ekey/comment_1_50e3555b3c68e65505e6b55b22320fd7._comment b/posts/ekey/comment_1_50e3555b3c68e65505e6b55b22320fd7._comment new file mode 100644 index 0000000..c4802bc --- /dev/null +++ b/posts/ekey/comment_1_50e3555b3c68e65505e6b55b22320fd7._comment @@ -0,0 +1,16 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlP5-nhWXTLulX-RU7X72Gng5bsM-MErvI" + nickname="Rob" + subject="entropy poolsize" + date="2011-09-06T20:47:00Z" + content=""" +The kernel has a maximum poolsize which is 4096 by default. + + $ sysctl kernel.random.poolsize + +might help you see if that is the setting on your system too. + + $ sysctl -w kernel.random.poolsize=8192 + +can be used to set it to other values. +"""]]
creating tag page tag/ekey
diff --git a/tag/ekey.mdwn b/tag/ekey.mdwn new file mode 100644 index 0000000..d92b5fd --- /dev/null +++ b/tag/ekey.mdwn @@ -0,0 +1,4 @@ +[[!meta title="pages tagged ekey"]] + +[[!inline pages="tagged(ekey)" actions="no" archive="yes" +feedshow=10]]
Blog about ekey.
diff --git a/posts/ekey.mdwn b/posts/ekey.mdwn new file mode 100644 index 0000000..7663662 --- /dev/null +++ b/posts/ekey.mdwn @@ -0,0 +1,55 @@ +[[!meta title="Entropy key: a review"]] +[[!tag ekey]] + +I recently bought a [Simtec Entropy Key](http://www.entropykey.co.uk/). +It's a hardware randomness generator encapsulated as a USB device, the +size of a flash drive. It's good for any host or cluster that uses +a lot of kernel-provided random numbers (`/dev/random`), or does things +that cause the kernel to use a lot of random numbers. Opening a lot +of network connections, or generating session keys for public key +encryption. That kind of stuff. If the kernel has too little entropy +available to generate good random numbers, things will stall until it +gets some more. + +The ekey gives you real random bits, not just software-produced +pseudo-random numbers. In other words, it's some kind of magic. + +I got the device in a DVD box, with a leaflet explaining how to install +stuff, a CD with the software, an envelope with a secret key, and the +actual device itself. + +I read the leaflet, which said to install the ekeyd daemon first. +I ignored the CD, since a) my laptop has no optical drive and b) the +software is in Debian stable (squeeze) anyway, so apt-getting it is +easier and faster. + +I was utterly disappointed with the software. I didn't have to edit +a single hard disk sector, or do a BIOS upgrade, or anything. Utterly, +totally, boring. I had reserved an afternoon and evening for this, +but I had to find something else to do instead. + +After plugging in the device in a spare USB port, I opened the +envelope and ran a command to specify the secret key. The key and +host processors use encryption to make sure nobody can eavesdrop +on the randomness -- that would give them an edge in cracking SSL +and GPG encryption, I guess. + +Here's an example of how it works. Before I plug in the device, +the kernel has some, but not too much entropy available: + + liw@havelock$ cat /proc/sys/kernel/random/entropy_avail + 1053 + +A few seconds after the ekey is inserted: + + liw@havelock$ cat /proc/sys/kernel/random/entropy_avail + 3968 + +And here's the kicker: pretty much regardless of what I do, the +entropy stays at about 4 kilounits (I don't know if it's bits or +bytes, and I'm too lazy to check). + +In the end, the only thing I can really complain about is the +typography and layout of the leaflet. The text is clear enough, but +it's laid out too densely. It needs more whitespace. +Some pictures of bunnies or kittens would be nice, too.
Added a comment: fastrm
diff --git a/posts/fileops-benchmark/comment_8_482b13de69e4f322b37f4bc8b4927988._comment b/posts/fileops-benchmark/comment_8_482b13de69e4f322b37f4bc8b4927988._comment new file mode 100644 index 0000000..b26eea0 --- /dev/null +++ b/posts/fileops-benchmark/comment_8_482b13de69e4f322b37f4bc8b4927988._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="fastrm" + date="2011-09-05T07:50:04Z" + content=""" +Joey found <http://linux.die.net/man/1/fastrm>, part of inn, which may be a faster thing than calling rm. +"""]]
Added a comment: genbackupdata features and storage tricks
diff --git a/posts/fileops-benchmark/comment_7_a278e2216142d550a60d4577c64a7844._comment b/posts/fileops-benchmark/comment_7_a278e2216142d550a60d4577c64a7844._comment new file mode 100644 index 0000000..0bbfd63 --- /dev/null +++ b/posts/fileops-benchmark/comment_7_a278e2216142d550a60d4577c64a7844._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="genbackupdata features and storage tricks" + date="2011-09-01T23:02:27Z" + content=""" +Javier, that would probably be a good addition to genbackupdata. However, I'm not needing it particularly much myself, so unlikely to add it anytime soon. Patch most welcome, though! + +Josh, those would make the benchmarking setup be faster, but also less realistic. I'd like the benchmarks not be too far away from reality, and backing up from or to a tmpfs does not seem a particularly common use case. Overlay filesystems or lvm copy-on-write setups would likewise affect how the filesystem acts underneath obnam, and I'd like to avoid those complications when considering what to optimize next. But for other situations, they're definitely a good thing to consider. +"""]]
Added a comment: tmpfs?
diff --git a/posts/fileops-benchmark/comment_6_7cfddc1127754bf4e181df3aad8bc472._comment b/posts/fileops-benchmark/comment_6_7cfddc1127754bf4e181df3aad8bc472._comment new file mode 100644 index 0000000..5edaf9c --- /dev/null +++ b/posts/fileops-benchmark/comment_6_7cfddc1127754bf4e181df3aad8bc472._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="JoshTriplett" + ip="50.43.15.19" + subject="tmpfs?" + date="2011-09-01T21:50:09Z" + content=""" +Have you considered testing via a tmpfs? You can then just umount when done, which will complete almost instantly. + +As for copying, you might consider one of the many transparent overlay filesystems, which would again allow more-or-less instantaneous operation. +"""]]
Added a comment: Suggestion about genbackupdata
diff --git a/posts/fileops-benchmark/comment_5_57131b95930ed3da3f2d6791d669f337._comment b/posts/fileops-benchmark/comment_5_57131b95930ed3da3f2d6791d669f337._comment new file mode 100644 index 0000000..7ae6d78 --- /dev/null +++ b/posts/fileops-benchmark/comment_5_57131b95930ed3da3f2d6791d669f337._comment @@ -0,0 +1,15 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl588yvpbe82tMHUpGJdaPmfbFBt8dtTXo" + nickname="Javier" + subject="Suggestion about genbackupdata" + date="2011-09-01T14:58:45Z" + content=""" +A wishlist (I cannot open a wishlist bug), make size of files variables and allow various directories levels + +-t range-size (-t 10k-50k) +-b range-size +--depth number-of-dirs depth +--max-count-dir max number of directories by directory + +Regards, +"""]]
Added a comment: Hardlinking isn't copying
diff --git a/posts/fileops-benchmark/comment_4_5b4857fad2be23c4710bc5c6bd67acb8._comment b/posts/fileops-benchmark/comment_4_5b4857fad2be23c4710bc5c6bd67acb8._comment new file mode 100644 index 0000000..c065d05 --- /dev/null +++ b/posts/fileops-benchmark/comment_4_5b4857fad2be23c4710bc5c6bd67acb8._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="Hardlinking isn't copying" + date="2011-09-01T13:00:20Z" + content=""" +Hardlinks aren't useful, when files are being modified in place. So yeah, I've thought of that obvious optimization, but it's not going to work. +"""]]
Added a comment
diff --git a/posts/fileops-benchmark/comment_3_5bda12e05dd3030372e7c605ba5a013f._comment b/posts/fileops-benchmark/comment_3_5bda12e05dd3030372e7c605ba5a013f._comment new file mode 100644 index 0000000..4e85e5e --- /dev/null +++ b/posts/fileops-benchmark/comment_3_5bda12e05dd3030372e7c605ba5a013f._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmJ_6Kxghy1Ne-Ypxmu-A85CFK9Tx3fnDo" + nickname="Aigars" + subject="comment 3" + date="2011-09-01T12:41:03Z" + content=""" +Have you tried using 'cp -al' for your testing purposes? It is much faster that actually copying the data around. +"""]]
Added a comment: Good point about nature of data
diff --git a/posts/fileops-benchmark/comment_2_0c6d8043988a77d6f90929acf8999d41._comment b/posts/fileops-benchmark/comment_2_0c6d8043988a77d6f90929acf8999d41._comment new file mode 100644 index 0000000..9ab020b --- /dev/null +++ b/posts/fileops-benchmark/comment_2_0c6d8043988a77d6f90929acf8999d41._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="Good point about nature of data" + date="2011-09-01T11:51:33Z" + content=""" +That's a good point about the nature of the test data. The [genbackupdata](http://braawi.org/genbackupdata/) produces the data, and it puts creates a directory hierarchy that is three deep by default, and has 128 files per leaf directory, by default. Files are 16 KiB. See the source for details. +"""]]
Added a comment: data set?
diff --git a/posts/fileops-benchmark/comment_1_505c548d1b1d6e75b93769ae55cc9de6._comment b/posts/fileops-benchmark/comment_1_505c548d1b1d6e75b93769ae55cc9de6._comment new file mode 100644 index 0000000..d8306f7 --- /dev/null +++ b/posts/fileops-benchmark/comment_1_505c548d1b1d6e75b93769ae55cc9de6._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawl5jTQLUi9FHyOFaFx5LyBvqbObTTwJMH0" + nickname="Juan Antonio" + subject="data set?" + date="2011-09-01T11:45:57Z" + content=""" +you mention in your post that the data set is 30GiB in size, but how is that distributed? how deep is the dir hierarchy? how many files per dir? etc... +"""]]
Markup fixes.
diff --git a/posts/fileops-benchmark.mdwn b/posts/fileops-benchmark.mdwn
index cd4b15e..4c507a5 100644
--- a/posts/fileops-benchmark.mdwn
+++ b/posts/fileops-benchmark.mdwn
@@ -13,17 +13,17 @@ to be copied and removed. The faster that goes, the better.
Overnight, I ran a little benchmark on those operations, to compare
a couple of ways to do them. The results are below:
- elapsed cmd
- (s)
- 107.2 rm -rf /scratch/liw/tmp/tmp.V3tGTVUbbS/data
- 98.4 find /scratch/liw/tmp/tmp.V3tGTVUbbS/data -delete
- 100.1 find /scratch/liw/tmp/tmp.V3tGTVUbbS/data -exec rm -rf {} +
- 116.2 find /scratch/liw/tmp/tmp.V3tGTVUbbS/data -depth -print0 | xargs -0 rm -rf
-
- elapsed cmd
- (s)
- 3567.5 cp -a /scratch/liw/tmp/tmp.V3tGTVUbbS/data /scratch/liw/tmp/tmp.V3tGTVUbbS/copy
- 3219.5 cd /scratch/liw/tmp/tmp.V3tGTVUbbS && mkdir copy && tar -C data -cf - . | tar -C copy -xf -
+ elapsed cmd
+ (s)
+ 107.2 rm -rf tmp/data
+ 98.4 find tmp/data -delete
+ 100.1 find tmp/data -exec rm -rf {} +
+ 116.2 find tmp/data -depth -print0 | xargs -0 rm -rf
+
+ elapsed cmd
+ (s)
+ 3567.5 cp -a tmp/data tmp/copy
+ 3219.5 cd tmp && mkdir copy && tar -C data -cf - . | tar -C copy -xf -
It is surprising, but it's clear that find is significantly
faster than rm in deleting files, by almost ten percent.
@@ -34,7 +34,7 @@ For file copying, the piping of two tars is a common trick, and
it really is faster, again by almost ten percent.
Obviously, there might also be a problem with the benchmark. I attach
-the [[script|doit.sh]], which uses `benchmark-cmd` in
+the [[script|doit.txt]], which uses `benchmark-cmd` in
[extrautils](http://liw.fi/extrautils/), which I wrote for this kind
of thing. If there is a problem with the benchmark, don't hesitate
to provide a patch to fix that.
diff --git a/posts/fileops-benchmark/doit.sh b/posts/fileops-benchmark/doit.sh
deleted file mode 100755
index 2c8c700..0000000
--- a/posts/fileops-benchmark/doit.sh
+++ /dev/null
@@ -1,26 +0,0 @@
-#!/bin/sh
-
-set -e
-
-amount="30g"
-
-tempdir="$(mktemp -d)"
-
-benchmark-cmd \
- --verbose \
- --setup="genbackupdata -c $amount $tempdir/data" \
- --verify="test ! -e $tempdir/data" \
- --command="rm -rf $tempdir/data" \
- --command="find $tempdir/data -delete" \
- --command="find $tempdir/data -exec rm -rf {} +" \
- --command="find $tempdir/data -depth -print0 | xargs -0 rm -rf"
-
-genbackupdata -c "$amount" "$tempdir/data"
-benchmark-cmd \
- --verbose \
- --verify="diff -rq $tempdir/data $tempdir/copy" \
- --cleanup="rm -rf $tempdir/copy" \
- --command="cp -a $tempdir/data $tempdir/copy" \
- --command="cd $tempdir && mkdir copy && tar -C data -cf - . | tar -C copy -xf -"
-
-rm -rf "$tempdir"
diff --git a/posts/fileops-benchmark/doit.txt b/posts/fileops-benchmark/doit.txt
new file mode 100755
index 0000000..2c8c700
--- /dev/null
+++ b/posts/fileops-benchmark/doit.txt
@@ -0,0 +1,26 @@
+#!/bin/sh
+
+set -e
+
+amount="30g"
+
+tempdir="$(mktemp -d)"
+
+benchmark-cmd \
+ --verbose \
+ --setup="genbackupdata -c $amount $tempdir/data" \
+ --verify="test ! -e $tempdir/data" \
+ --command="rm -rf $tempdir/data" \
+ --command="find $tempdir/data -delete" \
+ --command="find $tempdir/data -exec rm -rf {} +" \
+ --command="find $tempdir/data -depth -print0 | xargs -0 rm -rf"
+
+genbackupdata -c "$amount" "$tempdir/data"
+benchmark-cmd \
+ --verbose \
+ --verify="diff -rq $tempdir/data $tempdir/copy" \
+ --cleanup="rm -rf $tempdir/copy" \
+ --command="cp -a $tempdir/data $tempdir/copy" \
+ --command="cd $tempdir && mkdir copy && tar -C data -cf - . | tar -C copy -xf -"
+
+rm -rf "$tempdir"
Blog about fileops benchmark.
diff --git a/posts/fileops-benchmark.mdwn b/posts/fileops-benchmark.mdwn
new file mode 100644
index 0000000..cd4b15e
--- /dev/null
+++ b/posts/fileops-benchmark.mdwn
@@ -0,0 +1,49 @@
+[[!meta title="File copying and removal benchmarking"]]
+[[!tag benchmark]]
+
+As part of my development of a [backup application](http://braawi.org/obnam/),
+I run benchmarks on it, and that means creating a test data set, running
+some backups, and then removing the test data set. One of the test data
+sets I use is a 140 gibibytes of my real data. The benchmark first copies
+the data to a temporary location.
+
+In other words, a fair bit of my current life is spent waiting for files
+to be copied and removed. The faster that goes, the better.
+
+Overnight, I ran a little benchmark on those operations, to compare
+a couple of ways to do them. The results are below:
+
+ elapsed cmd
+ (s)
+ 107.2 rm -rf /scratch/liw/tmp/tmp.V3tGTVUbbS/data
+ 98.4 find /scratch/liw/tmp/tmp.V3tGTVUbbS/data -delete
+ 100.1 find /scratch/liw/tmp/tmp.V3tGTVUbbS/data -exec rm -rf {} +
+ 116.2 find /scratch/liw/tmp/tmp.V3tGTVUbbS/data -depth -print0 | xargs -0 rm -rf
+
+ elapsed cmd
+ (s)
+ 3567.5 cp -a /scratch/liw/tmp/tmp.V3tGTVUbbS/data /scratch/liw/tmp/tmp.V3tGTVUbbS/copy
+ 3219.5 cd /scratch/liw/tmp/tmp.V3tGTVUbbS && mkdir copy && tar -C data -cf - . | tar -C copy -xf -
+
+It is surprising, but it's clear that find is significantly
+faster than rm in deleting files, by almost ten percent.
+Since performance is a feature,
+this would indicate that that feature in rm is buggy.
+
+For file copying, the piping of two tars is a common trick, and
+it really is faster, again by almost ten percent.
+
+Obviously, there might also be a problem with the benchmark. I attach
+the [[script|doit.sh]], which uses `benchmark-cmd` in
+[extrautils](http://liw.fi/extrautils/), which I wrote for this kind
+of thing. If there is a problem with the benchmark, don't hesitate
+to provide a patch to fix that.
+
+There may be other ways to remove or copy files that should be compared,
+too. rsync? cpio? For file removal, a tool using Linux `getdents` directly
+would probably be faster than the portable code in GNU coreutils and
+findutils. Somebody should write that and compare.
+
+In all cases above, the test data set to be removed or copied is
+30 GiB. Copies happened to the same disk (that's what happens with
+my backup benchmarks too). The filesystem used was ext4.
diff --git a/posts/fileops-benchmark/doit.sh b/posts/fileops-benchmark/doit.sh
new file mode 100755
index 0000000..2c8c700
--- /dev/null
+++ b/posts/fileops-benchmark/doit.sh
@@ -0,0 +1,26 @@
+#!/bin/sh
+
+set -e
+
+amount="30g"
+
+tempdir="$(mktemp -d)"
+
+benchmark-cmd \
+ --verbose \
+ --setup="genbackupdata -c $amount $tempdir/data" \
+ --verify="test ! -e $tempdir/data" \
+ --command="rm -rf $tempdir/data" \
+ --command="find $tempdir/data -delete" \
+ --command="find $tempdir/data -exec rm -rf {} +" \
+ --command="find $tempdir/data -depth -print0 | xargs -0 rm -rf"
+
+genbackupdata -c "$amount" "$tempdir/data"
+benchmark-cmd \
+ --verbose \
+ --verify="diff -rq $tempdir/data $tempdir/copy" \
+ --cleanup="rm -rf $tempdir/copy" \
+ --command="cp -a $tempdir/data $tempdir/copy" \
+ --command="cd $tempdir && mkdir copy && tar -C data -cf - . | tar -C copy -xf -"
+
+rm -rf "$tempdir"
Obnam version 0.22.
diff --git a/posts/obnam-0.22.mdwn b/posts/obnam-0.22.mdwn new file mode 100644 index 0000000..5583c09 --- /dev/null +++ b/posts/obnam-0.22.mdwn @@ -0,0 +1,28 @@ +[[!meta title="Obnam version 0.22 (backup application)"]] +[[!tag obnam]] + +I have just released version 0.22 of [Obnam](http://braawi.org/obnam/), +my backup application. Snippet from the NEWS file below. +This version fixes a couple of bugs that could be said to be of +the brown paper bag variety. + +USER VISIBLE CHANGES: + +* Obnam now reports its current configuration in the log file at startup. + This will hopefully remove one round of "did you use the --foo option?" + questions between developers and bug reporters. + +BUG FIXES: + +* The repository is now unlocked on exit only if it is still locked. +* A wrongly caught `GeneratorExit` is now dealt with properly. +* Keyboard interrupts are logged, so they don't show up as anonymous errors. + +CHANGES RELEVANT TO DEVELOPERS ONLY: + +* `setup.py` has been enhanced to work more like the old `Makefile` did: + `clean` removes more artifacts. Instructions in `README` have been updated + to point at `setup.py`. +* Compiler warning about `_XOPEN_SOURCE` re-definition fixed. +* Tests are now again run during a Debian package build. +
Obnam 0.21.
diff --git a/posts/obnam-0.21.mdwn b/posts/obnam-0.21.mdwn new file mode 100644 index 0000000..0a23523 --- /dev/null +++ b/posts/obnam-0.21.mdwn @@ -0,0 +1,26 @@ +[[!meta title="Obnam 0.21 released (backup software)"]] +[[!tag obnam debian]] + +I've today released version 0.21 of +[Obnam](http://braawi.org/obnam/), my backup application. +This is the first version +that also got uploaded to Debian. It is currently waiting for +manual processing in the NEW queue. + +From the NEWS file: + +USER VISIBLE CHANGES: + +* Obnam will now unlock the repository if there's an error during a backup. + For the most part, the `force-lock` operation should now be unnecessary, + but it's still there in case it's useful some day. + +BUG FIXES: + +* Negative timestamps for files now work. Thanks to Jamil Djadala + for reporting the bug. +* The documentation for --checkpoint units fixed. Thanks, user weinzwang + from IRC. +* The connections to the repository and live data filesystem are now + properly closed. This makes benchmark read/write statistics be correct. +
Added a comment: Is Autotools a crutch?
diff --git a/posts/sw-roles-build-engineers/comment_4_4b14c0672008d83cd95f7e3cf9f9f051._comment b/posts/sw-roles-build-engineers/comment_4_4b14c0672008d83cd95f7e3cf9f9f051._comment new file mode 100644 index 0000000..f3cbea2 --- /dev/null +++ b/posts/sw-roles-build-engineers/comment_4_4b14c0672008d83cd95f7e3cf9f9f051._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawkCq9r-wSxMkVDrMm5VSxe8uP97MsJCKm4" + nickname="Jeff" + subject="Is Autotools a crutch?" + date="2011-08-21T22:40:06Z" + content=""" +A couple years ago I heard Doug McIlroy talk about the growth of complexity in Unix/Linux. His attitude was that Autotools did more harm than good because it allows different Unixes to do whatever they want. If an effort had been made to standardize things rather than letting everyone do their own thing, we wouldn't need Autotools to work around all the warts in all the systems. + +I'm no expert, but it sounds reasonable to me. That doesn't mean that there's any practical way to fix the problem. Most people would rather everyone else change to fit their system. Still it would be nice if there were more incentive to not reinvent the wheel. +"""]]
Remove test post.
diff --git a/posts/test.mdwn b/posts/test.mdwn deleted file mode 100644 index 2786da3..0000000 --- a/posts/test.mdwn +++ /dev/null @@ -1,4 +0,0 @@ -[[!meta title="test post"]] -[[!tag draft]] - -You should never see this in my feed.
Exclude drafts from english feed too.
diff --git a/englishfeed.mdwn b/englishfeed.mdwn
index 2608787..1554294 100644
--- a/englishfeed.mdwn
+++ b/englishfeed.mdwn
@@ -1,5 +1,5 @@
Feed for Planet Debian.
-[[!inline pages="posts/* and !posts/*/* and !link(tag/in-finnish) and
- !tagged(draft) created_after(posts/dd-again)"
+[[!inline pages="posts/* and !posts/*/* and !tagged(in-finnish) and
+ !tagged(draft) and created_after(posts/dd-again)"
description="liw's English language blog feed"]]
Exclude drafts from english feed too.
diff --git a/englishfeed.mdwn b/englishfeed.mdwn
index f3147cd..2608787 100644
--- a/englishfeed.mdwn
+++ b/englishfeed.mdwn
@@ -1,5 +1,5 @@
Feed for Planet Debian.
[[!inline pages="posts/* and !posts/*/* and !link(tag/in-finnish) and
- created_after(posts/dd-again)"
+ !tagged(draft) created_after(posts/dd-again)"
description="liw's English language blog feed"]]
creating tag page tag/draft
diff --git a/tag/draft.mdwn b/tag/draft.mdwn new file mode 100644 index 0000000..dd27d9d --- /dev/null +++ b/tag/draft.mdwn @@ -0,0 +1,4 @@ +[[!meta title="pages tagged draft"]] + +[[!inline pages="tagged(draft)" actions="no" archive="yes" +feedshow=10]]
Exclude drafts from front page and feed.
diff --git a/index.mdwn b/index.mdwn index a4c50f9..50b46f7 100644 --- a/index.mdwn +++ b/index.mdwn @@ -4,7 +4,7 @@ introduction. See the [[archive|posts]] page for all posts, and [[english language feed|englishfeed]] if you don't want to see Finnish.) See also [identi.ca](http://identi.ca/liw). -[[!inline pages="posts/* and !posts/*/*"]] +[[!inline pages="posts/* and !posts/*/* and !tagged(draft)"]] [[!edittemplate template="posttemplate" match="posts/* and !posts/*/*" diff --git a/posts/test.mdwn b/posts/test.mdwn new file mode 100644 index 0000000..2786da3 --- /dev/null +++ b/posts/test.mdwn @@ -0,0 +1,4 @@ +[[!meta title="test post"]] +[[!tag draft]] + +You should never see this in my feed.
Blog about my Linux at 20 years article.
diff --git a/posts/linux20.mdwn b/posts/linux20.mdwn new file mode 100644 index 0000000..0479bfa --- /dev/null +++ b/posts/linux20.mdwn @@ -0,0 +1,6 @@ +[[!meta title="Linux at 20"]] +[[!tag history linux]] + +Wrote up some of my memories about the early history of Linux. +See <http://liw.fi/linux20/>, but be warned that it's long and +rambling.
creating tag page tag/funny
diff --git a/tag/funny.mdwn b/tag/funny.mdwn new file mode 100644 index 0000000..8859c34 --- /dev/null +++ b/tag/funny.mdwn @@ -0,0 +1,4 @@ +[[!meta title="pages tagged funny"]] + +[[!inline pages="tagged(funny)" actions="no" archive="yes" +feedshow=10]]
Blog about Amazon and notebooks.
diff --git a/posts/amazon-notebooks.mdwn b/posts/amazon-notebooks.mdwn new file mode 100644 index 0000000..438c29f --- /dev/null +++ b/posts/amazon-notebooks.mdwn @@ -0,0 +1,14 @@ +[[!meta title="Oh Amazon..."]] +[[!tag funny]] + +I was considering buying a notebook. Here's part of the page on +Amazon: + +[[!img amazon-notebooks.jpg]] + +In other words: + +* Amazon wants to have offer search (and browsing) of an empty notebook. +* Amazon wants to have empty notebooks as e-books on the Kindle. + +I wonder if the warranty of Kindles covers writing on the screen in ink. diff --git a/posts/amazon-notebooks/amazon-notebooks.jpg b/posts/amazon-notebooks/amazon-notebooks.jpg new file mode 100644 index 0000000..2a7bcf8 Binary files /dev/null and b/posts/amazon-notebooks/amazon-notebooks.jpg differ
Added a comment: What about having long lasting implementation of the silencing of the speaker made optional ?
diff --git a/posts/beepless/comment_5_275dcc8efd45a0c099415a7c95a36fc0._comment b/posts/beepless/comment_5_275dcc8efd45a0c099415a7c95a36fc0._comment new file mode 100644 index 0000000..d28671e --- /dev/null +++ b/posts/beepless/comment_5_275dcc8efd45a0c099415a7c95a36fc0._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://www-public.it-sudparis.eu/~berger_o/" + nickname="obergix" + subject="What about having long lasting implementation of the silencing of the speaker made optional ?" + date="2011-08-16T06:03:36Z" + content=""" +I think it would be even better to have someone implement some long-lasting optional way to shutdown the speakers. Have you tried and filing a bug in order to suggest such a change ? +"""]]
creating tag page tag/history
diff --git a/tag/history.mdwn b/tag/history.mdwn new file mode 100644 index 0000000..e011cfa --- /dev/null +++ b/tag/history.mdwn @@ -0,0 +1,4 @@ +[[!meta title="pages tagged history"]] + +[[!inline pages="tagged(history)" actions="no" archive="yes" +feedshow=10]]
Blog about Linux Anecdotes.
diff --git a/posts/linux-anecdotes.mdwn b/posts/linux-anecdotes.mdwn new file mode 100644 index 0000000..cbd527e --- /dev/null +++ b/posts/linux-anecdotes.mdwn @@ -0,0 +1,12 @@ +[[!meta title="Linux 20 years: anecdotes from 1998"]] +[[!tag linux anecdote history]] + +Linux will be 20 years old this year. I've been drafting an outline +for an article of my memories of the early days, but that's +going to take a lot of writing before it is ready. + +Just in case I don't finish the article, I'll point at +[Linux Anecdotes](http://liw.fi/linux-anecdotes/), the transcript +of a talk I gave in 1998, when Linux was only seven years old. +It might still be entertaining to some. If nothing else, you can +laugh at my naivete about the future, now our past.
Obnam 0.20.
diff --git a/posts/obnam-0.20.mdwn b/posts/obnam-0.20.mdwn new file mode 100644 index 0000000..9798eed --- /dev/null +++ b/posts/obnam-0.20.mdwn @@ -0,0 +1,25 @@ +[[!meta title="Obnam version 0.20 (backup software)"]] +[[!tag obnam]] + +The trek from Andromeda is getting shorter every time. Hopefully +the galaxies won't collide. + +I've just released version 0.20 of [Obnam](http://braawi.org/obnam/) +(although the armel binary is still building). The important bits: + +BUG FIXES: + +* Non-ASCII filenames over SFTP root now work. (Thanks, Tapani Tarvainen, + for the reproducible bug report.) +* The count of files while making a backup now counts all files found, + not just those backed up. The old behavior was confusing people. + +USER VISIBLE CHANGES: + +* The output of `obnam ls` now formats the columns a little prettier, + so that wide values do not cause misalignment. +* The error message when trying to use an encrypted repository without + encryption is now better (and suggests missing encryption being the + reason). Thanks, chrysn. +* Obnam now supports backing up of Unix sockets. +
Added a comment: sane build systems and autotools
diff --git a/posts/sw-roles-build-engineers/comment_3_d54cae9e8bbf1019bc5840112e74bc88._comment b/posts/sw-roles-build-engineers/comment_3_d54cae9e8bbf1019bc5840112e74bc88._comment new file mode 100644 index 0000000..ad03206 --- /dev/null +++ b/posts/sw-roles-build-engineers/comment_3_d54cae9e8bbf1019bc5840112e74bc88._comment @@ -0,0 +1,14 @@ +[[!comment format=mdwn + username="http://terceiro.myopenid.com/" + nickname="terceiro" + subject="sane build systems and autotools" + date="2011-08-09T01:02:08Z" + content=""" +Nice post! + +One point that is important to make is that *projects with reasonable build systems makes the life of distro developers easier*, and therefore allows your software to reach a much wider audience. + +About autotools requiring boilerplate files: actually, you can disable that requirement by using a single line in configure.ac: + + AM_INIT_AUTOMAKE([foreign]) +"""]]
Added a comment: Informalistic blathering
diff --git a/posts/sw-roles-build-engineers/comment_2_e9b234bc2a1df8d23f1123b658fd569e._comment b/posts/sw-roles-build-engineers/comment_2_e9b234bc2a1df8d23f1123b658fd569e._comment new file mode 100644 index 0000000..208981b --- /dev/null +++ b/posts/sw-roles-build-engineers/comment_2_e9b234bc2a1df8d23f1123b658fd569e._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="Informalistic blathering" + date="2011-08-08T07:04:41Z" + content=""" +Hi, Egon. Unfortunately I don't know if there are formal models for roles or types of people in the computing world. I'm just blathering informally about this. The unappreciated status of build engineering has been bothering me for a while. + +I hope someone has done something already, or will do, though. +"""]]
Added a comment: User roles versus user types?
diff --git a/posts/sw-roles-build-engineers/comment_1_9c9e6262547f1ea555a7af7a6c74285a._comment b/posts/sw-roles-build-engineers/comment_1_9c9e6262547f1ea555a7af7a6c74285a._comment new file mode 100644 index 0000000..eb79029 --- /dev/null +++ b/posts/sw-roles-build-engineers/comment_1_9c9e6262547f1ea555a7af7a6c74285a._comment @@ -0,0 +1,24 @@ +[[!comment format=mdwn + username="http://chem-bla-ics.blogspot.com/" + ip="109.58.134.116" + subject="User roles versus user types?" + date="2011-08-08T05:54:02Z" + content=""" +Dear Lars, + +your blog post was very timely with respect to a discussion I have about user types in biology/bioinformatics. I posted a question here: + +http://biostar.stackexchange.com/questions/10991/models-for-biology-user-types + +Where I am looking for established models for user types in that field. Now, user types are probably very much related to user roles, on a theoretical modeling level, as a certain user type associates with a few typical roles. It probably is an N-to-N relationship, but have the same purpose: categorize requirements for a software product and/or process, like software development in your post. + +My software engineering skills are quite rusty. I have a formal training in it, though limited, as it was part of a cheminformatics study, and tend to use whatever comes up, rather than more formal models in the work I do (such as development of the Chemistry Development Kit, libcdk-java in Debian). So, I very much appreciate your post. + +Therefore, I was wondering about more modern theoretical approaches from the ICT sciences. For example, are users always modeled by their roles, or are there models too that define them by type too. + +The reason for me to think of users as types is that captures what capacities they have. The role of data uploader or downloader in biology is hard to capture accurately. A user type looks more appropriate to me: a bench biologist will have different requirements for the system than a data analyst, despite having the same role with respect to the system functionality. + +With kind regards, + +Egon +"""]]
Blog about build engineers.
diff --git a/posts/sw-roles-build-engineers.mdwn b/posts/sw-roles-build-engineers.mdwn new file mode 100644 index 0000000..359e4ce --- /dev/null +++ b/posts/sw-roles-build-engineers.mdwn @@ -0,0 +1,117 @@ +[[!meta title="Missing roles in software: build engineers"]] +[[!tag programming debian rant]] + +The construction, deployment, and use of software involves a number of roles. +I do not have an exhaustive list, but I have a couple in my list that +seem to be surprising to some developers. + +In the traditional Unix (mainframe) model, there are three roles: + +* **developers** actually create the software; they include architects, + coders, documenters, testers, maintenance programmers, etc; + their deliverable is a release tarball +* **sysadmins** install the software on a host, and install new versions + when appropriate; often, the sysadmin is expected to carefully read + `README` files, and construct or tweak configuration files accoriding to + the specific needs of their users; + their deliverable is some files in `/usr/local` and `/etc`, plus perhaps + an announcement e-mail to an internal mailing list +* **users** use the software; they have no deliverable in this context + +In my personal reality distortion bubble, this model is badly broken, and +has been since the middle of the 1990s. Here's an updated model: + +* **users** control their own machines, and are not experts on the software, + or on using computers at all, and requiring them to be would be to limit + their freedom in important ways; the big change is that users now have + much more control of their computing environment than before +* **sysadmins** mainly work for companies or other organizations; they need + not be experts on what their users do, and in fact often aren't, but they + know how to handle systems that are by nature too complicated or hard for + the layman to handle, because of performance, reliability, security, or + other such needs +* **distro developers** create a system consisting of thousands or tens + of thousands of components (mostly provided by upstream developers), + and integrate and adjust things so that they work together; they also + monitor upstream projects and other news sources for updates, and provide + those +* **upstream developers** actually create the software; they include + architects, coders, documenters, testers, maintenance programmers, etc; + their deliverable is a release tarball (no change from earlier) + +A common misconception is that distro developers merely compile software +and shove the binaries into glorified tarballs. That is, of course, part +of the job, but it's also the part that most distro's tools completely +automate. There's a lot more to do, though. For example, just keeping +track of known security problems among thirty thousand packages, and +making or finding fixes for them, and testing the fixes, and releasing +updates for them, takes up quite a bit of effort. + +Another thing distro developers have to spend effort on is dealing +with the multitude of build systems upstream projects use. The freedom +to experiment with different solutions is one of the cornerstones of +free software development. However, when it comes to build systems, +many of them are not serious experiments, and instead are quick hacks +thrown together to imperfectly solve problems that sometimes should not +exist at all. + +This highlights one role prominently still missing: **build engineer**. +Their task is to solve +the problem of taking the upstream source code, dealing with +build-time configuration issues, actually building the software, and +then installing it in whatever location the person installing the +software wants it installed. Further, their task is to provide the +software to do this in a sensible manner. + +Build engineers already exist, of course. For example, the developers +of the autotools suite are such. Every upstream project also has someone +who sets up autotools for their project, or some other build tools, or +writes them from scratch for that project. + +However, what's lacking, perhaps, is recognition that build engineering +is an important part of software development. Mostly, build systems +are set up (or, worse, written) by upstream developers to fit their +narrow view of how things should work, a view anchored in the old, +broken model of roles. + +This attitude matters, because for the most part, build systems do not +seem to be written to the same quality standards as the rest of the +upstream software. For example, when an upstream developer writes a +`Makefile`, and does not consider distro development, they'll often +make it hard to differentiate between install-time location and +run-time location of various components of the software. Thus, it +might be hard for the Debian packager to get the software installed +into `debian/tmp` rather than `/usr/local`, because the build system +hardcodes `debian/tmp/usr/lib/foo/plugins` as the location of the +plugins, since that's where the software gets installed, even if, in +the Debian scenario, it's not where it gets run from. + +Luckily for much software these problems have been solved or avoided. +Indeed, as crufty as autotools is, it's one of the things that has kept +upstreams sane about their build systems, by providing such good tools that +it's more work to write a new, bad one than to use autotools. + +We currently have roving bands of translators, who move from project +to project and translate them into various natural languages. I would +like to see a future where there are people dedicated to build engineering, +who go from project to project and set up or fix their build problems. +A good build system, once set up, should require a minimum of work +to keep working. + +I'd also like to see much improvement in the build software. +As good as autotools has been to free software, I'm sure there's a lot +of room for improvement. For example, autotools requires setting +up a bunch of boilerplate files in each project, which seems like +something that could be avoided by relying on specific conventions. +However, those improvements are a separate topic. I might go into +them some day, if I decide to start developing build tools myself, +but until then, I'll let those who do that decide how they want to do it. + +PS. There are lots more missing roles in my model of +free software development, of course. For example, +security engineers (find and fix problems, but also provide +ways to prevent problems, or limit their scope), internal politicians +(keep the project's human wheels from squeaking too much), +diplomats (keep different projects from clashing too much), +release engineers, etc. Someone should make an exhaustive list. +
Post about subkeys doc.
diff --git a/posts/subkeys.mdwn b/posts/subkeys.mdwn new file mode 100644 index 0000000..e36791d --- /dev/null +++ b/posts/subkeys.mdwn @@ -0,0 +1,12 @@ +[[!meta title="Using OpenPGP subkeys in Debian development"]] +[[!tag debian]] + +I don't want to keep my GPG private keys on my laptop, so I've been +keeping them on an encrypted USB drive instead, and loading them onto +the laptop only when needed. That's cumbersome, when making Debian +packages, and needing to sign the packages frequently. So I'm going +to start using subkeys instead. However, I couldn't find good +documentation for that, so after figuring things out, I wrote +something that will hopefully be useful for others. + +It's at <http://wiki.debian.org/subkeys> for your fixing pleasure.
Obnam 0.19.
diff --git a/posts/obnam-0.19.mdwn b/posts/obnam-0.19.mdwn new file mode 100644 index 0000000..5450f6b --- /dev/null +++ b/posts/obnam-0.19.mdwn @@ -0,0 +1,61 @@ +[[!meta title="Obnam 0.19 (backup software)"]] +[[!tag obnam]] + +From the depths of the Andromeda galaxy I bring you version 0.19 of +the backup program called +[Obnam](http://braawi.org/obnam/), just released and travelling to the +Terran Internet faster than light! + +This is a BETA release. Some day soon it may well end up being in +Debian, unless you prevent that by reporting bugs faster than I can +fix them. Meanwhile, my [code.liw.fi](http://liw.fi/code/) has +.debs for amd64, i386, and armel. Or you can install it from source, +presumably (I never do). + +The NEWS items as listed below. + +INCOMPATIBILITY CHANGES: + +* We now require version 0.21 of the `larch` library, and this requires + bumping the repository format. This means old backup repositories can't + be used with this version, and you need to back up everything again. + (Please tell me when this becomes a problem.) + +BUG FIXES: + +* Found one more place where a file going missing during a backup may + cause a crash. +* Typo in error message about on-disk formats fixed. + (Thanks, Tapani Tarvainen.) +* The `--trace` option works again. +* `fcntl.F_SETFL` does not seem to work on file descriptors for files + owned by root that are read-only to the user running obnam. Worked + around by ignoring any problems with setting the flags. +* The funnest bug in this release: if no log file was specified with `--log`, + the current working directory was excluded from the backup. + +USER VISIBLE CHANGES: + +* `obnam(1)` manual page now discusses how configuration files are used. +* The manual page describes problems using sftp to access live data. +* The documentation for `--no-act` was clarified to say it only works + for `forget. (Thanks, Daniel Silverstone.) +* `obnam-benchmark` now has a manual page. +* The backup plugin logs files it excludes, so the user can find out what's + going on. A confused user is an unhappy user. + +INTERNAL STUFF: + +* Tracing statements added to various parts of the code, to help debug + mysterious problems. +* All exceptions are derived from `obnamlib.AppException` or + `obnamlib.Error`, and those are derived from `cliapp.AppException`, + so that the user gets nicer error messages than Python stack traces. +* `blackboxtests` is no longer run under fakeroot, because Debian packages + are built under fakeroot, and fakeroot within fakeroot causes trouble. + However, the point of running tests under fakeroot was to make sure + certain kinds of bugs are caught, and since Debian package building runs + the tests anyway, the test coverage is not actually diminished. +* The `Makefile` has new targets `fast-check` and `network-tests`. The + latter runs tests over sftp to localhost. +
Added a comment: Erm?
diff --git a/posts/obnam-ubuntu-ppa/comment_2_9b98da5fee8251aadcf8f20b7aceef0d._comment b/posts/obnam-ubuntu-ppa/comment_2_9b98da5fee8251aadcf8f20b7aceef0d._comment new file mode 100644 index 0000000..9584f2a --- /dev/null +++ b/posts/obnam-ubuntu-ppa/comment_2_9b98da5fee8251aadcf8f20b7aceef0d._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="Erm?" + date="2011-07-28T20:06:04Z" + content=""" +That's a weird comment on a PPA announcement. :) + +I have something cooking, and if I get it done in time, I'll publish it. We'll see. Meanwhile, [Linux Anecdotes](http://liw.fi/linux-anecdotes/) is a related talk I gave in 1998. +"""]]
Added a comment: Hello Lars
diff --git a/posts/obnam-ubuntu-ppa/comment_1_0cc0de588f4d2aca986b95401e8b2869._comment b/posts/obnam-ubuntu-ppa/comment_1_0cc0de588f4d2aca986b95401e8b2869._comment new file mode 100644 index 0000000..9c71038 --- /dev/null +++ b/posts/obnam-ubuntu-ppa/comment_1_0cc0de588f4d2aca986b95401e8b2869._comment @@ -0,0 +1,11 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawn6FUCYCWFrPTd1nTcEhvJhQgsAKt2LD9A" + nickname="Tom" + subject="Hello Lars" + date="2011-07-28T19:38:20Z" + content=""" +Are you going to tell the story of Linux' birth from your POV for the 20th birthday? I think a lot of people would be really interested. + +thanks +Thomas +"""]]
Obnam Ubuntu PPA.
diff --git a/posts/obnam-ubuntu-ppa.mdwn b/posts/obnam-ubuntu-ppa.mdwn new file mode 100644 index 0000000..095cd1d --- /dev/null +++ b/posts/obnam-ubuntu-ppa.mdwn @@ -0,0 +1,14 @@ +[[!meta title="Obnam Ubuntu PPA"]] +[[!tag obnam ubuntu]] + +There is now an [Ubuntu PPA for +Obnam](https://launchpad.net/~chris-bigballofwax/+archive/obnam-ppa), +with packages uploaded by the awesome +Chris Cormack of [Koha](http://koha-community.org/) fame. + +His mail to the (very low volume) Obnam mailing list has details: +<http://lists.liw.fi/obnam@braawi.org/msg00013.html>. + +If you experience any problems with the packages, don't hesitate +to mail me, or the Obnam mailing list, or tell us on the +`#obnam` IRC channel on irc.oftc.net.
Obnam 0.18 announcement.
diff --git a/posts/obnam-0.18.mdwn b/posts/obnam-0.18.mdwn new file mode 100644 index 0000000..94eb108 --- /dev/null +++ b/posts/obnam-0.18.mdwn @@ -0,0 +1,51 @@ +[[!meta title="Obnam version 0.18"]] +[[!tag obnam]] + +I have released version 0.18 of [Obnam](http://braawi.org/obnam/), +my backup application. From the [NEWS](http://braawi.org/obnam/NEWS/) +file: + +* The repository format has again changed in an incompatible manner, + so you will need to re-backup everything again. (If this is a problem, + tell me, and I'll consider adding backwards compatibility before 1.0 + is released.) +* New option `--exclude-caches` allows automatic exclusion of cache + directories that are marked as such. +* Obnam now makes files in the repository be read-only, so that they're + that much harder to delete by mistake. +* Error message about files that can't be backed up now mentions the + correct file. +* Bugfix: unreadable files and directories no longer cause the backup + to fail. The problems are reported, but the backup continues. + Thanks to Jeff Epler for reporting the bug. +* Speed improvement from Jeff Epler for excluding files from backups. +* Various other speed improvements. +* Bugfix: restoring symlinks now works even if the symlink is restored + before its target. Also, the permissions of the symlink (rather than its + target) are now restored correctly. Thanks to Jeff Epler for an + exemplary bug report. +* New option `--one-file-system`, from Jeff Epler. +* New benchmarking tool `obnam-benchmark`, which is more flexible than + the old `run-benchmark`. +* When encrypting/decrypting data with GnuPG, temporary files are no + longer used. +* When verifying, `.../foo` and `.../foo/` now work the same way. +* New option `--symmetric-key-bits`. +* The chunk directory uses more hierarchy levels, and the way chunks + are stored there is now user-configurable (but you'll get into trouble + if you don't always use the same configuration). This should speed + things up a bit once the number of chunks grows very large. +* New `--chunkids-per-group` option, for yet more knobs to tweak when + searching for optimal performance. +* Local files are now opened using `O_NOATIME` so they can be backed + up without affecting timestamps. +* Now uses the `cliapp` framework for writing command line applications. + The primary user-visible effect is that the manpage now has an + accurate list of options. +* Bugfix: Obnam now again reports VFS I/O statistics. +* Bugfix: Obnam can again back up live data that is accessed using sftp. + Thanks to Tapani Tarvainen for reporting the problem. + +I alse made releases of some dependencies of Obnam. They're all in +my code.liw.fi repository, and some of them are also now in Debian +unstable.
Added a comment: problem solved 

diff --git a/posts/vmdebootstrap-2/comment_9_2e1de903aaa9e78e452abf2bb63ed873._comment b/posts/vmdebootstrap-2/comment_9_2e1de903aaa9e78e452abf2bb63ed873._comment new file mode 100644 index 0000000..9fa7d55 --- /dev/null +++ b/posts/vmdebootstrap-2/comment_9_2e1de903aaa9e78e452abf2bb63ed873._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmvt1RdNbCoHvKoxPnYNxvFw9228wYA_ms" + nickname="Angelo" + subject="problem solved :)" + date="2011-07-23T19:21:29Z" + content=""" +i have solved the problem with liw's help :) vmdebootstrap only works with extlinux from squeeze. the extlinux on sid and wheezy is broken. +"""]]
Added a comment: doesn't work for me
diff --git a/posts/vmdebootstrap-2/comment_8_2c7780f6d396759a072890a23d1d9284._comment b/posts/vmdebootstrap-2/comment_8_2c7780f6d396759a072890a23d1d9284._comment new file mode 100644 index 0000000..50fb18a --- /dev/null +++ b/posts/vmdebootstrap-2/comment_8_2c7780f6d396759a072890a23d1d9284._comment @@ -0,0 +1,23 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawmvt1RdNbCoHvKoxPnYNxvFw9228wYA_ms" + nickname="Angelo" + subject="doesn't work for me" + date="2011-07-23T18:51:39Z" + content=""" +Hi. + +I run everytime in this error: + + +Installing extlinux +EEEK! Something bad happened... +Cleaning up +ERROR: command failed: ['extlinux', '--install', '/tmp/tmp0IIOI0'] + +/tmp/tmp0IIOI0 is device /dev/mapper/loop0p1 + + +my os is a debian sid and I have installed all the dependencies + +here are the command I am using to create the VM's: ./vmdebootstrap --image test.img --size 2G --verbose --root-password=xxx --distribution=sid --log test.log +"""]]
Added a comment: Speeding up rm (from Russell Coker)
diff --git a/posts/rm-is-too-slow/comment_11_fdb86c0adae354e6b4294816cd258c54._comment b/posts/rm-is-too-slow/comment_11_fdb86c0adae354e6b4294816cd258c54._comment new file mode 100644 index 0000000..43e1ae7 --- /dev/null +++ b/posts/rm-is-too-slow/comment_11_fdb86c0adae354e6b4294816cd258c54._comment @@ -0,0 +1,30 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="Speeding up rm (from Russell Coker)" + date="2011-07-23T11:07:25Z" + content=""" +**Russell Coker sent me the following by e-mail, and said it is OK to add it to the blog as a comment. Thanks!** + +The problem with rm is going to be latency, as you note you have to read the +inode data before doing anything, this happens sequentially. A kernel API +that allowed multiple files to be unlinked at once wouldn't do any good unless +the filesystem driver supported batch operations, using a single transaction +for the meta-data changes and a single set of lookups to read the data. The +latency of user to kernel space is nothing compared to disk IO. + +One thing that could be done for a fast rm program would be to have a child +process stat each file while the parent unlinks them. You can synchronise +this by using a pipe and having the child write a byte for every file that is +stat'd and the parent read a byte for every file that is unlinked, this will +prevent the child from getting more than 8192 files ahead (and the kernel +dentry cache should contain a lot more than that). For a RAID array (which +I'm sure you are using given the size) this has the potential to provide some +significant benefits. For a single disk it probably won't do much good. + +At Steve Tweedie's suggestion I implemented something similar in the SE Linux +setfiles program and it provided a significant difference, however one thing +to note is that setfiles did a moderate amount of CPU operations which made it +more important to do the seeks for reading concurrently and gave benefits on a +single-CPU single-disk system. +"""]]
Added a comment: My rm is from squeeze.
diff --git a/posts/rm-is-too-slow/comment_10_36dc0dc3b76c4369f5ed6d05bcdc2694._comment b/posts/rm-is-too-slow/comment_10_36dc0dc3b76c4369f5ed6d05bcdc2694._comment new file mode 100644 index 0000000..712f526 --- /dev/null +++ b/posts/rm-is-too-slow/comment_10_36dc0dc3b76c4369f5ed6d05bcdc2694._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="My rm is from squeeze." + date="2011-07-22T16:25:29Z" + content=""" + ii coreutils 8.5-1 GNU core utilities + +Further, the filesystem is using `dir_index`. And, again, it's not a problem of a million files in one directory, it's a question of having 60+ million files and hardlinks to delete, period. +"""]]
Added a comment: rm 7.0 may be faster
diff --git a/posts/rm-is-too-slow/comment_9_b00e8ad7eaf8469556f8bf4c7f8658dc._comment b/posts/rm-is-too-slow/comment_9_b00e8ad7eaf8469556f8bf4c7f8658dc._comment new file mode 100644 index 0000000..879ebed --- /dev/null +++ b/posts/rm-is-too-slow/comment_9_b00e8ad7eaf8469556f8bf4c7f8658dc._comment @@ -0,0 +1,20 @@ +[[!comment format=mdwn + username="rwp" + ip="216.17.153.62" + subject="rm 7.0 may be faster" + date="2011-07-22T14:47:05Z" + content=""" +What version of rm is your machine running? Lenny has coreutils 6.10 while Squeeze has coreutils 8.5. Depending upon many things Squeeze may be significantly faster because of this change: +<pre> +* Noteworthy changes in release 7.0 (2008-10-05) [beta] + +** New features + + chgrp, chmod, chown, chcon, du, rm: now all display linear performance, + even when operating on million-entry directories on ext3 and ext4 file + systems. Before, they would exhibit O(N^2) performance, due to linear + per-entry seek time cost when operating on entries in readdir order. + Rm was improved directly, while the others inherit the improvement + from the newer version of fts in gnulib. +</pre> +"""]]
Added a comment: file removal is slow, not inode lookup
diff --git a/posts/rm-is-too-slow/comment_8_ef82ad097b5472ddb90ca4a89a88930a._comment b/posts/rm-is-too-slow/comment_8_ef82ad097b5472ddb90ca4a89a88930a._comment new file mode 100644 index 0000000..a9a948d --- /dev/null +++ b/posts/rm-is-too-slow/comment_8_ef82ad097b5472ddb90ca4a89a88930a._comment @@ -0,0 +1,10 @@ +[[!comment format=mdwn + username="cascardo" + ip="201.80.51.35" + subject="file removal is slow, not inode lookup" + date="2011-07-19T12:49:27Z" + content=""" +Hello, Lars. + +My guess is that the file removal itself is slow. As poisonbit has pointed out, rm is likely using unlinkat, which saves a lot of inode lookups. After the lookup, the filesystem has to mark blocks as unused, clean the inode and rewrite the directory table. And all that must go through the journal. Since it's metadata, it's likely it requires a journal that is flushed to disk. So, yes, changing filesystems may really help in this case, like you pointed out with ext4, for example. +"""]]
Added a comment: Tree, not single directory
diff --git a/posts/rm-is-too-slow/comment_7_814f3be0f79a2e71f6ee3b285095759c._comment b/posts/rm-is-too-slow/comment_7_814f3be0f79a2e71f6ee3b285095759c._comment new file mode 100644 index 0000000..3db5364 --- /dev/null +++ b/posts/rm-is-too-slow/comment_7_814f3be0f79a2e71f6ee3b285095759c._comment @@ -0,0 +1,12 @@ +[[!comment format=mdwn + username="http://liw.fi/" + nickname="Lars Wirzenius" + subject="Tree, not single directory" + date="2011-07-19T10:42:34Z" + content=""" +Mariusz, ah, a slight misunderstanding: the problem is removing lots of files spread over lots of directories, not a single directory with 60+ million files. + +juliank: Btrfs subvolumes (or LVM copy-on-write volumes, or other similar tricks) would perhaps work quite well, but only for the specific case of backups. There are lots of situations where you may need to delete lots of files, and where those tricks are not a suitable option. + +poisonbit: It's probably true rm is using only one system call per file. In fact, I would expect it to be. However, the total time spent deleting those files is ridiculously slow. My unsubstantiated guess would be that it is because every unlink (or unlinkat) call will need to do an inode lookup from the name, and that's a lot of repetitive work. Having a system call that would say \"unlink everything in this directory\" would give the kernel the option of avoiding the repetitive work, I suspect. +"""]]
Added a comment
diff --git a/posts/rm-is-too-slow/comment_6_ed56f00b2290a83c88e53de9f78b17c0._comment b/posts/rm-is-too-slow/comment_6_ed56f00b2290a83c88e53de9f78b17c0._comment new file mode 100644 index 0000000..350475e --- /dev/null +++ b/posts/rm-is-too-slow/comment_6_ed56f00b2290a83c88e53de9f78b17c0._comment @@ -0,0 +1,9 @@ +[[!comment format=mdwn + username="https://www.google.com/accounts/o8/id?id=AItOawlpM8Zxaq1W2BHytOX0M1u_bEAD_9Nj4Ts" + nickname="Mariusz" + subject="comment 6" + date="2011-07-19T09:53:45Z" + content=""" +That's not a solution, only workaround ;) if you can use subvolumes with btrfs you can just use subdirectories with any other FS, problem is \"a lot of files in one directory\" not \"a lot of files\" + +"""]]
Added a comment: btrfs subvolumes
diff --git a/posts/rm-is-too-slow/comment_5_12a7b7430b764471ff151fc13285adf3._comment b/posts/rm-is-too-slow/comment_5_12a7b7430b764471ff151fc13285adf3._comment new file mode 100644 index 0000000..abb6570 --- /dev/null +++ b/posts/rm-is-too-slow/comment_5_12a7b7430b764471ff151fc13285adf3._comment @@ -0,0 +1,8 @@ +[[!comment format=mdwn + username="http://juliank.wordpress.com/" + nickname="juliank" + subject="btrfs subvolumes" + date="2011-07-18T09:26:02Z" + content=""" +In btrfs, you don't have this problem, if each backup directory is a subvolume. You can then simply remove the sub volume which happens in constant time. +"""]]
Added a comment: Slow in relation with ?
diff --git a/posts/rm-is-too-slow/comment_4_4971dbcd63b73985e31c4e4f4632a68e._comment b/posts/rm-is-too-slow/comment_4_4971dbcd63b73985e31c4e4f4632a68e._comment
new file mode 100644
index 0000000..dc9fa77
--- /dev/null
+++ b/posts/rm-is-too-slow/comment_4_4971dbcd63b73985e31c4e4f4632a68e._comment
@@ -0,0 +1,22 @@
+[[!comment format=mdwn
+ username="http://poisonbit.wordpress.com/"
+ nickname="poisonbit"
+ subject="Slow in relation with ?"
+ date="2011-07-17T20:13:17Z"
+ content="""
+ # mkdir -p fooo{1,10,100}
+ # touch foo1/1
+ # touch foo10/{1..10}
+ # touch foo100/{1..100}
+ # strace -c rm -rf foo1 2>&1| awk '/total/{print $3}'
+ 52
+ # strace -c rm -rf foo10 2>&1| awk '/total/{print $3}'
+ 61
+ # strace -c rm -rf foo100 2>&1| awk '/total/{print $3}'
+ 151
+
+I see a sinlge *unlinkat* call more for each file... I think it's pretty efficient if I compare to other solutions...
+
+*rm* can take *multiple arguments*, and *xargs* can *parallelize*, but I think no *magic* will end under 1 system call per file, that is what currently *GNU/rm* uses for this filesystem operation.
+
+"""]]