Welcome to my web log. See the first post for an introduction. See the archive page for all posts. (There is an english language feed if you don’t want to see Finnish.)

Archives Tags Moderation policy Main site

Me on Mastodon, for anything that is too small to warrant a blog post.

All content outside of comments is copyrighted by Lars Wirzenius, and licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License. Comments are copyrighted by their authors. (No new comments are allowed.)


Unix command line conventions over time

ETA, 2022-05-19: I’m happy this blog post has gathered a fair bit of interest. However, this post is as much effort as I’m prepared to put into the topic. I think it would be a good idea to write an essay, article, or even a book, on how syntax of the Unix command line has varied over the years, and in different subcultures. Something semi-scholarly with cited sources for claims, and everything. I’d be happy to see this post be used as a basis: the CC license makes that easy. However, such a project would be quite a bit of work that I’m not interested in doing, I’m afraid.

This blog post documents my understanding of how the conventions for Unix command line syntax have evolved over time. It’s not properly sourced, and may well be quite wrong. I’ve not been using Unix until 1989, so I wasn’t there for the early years. Maybe someone has written a proper essay on this, with citations. I’m too lazy to dig them up.

Early 1970s

In the beginning, in the first year or so of Unix, an ideal was formed for what a Unix program would be like: it would be given some number of filenames as command line arguments, and it would read those. If no filenames were given, it would read the standard input. It would write its output to the standard output. There might be a small number of other, fixed, command line arguments. Options didn’t exist. This allowed programs to be easily combined: one program’s output could be the input of another.

There were, of course, variations. The echo command didn’t read anything. The cp, mv, and rm commands didn’t output anything. However, the “filter” was the ideal.

$ cat *.txt | wc

In the example above, the cat program reads all files with names with a .txt suffix, writes them to its standard output, which is then piped to the wc program, which reads its standard input (it wasn’t given any filenames) to count words. In short, the pipeline above counts words in all text files.

This was quite powerful. It was also very simple.

Options

Fairly quickly, the developers of Unix found that many programs would be more useful if the user could choose between minor variations of function. For example, the sort program could provide the option to order input lines without consideration to upper and lower case of text.

The command line option was added. This seems to have resulted in a bit of a philosophical discussion among the developers. Some were adamant against options, fearing the complexity it would bring, and others really liked them, for the convenience. The side favoring options won.

To make command line parsing easy to implement, options always started with a single dash, and consisted of a single character. Multiple options could be packed after one dash, so that foo -a -b -c could be shortened to foo -abc.

If not immediately, then soon after, an additional twist was added: some options required a value. For example, the sort program could be given the -kN option, where N is an integer specifying which word in a line would be used for sorting. The syntax for values was a little complicated: the value could follow the option letter as part of the same command line argument, or be the next argument. The following two commands thus mean the same thing:

$ sort -k1
$ sort -k 1

At this point, command line parsing became more than just iterating over the command line arguments. The dominant language for Unix was C, and a lot of programs implemented the command line parsing themselves. This was unfortunate, but at this stage the parsing was still sufficiently simple that most of them did it in sufficiently similar ways that it didn’t cause any serious problems. However, it was now the case that one often needed to check the manual, or experiment, to find out how a specific program was to be used.

Later on, Wikipedia says 1980, the C library function getopt was written. It became part of the Unix C standard library. It implemented the command line parsing described above. It was written in C, which at that time was quite a primitive programming language, and this resulted in a simplistic API. Part of that API is that if the user used an unknown option on the command line, the getopt function would return a question mark (?) as its value. Some programs would respond by writing out a short usage blurb. This led to -? being sometimes used to tell a program to show a help text.

Long options

In the late 1970s Unix spread from its birthplace, Bell Labs, to other places, mostly universities. Much experimentation followed. During the 1980s some changes to command line syntax happened. The biggest change here was long options: options whose name wasn’t just a single character. For example, in the new X window system, the -display option would be used to select which display to use for a GUI program.

Note the single dash. This clashed with the “clumping together” of single character option. Does -display mean which display to use, or the options -d -i -s -p -l -a -y clumped together? This depended on the program and how it decided to parse the options.

A further complication to parsing the command line was that single-dash long options that took values couldn’t allow the value to be part of the same command line argument. Thus, -display :0 (two words) was correct, but it could not be written as -display:0, because a simple C command line parser would have difficulty figuring out what was the option name and what was the option’s value. Thus, what previously might have been written as a single argument -d:0 now became two arguments.

The world did not end, but a little more complexity had landed in the world of Unix command line syntax.

The GNU project

The GNU project was first announced in 1983. It was to be an operating system similar to Unix. One of the changes it made was to command line syntax. GNU introduced another long option syntax, I believe to disambiguate the single-dash long option confusion with clumped single-character options.

Initially, GNU used the plus (+) to indicate a long option, but quickly changed to a double dash (--). This made it unambiguous whether a long option or clumped short options were being used.

I believe it was also GNU that introduced using the equals sign (=) to optionally add a value to a long option. Values to options could be optional: --color could mean the same as --color=auto, but you could also say --color=never if you didn’t like the default value.

GNU further allowed options to occur anywhere on the command line, not just at the beginning. This made things more convenient to the user.

GNU also wrote a C function, getopt_long, to unify command line parsing across the software produced by the project. I believe it supported the single-dash long options from the start. Some GNU programs, such as the C compiler, used those.

Thus, the following was acceptable:

$ grep -xi *.txt --regexp=foo --regexp bar

The example above clumps the short options -x and -i into one argument, and provided grep with two regular expression patterns, one with an equals, and one without.

The GNU changes have largely been adopted by other Unix variants. I’m sure those have had their own changes, but I’ve not followed them enough to know.

GNU also added standard options: almost every GNU program supports the options --help, --version, and --mail=ADDR.1

Double dash

Edited to add: Apparently the double-dash was supported already in about 1980 in the first version of getopt in Unix System III. Thank you to Chris Siebenmann.

Around this time, a further convention was added: an argument of two dashes only (--) as a way to say that no further options to the command being invoked would follow. I believe this was another GNU change, but I have no evidence.

This is useful to, say, be able to remove a file with name that starts with a dash:

$ rm -- -f

For rm, it was always possible to provide a fully qualified path, starting from the root directory, or to prefix the filename with a directory—rm ./-f—and so this convention is not necessary for removing files. However, given all GNU programs use the same function for command line parsing, rm gets it for free. Other Unix variants may not have that support, though, so users need to be careful.

The double dash is more useful for other situations, such as when invoking a program that invokes another program. An example is the cargo tool for the Rust language. To build and run a program and tell it to report its version, you would use the following command:

$ cargo run -- --version

Without the double dash, you would be telling cargo to report its version.

Subcommands

I think at around the late 1980s, subcommands were added to the Unix command line syntax conventions. Subcommands were a response to many Unix programs gaining a large number of “options” that were in fact not optional at all, and were really commands. Thus a program might have “options” --decrypt and --encrypt, and the user was required to use one of them, but not both. This turned out to be a little hard for many people to deal with, and subcommands were a simplification. Instead of using option syntax for commands, just require commands instead.

I believe the oldest program that uses subcommand is the version control system SCCS, from 1972, but I haven’t been able to find out which version added subcommands. Another version control system, CVS, from 1990, seems to have had them the beginning. CVS was built on top of yet another version control system, RCS, which had programs such as ci for “check in”, and co for “check out”. CVS had a single program, with subcommands:

$ cvs ci ...
$ cvs co ...

Later version control systems, such as Subversion, Arch, and Git, follow the subcommand pattern. Version control systems seem to inherently require the user to do a number of distinct operations, which fits the subcommand style well, and also avoids adding large numbers of individual programs (commands) to the shell, reducing name collisions.

Subcommands add further complications to command line syntax, though, when inevitably combined with options. The main command may have options (often called “global options”), but so can subcommands. When options can occur anywhere on the command line, is --version a global option, or specific to a subcommand? Worse, how does a program parse a command line? If an option is specific to a subcommand, the parsing needs to know which subcommand, if only so it knows whether the options requires a value or not.

To solve this, some programs require global options to be before the subcommand, which is easy to implement. Others allow them anywhere. Everything seems to require per-subcommand options to come after the subcommand.

Summary

The early Unix developers who feared complexity were right, but also wrong. It would be intolerable to have to have a separate program for every combination of a program with options. To be fair, I don’t think that’s what they would’ve advocated: instead, I think, they would’ve advocated tools that can be combined, and to simplify things so that fewer tools are needed.

That’s not what happened, alas, and we live in a world with a bit more complexity than is strictly speaking needed. If we were re-designing Unix from scratch, and didn’t need to be backwards compatible, we could introduce a completely new syntax that is systematic, easy to remember, easy to use, and easy to implement. Alas.

None of this explains dd.


  1. The --email bit is a joke.↩︎

Release notes: why and how?

Every free and open source project should probably make a release from time to time. Every release should be accompanied by release notes so that people interested in the software know what has changed, and what to expect from the new version.

Release notes can be massive enough to sub-projects of their own. This can be necessary for very large projects, such as Linux distributions. For projects more on the individual developer scale, release notes can be small and simple. My own preference is to include a file called NEWS (or NEWS.md) in the source tree, which lists the major, user-visible changes in each release. I structure mine as a reverse log file, pre-pending the newest release at the beginning. I don’t delete old releases, unless the file grows very big.

I am not fond of a “git commit log” as release notes. As a user I am really not interested in the meandering route by which the developers came to the new version. It may be possible to produce release notes by filtering strictly structured commit messages, but I’ve not seen it done well enough. Your experience may be different.

As a user, what I want to see in release notes are:

  • a list of new or changed features, with short descriptions of why I would want to use them
  • a list of fixed bugs, with short explanations of what the problem was so I know if I have been affected
  • a list of breaking changes, whether features or bug fixes, with some indication of what is needed to cope with the breakage, so I can make an informed decision of when and how to upgrade
  • anything else I as a user, system administrator, or packager need to know about, when upgrading

If I contribute to the project and want more detail, I’ll be happy to look at the actual version control history, with detailed commit messages and unified diffs of each change.

Examples from my projects:

  • Subplot
    • Subplot is tooling for documenting and verifying acceptance criteria in ways that all project stakeholders understand them
  • Obnam
    • Obnam is an encrypting backup program
  • vmdb2
    • vmdb2 builds virtual machine images with Debian installed
On Linus's Law

Linus’s Law is “given enough eyeballs, all bugs are shallow”. A lot of people seem to interpret this as follows:

The more people use some software, the faster any bugs in it will be found.

They are then upset when some popular software has bugs. How can that be, when Linus’s Law says they’re shallow?!

I think this interpretation is wrong, and have always instead understood it to mean this:

Once a bug is known to exist, the more people are looking for what causes it, the faster it is likely to be found, and then fixed.

The law is from Eric Raymond’s “Cathedral and the Bazaar”, see the Release early, release often section, where the full description is:

Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone

I think that supports my interpretation.

See also Wiio’s laws.

Posted
A year of README reviews

A year ago I published an offer to review the README of free and open source projects. I didn’t expect much interest, but someone posted the link to Hacker News, and I got enough requests that it was a little overwhelming. I’ve now reviewed 196 READMEs, my queue is empty, and I’m suspending the offer for now, even if it has been fun.

I especially wanted to help people new to FOSS development, and I know that for one’s first project it can be scary to open oneself up for critique. Thus I made it possible to request a review in private, and sent my feedback in private, and I haven’t published the projects I reviewed. However, I feel it might be of interest to read a summary of my experience doing this.

Overall, I was pleasantly surprised at how good the READMEs were. People put in a lot of effort into them. There might be some selection bias here: someone who doesn’t care about making a good README probably won’t ask for a review, either.

It seems to be common now to have a screenshot or an animated GIF of the user interface of software, in addition to a textual description. I find this to be an excellent idea: it gives the reader an immediate look at what the software is like. For code libraries examples of using the library serves the same purpose. A link to a demo is often quite illuminating to the reader.

I find it quite useful for a README to start with an explanation of what the software does, and what one could achieve with it. It seems worth it to me to be explicit and not assume the reader will figure it out themselves.

The READMEs I was asked to review were divided into roughly four categories:

  • A short calling card that just tells you what the aim of the project is.
  • A “getting started using the software” type of document.
  • A more thorough introduction to the project as a project: who works on it, what it’s background is, how to contribute, etc.
  • An in-depth manual.

I tend to prefer the shorter approaches, but it is a matter of taste and goal.

It’s important to consider the audience. For a README, the reader may land on it via a web search, and might not know the topic area of the project. Even so, they might be interested in it, once they do understand. I find it useful for a README to explain things to a random passer-by in a few words, or to link to an explanation elsewhere. As an example, if a README says the project is a library of some kind, it may be useful to say it’s a code library rather than, say, a collection of cultural works. The sooner one makes that clear, the less time others will spend figuring out if the project is what they’re looking for.

I’m suspending my offer to review READMEs for now. I feel this is something that anyone could do: you, too, can do it. Why don’t you? It’s easy, and it turns out people appreciate it.

v-i 0.1, a Debian installer using vmdb2

I’ve published release 0.1 of v-i, my installer for Debian based on vmdb2. It’s entirely non-interactive, dangerous, unhelpful, but fairly fast. It will remove all data from your drives. See the initital announcement for more, or the README. There is now a pre-built image available, see the tutorial.

I’d be curious to hear if anyone else is interested in this.

This is the target spec file I use for my VM host:

drive: /dev/nvme0n1
extra_drives:
  - /dev/nvme1n1
hostname: exolobe5
extra_playbooks:
  - exolobe5-ansible.yml
ansible_vars:
  user_pub: |
   ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPQe6lsTapAxiwhhEeE/ixuK+5N8esCsMWoekQqjtxjP liw personal systems
extra_lvs:
  - name: vms
    size: 1T
    fstype: ext4
    mounted: /mnt/vms
Insert a million rows into Sqlite, from Rust

For my backup program, Obnam, I needed to find a way to insert many rows into an SQLite database table as quickly as possible. Obnam uses an SQLite database, in its own file, for each backup generation, to store the names and other metadata about the backed up files. The file content is stored separately. Thus, one row per backed up file, and I have over a million files.

Here’s where I started from, a very simple, obvious approach:

for i in 0..N {
    let t = conn.transaction()?;
    t.execute(
        "INSERT INTO files (fileno, filename, json, reason, is_cachedir_tag) VALUES (?1, ?2, ?3, ?4, ?5)",
        params![i, filename, "", "", false],
    )?;
}

I looked on the web for suggestions for making this faster. I found a Jason Wyatt article with some good tips. Specifically:

  • use one transaction, not one transaction per insert
    • in Obnam, either all inserts succeed, or the backup fails
    • thus, transactions aren’t needed for this case
    • for me, this almost doubled the speed
  • use prepared statements
    • for me, this was an almost 5x speed up

Here is what I ended up with:

let mut stmt = conn.prepare_cached("INSERT INTO files (fileno, filename, json, reason, is_cachedir_tag) VALUES (?1, ?2, ?3, ?4, ?5)")?;
let t = conn.unchecked_transaction()?;
for i in 0..N {
    let filename = format!("file-{}", i);
    stmt.execute(params![i, filename, "", "", false])?;
}
t.commit()?;

The results:

  • initial simple approach: 117509 inserts/s
  • one transaction: 209512 inserts/s
  • also prepared: 970874 inserts/s

It might be possible to improve on this further, but with this is already so fast it’s not even close to being a bottleneck for Obnam.

I then re-did the Obnam database abstraction, and it improves the speed for Obnam in one benchmark quite dramatically. The merge request is under review now.

Subplot talk at FOSDEM

Daniel gave an online talk at FOSDEM about Subplot, our tool for documenting acceptance criteria and their verification. I helped in the accompanying chat. The video is now public on Youtube.

We also have a Matrix room for Subplot now, as well as a new domain for the project: subplot.tech.

If acceptance testing, integration testing, or documenting either, interests you at all, please have a look!

Posted
Security ratchet

A ratchet is a wonderful device and a powerful metaphor. For the device, from Wikipedia:

A ratchet (occasionally spelled rachet) is a mechanical device that allows continuous linear or rotary motion in only one direction while preventing motion in the opposite direction. Ratchets are widely used in machinery and tools. The word ratchet is also used informally to refer to a ratcheting socket wrench.

The Wikipedia page has illustrations and even an animation (used below). The operating principle is simple enough.

animated image of a simple ratchet

The metaphor is similar: circumstances are set up so that if you do a thing, your situation changes, and it’s hard to go back. As an example, if you buy a machine that lets you produce more widgets per month, you can use the profit from selling those widgets to buy a second machine, and then a third, and so on. As long as you can sell all the widgets you produce, buying machines is a way to turn your weatlh ratchet. Seth Godin, the author, podcaster, entrepreneur, and marketing thinker, tends to use the ratchet metaphor often.

I’ve become fond of using the metaphor for computer security. If you make a computer or system more secure, that’s an improvement, even if the system is still not perfectly secure. Once you’ve made a small improvement, you can build on that to make the next improvement. Turn by turn, the ratchet tightens your security more.

I think of the security ratchet every time I talk to one of the security-minded geeks for whom security must be absolute lest it be pointless. You may have encountered such: there’s not point in using PGP unless you keep your private key on a hardware token, inside a vault, buried under a mountain, guarded by former US Navy Seals turned mercenaries.

Whereas I think using PGP at all is an improvement over not using it, even if you use a weak key without a passphrase. Once you’ve got that far, you can, if you want or need to, easily turn the ratchet a few times: add a passphrase, switch to a stronger key, use a hardware security module to store it, use a new subkey for every encrypted file, and so on.

PGP is an example here. If you don’t like PGP, consider SSH instead. Using SSH at all is better than using rsh or telnet, even if you still user passwords to authenticate. Turning the ratchet improves things: use keys, use stronger keys, use keys that require a hardware security token, use SSH CA user certificates. And so on.

The goal is to have a threat model, implement defenses based on that model, and then iterate to improve as needed.

Posted
My split external brain

I use a laptop as my primary computer: it’s hooked to my nice keyboard, my nice mouse, and my two monitors. It’s a laptop so that if I need to go on a trip, it’s easy for me to bring my external brain with me. (Admittedly, this is less relevant than it used to be, but this setup is a few years old by now, and it’s also plausible that I’ll need to travel for work again some day.)

The laptop is a fairly nice computer, especially for a laptop. However, I also have really fast desktop computer, without a monitor or keyboard, in a corner of my office, acting as a server. It has a lot more CPU and memory, and way faster storage, than my laptop does. The speed difference between these two machines is quite remarkable.

I use the desktop for heavy tasks, such as compiling Rust programs and running test suites. The desktop has a separate virtual machine for each of my projects, set up just-so for that project. Some of my projects want a specific version of Debian, others need dependencies installed. It’s convenient for me to not have to try to satisfy the environmental requirements for all my projects with one system.

I can create and configure each environment with two commands: one to create it, and one to run Ansible to configure it. I could change my tooling to allow me to achieve both of those in one command. I’ll do that when the two-command approach starts chafing.

It takes me a minute or two to create a new virtual machine from scratch, but it’s all just waiting. I tend to make sure I have tea in my mug during that time.

The actual building and test running is done using a little script that copies my source to the target virtual machine, with rsync, and then runs a command there, with SSH. This turns out to have very little overhead compared to just running the command locally: only a fraction of a second. The build times on the faster machine would make up for the transfer overhead even if it was significantly larger, though.

(I’m not fond of container tooling, so I prefer virtual machines over containers. It’s my external brain, so I get to decide.)

Posted