This month in Radicle CI, March 2026

This is a monthly newsletter about the current state of Radicle CI, what has happened recently, and near future plans.

Current status

Radicle CI is in production use. There are several CI nodes, and Lars runs a public one for open source Rust projects at https://callisto.liw.fi/.

Radicle CI use metrics

I now track Radicle CI use by counting job COBs that reach a public seed node run by Adrian. In the table below, "day" is day offset since the tracking started in early March. The data below excludes CI runs on my own CI nodes, to avoid skewing the numbers. I run CI a lot, as part of developing Radicle CI.

I'm not sure I trust these numbers, yet. There's weirdness in the data, such as counts going downward, which I haven't investigated yet.

The data collection and reporting is done by job-cob-counts.

Issues and patches

As of 2026-03-16.

Office hours

The Radicle CI office hour is a place where anyone can show up and ask questions about Radicle CI or Radicle or anything related to them. Sometimes Lars or someone else will have a demo or a short talk.

This happens weekly at https://meet.ffmuc.net/radicleciofficehour. It's Jitsi and you only need a web browser and mic, camera is optional.

Each office hour is announced on #Annoncements and #radicle-ci on the Radicle Zulip, and on Lars' fediverse account (@liw@toot.liw.fi).

Upcoming:

• March 18 at 17:00 UTC
• March 25 at 08:00 UTC
• April 1 at 17:00 UTC
• April 8 at 08:00 UTC
• April 15 at 17:00 UTC

Overview of the past month

For the next month, Lars has worked on:

• Tooling collect data on CI runs in Radicle repositories and report on it.
• Improving Ambient run logs.
• Supporting generic cloud images.
• Download deb package dependencies in Ambient.
• Download Rust toolchains for use in the Ambient VM.
• Preparation for talk on Radicle CI at the foss-north 2026 conference late April.

Lars meant to also work on improving Radicle CI installation and reproducible builds, but didn't. These will be picked up again in the future.

Radicle CI component releases

Ambient CI 0.13.0, released 2026-03-04

Other changes:

• The HTML run log has been improved in various ways.

  • Custom action output is included.
  • Log messages say which part of a CI run they come from.
  • The log page is also divided into sections, based on log message source.
  • The HTML log uses synthesized log message to hide repetitive steps like "prepare to create directory" followed by "successfully created directory". The user now only sees "created directory" by default, but can click that open to see all the details. As part of this change, the library part for non-synthetic log messages is gone.
  • The raw JSON log messages are included in the HTML log, just in case they are helpful some day.
  • The VM console log is included in the HTML log to satisfy curious people.
  • Log messages and sections have HTML element IDs to allow hyper- to them.

• The HTML log now passes the W3C HTML validator.

• The virtual machine console log is captured in the project state directory and HTML run log. In the Ambient crate library part, the ambient_ci::qemu::QemuBuilder now requires setting the path to the console log.

• Ambient now has a lint for plan having a deb action and post-plan having a dput2 action or deb2 and dput. The version of the actions are incompatible: they put the built packages in different directories. The lint catches this when a CI run starts, making the problem evident quickly.

• There is also a lint for shell actions referring to the old name for the workspace, /workspace, instead of the new one, /ci. The old name will work until further notice, but the lint is there to nudge people to switch so that some day the old name can be retired safely.

• The Debian packaging now depends on shellcheck, for the linting Ambient does for shell actions.

• The main error type is now called AmbientError, renamed from AmbientDriverError. The old name referred to an old name for Ambient.

Ambient CI 0.13.1, released 2026-03-06

• A bug that caused a panic in how the sequence of run log messages is parsed to create synthetic log messages has been fixed.

Radicle CI broker 0.26.0, released 2026-03-06

• Defelo added support for the LocalRefsAnnounced events. This means you can CI on your local machine and its Radicle node. If you push a change to your node, CI runs. This means it's not necessary to set up a separate CI node. A separate CI node is and will always be supported, though.

• The CI broker now depends on crates for Radicle 1.17, which is about to be released.

• The installation documentation on a Debian system have been updated to refer to the Radicle APT repository, with release versions. Thank you to Yorgos for finding the problem.

Ambient adapter 0.15.0, released 2026-03-06

• Depend on Ambient 0.13.1.

• The repository name is now in the HTML log page title.

• The HTML log has many changes that are meant to be usability improvements, which mostly come from Ambient CI 0.13.

Radicle native CI adapter 0.13.0, released 2026-03-06

• No user visible changes, as such, but depends on new Radicle and CI broker.

rad-ci 0.8.0, released 2026-03-06

• No user-visible changes, but depends on new versions of Radicle and CI components.

Goals for 2026 (unchanged)

The primary goal for Radicle CI in 2026 is to get more people using it. To achieve this, much work is needed to improve awareness, documentation, installation, configuration, and operation. Some feature work and bug fixing will also be needed. While the basics work and are in use, things could be much more polished.

In addition, a supporting goal is to start tracking how much Radicle CI is used, based on job COBs.

Future plans

For the next month, Lars plans to work on:

• Support for more language and packaging ecosystems for Ambient.
  • npm at least
• Improve remote troubleshooting.
  • include adapter stderr in CI broker report pages
• Improve Ambient image handling.
  • allow Ambient to manage a set of images and projects to declare the image (and version) they want to use
• Add a fake network interface to Ambient to make generic cloud images boot faster.
• Making installation of Radicle CI easier.
  • systemd units for the components
  • a shell script to provision a fresh Debian host as a CI node
• Improve development process so in-progress versions of Radicle CI components can be tested together.
• Prepare a demo for the foss-north conference at end of April.
• Make releases of CI components so that recent improvements can be deployed.

Notes

If you have an open source Rust project and want to try Radicle and Ambient on callisto, see https://callisto.liw.fi/callisto/ for instructions.

Once Ambient supports more than Rust, callisto will accept non-Rust projects, too. But still only open source projects.

Links

• Radicle home page
• Radicle CI home page, unofficial
• Radicle CI broker repository
• Radicle CI Ambient adapter repository
• Radicle CI integrations documentation

My computing life has often been difficult and complicated. I've gone against the mainstream for most of it. In the very early 1990s I chose to use Linux, when it was very new and have stuck with it. I could've chosen MS-DOS and then Windows, or a Mac, and had the benefit of more things working more of the time. It would have been easier to collaborate with people. It would have been easier to do many things I either couldn't realistically do, had to wait for someone to build software for, or had to build it myself. I might have done more ambitious, technically interesting things.

If I and others like me had chose differently, Linux and other free operating systems would probably have withered, never gaining much traction. Everyone would be much more strongly controlled by Microsoft and Apple.

Thanks to thousands of people, and to a small degree I, spending a lot of time and effort and being stubborn and contrarian, software freedom is still here, despite the best efforts of monopolists to kill it.

I am happy I made this choice.

A Rust function that takes a string argument might look like this:

fn foo(s: &str) { ... }

This works well, if the the function merely uses the string. If the string is needed after the function returns, e.g., because it's stored somewhere, either the function needs to manage lifetimes, or make a copy. Correctly managed lifetimes are good for avoiding unnecessary copies, but they can be tricky to manage.

Sometimes a new value is necessary. There are three option:

• fn foo(s: &str)
• fn foo(s: String)
• fn foo(s: impl Into<String>)

In the first case, the function creates the new value itself. This is convenient for the caller, but can cause an unncessary new value to be made if the caller already has one it can give. This matters if there's a need to manage memory tightly.

In the second case the caller creates the new value, if a new one is needed. This avoids creating unnecessary new values, but is inconvenient to the caller.

In the third case, we make use of Rust monomorphism and traits and have the Rust compiler generate code that creates a new value, if one is needed. This makes it possible to be both convenient and manage memory tightly.

I learnt this trick from Daniel.

This month in Radicle CI, 2026-02

This is a monthly newsletter about the current state of Radicle CI, what has happened recently, and near future plans.

Current status

Radicle CI is in production use. There are several CI nodes, and Lars runs a public one for open source Rust projects at https://callisto.liw.fi/.

Issues and patches

As of 2026-02-17.

Office hours

The Radicle CI office hour is a place where anyone can show up and ask questions about Radicle CI or Radicle or anything related to them. Sometimes Lars or someone else will have a demo or a short talk.

This happens weekly at https://meet.ffmuc.net/radicleciofficehour. It's Jitsi and you only need a web browser and mic, camera is optional.

Each office hour is announced on #Annoncements and #radicle-ci on the Radicle Zulip, and on Lars' fediverse account (@liw@toot.liw.fi).

Upcoming:

• February 18 at 17 UTC
• February 25 at 08 UTC
• March 4 at 17 UTC
• March 11 at 08 UTC

Overview of the past month

Ambient CI version 0.12.0, released 2026-02-05

From NEWS.md:

New feature:

• Ambient (ambient run) now produces run logs in JSON Lines format (one JSON object per line of output). This can be turned into HTML pages using ambient log --format=html. The HTML page is supposed to easier to navigate than tens to thousands of kilobytes of unstructured text. It is expected that the HTML logs can still be improved. They are included in a release so that the Ambient developers can get feedback on how to improve them.

  The parts where things fail are colored pink in the HTML log. Advice on better colors and other ways to distinguish is most welcome.

  To see the new logs, see https://callisto.liw.fi/ after this release has been deployed there. That should happen within an hour or two after it's been uploaded to crates.io.

  Old run logs (run.log in the project state directory) are no longer understood by this version of Ambient.

Changed features:

• The deb and deb2 actions log the package directory they create or that it already exists. This helps debugging via CI run logs.

Bug fixes:

• In the deb2 action, if the /ci/artifacts/debian directory doesn't exist, create it. Although the deb and deb2 actions share code, this doesn't change anything for deb, as it puts the built package in /ci/artifacts, which always exists.

The end of year holidays happened, and fairly little has happened in Radicle CI development over the past month.

Goals for 2026

The primary goal for Radicle CI in 2026 is to get more people using it. To achieve this, much work is needed to improve awareness, documentation, installation, configuration, and operation. Some feature work and bug fixing will also be needed. While the basics work and are in use, things could be much more polished.

In addition, a supporting goal is to start tracking how much Radicle CI is used, based on job COBs.

Future plans

For the next month, Lars plans to work on:

• A prototype for a new tool to collect data on CI runs in Radicle repositories.
• Improve Ambient run logs.
• Improve installation of Radicle CI with Ambient.
• Change Ambient to support generic cloud images well.
• Make Radicle CI components build in a reproducible way.

Support for generic cloud images means using an image meant to run a virtual machine in the cloud with Ambient. Many Linux distributions publish such images. They are meant to be set up on first boot using cloud-init, which Ambient supports. However, there are missing bit that make using them not a good option yet.

Notes

If you have an open source Rust project and want to try Radicle and Ambient on callisto, see https://callisto.liw.fi/callisto/ for instructions.

Links

• Radicle home page
• Radicle CI home page, unofficial
• Radicle CI broker repository
• Radicle CI Ambient adapter repository
• Radicle CI integrations documentation

Goal

Last time I added positional parameters to directives. Since then I've added a type to represent shortcuts. The goal for today is to add metadata to the Site type, which will include site shortcuts.

Plan

• Add a SiteMetadata type, initially empty, and corresponding field to Site.
• Change directives so they get a reference to Site so that they have a way to update site metadata.
• Add shortcuts to site metadata.
• Change shortcut directive so it adds the shortcut to site metadata.

Notes

• Added the SiteMetadata type. It's currently empty, as that's enough to play with the Rust type system and to get interfaces right.
• Tried passing a reference to a mutable Site, but this runs into Rust borrow checking rules. The problem is that Site::process needs to mutate itself but also let a directive do that. That's what the Rust rules are meant to prevent.
• I can work around that in various ways, such as providing internal mutability for a site. Except that also fails: a site asks a page to mutate itself, and needs to give it a reference to the site. The mutable page reference is to a mutable site, and then there's both a mutable and immutable site reference at the same time.
• An interesting problem, too, is that there's a riki render-page command that operates on an individual page that is not part of a site. I'll just drop that, first.
• Ah! At this time I only need to modify site metadata when processing the shortcut directive, and that only needs to do that at the very start. In general, I suspect directives don't need to do that.
• I'll add a prepare method to the DirectiveImpl trait that only shortcut will implement. Every other directive will get a default no-op implementation provided for them.
• This also means the site processing will iterate over pages twice. Once to call the prepare method, and other to call the process method.
• Further, when looking up directives that match invocations, we'll have to check shortcuts, too, if a directive doesn't match.
• Created a test site that defines and uses a shortcut, for manual testing, while I experiment.
• Hm, the prepare step also runs into the problem if mixing mutable and immutable references. Of course it does. I need to do a rethink.
• Hmm.
• Hmm.
• Repeat a few times while I do the dishes.
• I can sidestep the mixed types of references by making a copy (clone) of the site metadata, passing a mutable reference to that to `Page::prepare, and copying back into the site. It's not entirely elegant, but it's good enough.
• Phew. I almost felt like the borrow checker was going to hold a grudge against me.
• Next I'll make Page::prepare actually do wikitext parsing and call directive implementations that match invocations, but ignore ones that don't.
• Another borrow checking situation: if I want to store the wikitext parsing result in the page, I need to make the method get a mutable reference to self. Luckily the sidestep makes this not be a problem. So very exciting!
• Added a prepare step to directives. Called it at the opportune time.
• Hmm, the shortcut directive can have both positional and named parameters. Will need to fix the invocation code, but I'll stick with just positionals, for now.
• After some tired debugging, got things to work.
• But now I'm too tired to add a test suite for this. The test suite will need to create a test site that defines and uses a shortcut and build the HTML and make sure the shortcut is expanded correctly.
• I'll merge this but open issues to fix the desc parameter and add a test suite.

Summary

This was more complicated than I wished, but the way the shortcut directive interacts with other directives and site building turned out to be complicated.

Goal

Last time I tried, but failed, to implement the shortcut directive. I failed, because I was trying to change many things at a time. One of the problems was that riki currently assumes parameters to directives are always named. That is not actually true for all directives, including shortcut.

My goal today is to change riki so that parameters can be named without a value, valued without a name, or be name/value pairs.

I'll have succeded if riki can handle all the following forms, as far as parsing is concerned. I don't care about actually implementing the directives yet.

• [[!foo bar]] - positional parameter, could be name or value
• [[!foo yo=yoyo]] - name/value parameter

The usual unquoted, quoted, or triple quoted variants for positional parameters or values are expected. Parsing the variants is already implemented.

Plan

• Write down a detailed prose specification of how parameters should work. Add it to doci/directifes.md.
• Start implementing the specification, first in the wikitext parser.

Notes

Specification

• At wikitext parsing time, name/value pairs are easy to recognize unambiguously.
• A quoted value without a name can also be distinguished.
• However, if a parameter that is only a name or an unquoted value, the parser doesn't know which it is. The directive implementation needs to do that.
• There are thus three cases:
  • a name/value pair
  • a quoted value
  • a name or unquoted value
• Wrote that up in doc/directives.md in informal prose.

Wikitext parser

• The current struct Parameter type assumes each parameter has a name. It does not allow for positional parameters. This is the cause of all the pain in this area.
• I will change that to be an enum with variants for name/value pairs and positional parameters. The wikitext parser needs to be changed too, to allow for quoted or triple quoted values without names as parameters. And the directive module needs to be changed for the new enum.
• This is too many changes at once. I'll do the minimum, which is change the parameter type into an enum, but not the parser. I'll gut the directive module parameter handling, for now. I'll need to change all the directive stuff anyway for the parameter handling changes.
• That made things easier.
• Adding parsing of value-only parameters. Easy enough, but reminded that I still don't have a good way to debug winnow parsers, except adding prints in opportune places.

Directives

• Most directives only have named parameters. They might not have values.
• Others will have only positional parameters, such as tag.
• For full generality, a directive needs to be able to iterate over the parameters, regardless of kind.
  • but I'll add this only once it's needed
• The directive implementations need to easily check errors in parameters.
• I may later add a declarative way for directives to specify what parameters they support, but for now I'll stick with repetitive code.
• I may want to later distinguish between unnamed quoted and unquoted values. For now, the quoted value is treated as if it were a name.

Summary

Making one change at a time sure does make things easier. Merged.

This month in Radicle CI, 2026-01

This is a monthly newsletter about the current state of Radicle CI, what has happened recently, and near future plans.

Current status

Radicle CI is in production use. There are several CI nodes, and Lars runs a public one for open source Rust projects at https://callisto.liw.fi/.

After the Ambient release in December, the Ambient adapter for Radicle CI needs a release to allow the new actions to work. CI plans that don't use the new actions work fine.

Links

• Radicle home page
• Radicle CI home page, unofficial
• Radicle CI broker repository
• Radicle CI Ambient adapter repository
• Radicle CI integrations documentation

Office hours

• Lars holds a weekly office hour for Radicle CI. Alternating weeks at 08 or 17 UTC at https://meet.ffmuc.net/radicleciofficehour (it's Jitsi and you only need a web browser and mic). Each office hour is announce on #radicle-ci on the Radicle Zulip, and on Lars' fediverse account. The office hour is a place where anyone can show up and ask questions about Radicle CI or Radicle or anything related to them. Sometimes Lars or someone else will have a demo or a short talk.

  The next ones are January 23 at 17:00 UTC, and January 30 and 08:00 UTC.

The past month

The end of year holidays happened, and fairly little has happened in Radicle CI development over the past month.

Radicle CI broker version 0.24.0 on 2025-12-09

From NEWS.md:

• The way cib looks up job COBs for specific commits has been optimized to use a cache. This speeds things up a lot in some circumstances: in one benchmark, creating 100 new runs for a commit went down from 600 seconds to 40.

• The log messages for a CI run, at the INFO level, are now easier to follow without being familiar with the code base. Especially the decision on whether an event filter should trigger CI to run is logged as one message and contains the whole decision tree as JSON.

• When it loads the configuration file, cib now checks that a default adapter is set, if the filters field is used. Previously a missing default adapter was only noticed later when starting a CI run. The new behavior exposes problems much earlier.

Ambient CI version 0.11.0 on 2025-12-20

From NEWS.md:

Breaking changes

• The Rust Cargo related actions now use the cargo-target subdirectory in the workspace cache directory (/ci/cache/cargo-target). This is invisible to all common uses of Rust in Ambient projects, but allows dividing the cache between different kinds of use in the future. This does mean that existing caches become obsolete and should be deleted. That makes this technically a breaking change.

• Ambient now checks for common problems when it loads the projects file. This is know as "the linter". Currently it checks that an rsync target has been configured if an rsync or rsync2 action is used, that each file downloaded by a http_get action has a unique filename, and that all shell script snippets in shell actions are OK according to the shellcheck program.

  Linting can be prevented by setting lint to false in the configuration. This may be necessary if, say, one of the checks is wrong.

  This is a breaking change because most shell script snippets will be found wanting.

• The pre- and post-plan now only allow actions that are actually meant for them. Previously both allowed the same set of actions. However, actions like cargo_fetch don't really make sense for post-plan. Now the separation is more strict.

  This is technically a breaking change, but hopefully doesn't actually break anything for anyone. If you have a legitimate use for a pre- or post-plan action that is now not allowed, let us know.

Problems fixed

• Some portability fixes for NixOS by invoking the Bash shell by name instead of path. NixOS does not put Bash at /bin/bash, so using a full path doesn't work reliably. Using the name should work anywhere.

• Ambient now checks, when loading the projects file, that the source location for a project is a directory (following symlinks) and gives an error if it is not. This means the problem is found when Ambient starts and not much later when it starts running CI for a specific project. If there are many projects, that might be hours later.

New features added

• The workspace in the VM is now /ci. The old name /workspace will work indefinitely. The new name is shorter and arguably clearer. The workspace is set up by ambient-execute-plan and so this change does not affect any VM images.

• There is now a user guide for Ambient, published by CI at https://doc.liw.fi/ambient-ci/userguide.html and included in the deb package at /usr/share/doc/ambient-ci/userguide.html. The user guide contains a description of each action that a project CI plan can use. The guide is woefully lacking, but it's easier to add things to something that exists than starting from an empty directory.

• The Ambient subplot document is formatted and published at https://doc.liw.fi/ambient-ci/ambient.html. It may be useful for checking how a specific aspect of Ambient is used. The subplot is the test suite that verifies most aspects. That means it's continually run and does not easily get out of date.

• New action setenv allows setting environment variables for later actions. Using the shell action does not work for this, because each shell forgets any changes to its environment variables when it terminates.

  - action: setenv
    set:
      foo: bar

• New plan action deb2 and post-plan actions dput2 and rsync2 use subdirectories debian and rsync in the artifacts directory. This means if a project both builds documentation and a deb package, they don't get mixed into the same directory. Instead, documentation goes into /ci/artifacts/rsync and the package into /ci/artifacts/debian.

  The old actions deb, dput, and rsync continue to work as before and use the whole artifacts directory. The new actions were added to avoid changing the existing actions in an incompatible, breaking way. The old actions are not deprecated.

  The runnable plan versions of the old actions have changed. The plan and post-plan actions result in the same runnable plan action. Changes to runnable actions are not currently considered breaking changes in Ambient.

• In the VM, the git command is now configured by default to have "Ambient CI" as the name for the user and "ambient@example.com" as the email address. This removes the need for each project to do that in their CI plan just to use git.

• New subcommand ambient state lists the projects in the Ambient state directory (configuration field state) and the sizes of the files and subdirectories they contain. The output looks like this:

  {
    "projects": {
      "dummy": {
        "latest_commit": "09d6a5d81a5001bf210df2bf80e871e3731f6e9f",
        "run_log": 21370,
        "dependencies": 472923464,
        "cache": 2074946410,
        "artifacts": 4096
      },
    },
    "project_count": 6
  }

• The ambient qemu subcommand has been added to execute a runnable plan in a virtual machine, with or without networking. This is primarily a utility command to help develop Ambient by making it easier to experiment.

• The configuration file now allows enabling UEFI use for an image. The run and qemu subcommands additionally have a --uefi option for that.

• The ambient qemu --persist options allows creating a variant of an image. This can be used, for example, to change a generic cloud image from Debian or Arch Linux to boot fast even if the VM has no network access. Together with the optional UEFI support this paves way for using generic images instead of custom images for Ambient. That, in turn, should enable Ambient users to run CI under other operationg systems in the VM. However, Ambient needs further changes to make this convenient.

• Ambient now gives an error message if a virtual drives is too big. The virtual drives are created before the virtual machine starts. Previously, there was no helpful error message, only an "assert" error that only makes sense to Ambient developers.

• The exported parts of the Ambient library part now all have documentation. This makes the library usable from other programs, but more importantly, makes it harder for Lars to forget what a type or method is for. Many typo fixes and other changes were made to exported names.

  Note that Ambient is probably not very useful to use as a library. If you use it that way, or would like to, please be in touch and let us know so can try to avoid breaking it for you.

Ambient CI version 0.11.1 on 2016-01-14

From NEWS.md:

Bug fix:

• Always pass UEFI OVMF firmware file to QEMU, not just when UEFI support is on. The change for optional UEFI support in 0.11.0 broke some uses of Ambient. This change restores functionaliy at least for my own uses.

Future plans

Lars is still getting back up to speed with Radicle CI work and has no concrete plans as yet. Two clear needs are making it easier to set up Radicle CI both on a server and for local use (see rad-ci), and also automate the release process.

New releases of the CI broker, Ambient adapter, and native CI adapter are pending and will happen soon.

The overall goal is to make CI a joy to use with Radicle.

Notes

If you have an open source Rust project and want to try Radicle and Ambient on callisto, see https://callisto.liw.fi/callisto/ for instructions.

Quotes

No quotes yet, but please suggest something for the next issue.

Goal

My goal for today is to implement the shortcut directive from ikiwiki in riki. The plugin allows the page shortcuts in the site define shortcuts that can be used on page. A shortcut is defined like this:

[[!shortuct  wikpedia https://en.wikipedia.org/wiki/%W]]
[[!wikipedia   War_of_1812]]

The shortcut directive defines a new directive, which can be invoked as if it were a normal directive. The first nameless parameter is the name of the new directive, and the second is the URL pattern.

In the URL pattern, %s is replaced by the nameless argument to the invocation of the directive added as the shortcut, with URL encoding. %S (upper case) is the same, but without encoding. %W is encoded in a way suitable for Wikipedia. The desc parameter in the definition or invocation sets a description for the link, i.e., link text.

Plan

I don't quite like the way ikiwiki implements shortcuts, so my plan is to be compatible, but different.

• The shortcut directive can be used anywhere on the site, not just in the page called shortcuts.
• It's an error to define a shortcut with the name of a directive, or another shortcut.
• I don't want a dynamic set of directives. I think ikiwiki does things that because it was easy to do in Perl. I'll implement something different.
• riki does not currently support nameless parameters. I will add those, but I'll tackle the shortcut directive first. While I do that, I'll require named parameters. Once I add support for nameless parameters, I'll change the riki directive to define shortcuts.

Notes

Add phases for directives

• Some directives depend on other directives having been executed first. The shortcut directive needs to be executed before any shortcut is processed. The inline directive requires other pages to have been processed into HTML first.
• I can deal with this by adding explicit dependencies on directives, and pages using them, but that seems like a lot of work and prone to bugs.
• Instead, I'll add the concept of "phases" to execution of directives. Each directive execution will be given the "phase number" currently executing. After phases 0 (the first phase), any invocation of a directive must be either a known builtin directive, or a known shortcut for the site.
• Each directive will return a value indicating it has been executed in a phase, and does not need to be executed again.
• The site build process will iterate over invocations of directives, executing them until all directives have been executed.
• Actually, that seems unnecessarily tricky. New plan: each directive can define three methods, execute_0, execute_1, and execute_2. The site build process will loop over directive invocations three times, calling a successive execution method in each loop. Further, all invocations on all pages are executed in each phase.
• Adding that was easy and quick.
• Next I'll implement a datatype for holding known shortcuts. I'll add a field to the type that represents the site to use this. The shortcut directive will be able to add to that.
• I don't know what the Wikipedia encoding is, so I'll skimp on that. In fact, I'll only implement %s as that's the only thing I use. I'll get back to this later.
• All directives will need to get a mutable reference to the site.

Ran out of time implementing this today, will continue tomorrow.

Next morning

• I've coded myself into a corner. I want directives to be able to modify the site, but also the pages, and this results in more than one mutable reference to the site.
• I'll put the shortcut set in an arc box, or something.
• But not this early in the morning.
• I'm juggling too many changes at the same time. That might be a signal that I should start over and do one thing at a time.

Summary

Things did not end well. Mistakes were made. I will try again another day.

Goal

I had code and test for internal linking. I rewrote the code to be in a Site type, but lost the tests, as they were hard to add. Now I don't know if my code works. Actually, I know there's problem.

The goal for today is to add tests for internal linking, following the specification in doc/linking.md.

Plan

• I have helper functions to produce candidate pages for resolving a link on a page. Add unit tests for these. Each function is easy to test in isolation.
• Add unit tests for the Site::resolve method to test the order in which candidate target pages are tried.

Notes

• To make it easier to set up a Site for testing, add a Site::fake_page method, only usable for tests. Also add a testing-only Site::empty constructor.
• Sibling test, easy.
• Direct subpage, tricky. Apparently the code I have is broken. It's the code I rewrote from what I thought was working code, but I'm not convinced that actually worked either. This is why tests are important.
• A tricky bit: the origin page is, for example, /src/foo.mdwn, which means its internal name is /foo. There is also the target page /src/foo/bar.mdwn, with internal name /foo/bar. A link bar on the origin page should resolve to the target page. I can join bar (the link) with the origin internal name (/foo) to get /foo/bar. That's not a full page name, though. It lacks the fully qualified path the source file: I can get the source directory from the Site, but not the suffix (.mdwn).
• Maybe the internal name of the target would be enough? I can have Site::resolve do a lookup with that. The hash map of pages uses internal names for lookup, anyway.
• Now that I think about it, the helper functions that construct possible targets for links should return a PathBuf representing the internal name, not a full PageName. The full names are trickier to construct in all cases. Ideally it'd be a dedicated type, but I'll start with the generic PathBuf.
• Actually, an optional PathBuf - sometimes it's not possible to construct the candidate
• I'll throw away all the code and start over.
• I do, however, want to have my own type for the set of pages, instead of a raw HashMap, since I have to change everywhere it's used. Future me will thank me and so will present me.
• That was a lot of furious rewriting and refactoring, but it works now.
• Merged.

I'm making progress with riki, although very slowly. It's a program I'd really like to have, but it's so far down my priorities that I don't work on it often. But I'm currently on holiday and have more degrees of freedom. Here are notes from today's develoment session.

Goal

My goal today is to make riki be able to render a simple site. I want this so that I can start trying riki on my various ikiwiki sites.

riki can already render a single page. Today I want to make a command like this work:

riki build src dest

This will read the directory src, load all files, process every .mdwn file as Markdown with wikitext and produce HTML for that, and write the output to dest. All other files get copied verbatim.

This is only about processing a directory tree. It's OK if links within the site do not work in the output. It's OK to not implement any additional directives or other functionality. Once I can try to render sites, I can run riki on my various sites, and other chosen test sites, and find out what I need to fix.

Plan

• Create a minimal test site to verify riki build works even for the easiest case.
• Implement the scaffolding for the riki build command.
• Implement scanning the source tree for files.
• Implement processing of .mdwn files into HTML.
• Implement writing out of output files.

Notes

Test site

• For the test site I want about two files. One is not enough. More than two seems unnecessary.
• index.mdwn and other.mdwn will both contain a meta title and some text to make it easy to verify the page contains the right stuff.

Scan source folder

• I'll use the walkdir, as it's good and I'm familiar with it.
• I'll ignore everything that isn't a regular file. Directories will be created from source file paths, symlinks are dangerous. I might add support for symlinks some day, if there's a request, but for now I'm ignoring them.
• I'll only process .mdwn files as wikitext, everything else is a blob.
• I'm reading in blobs into memory. I may later only remember the pathname and copy files without keeping them in memory, but for now this is easier.

Process .mdwn files

• This is basiclly what the render-page command I implemented earier does. A bit of copy-pasta, and done! I may later refactor this to avoid code duplication, but it's literally two lines, so I'm not too bothered today.

Write out files

• For any input file foo/bar/yo.mdwn under the source directory, write foo/bar/yo/index.html in the output directory, creating the output directory and subdirectories as needed. But index.mdwn in a directory should result in index.html in the corresponding output directory.

• Hmm, my plan was to hack up something quickly and then tidying that up. But this turns out be ever so slightly more complicated than I thought. This is my third time trying to implement riki and I've run into this kind of surprise compoication before. But I'll push through. I should take notes of what the complexities are. Right now it's keeping track of page content (blob vs HTML), and filename. I want to be consistent, and avoid possibilities for coding mistakes, when I can.

• OK, I have something that works with my silly little test site.

• Did a little refactor to simplify code.

• Figuring out the output file name is a little tricky, but not too bad. Prime target for unit tests, too.

Trying this on my sites

• Some of my very simples sites work.
• Others have problems.
  • failing to handle some Markdown combinations
  • missing directives: inline, map
• I've noted those in issues.

End

• I've merged the changes.