I've mentioned I work on Baserock for Codethink. I've gotten out of practice writing, so I'm going to talk about something I know well, how we build software in Baserock.

Morpohologies

The software recipes are called morphologies. The unusual name choice was deliberate to prevent the name imposing design choices. If we called them recipes we would be tempted to implement them like bitbake recipes.

Chunk morphologies

Chunk morphologies describe how to build "packages". We don't call them packages since they aren't independently usable.

These compose a set of commands to configure, build then install the chunk.

Stratum morphologies

Strata are a set of descriptions on where to find chunks, and their build-dependencies.

These are specified externally to the chunk, as opposed to with them, as packages are usually built, since a chunk builds differently with different packages available but the same commands.

As an example, vim will build gvim if graphics are available, but only the command line vim if they aren't.

As a far more extreme example, cpython builds python bindings for pretty much every library it finds installed, since the standard python distribution has a reputation for having batteries included.

This is somewhat analogous to layers in Yocto.

Strata describe where to find a chunk by the git repository the chunk is located in, the branch to use, and the name of the chunk morphology file to use.

System morphologies

System morphologies describe which strata to install on the system. These can be combined from different sources to produce different systems, while reducing repetition for common elements.

Building

Resolving morphologies and fetching sources

The first step is to load all the disjoint morphologies, so that they can be brought together to describe how to build the System.

The strata listed in the system are found by fetching the specified repository and reading the morphology file from it.

This information is also used to compute hashes of all the information required to build, so that if a part of the system has already been built, it can be reused without having to build everything again.

Building Chunks

The hash of dependencies is used to find out if a chunk is already built. If it is then the following steps are not performed.

Constructing a staging area

The cached build artifact of all the chunks that are build dependencies of the target chunk are extracted into a temporary directory. This temporary directory is named the same as the cache key, and is not extracted if it already exists.

This acts as a cache so that chunks do not need to be extracted multiple times.

The staging area itself it constructed by hardlinking all of the chunks into a new temporary directory.

The git repository with the chunk's source code is then cloned into place for the builds to be run in.

Running configure and build commands

Since the staging area is constructed from hardlinks, builds must not be able to alter the staging area except for directories deemed safe.

To do this we use linux-user-chroot, a command written by Colin Walters, which uses kernel namespaces to isolate a subprocess. It can be used to make subtrees read-only.

Configure commands are expected to prepare the build. It is told things like the prefix to install into and the gnu build triplet.

Build commands are similar, but also have MAKEFLAGS set to make use of all available compute resources.

Installing chunks

Chunks are expected to fill the directory pointed to by the environment variable DESTDIR.

The contents of this directory is archived and placed into the artifact cache.

Building Strata

Stratum artifacts are just the list of chunks they are made of, plus a little metadata.

Building Systems

Systems are made of all the files of all the chunks in all the strata listed. They do not include all the strata built, as a stratum can depend on another stratum that is not part of the System. This is currently the only way to handle pure build dependencies.

Bootstrapping

The staging area needs a way to be filled. Previously we used to have a staging filler, which was a tarball to extract first.

The filler was constructed by a complicated bootstrap script and was hard to maintain.

This caused many problems, such as chunks linking against things they would not have in the final system, since they were in the filler but were removed from the morphologies.

Now, we have a carefully constructed stratum, called build-essential, which contains chunks built in what's called "bootstrap mode", where while the staging area is constructed, we do not chroot into it. We do however use linux-user-chroot to make the host's root file system read-only for protection.