Systemd dependencies

There is a lot of hate around Systemd in unixy circles. Like, a lot. There are many reasons for this, a short list:

• For some reason they felt the need to reimplement daemons that have existed for years. And are finding the same kinds of bugs those older daemons found and squashed over a decade ago.
  • I'm looking at you Time-sync and DNS resolver.
• It takes away an init system that everyone knows and is well documented in both the official documentation sense, and the unofficial 'millions of blog-posts' sense. Blog posts like this one.
• It has so many incomprehensible edge-cases that make reasoning about the system even harder.
• The maintainers are steely-eyed fundamentalists who know exactly how they want everything.
• Because it runs so many things in parallel, bugs we've never had to worry about are now impossible to ignore.

So much hate. Having spent the last few weeks doing a sysv -> systemd migration, I've found another reason for that hate. And it's one I'm familiar with because I've spent so many years in the Puppet ecosystem.

People love to hate on puppet because of the wacky non-deterministic bugs. The order resources are declared in a module is not the order in which they are applied. Puppet uses a dependency model to determine the order of things, which leads to weird bugs where a thing has worked for two weeks but suddenly stops working that way because a new change was made somewhere that changed the order of resource-application. A large part of why people like Chef over Puppet is because Chef behaves like a scripting language, where the order of the file is the order things are done in.

Guess what? Systemd uses the Puppet model of dependency! This is why its hard to reason. And why I, someone who has been handling these kinds of problems for years, haven't spent much time shaking my tiny fist at an uncaring universe. There has been swearing, oh yes. But of a somewhat different sort.

The Puppet Model

Puppet has two kinds of dependency. Strict ordering, and do this if that other thing does something. Which makes for four ways of linking resources.

• requires => Do this after this other thing.
• before => Do this before this other thing.
• subscribes => Do this after this other thing, but only if this other thing changes something.
• notifies => Do this before this other thing, and tell it you changed something.
  

This makes for some real power, while also making the system hard to reason about.

Thing is, systemd goes a step further

The Systemd Model

Systemd also has dependencies, but it was also designed to run as much in parallel as possible. Puppet was written in Ruby, so has a strong single-threaded tendencies. Systemd is multi-threaded. Multi-threaded systems are harder to reason about in general. Add on dependency ordering to multi-threaded issues and you get a sheer cliff of learning before you can have a hope of following along. Even better (worse), systemd has more ways of defining relationships.

• Before= This unit needs to get all the way done before the named units are even started. And, the named units only get started if this unit finishes successfully.
  
• After= This unit only gets started if the named units run to completion first, successfully.
  
• Requires= The named units will get started if this one is, and do so at the same time. Not only that, but if the named units are explicitly stopped, this one will be stopped as well. For puppet-heads, this breaks things since this works backwards.
• BindsTo= Does everything Requires does, but will also stop this unit if the named unit stops for any reason, not just explicit stops.
• Wants= Like Require, but less picky. The named units will get started, but not care if they can't start or end up failing.
• Requisite= Like Require, but will fail immediately if the named services aren't started yet. Think of mount units not starting unless the device unit is already started.
• Conflicts= A negative dependency. Turn this unit off if the named unit is started. And turn this other unit off if this unit is started.

There are several more I'm not going into. This is a lot, and some of these work independently. The documentation even says:

It is a common pattern to include a unit name in both the After= and Requires= options, in which case the unit listed will be started before the unit that is configured with these options.

Using both After and Requires means that the named units need to get all the way done (After=) before this unit is started. And if this unit is started, the named units need to get started as well (Require=).

Hence, in many cases it is best to combine BindsTo= with After=.

Using both configures a hard dependency relationship. After= means the other unit needs to be all the way started before this one is started. BindsTo= makes it so that this unit is only ever in an active state when the unit named in both BindsTo= and After= is in an active state. If that other unit fails or goes inactive, this one will as well.

There is also a concept missing from Puppet, and that's when the dependency fires. After/Before are trailing-edge triggers, they fire on completion, which is how Puppet works. Most of the rest are leading-edge triggered, where the dependency is satisfied as soon as the named units start. This is how you get parallelism in an init-system, and why the weirder dependencies are often combined with either Before or After.

Systemd hate will continue for the next 10 or so years, at least long enough for most Linux engineers to have been working with it to stop grumbling about how nice the olden days were.

It also means that fewer people will be writing startup services due to the complexity of doing anything other than 'start this after this other thing' ordering.