My dayjob hasn't brought me into much contact with containers, which is becoming a problem. I mean, I've been doing virtualization for a decade and a half at this point so the concepts aren't new. But how containers work these days isn't something I'm all that up on.
So when the OpenSuSE factory mailing list exploded recently with news of what the next versions of what's now Leap would look like, I paid attention. To quote a weekly status update that caused a lot of debate:
we have mostly focused this week's meeting on how the ALP Desktop is going to look like and how we can dispel fears around flatpaks and containers when it comes to the ALP desktop.
They're moving away from a traditional RPM-based platform built on top of the SuSE Linux Enterprise Server (SLES) base, and are doing it because (among other reasons) Python on SLES is an officially unsupported version. Most Leap users want more recent Python, which the OpenSuSE project can't do due to a lack of volunteer support to rewrite all the OS-related Python bits.
What they're replacing it with is something called Adaptable Linux Platform (ALP) which is built in part on the work done in OpenSUSE MicroOS. It will use Flatpak packages (or RPM wrappers around Flatpaks), which is a contentious decision let me tell you. This debate is what made me look into Flatpak and related tech.
The hell no side of the debate are old-guard sysadmins from my era who have been doing vulnerability management for a long, long time and know that containers make that work problematic. Building a Configuration Management Database (CMDB) that lists all of your installed packages is a core competency for doing vulnerability management, because that's the list you consult when yet another OpenSSL vulnerability arrives and you need to see how bad it'll be this time. This work is made way harder when every container you have comes bundled with its own version of the OpenSSL libraries. We now have emerging tooling that will scan inside containers for bad packages, but that's only half the problem; the other half is convincing upstream container providers to actually patch on your vulnerability program's schedule. Many won't.
On the well actually, that's kind of a good idea side of the debate are the package maintainers. For a smaller project like OpenSUSE, package maintainers need to support packages for a few different distributions:
- Tumbleweed, the rolling release that is kept up to date at all times
- The current Leap release, which has base OS libraries getting older every year (and might be old enough to prevent compiling 'newest' without lots of patches)
- Leap Micro (based on SLES Micro), has a read-only file system assumption, using Flatpaks, is designed to be container-first, and isn't meant to be a desktop distribution
- Any additional architectures beyond x86_64/amd64 they want to support (various ARM flavors are in demand, but OpenSuSE also has s390 support)
For some packages this is no big deal. For others that involve touching the kernel in any way, such as VirtualBox, each distribution can have rather different requirements. This adds to the load of supporting things, and consumes more volunteer time.
Flatpak lets you get away with making a single package that will support all the distributions. Quite the labor saving device for a project strapped for volunteers. Of course, this leads to the usual 54 versions of openssl libraries problem, but the distributions can all still be made. That's valuable.
This also makes it easier to sandbox desktop utilities. I say sandbox instead of containerize because the key benefit you're getting from this isn't the containers, but the sandboxing that comes with them. I've spent time in the past attempting to build an AppArmor profile for Firefox (it mostly worked, but took too much maintenance to tune). These days you can use systemd user units to apply cgroups to processes and get even more control of what they're allowed to do, including restricting them to filesystem namespaces. This isn't a bad thing.
Also in the not a bad thing camp is ALP's stance that most of the filesystem will be marked read-only. This improves security, because malicious processes that break out of their sandbox will have a harder time replacing system binaries. Having partitions like /usr mounted as read-only has shown up on hardening guides for a couple of decades at this point.
The final thing is that the Open Build Service, which creates all the OpenSUSE packages, already has support for creating Flatpaks along side traditional RPMs. This will make maintenance even easier.
What do I think of all of this?
I'm still making up my mind. I'm going to have to get over no longer having "tarball with some added scripts and metadata" style packagers like I've used since the 1990s, that writing is on the wall. We'll still have some of that around, but major application-packages are going to get shipped as functional filesystem images for even "base" Linux installs. It won't be all that space-efficient, a package needing modern Python will end up shipping an entire Python 3.10 interpreter in the Flatpak for a SLES server, but that's less important in the days of 1TB microSD cards.
Ubuntu has this idea in their Snaps, flatpack and snap are highly similar in goals and features, which has been controversial all by itself. At DayJob we've had the vulnerability management conversation regarding snaps and our ability to introspect into snaps for managing CVEs, which isn't great yet, and decided not to go there at this time.
All in all, the industry is slowly shifting from treating the list of system-packages as the definitive list of software installed on a server towards a far more nuanced view. NPM is famous for building vast software dependency trees, and we've had most of a decade for vulnerability scanning tooling to catch up to that method. For all that OS-side updates were among the first to create dependency resolvers (all that metadata on .deb and .rpm packages is there for a reason) the not-developing-on-C software industry as a whole has taken this approach to whole new levels, which in turn forced vulnerability management tooling to adapt. We're about at the point where SLES Micro makes a good point; the use of traditional methods for a minimal OS and container-blobs for everything else is just about supportable given today's tooling.
The only constant in this industry is change, and this is feeling like one of those occasional sea changes.