Immutable servers and containers

| 1 Comment

The immutable server pattern has been around for a long time The whole idea around it is you simplify your change-management procedures by making it very hard if not impossible to change system-state after it is launched. It makes that famous auditor question...

If an engineer Leeroy Jenkins!!!  in a change, how do you ensure your approved state is maintained?

...easier to answer.

They're not allowed to log in at all. That should never happen.

No having to explain your puppet/chef application interval and making sure your config-management platform covers everything an engineer can change on the system.

No having to go over your detective controls to make sure sudo usage is monitored and tracked back to change-requests.

No setting up alarms for detecting changes going unapproved.

Turn off SSH on the box, you just can't do it.


Now, this definitely makes the compliance/regulation processes much easier to endure.

It also has benefits that the Docker side of the tech industry has been talking about for years now.

Which is also true of the immutable-server pattern. If it carries state, it shouldn't be immutable. Because I'm well known for running Logstash, I'll use ElasticSearch as an example of how it could be used with either immutable pattern here (server or docker).

  • client-only nodes: Dockerize/immutable that thing. Carry no state, negotiate with the rest of the cluster to direct traffic. Go for it.
  • ingestion-only nodes: These carry local state, so losing one can lose some transactions. But it isn't persistent across more than a few seconds, or a minute. Safe to immutable-ize.
  • master-only nodes:These do maintain state, but in the sense of being the final judge of what gets committed to the cluster's state. These are safe-ish to immutable-ize, but care must be taken to ensure a minimum number of them are up and joined to the cluster. This makes them more complicated to integrate into an immutable framework (min: 3 isn't quite enough assurance).
  • data-only nodes: Filled to the gills with state. So much state, it can take hours for the cluster to fully recover from a loss. Subject these to your full change-management, detective-controls config.

Databases of any kind should be exempted from immutable-ness or containerization.

It turns out much of my career has been in maintaining state-containing machines. It's only recently that I've started working in architectures that even have logic-only nodes. I don't have the reflexes for it yet, but I do know what shouldn't go into a container.

These are good patterns to follow, but know where they won't apply cleanly and accept that.

1 Comment

I don't understand the auditor quote?

"If an engineer Leeroy Jenkins!!! in a change, how do you ensure your approved state is maintained?"