Last Tuesday Patricia Aas asked the question:
Devs, what is the weirdest hurdle Ops has put in your way?
-- Patricia Aas (@pati_gallardo) November 12, 2019
Ops, what is the craziest thing you've seen Devs do?
(I have a talk I'm mulling over)
It got a lot of responses, but it did make me think. It took me a while to find an answer to this because so many of these are stemming from ignorance. My own answer came from when I worked at my last startup, before we hit launch (the best time to catch this problem).
Splitting an MD5 sum hash by each digit... for a file hosted on an actual-filesystem. Think a/1/b/3/c/ogodmakeitstop/d/e/original.pdf. I had to explain how inodes worked, and why that was a terrible idea.
-- DevSecProdOps1138 (@SysAdm1138) November 12, 2019
If you've done filesystem work on Linuxy systems, you're probably cringing right now.
It's what happens next that is the interesting part. If you're from Old School Operations, which is where I spent the first 14 years of my career, seeing something like this would spin the sarcasm engines up to speed and you'd roast that poor developer so hard they'd think twice about writing files forever more. So much sarcasm, but I meant well.
New School DevOps is a different world, though. Because I caught this pattern before a fix would require handling multiple file-system layout schemas in code (we could just nuke our existing layout and start afresh), it was a teaching moment. This developer knew that large directory sizes were a problem, and fixed it by making sure they were always small. Their intent was good, even though they didn't understand the problem space.
A few background details that were implied, but not stated:
- This was for a filesystem I fully expected to crack 10 million files within the first year.
- Probably would have crossed 1 billion before AWS came up with what would be called Elastic File Services.
- Couldn't use S3 for technical reasons relating to a need for full POSIX during certain operations (could have been done without, but those pipeline stages would have taken a lot longer to finish, like an order of magnitude or more slower)
My understanding of how filesystems worked on the inside, the details of the filesystem I picked, needed to mesh with what the developer needed to do to perform application work. Sarcasm, belittling, and biting humor about RTFM merely makes it less likely that they'll talk to me. I need them to talk to me more. Together we came up with a much more healthy compromise between inode size and directory tree sizes. A compromise that got them past 200 million files before I left.