January 2015 Archives

The sysadmin skills-path.

| No Comments

Tom Limoncelli posted a question today.

What is the modern rite of passage for sysadmins? I want to know.

That's a hard one, but it got me thinking about career-paths and skills development, and how it has changed since I did it. Back when I started, the Internet was just becoming a big source of information. If it wasn't on Usenet, the vendor's web-site might have a posted knowledge-base. You could learn a lot from those. I also learned a lot from other admins I was working with.

One of the big lamentations I hear on ServerFault is that kids these days expect a HOWTO for everything.

Well, they're right. I believe that's because of how friendly bloggers like myself have trained others into finding out how to do stuff. So I posit this progression of skill-set for a budding sysadmin deploying a NewThing.

  1. There is always a checklist if you google hard enough. If that one doesn't work, look for another one.
    • And if that doesn't work, ask a patch of likely experts (or bother the expert in the office) to make one for you. It works sometimes.
    • And if that doesn't work, give up in disgust.
  2. Google for checklists. Find one. Hit a snag. Look for another one. Hit another snag. Titrate between the two to get a good install/config.
    • If that doesn't work, follow the step-1 progression to get a good config. You'll have better luck with the experts this time.
  3. Google for checklists. Find a couple. Analyze them for failure points and look for gotcha-docs. Build a refined procedure to get a good install/config.
  4. Google for checklists. Find a few. Generalize a good install/config procedure out of them and write your own checklist.
    • If it works, blog about it.
  5. Google for checklists. Find a few, and some actual documentation. Make decisions about what settings you need to change and why, based on documentation evidence and other people's experience. Install it.
    • If it works, write it up for the internal wiki.
  6. [Graduation] Plunder support-forums for problem-reports to see where installs have gone wrong. Revise your checklist accordingly.
    • If it works, go to local Meetups to give talks about your deploy experience.

That seems about right. When you get to the point where your first thought about deploying a new thing is, "what can go wrong that I need to know about," you've arrived.

Smartphone ecosystems have definitely reached the level of complexity where we have to worry about hostile apps. And they're following the pattern shown by the Internet over the years in that there are classes of hostile actions:

  • Known/Allowed, also known as ad/revenue streams. App owners have to pay the bills somehow, and purchase fees only go so far.
  • Known/Disallowed, also known as malware following known exploits. For this we have scanners.
  • Unknown, apps doing things they shouldn't, by ways that aren't in the scanners yet. Evil, evil little beasties.

If there is one lesson about information security that has been true since the beginning, is that it's the victim's fault for getting owned. Really, look at the press following hacks: hacks are entirely the fault of the defending entity for not being good enough. If you just followed accepted security standards, this would never happen. Never mind that transitive trust models in very complex IT infrastructures are nearly impossible to fully secure, especially ones that involve humans, it's still the victim's fault.

Those 'accepted security standards' are somehow lacking in the app-stores, especially Android. It's like the app-owners don't really want you to secure yourself.

What would be very nice in these phone OS security system would be selectable permission filters. Don't want to allow bluetooth-access to any applications except those you whitelist? Don't want to share your contacts with an app that seemingly has no need for it? A limited version of this is in iOS, but as I'll get to in a moment it only goes so far.

There are two methods of denying access to capabilities, and we already have a good example of this two-tier model in the firewall world:

  • Notifies connections of no-connection.
  • Pretends there is nothing there.

The first method is nice for applications since they learn quickly to stop trying. The second is nice for defenders because it means potential attackers have to wait for timeouts before marking a IP:Port tuple as up/down. When it comes to phones, there are two ways to deal with selectable permissions:

  • Notify the app that they don't have rights to that thing. Apps know they're being banned.
  • Lie to the app and provide a stub service that returns nothing or a simple carrier-signal. Apps will have to do tests to see if they're banned.

IOS uses the first model. If you've ever seen a, "turn on bluetooth for an enhanced user experience," modal, that's what happened. I believe that Apple standards say that applications have to honor those settings in that they still run and don't quit in a huff over not getting your identity goodies. You may not be able to do much, but they'll still run.

Android currently doesn't have selectable permissions (out of the box; there are some apps that try to provide it), you decide whether or not an app can be allowed to do it's full list at the time you install it. This can be problematic, especially if circumstances require that you install certain apps, but you want to disable certain capabilities. Such as having only one phone with both work and email on it, and you'd rather they didn't wipe it when they fire you.

That's where things like XPrivacy can come in handy. This only runs on a rooted device, but it provides the stub-services needed to prevent apps from quitting in a huff over not getting the ability to remove accounts on the device, lie about Bluetooth/NFC/Wifi access and state, or falsify 'network' location data. Things like XPrivacy allow us to provide those very 'accepted security standards' that reduce victim-blaming after incidents. It would be awesome if this came stock, but we can't have everything.

Way back when I first got into Group Policies, which was just after Group Policies were released, one of the things we mooted about the BoF den was a simple thing we could do to tell users that they were on a managed station. What we came up with was pretty simple: manage the desktop background.

No, we didn't put an all-seeing-eye on it. That would be creepy, don't be silly. We used a logo of the company.

It made sense! A simple cue, and we'd save RAM (back in those days the desktop background took more than trivial RAM). We were happy.


It turns out, that's not how you build a happy user-base. By doing so, we told people explicitly everything you do can and will be used against you in an HR action. People don't like to be told they're being monitored.

You know who likes to be told they're being monitored like that? No one.

You know who we want to be monitored that way? Prisoners and people likely to become prisoners.

No one wants to be thought of as a prisoner, or likely to be one.

In fact, later GPO guides specifically discouraged doing things like managing the desktop background or theme. It could be done, but... why would you want to? Desktop theme is one very low impact thing on the system and the single biggest thing the user can customize to their preferences. It's a very low challenge to the system to increase user experience by a great amount. Let them customize and don't worry about it.

But still manage their IE zones, certificate enrollment policies, software distribution methods, and event-log reporting.

They can make their jail-cell a pink polka-dot wonder, far better than bare cinder-block! It's still a cell, but without that camera in their face, they're happier about living in it.


It looks like consumer-focused big-data stuff is suffering the same faults as early GPOs did: they're being too obvious about the surveillance.

"Hello, Mister ${mispronounced last name}," said the sales-clerk I'd never met before. I sighed in resignation, vowing to factory reset my cell-phone. Again. One of these days I'm just going go cash only.

Or another one I almost guarantee will happen:

TSA Customer Service
@sysadm1138 We noticed you were in DFW security line for 49 minutes. We would like some feed back about that, https://t.co/...

Er, wait. That's Big Brother. Sorry, dial slipped. Let's try again.

VIctorias Secret
@sysadm1138 We noticed you spent time in our DES MOINES, IA store. If you have time, please take a short survey about your visit. https://t.co/...

You've probably run into this one, but hitting a random website, and then that site haunts your web-ads (for those of you who don't run on AdBlock-Strict) for weeks.

They haven't figured out that a large percentage of us don't like being reminded we live in a panopticon. Give me my false illusion of anonymity and I'm happy!

It's all about the user-factors. What's good for the retailer, is not always good for their consumers. Obviously. But the best kind of thing like that are things that aren't obviously not-good for the consumer.

User-factors, people!

Other Blogs

My Other Stuff

Monthly Archives