The sysadmin skills-path.

| No Comments

Tom Limoncelli posted a question today.

What is the modern rite of passage for sysadmins? I want to know.

That's a hard one, but it got me thinking about career-paths and skills development, and how it has changed since I did it. Back when I started, the Internet was just becoming a big source of information. If it wasn't on Usenet, the vendor's web-site might have a posted knowledge-base. You could learn a lot from those. I also learned a lot from other admins I was working with.

One of the big lamentations I hear on ServerFault is that kids these days expect a HOWTO for everything.

Well, they're right. I believe that's because of how friendly bloggers like myself have trained others into finding out how to do stuff. So I posit this progression of skill-set for a budding sysadmin deploying a NewThing.

  1. There is always a checklist if you google hard enough. If that one doesn't work, look for another one.
    • And if that doesn't work, ask a patch of likely experts (or bother the expert in the office) to make one for you. It works sometimes.
    • And if that doesn't work, give up in disgust.
  2. Google for checklists. Find one. Hit a snag. Look for another one. Hit another snag. Titrate between the two to get a good install/config.
    • If that doesn't work, follow the step-1 progression to get a good config. You'll have better luck with the experts this time.
  3. Google for checklists. Find a couple. Analyze them for failure points and look for gotcha-docs. Build a refined procedure to get a good install/config.
  4. Google for checklists. Find a few. Generalize a good install/config procedure out of them and write your own checklist.
    • If it works, blog about it.
  5. Google for checklists. Find a few, and some actual documentation. Make decisions about what settings you need to change and why, based on documentation evidence and other people's experience. Install it.
    • If it works, write it up for the internal wiki.
  6. [Graduation] Plunder support-forums for problem-reports to see where installs have gone wrong. Revise your checklist accordingly.
    • If it works, go to local Meetups to give talks about your deploy experience.

That seems about right. When you get to the point where your first thought about deploying a new thing is, "what can go wrong that I need to know about," you've arrived.

Smartphone ecosystems have definitely reached the level of complexity where we have to worry about hostile apps. And they're following the pattern shown by the Internet over the years in that there are classes of hostile actions:

  • Known/Allowed, also known as ad/revenue streams. App owners have to pay the bills somehow, and purchase fees only go so far.
  • Known/Disallowed, also known as malware following known exploits. For this we have scanners.
  • Unknown, apps doing things they shouldn't, by ways that aren't in the scanners yet. Evil, evil little beasties.

If there is one lesson about information security that has been true since the beginning, is that it's the victim's fault for getting owned. Really, look at the press following hacks: hacks are entirely the fault of the defending entity for not being good enough. If you just followed accepted security standards, this would never happen. Never mind that transitive trust models in very complex IT infrastructures are nearly impossible to fully secure, especially ones that involve humans, it's still the victim's fault.

Those 'accepted security standards' are somehow lacking in the app-stores, especially Android. It's like the app-owners don't really want you to secure yourself.

What would be very nice in these phone OS security system would be selectable permission filters. Don't want to allow bluetooth-access to any applications except those you whitelist? Don't want to share your contacts with an app that seemingly has no need for it? A limited version of this is in iOS, but as I'll get to in a moment it only goes so far.

There are two methods of denying access to capabilities, and we already have a good example of this two-tier model in the firewall world:

  • Notifies connections of no-connection.
  • Pretends there is nothing there.

The first method is nice for applications since they learn quickly to stop trying. The second is nice for defenders because it means potential attackers have to wait for timeouts before marking a IP:Port tuple as up/down. When it comes to phones, there are two ways to deal with selectable permissions:

  • Notify the app that they don't have rights to that thing. Apps know they're being banned.
  • Lie to the app and provide a stub service that returns nothing or a simple carrier-signal. Apps will have to do tests to see if they're banned.

IOS uses the first model. If you've ever seen a, "turn on bluetooth for an enhanced user experience," modal, that's what happened. I believe that Apple standards say that applications have to honor those settings in that they still run and don't quit in a huff over not getting your identity goodies. You may not be able to do much, but they'll still run.

Android currently doesn't have selectable permissions (out of the box; there are some apps that try to provide it), you decide whether or not an app can be allowed to do it's full list at the time you install it. This can be problematic, especially if circumstances require that you install certain apps, but you want to disable certain capabilities. Such as having only one phone with both work and email on it, and you'd rather they didn't wipe it when they fire you.

That's where things like XPrivacy can come in handy. This only runs on a rooted device, but it provides the stub-services needed to prevent apps from quitting in a huff over not getting the ability to remove accounts on the device, lie about Bluetooth/NFC/Wifi access and state, or falsify 'network' location data. Things like XPrivacy allow us to provide those very 'accepted security standards' that reduce victim-blaming after incidents. It would be awesome if this came stock, but we can't have everything.

Way back when I first got into Group Policies, which was just after Group Policies were released, one of the things we mooted about the BoF den was a simple thing we could do to tell users that they were on a managed station. What we came up with was pretty simple: manage the desktop background.

No, we didn't put an all-seeing-eye on it. That would be creepy, don't be silly. We used a logo of the company.

It made sense! A simple cue, and we'd save RAM (back in those days the desktop background took more than trivial RAM). We were happy.


It turns out, that's not how you build a happy user-base. By doing so, we told people explicitly everything you do can and will be used against you in an HR action. People don't like to be told they're being monitored.

You know who likes to be told they're being monitored like that? No one.

You know who we want to be monitored that way? Prisoners and people likely to become prisoners.

No one wants to be thought of as a prisoner, or likely to be one.

In fact, later GPO guides specifically discouraged doing things like managing the desktop background or theme. It could be done, but... why would you want to? Desktop theme is one very low impact thing on the system and the single biggest thing the user can customize to their preferences. It's a very low challenge to the system to increase user experience by a great amount. Let them customize and don't worry about it.

But still manage their IE zones, certificate enrollment policies, software distribution methods, and event-log reporting.

They can make their jail-cell a pink polka-dot wonder, far better than bare cinder-block! It's still a cell, but without that camera in their face, they're happier about living in it.


It looks like consumer-focused big-data stuff is suffering the same faults as early GPOs did: they're being too obvious about the surveillance.

"Hello, Mister ${mispronounced last name}," said the sales-clerk I'd never met before. I sighed in resignation, vowing to factory reset my cell-phone. Again. One of these days I'm just going go cash only.

Or another one I almost guarantee will happen:

TSA Customer Service
@sysadm1138 We noticed you were in DFW security line for 49 minutes. We would like some feed back about that, https://t.co/...

Er, wait. That's Big Brother. Sorry, dial slipped. Let's try again.

VIctorias Secret
@sysadm1138 We noticed you spent time in our DES MOINES, IA store. If you have time, please take a short survey about your visit. https://t.co/...

You've probably run into this one, but hitting a random website, and then that site haunts your web-ads (for those of you who don't run on AdBlock-Strict) for weeks.

They haven't figured out that a large percentage of us don't like being reminded we live in a panopticon. Give me my false illusion of anonymity and I'm happy!

It's all about the user-factors. What's good for the retailer, is not always good for their consumers. Obviously. But the best kind of thing like that are things that aren't obviously not-good for the consumer.

User-factors, people!

A minimum vacation policy

| No Comments

A, "dude, that's a cool idea," wave has passed through the technology sector in the wake of an article about a minimum vacation policy. This was billed as an evolution of the Unlimited Vacation Policy that is standard at startups these days. The article correctly points out some of the social features of unlimited-vacation-polices that aren't commonly voiced:

  • No one wants to be the person who takes the most vacation.
  • No one wants to take more vacation than others do.
  • Devaluing vacation means people don't actually take them. Instead opting for low-work working days in which they only do 2 hours remotely instead of a normal 10 in the office.

These points mean that people with an unlimited policy end up taking less actual vacation than workplaces with an explicit 15 days a year policy. Some of the social side-effects of a discrete max-vacation policy are not often spelled out, but are:

  • By counting it, you are owed it. If you have a balance when you leave, you're owed the pay for those earned days.
  • By counting it, it has more meaning. When you take a vacation day, you're using a valuable resource and are less likely to cheapen it by checking in at work.
  • There is never any doubt that you can use those days, just on what days you can use them (maintain coverage during the holidays/summer, that kind of thing).

Less stress all around, so long as a reasonable amount is given. To me, this looks like a better policy than unlimited.

But what about minimum-vacation? What's that all about?

The idea seems to be a melding of the best parts of unlimited and max. Employees are required to take a certain number of days off a year, and those days have to be full-disconnect days in which no checking in on work is done. Instead of using scarcity to urge people to take real vacations, it explicitly states you will take these days and you will not do any work on them. For the employer it means you do have to track vacation again, but they're required days, don't create the vacation-cash-out liability that max-vacation policies create, and you only have to track up to the the defined amount. If an employee takes 21 days in a year, you don't care since you stopped tracking one they hit 15.

The social factors here are much healthier than unlimited:

  • Explicit policy is in place saying that vacations are no-work days. People get actual down-time.
  • Explicit policy is in place that N vacation days shall be used, so everyone expects to use at least those days. Which is probably more than they'd use with an unlimited policy.
  • Creates the expectation that when people are on vacation, they're unreachable. Which improves cross-training and disaster resilience.

I still maintain that a max-vacation policy working in-hand with a liberal comp-time policy is best for workers, but I can't have everything. I like min-vacation a lot better than unlimited-vacation. I'm glad to see it begin to take hold.

Categories

| No Comments

Humans are curious critters. We keep trying to pick apart reality to figure out how it works. Part of that is to break reality up into smaller chunks so it makes sense. Abstractions improve understanding and allow further refinements to the model. It's what science is based on.

Biology is a continually vexing problem, though. In many ways, it's a continuous function we keep attempting to turn into a discrete maths problem. Taxonomy, the naming of species, is a great example of this. Early classification methods relied on similar morphology to determine relatedness, and that gave us a nice family tree. Then we figured out how to sequence genomes and we learned how wrong we were; they're now moving whole species/phylum branches around. It turns out nature sometimes solves the same problem the same way through completely unrelated species.

What sparked the rearrangement? A new way to classify. A new method was picked to be more accurate, and changes had to be made.

Topically, take a look at OS classification of non-mobile consumer computing devices (what used to be called desktop-OS). You can see this on any web-visitor analytics platform. Some break it down like:

  • Windows
  • OSX
  • Linux

Others get more specific, breaking it down to versions within OS:

  • Window XP
  • Windows 7
  • Windows 8
  • OSX 10.6
  • OSX 10.7
  • OSX 10.8
  • OSX 10.9
  • OSX 10.10
  • Linux

For some reason they don't break apart the Linux versions. Perhaps because it's such a small segment of the market and highly fragmented at that. Still more detailed charts go down to Windows service-pack levels. OS version is a discrete space, but in order to provide a brief chart some simplifications are made. Each analytics application makes its own classification decisions.

Less topically, lets take a look at a fictional made-up species, the Variegated Civet. Take the physical sex of this critter. The original population study was done in 1906 and an odd sex ratio was observed, 1.3 females to every male. As with all studies of the time, external morphology studies were used to determine sex with a few dissections as a cross-check.

Fast forward a bunch of years and genetic studies become financially doable for an appropriate sample-size of the population. It reveals a funny thing. Some of the females are genetically male. This raises eyebrows and further studies reveal the cause. A significant percentage of males undergo gonadogeneis at puberty, not in-utero, which skewed the original study's sex ratio.

A new classification technique, genetics, reveals an interesting feature in a specific population. It also raises the question of what how to differentiate pre-puberty males with fully formed gonads from those who will do so later. A third sex may need to be created to explain this species.

We're undergoing an attempt to change the cultural classification method for gender in humans. For ages it has been based on physical morphology and came in two types. Nature being nature, there are plenty of ambiguous presentations to make the classifier problem harder (intersex); but not enough to prompt the creation of a third gender. Those weird-cases were assigned into one or the other, which ever was closer, in the opinion of the classifier (sometimes nudging things along with a bit of surgery). For ages gender was a synonym of sex.

That's beginning to change, and it's not been an easy thing to bring about. For one, gender is becoming more widely seen as discrete from sex. For another, gender is at the early stages of redefining its classifier away from external morphology and/or chromosomes and into self description. Self-description brings it away from a discrete function (binary, trinary) and into more of a multi-axis graph.

It takes a long time for a change like this to take hold, and there are fights being had. Vital Records only record one of these, and it's still legally entangling both sex and gender. Maybe in some future time driver's licenses will have both sex and gender fields on it. Or maybe those fields will be left off all together (the better option, in my opinion). Chromosomes are not truth, as nature's continuous function ensures there will always be an exception (complete Androgen Insensitivity Syndrome is the big exception to the XY = Male 'truth').

The work continues.

Over a month, and nothing

| No Comments

This is what busy looks like. I had an interesting puppet thing happen that I wanted to write about, but I couldn't grab the needed log-file in time. Dammit. It was about an odd message that shows up in user resources when it only goes part way, and was an interaction with VMWare OS Customization scripts. Sorry!

In the last month I've:

  • Survived a Great Rearranging in our cubicals. They kept me with my friends which is ♥.
  • Been given the project to deploy our product on [cloud provider] since we actually have a client who will pay us to do it.
  • Written a LOT of puppet-code to refactor our setup into something that can deal with our product on multiple infrastructures.

BusyBusy.

Breath-taking honesty

| No Comments

An essay has been making the rounds about how much you (developer type person) are worth for hourly-rate. It is breath-taking in its honesty. This person sets out a pay-rate scale based strictly on public-access reputation markets and evidence of community activity, with a premium on community contributions. To get top dollar, you will need to tick all of these boxes:

  • Have high-rated open-source libraries on GitHub.
  • Have a StackOverflow reputation over 20K.
  • Vendor-based code certifications, such as those from Oracle (Java) or Zend (Php)
  • Evidence of mastery in multiple languages. So, Ruby AND Erlang, not Erlang and maybe Ruby if you have to.
  • Published talks at conferences

If you don't have all those boxes ticked, you can still get paid. It just won't be enough to live on in most hot technical job-markets. The author is also very explicit in what they don't care about:

  • Cost-of-living. With fully remote work, location is elective. Want to make boatlads of cash? Move to the Montana prairie. You won't get more money by living in London.
  • Education. Masters, BA, BS, whatever. Don't care.
  • Past employment. Blah blah blah corporate blah.
  • Years of experience. I call bull on this one, since I'm dead certain that if they see "10 years of experience in Blah" this job-reviewer is going to look more critically on the lack of an auditable career than someone with 2 years.

Before long we're going to get a startup somewhere that will take evidence of all of the first list of bullet-points and distil it down to a Klout-like score.

One not-mentioned feature of this list is it means there are a variety of career suicide moves if more companies start adopting this method of pricing developer talent:

  • Working on closed-source software.
  • Working for a company that doesn't contribute to open source projects.
  • Working for a company that doesn't pay to present at conferences.
  • Working for a company that doesn't pay for continuing education.
  • Working for a company that has strict corporate communications rules, which prevent personal blogging on techical topics.
  • Working for a company with employment contracts that prohibit technical contributions to anything, anywhere that isn't the company (often hidden as the no-moonlighting rule in the employment contract).

Career suicide, all of it. I'm glad the systems engineering market is not nearly as prone to these forces.

No, I'm talking about that fancy wristband some of you wear, the one that talks to a smartphone. That's a monitoring system, but for your body.

We're IT. We do monitoring systems, so lets take a look at this one!

Getting stuck in Siberia

| No Comments

I went on a bit of a twitter rant recently.

Good question, since that's a very different problem than the one I was ranting about. How do you deal with that?


I hate to break it to you, but if you're in the position where your manager is actively avoiding you it's all on you to fix it. There are cases where it isn't up to you, such as if there are a lot of people being avoided and it's affecting the manager's work-performance, but that's a systemic problem. No, for this case I'm talking about you are being avoided, and not your fellow direct-reports. It's personal, not systemic.

No, it's not fair. But you still have to deal with it.

You have a question to ask yourself:

Do I want to change myself to keep the job, or do I want to change my manager by getting a new job?

Because this shunning activity is done by managers who would really rather fire your ass, but can't or won't for some reason. Perhaps they don't have firing authority. Perhaps the paperwork is too much to bother with firing someone. Perhaps they're the conflict-avoidant type and pretending you don't exist is preferable to making you Very Angry by firing them.

You've been non-verbally invited to Go Away. You get to decide if that's what you want to do.

Going Away

Start job-hunting, and good riddance. They may even overlook job-hunt activities on the job, but don't push it.

Staying and Escalating

They can't/won't get rid of you, but you're still there. It's quite tempting to stick around and intimidate your way into their presence and force them to react. They're avoiding you for a reason, so hit those buttons harder. This is not the adult way to respond to the situation, but they started it.

I shouldn't have to say that, but this makes for a toxic work environment for everyone else so... don't do that.

Staying and Reforming

Perhaps the job itself is otherwise awesome-sauce, or maybe getting another job will involve moving and you're not ready for that. Time to change yourself.

Step 1: Figure out why the manager is hiding from you.
Step 2: Stop doing that.
Step 3: See if your peace-offering is accepted.

Figure out why they're hiding

This is key to the whole thing. Maybe they see you as too aggressive. Maybe you keep saying no and they hate that. Maybe you never give an unqualified answer and they want definites. Maybe you always say, 'that will never work,' to anything put before you. Maybe you talk politics in the office and they don't agree with you. Maybe you don't go paintballing on weekends. Whatever it is...

Stop doing that.

It's not always easy to know why someone is avoiding you. That whole avoidant thing makes it hard. Sometimes you can get intelligence from coworkers about what the manager has been saying when you're not around or what happens when your name comes up. Ask around, at least it'll show you're aware of the problem.

And then... stop doing whatever it is. Calm down. Say yes more often. Start qualifying answers only in your head instead of out loud. Say, "I'll see what I can do" instead of "that'll never work." Stop talking politics in the office. Go paintballing on weekends. Whatever it is, start establishing a new set of behaviors.

And wait.

Maybe they'll notice and warm up. It'll be hard, but you probably need the practice to change your habits.

See if your peace-offering is accepted

After your new leaf is turned over, it might pay off to draw their attention to it. This step definitely depends on the manager and the source of the problem, but demonstrating a new way of behaving before saying you've been behaving better can be the key to get back into the communications stream. It also hangs a hat on the fact that you noticed you were in bad graces and took effort to change.

What if it's not accepted?

Then learn to live in Siberia and work through proxies, or lump it and get another job.

If you're wondering why comments aren't working, as I was, and are on shared hosting, as I am, and get to looking at your error_log file and see something like this in it:

[Sun Oct 12 12:34:56 2014] [error] [client 192.0.2.5] 
ModSecurity: Access denied with code 406 (phase 2).
Match of "beginsWith http://%{SERVER_NAME}/" against "MATCHED_VAR" required.
[file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "1425"] [id "340503"] [rev "1"]
[msg "Remote File Injection attempt in ARGS (/cgi-bin/mt4/mt-comments.cgi)"]
[severity "CRITICAL"]
[hostname "example.com"]
[uri "/cgi-bin/mt/mt-comments.cgi"]
[unique_id "PIMENTOCAKE"]

It's not just you.

It seems that some webhosts have a mod_security rule in place that bans submitting anything through "mt-comments.cgi". As this is the main way MT submits comments, this kind of breaks things. Happily, working around a rule like this is dead easy.

  1. Rename your mt-comments.cgi file to something else
  2. Add "CommentScript ${renamed file}" to your mt-config.cgi file

And suddenly comments start working again!

Except for Google, since they're deprecating OpenID support.

Other Blogs

My Other Stuff

Monthly Archives