Recently in lopsa Category

Burocracy ~= Automation

That was one of the big points in this morning's keynote by Mickey Dickerson. Laws and the federal system is designed to be a big hunk of automation, executed by humans. Failures in the processes are entirely predictable, and like CPUs, the human processors likely can't do much about it.

So, so true.

And also pointed out that the US Federal System is a huge open-source project kicked off a couple hundred years ago with 538 comitters at the moment. There are lessons in big open-source project management in here somewhere, I'm sure.

It turns out one of the biggest goals of the US Digital Service is to ferret out the worst of the bad automation and try to turn it around. was their first big project, and right now they're making inroads with the Veterans Administration. They're doing some other things they can't talk about yet because government, but this is all worthy cause kinds of things.

During my last job-hunt I considered applying, but didn't go there due to where I wanted to live and an unclear answer on if they'd take a remote worker. Still, this is good stuff and I wish them all the luck and support.

Unions and tech-workers

My fellow technologists are some of the most rabid anti-union people I know. However, I believe there is a kind of union that we can actually benefit from. Read on.

By far the biggest complaint I hear against unions is this one:

BozoBob does a fifth of the work I do and he gets paid the same. How the **** is that fair? ****ing unions!

With a distant second of:

BozoBob has been here fifteen years and does half of what I do. So why the **** am I getting laid off and not him? Seniority?? ****ing unions!

Now that the big critiques are on the table, to look at what unions could bring to the the tech worker market I'll need to set some definitions.

Tech Worker(a): The intuitive definition of this is, "People who work in tech the way I do", which for readers of this blog is primarily the systems administration space and closely adjacent areas like software engineering and network engineering. These are people who work with tech, tend to be highly skilled, and also tend to be highly mobile; if a job is crappy, get a new one, there is always one out there for you.

Tech Worker(b): Now there is a different class of worker out there. These are people who work for tech, so the food service workers at the Google Cafeteria, the massage therapist working for a tech startup, the accountants who do the taxes, the travel specialists who arrange conference travel, the driver of the shuttle bus that brings workers from the affordable part of town to the company campus. These are the people who work for tech who aren't getting six figure salaries and can't necessarily leave a crappy job for a new one at the drop of a hat.

Exempt Employees: A bit of a labor-relations term, but this refers to the Fair Labor Standards Act definition of an employee who is exempt from mandatory overtime. There is a lot of labor law, and worse, case-law, that goes into defining who is owed what for hours worked and I'm not going to get into that here. Employers really like to employ nothing but exempt employees because it means they don't have to track hours worked, and don't have to pay anything extra if everyone is putting in 60 hour weeks.

Tech Worker(a) is almost always Exempt.

Non-Exempt Employees: As you'd expect, these employees earn overtime and are almost always hourly. And frequently part-timers. In many cases employers set rules in place to prevent the earning of overtime unless specifically cleared, which in turn guarantees a maximum 40 hour work-week for these employees.

Tech Worker(b) is often Non-Exempt.

Independent Contractors: a.k.a. "1099 workers". These are not employees of the company, they're employes of another company paid to do something the company wants. That 'other company' may be the employee themselves acting in a corporate way, or it could be a personal-resource firm. Such workers rarely have benefits provided by the company itself, expecting the contracted company to provide for it. They only get 'overtime' if it has been specifically negotiated.

(If you've ever wondered why in blazes contractors charge $200/hour for their work, it's because they're having to pay for their own health coverage, both the employee and employer FICA payment, and other such 'invisible' costs of employing someone).

Traditional Unions

The ones everyone loves to hate (see above critiques for the biggest hate-sources) came about as a push-back to excesses business took during industrialization. Textile work, assembly-line work, and all sorts of other big-headcount industries needed workers, and they didn't need particularly skilled workers since most of it was learned on the job. Which meant that if an employer didn't like a worker for some reason, they fired them and got a new one since there was always another hungry mouth to take the job and wouldn't be such a problem.

So if you wanted to keep your job, you kept your mouth shut and didn't make waves. Which in turn meant putting up with working 6 days a week, being asked to work late for big jobs, possibly buying all of your work-equipment at the company store (at company prices), and a whole bunch of other things. When you are a faceless cog in a sea of cogs, saying "I am special!" means you just get replaced with another faceless cog.

DevOps is all about removing "I am special!" virtual-machines from a sea of identical cogs; er, virtual machines. Yes, I do sometimes worry about the robot revolution.

Traditional unions came about as a way to push back against the exploitation business owners found so profitable. If all the faceless cogs band together and say, "Oy, enough with the 12 hour work days already," it's a lot harder to ignore them. The owners sure as hell tried, but once union penetration in a given industry is high enough there isn't a big enough pool of faceless cogs to ruthlessly exploit without side-effects, which in turn means they have to improve working conditions to compete with the unionized workplaces.

Traditional unions as most people understand them work best in industries where workers are easily interchangeable. The power each individual worker has is so small that there is no leverage for them to exploit other than banding together to pool their power. Seniority is a fair proxy of effectiveness as a worker since actual training only goes so far.

Traditional Unions and Tech Worker(a)

Tech Worker(a) is:

  • Highly skilled.
  • Specialized in diverse technologies.
  • There is a shortage of such workers in certain markets.

Tech Worker(a) is not:

  • Interchangeable with each other.
  • Quick to train up from scratch.

Tech Worker(a) is a very poor fit for traditional unions, as they do not fit the profile of 'exploitable class'. This is a problem for the US Civil Service, which is largely unionized; they have a hell of a time retaining this class of tech worker. Those that do stick around in Civil Service do so for reasons other than monetary.

Traditional Unions and Tech Worker(b)

These are workers who could benefit from a traditional union. These workers are working for companies that grew up thinking they only employed type A tech-workers and have a second class of workers supporting them. A lot of the 'support' roles may be filled by Independent Contractors, who in turn just may belong to a union such as the SEIU, IBEW, or CWA; the labor relations part of HR is handled by the contracted agency not the tech-company itself

Tech companies frequently do 'outsource' such services as the ethos of:

Do what you do best,
If you're not the best, pay someone who is the best to do it,
If you can't find someone who is the best, find someone who can be the best and help them become the best.

Is very prominent. This is why the -as-a-service movement is taking off, and the same model works for staffing the company cafeteria and running the employee-shuttle fleet. In many cases, it's not Google we should be pushing to allow unions in their support tier, it's the contractor Google pays to provide the support services. And if it so happens that Google is directly employing these people, then definitely push for collective bargaining.

Tech Worker(b) is the traditionally unionizable class of technical worker. Tech companies resist unions for many reasons, chief among them being they haven't had to deal with them before and they're good at increasing costs; very much against the lean concept.

Non-Traditional Unions

Or, the unions people don't think of as unions.

These come in a couple of different types, from Professional Organizations like AICPA for accountants and NCBTMB for massage therapists, to the wobblies (IWW) which push for unionization across the entire workforce rather than just one sector of it.

LOPSA is starting down the Professional Organization path with the LOPSA Recognized Professional program they announced at LISA 2013.

Professional Organizations often have some interaction with state/county/local government license boards, which gives them some geographic cover. AICPA manages the CPA exam. NCBTMB manages the national certification test, and ABMP does legislative advocacy and provides insurance support for massage therapists. There are others out there, and these tests/exams are in place to ensure that people who claim to be a certain type of professional actually have a minimum amount of knowledge and commitment to continuing education.

Such organizations are less concerned with working conditions than they are with ensuring the profession as a whole is viewed positively, and people know what to expect when they hire that kind of professional. Some do work against professional exploitation and can even bring sanctions down on workplaces who violate codes of conduct. These are unions, they're just named differently.

Professional Organizations and Tech Worker(a)

As I mentioned earlier, LOPSA is taking a step down this road for the Systems Administration profession with their LOPSA Recognized Professional program. This is not without controversy as the audience at LISA13 demonstrated, and the ongoing debate in the mailing lists and elsewhere continue to show. It's a move by a professional organization that represents a minority of the profession attempting to set a standard for the whole profession, for the betterment of that profession.

Vendors have been doing this for years, though not under the guise of a profession. The Cisco series of certifications are commonly used as gatekeeping tests for hiring network engineers. LOPSA is attempting something similar with the profession, we'll see if there is any traction there.

Professional Organizations don't do collective bargaining the way traditional unions do, they're more about indirect influence. LOPSA won't call a strike against, say, ACME Widgets; and even if they did it wouldn't be legal so ACME Widgets would be perfectly in the right to seek legal action. No, they're much more about making sure the profession as a whole is supported and has some form of group advocacy pushing for improvement.

Like traditional unions organizing an industry, Professional Organizations are more effective the more people belong to them. LOPSA represents a minority of systems administrators right now, so they face an uphill battle to professionalize it. In 10 years that may be different, and LOPSA membership may be on most system-adminsitrator resumes.

The Professional Organization is the kind of collective action entity that Tech Worker(a) can actually benefit from. It leaves individual salary and compensation negotiations in the hands of individual workers, and doesn't have any bearing on who gets the axe in a layoff. It can, however, recommend minimum acceptable benefits and compensation levels for these workers which will aid in salary negotiations by individuals. They can provide backstop health and/or liability insurance for independent contractors at rates much better than individuals can obtain. They can provide industry-proficient legal support, or referrals to such. They work with the educational sector to ensure up-and-comers get what they need to join the profession.

Professional Organizations and Tech Worker(b)

Some of these workers already belong to Professional Organizations. The CPAs that do the company books? AICPA. The massage therapist in the headquarters Health Center? ABMP or AMTA. The tech companies that employ them are already used to working with the members of these organizations, and by extension the organizations themselves. Individual barganing is very much a lean concept, and these organizations maintain that.

Tech workers like myself can benefit from collective action. The best template for us is the Professional Organization: active in advocating for the industry, provides support for members, and helps ensure the educational pipeline is adequate for supporting the profession.

Systems Administration can benefit from professional organizations, and so could Software Engineering.

tl;dr: Unions bad. Professional organizations, a kind of union, good. We should get some.

Oh look, recognition

My LOPSA membership is up for renewal so I was dealing with that today. I practically never log in over there, so I missed an addition to their profile page.


They've got more than one option for gender in there! Demographically, I'm likely not the first one to check that box. Likely not even in the first five (jokers not withstanding), though first ten is much more probable. But it's there! And LOPSA is big enough that there are probably more of us than statistical noise! Yay!

Confidence vs. Competence

| 1 Comment

Tech Competence: How good you actually are at whatever it is that you're doing.
Tech Confidence: How good you think you are at whatever it is that you're doing.

This post is inspired by a recent article that made a lot of very good points relating to the grass-roots of Open Source projects. As it happens, the same points also apply to closed source products (such as what I do for a living) and to the Systems Administration community as a whole.

For the tl;dr crowd, a bulleted list of key points:

  • Building a culture of elitism (also known as a robust meritocracy) means:
    • You risk not being able to replace retiring gurus with fresh ones.
    • You risk not being able to grow the community as a whole due to the barriers in place to keep out the non-elite.
    • Efforts to change up the diversity in the community will largely fail due to those same barriers.
  • Communities defined by tech competence are more often actually based around tech confidence.
    • Building barriers based on competence therefore do more harm than good in sustaining that community.
    • To grow such communities, it is far more effective to focus on building new member confidence.

For fully volunteer efforts like an Open Source project, these are very key things to keep in mind. If a project gets too cliquey it risks dying from lack of investment. Such projects live and die by their community.

For professional efforts like what I do for a living (we get paid for this so we kind of have to work together) you can get away with not doing this kind of thing, but it's still a bad idea. For one, new-hires will have to be shoe-horned in somehow and if the going gets rough they'll just jump ship to a less difficult company. As always, office culture matters quite a lot.

For outreach efforts advocating for a specific community it also matters quite a lot. For instance, LOPSA is such an advocate for the Systems Administration community as a whole and has faced significant headwinds in part from Systems Administrators generally being perceived as an elitist bunch of assholes (the dragon in the datacenter; do not poke unnecessarily, or your User Picture will be turned into smoking boots ). Changing an entire profession is a very hard problem.

Now for some specifics.


Like, really early. And this is a good thing! In previous years it opened just about the time my employers stopped accepting travel requests, since they both wanted several months lead time to get the best deals on flights. This is a good thing they're finally doing!

Not that I need it this year since LISA is in DC this time around. And on the Metro, so I won't even need a hotel.

But still, a trend I encourage!

The push for IPv6


This is inspired from last night's LOPSA DC meeting. The topic was IPv6 and we had a round-table.

One of the big questions brought up was, "What's making me go IPv6?"

The stock answer to that is, "IPv4 addresses are running out, we'll have to learn at some point or be left behind."

That's all well and good, but for us? Most of us are working in, for, or with the US Government, an entity that is not going to be experiencing v4 address scarcity any time soon. What is going to push us to go v6 (other than the already existing mandate to have support that is)?

In my opinion, it'll come from the edges. IPv6 is a natural choice for rapidly expanding networks such as wireless networks, and extremely large networks like Comcast/Verizon run for their kit. These are two areas where sysadmins in general don't deal with much at all (VPN and mobile-access being the two major exceptions).

If your phone has an IPv6 address and accesses the IPv4 internet through a carrier-grade NAT device, you may never notice. Joe Average User is going to be even less likely to notice so long as that widget just works. Once v6 is in the hands of the "I don't care how it works so long as it works" masses, it'll start becoming our problem.

Once having a native v6 site means slightly better perceived mobile performance (those DNS lookups do cause a bit of latency you know), you can guarantee that hungry startups are going to start pushing v6 from launch. Once that ecosystem develops it'll start dragging the entrenched legacy stuff (the, er, government) along with it.  Some agency sites are very sensitive to performance perception and will adapt early. Others only put their data online because they were told to and will only move when the pain gets to be too much.

Business-to-business links (or those between .gov agencies, and their .com suppliers) will likely stay v4 for a very, very long time. Those will also be subject to pain-based mitigation strategies.

But the emergence of v6 on mobile will likely push a lot of us to get v6 to at least our edges. Internal use may be long time coming, but it'll show up at all because of the need to connect with others.

This was posted earlier this year, but I only just ran across it.

Summary: Alcohol seems to pervade (javascript) culture. Big boozy events are at every conference, and this is exclusionary.

Is this a problem at Sysadminly conferences?

Founding a chapter takes people

Two kinds of people are needed for this DC chapter:

  1. LOPSA members, or people willing to become one. These will be our Charter Members.
  2. People willing to present on things.

The two can be one and the same person, but we still need to cover both types.

Presenting really isn't that hard, we're not holding you up to the standard shown at industry conferences. What we need is interesting. We already do interesting in the halls at conferences as we swap stories, so plenty of us already have the technical story to tell. The next step is working up some slides, or a demo, to show off whatever that is.

What qualifies as a good topic? Plenty!

  • Solving a tricky cross-platform issue in Puppet/cfengine/chef/bcfg.
  • Adding Windows to an existing Linux-only configuration-management system.
  • Disaster recovery post-mortem notes.
  • Office campaigns to convince financial powers-that-be that a major upgrade really is in their best interest, how the campaign was fought and won.
  • Anything having to do with scaling out systems, and problems encountered.
  • Creative ways of dealing with bring-your-own-device policies.

We have tech-startups, big government, and mid-size private companies all around the area, so there is a lot of potential audience for your story. And these meetings are the kind of place you can share just those stories.

Think you'd like to listen to these stories?

Think you could maybe share one or two?

Fill out this form!

Or drop a comment on this post! They're screened so I'll see them before they go public, and I promise I won't publish the comment if you ask me not to.

Starting a local LOPSA chapter

One thing lead to another and I'm now helping to co-found a Washington DC LOPSA chapter with LOPSA board-member Evan Pettery. We've had a chapter in the area for some time, the Crabby Admins, but I've yet to make a meeting since getting from downtown DC to where those meetings are is quite a challenge. Of the "get home from work early then drive there through Rush Hour traffic to a spot equidistant between DC and Baltimore" kind of challenge. We expect this new chapter will draw from the central DC and Northern Virginia areas.

It also helps that our meeting site is on Metro, just off Franklin Square. In fact, it's the same spot as the DC RUG and some MongoDC meetings.

What we need right now are two big things:

  • People who want to attend
  • People who want to show off what they've been working on

Drop a note here, or on the website and we'll get in touch.

A local sysadmin conference

I had heard of it before, but it looks like they managed to get it off the ground. The Cascadia IT Conference is a LOPSA-sponsored event, focusing on System Administration. It being within driving-range of my house and having a conference fee of under $1000 means that my ability to go is not zero. This is exciting. I haven't been to a conference since my last BrainShare in 2008, people are going to forget I exist.

The training schedule is pretty light, but they're in their first year so attracting both talent and people confident enough to present is tricky at this stage.