Burocracy ~= Automation

| No Comments

That was one of the big points in this morning's keynote by Mickey Dickerson. Laws and the federal system is designed to be a big hunk of automation, executed by humans. Failures in the processes are entirely predictable, and like CPUs, the human processors likely can't do much about it.

So, so true.

And also pointed out that the US Federal System is a huge open-source project kicked off a couple hundred years ago with 538 comitters at the moment. There are lessons in big open-source project management in here somewhere, I'm sure.

It turns out one of the biggest goals of the US Digital Service is to ferret out the worst of the bad automation and try to turn it around. Healthcare.gov was their first big project, and right now they're making inroads with the Veterans Administration. They're doing some other things they can't talk about yet because government, but this is all worthy cause kinds of things.

During my last job-hunt I considered applying, but didn't go there due to where I wanted to live and an unclear answer on if they'd take a remote worker. Still, this is good stuff and I wish them all the luck and support.

Security profiling: TSA

| No Comments

Being of a gender-nonconforming nature has revealed certain TSA truths to me.

Yes, they do profile.

It's a white-list, unlike the police profiling that gets people into trouble. There is a 'generic safe-traveler' that they compare everyone to. If you conform, you get the minimum screening everyone gets. If you don't conform, you get some extra attention. Some ways to earn extra attention:

  • Don't look like your government ID.
  • Wear your hair up, or in braids (they've seen those kung-fu movies too)
    • Yes, they put their gloved hands in your hair and feel round. Anyone with dreads knows this all too damn well.
  • Fly with a name other than the one on your government issued ID.
  • Have body-parts replaced with things, such as a prosthetic leg, or knee (if going through metal detectors).
  • Have junk when there shouldn't be junk (or so they think).
  • Have breasts when there shouldn't be breasts (or so they think).
  • Have breast prosthesis instead of actual breasts (mastectomy patients love this).
  • And many more.

Here is an exercize you can try the next time you fly in the US. When you get to the other side of the scanner (this only works for the porno-scanners, not the metal-detectors), while you are waiting for your stuff to come out of the X-ray machine, look at the back of the scanner. Watch the procedure. Maybe put your shoes on slow to catch it all. You'll notice something I've noticed:

There are always two officers back there, a man and a woman. When someone steps in to get scanned, they have to either hit a button to indicate the gender of the person being scanned, or are presented with a side-by-side with both genders and the officer has to chose which to look at. They have a second, maybe two, to figure out which body baseline to apply to you, and those of us who are genderqueer confuse things. I fail the too-much-junk test all the time and get an enhanced patdown in my inner-thighs.

Yes, but with PreCheck you can skip that.

This actually proves my point. By voluntarily submitting to enhanced screening, I can bypass the flight-day screen annoyances. It's admitting that I no longer fit the profile of 'generic safe traveler' and need to achieve 'specific safe traveler' status. That, or I can have my bits rearranged and conform that way. Whichever.

The H-1B and I-140 labor-market

| No Comments

A former employer of mine was a big user of H-1B and I-140 visa-workers. They were big enough we had international development centers, and often those employees came to the US HQ. As an Ops-type, this was pretty awesome since we had follow-the-sun support for our stuff. It also meant holidays in other countries impacted our operations in ways that I'd never experienced before. It was a very diverse workplace, which I enjoyed immensely.

However, the big flaw revealed itself when said company had a really bad quarter and decided to reorganize.

What 'reorganization' involved:

  • A near complete musical-chairs round in the Executive VP, and VP offices.
  • A restructuring of the company into completely new divisions.
  • Completely closing one of our foreign development offices.
  • A serious downsizing of the US workforce.


For those of us who are full citizens, we have a completely fluid job-market. This company was in one of the major tech-hubs (the DC metro area, a.k.a. NOVA, a.k.a. 'Ashburn VA', a.k.a. "us-east-1") where cloud-talent was experiencing a shortage and salaries were rising quickly. Unsurprisingly, a large percentage of our senior cloud-trained talent left the company rather than deal with working for a convulsing organization. Me and most other senior systems-engineer types had cold-calls from both Google and Amazon. Some of us took them up on that offer (not me).

It was during this time that I learned about how different the labor-market was for visa-workers.

Both the H-1B and I-140 visa require employer sponsorship. If you lose your employer, you need to get another job (transfer the visa sponsorship) within a specific time or you have to leave the US. Per my fellow coworkers, that time is four weeks. And that's four weeks to first-day-on-the-job, not four weeks to acceptance-letter-is-signed. Four weeks in which to do a complete job hunt with a company big enough to be able to deal with visas and the Department of Immigration. That is nearly impossible.

As a result, all of my coworkers here on visa were job-hunting just in case they got a layoff. All of them. Half of them had married (always other visa-workers) and bought houses here. Due to the DC metro area being one of the most expensive housing markets in the US, you need two full-time incomes in order to afford anything of any size; so having one of the earners leave the country was a recipe for financial disaster.

My department was one making buckets of money for the company, so we didn't take a layoff at all. Even so, we lost about a quarter of our people due to better offers coming in on that just-in-case job-hunt. The other quarter left due to not wanting to put up with the reorganizations.

Some visa-workers in other departments got layoffs. The rumor mill said they were offered a paid ticket back to their home country in addition to whatever other severance benefits they were giving out. No help with the expense of an international move.


After the layoffs were done and the reorganizations had completed to the point that the org-chart wasn't being updated on a weekly basis, upper management got down to the serious business of trying to keep the people they had. Part of this effort was to rebase salaries to market averages. There were some big movements.

One coworker had been out of the job market for about seven years to raise her son. When she got back into it, she was hired on as an 'associate systems engineer' at a salary of around $70K/year. Fast-forward five years, and she was still at that job-title, but doing a senior systems-engineer's work and getting paid $73K. They reallocated her up to a 'lead systems engineer' title and a salary of $100K.

Another coworker was here on visa and hired as a 'systems engineer'. He was being paid $85K. The full citizen in the chair next to him, also a 'systems engineer', was being paid $102K. After discussing salaries one memorable afternoon (this is legal, don't let anyone tell you otherwise), we learned that all of our visa-people were underpaid versus the citizens. When the reallocation came around, most of this disparity went away.

Why did it get that bad?

Capitalism, of course. The job-market for visa-workers is limited to those companies with skilled enough HR departments to deal with immigration paperwork. This greatly reduces the number of companies they can work for, which in turn reduces upward pressure on salaries. It also means that those companies who have figured out the paperwork problem have access to a skilled job market with less salary pressure than the greater one, so they tend to go deep into it.

The other factor at play here is the internal raise process. As has been mentioned elsewhere, you get your best raises when you change companies. While salaries in the overall job market had been raising, internal ones were not raising as much. People who don't job-hunt until provoked were falling behind their peers and not noticing because talking salary is taboo.

How can we prevent this disparity?

By talking salaries more often. The visa job-market is fundamentally different than the one for full citizens, but our costs-of-living are still the same. By talking about salaries and visa-status the labor market's supply, us workers, can learn which companies are known to exploit visa workers and which are more likely to treat them the same as the citizens. It means being pushy during negotiations, but that's how you stop this kind of exploitation.

Groking audit

| No Comments

I've been working with Logstash lately, and one of the tasks I was given was attempting to improve parsing of audit.log entries. Turning things like this:

type=SYSCALL msg=audit(1445878971.457:6169): arch=c000003e syscall=59 success=yes exit=0 a0=c2c3a8 a1=c64bc8 a2=c34408 a3=7fff44e370f0 items=2 ppid=16974 pid=18771 auid=1004 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=5 comm="compiled_evil" exe="/home/justsomeuser/bin/compiled_evil" key="hinkystuff"

Into nice and indexed entries where we can make Kibana graphs of all commands caught with the hinkystuff audit ruleset.

The problem with audit.log entries is that they're not very regexible. Oh, they can be. But optional sometimes-there-sometimes-not fields suck a lot. Take for example, the SYSCALL above. Items a0 through a3 are arguments 1-3 of the syscall, and there may be 1 to 3 of them. Expressing that in regex/grok is trying.

So I made a thing:

Logstash-auditlog: Grok patterns and examples for parsing Audit settings with Logstash.

May it be useful.

Asynchronus brick walls

| No Comments

I had the dubious honor of running into a brick wall that has so far resisted my skill in both search-engines and asking for help. It went so far that I asked on StackOverflow and got nothing (12 views as of this writing).


So far, trying to do this thing in coffee has taken two days. This is not a language set up to deal with "do this procedure one or more times until you don't get a thing, and then continue with execution," It can be done, I have no doubt about that. I have simply failed to figure out how.

Yesterday I gave up and wrote a ruby script that did what I needed. It took 6 hours to get the features I needed, including various checks to make sure I'm not breaking things by doing it. It worked the way I needed it to, and it was awesome.

And unfortunately proved that I definitely need to figure out the 'one or more' thing in coffee, going forward. My solution to that may very well be 'shell out to an external script that can do that kind of thing and accept the results.'

I think I now know why the Amazon SDK for Javascript doesn't include paging support like the CLI tools written in python do.

Paying for the web

| No Comments

An uncomfortable realization

| No Comments

I have a thing against Apple.

They're not the first tech company I have felt that way towards, but it is a definite thing now.

I've been snarking for a while now that...

But it's more complex than that. I've made quite a lot of money off of Microsoft; the first 13 years of my resume read like a Microsofty (the Novell stuff is ignored these days). Exchange. Web-serving. Clustering. Scaled up file-serving. Yep, made a lot of money with it.

And yet, when the most recent iPhone announcement hit my inbox(*) my reaction was definitely not the one I had about, say, Windows XP back in the day. Apple is not actually a Microsoft, and there are many people out there who will gladly fight me for saying it might be. But in terms of market-share for desktops... 90%? Those are numbers Microsoft was rocking for a long, long time. Admittedly, this is at a startup, not at a 1000+ employee company where manageability trumps individual employee preferences. But small companies become big companies...

My reaction to that email was a vengeful unsubscribe. Apparently I didn't want to be seen to be one of them.

I have Apple in the home. There is an iMac. We have Wifi from Apple because the iMac wouldn't stay attached to anything else. But it isn't anything I use on a daily basis(*).

At work I'm one of four non-Apple users. Three of us are Linux-desktop people, and we have a lone Windows user (who is a pretty chill dude). At least one of the other Linux users seems to feel the same way I do about Apple. So we're out there.

I used to be pretty cool with people using whatever, and believed I could use anything if given time. Heck, I was dual-stack Microsoft/Novell at a time when you were either one or the other. And I was dual-stack Linux/Microsoft at my last startup, which is a kind of unholy union people don't admit to but are actually kind of in demand.

But apparently I draw the line at OSX and iOS. When did that happen?

(*): I have an iPhone. It was a gift from a previous employer. Current Employer doesn't pay for telecoms, so I'm using it as my workphone. Thus, I get Apple marketing mail. And use it daily, because it was a free phone. When Apple decides this model is gauche and it becomes unusable, I'll replace it with an Android and feel relieved.

Or, a post I'd never thought I'd make seeing as I'm a sysadmin.

But it seems I'm the senior git expert in my team, so I'm making it. So odd.

There are a series of questions you should ask among your team before moving a repo over to git. Git is a hell of a toolbox, and like all toolboxes there are nearly infinite ways of using it. There is no one true way, only ways that are better for you than others. These are a series of questions to help you figure out how you want to use it, so you can be happier down the road.

Q: How do you use the commit-log?

History is awesome. Looking back five years in the code repository to figure out WTF a past developer was thinking about writing that bit of spaghetti code is quite useful if that commit includes something like, "found weird-ass edge case in glib, this is the workaround until they get a fix." That's actionable. Maybe it's even tied to a bug number in the bug tracking system, or a support ticket.

Do you ever look through the history? What are you looking for? Knowing this allows you to learn what you want out of your source-control.

Q: What is the worth of a commit?

A commit in Git is not the same thing as in SVN, Fog, or ClearCase. In some, a commit, or checkin, is a pretty big thing. It takes reviews, or approvals before it can be made.

This question is there to get you thinking about what a commit is. Commits in git are cheap, that changes things. Knowing that you will be facing more of then than you had in the past will help guide you in the later questions.

Q: Is every commit sacred, or you do you value larger, well documented commits more?

Practically everyone I know has made a commit with the message of 'asdf'. If you're grinding on a stupid thing, it may take you 19 commits to come up with the two lines of code that actually work. In five years, when you come back to look at that line of code, the final commit-message on those lines might be '

a1bd0809 maybe this will work

Not exactly informative.

bdc8671a Reformat method calls to handle new version of nokogiri

That is informative.

Most projects value more informative commits over lots of little, iterative ones. But your team may be different. And may change its mind after experience has been had.

Q: Should new features be all in one commit, or in a few modular commits?

Some features are quite large. So large, that rebasing them into a single commit leads to a diff of hundreds of lines. Such a large feature means that the history on those files will be slathered with the same initial-feature-commit with no context for why it is that way.

Is that good enough? Mabe it is, maybe you're more interested in the hotfix commits that are fixing bugs and explain non-intuitive behavior and workaround. Maybe it isn't, and you need each sub-feature in its own. Or maybe you want every non-fixup commit.

This is where your approach to the history really informs your decision. If you know how you deal with the past, you will be better able to put process in place to be happier with your past self.

Once you've thought about these questions and your answers to them, you'll be better able to consider the deeper problem of branching strategy. Git is notoriously lacking in undo features, at least in shared repos, so getting this out of the way early is good.

Public Cloud (AWS, Azure, etc) is a very different thing than on-prem infrastructures. The low orbit view of the sector is that this is entirely intentional: create a new way of doing things to enable businesses to focus on what they're good at. A lot of high executives get that message and embrace it... until it comes time to integrate this new way with the way things have always been done. Then we get some problems.

The view from 1000ft is much different than the one from 250 miles up.

From my point of view, there are two big ways that integrating public cloud will cause culture problems.

  • Black-box infrastructure.
  • Completely different cost-model.

I've already spoken on the second point so I won't spend much time on it here. In brief: AWS costing makes you pay for what you use every month with no way to defer it for a quarter or two, which is completely not the on-prem cost model.

Black-box infrastructure

You don't know how it works.

You don't know for sure that it's being run by competent professionals who have good working habits.

You don't know for sure if they have sufficient controls in place to keep your data absolutely out of the hands of anyone but you or nosy employees. SOC reports help, but still.

You may not get console access to your instances.

You're not big enough to warrant the white glove treatment of a service contract that addresses your specific needs. Or will accept any kind of penalties for non-delivery of service.

They'll turn your account off if you defer payment for a couple of months.

The SLA they offer on the service is all you're going to get. If you need more than that... well, you'll have to figure out how to re-engineer your own software to deal with that kind of failure.

Your monitoring system doesn't know how to handle the public cloud monitoring end-points.

These are all business items that you've taken for granted in running your own datacenter, or contracting for datacenter services with another company. Service levels aren't really negotiable, this throws some enterprises. You can't simply mandate higher redundancies in certain must-always-be-up single-system services, you have to re-engineer them to be multi-system or live with the risk. As any cloud integrator will tell you if asked, public cloud requires some changes to how you think about infrastructure and that includes how you ensure it behaves the way you need it to.

Having worked for a managed services provider and a SaaS site, I've heard of the ways companies try to lever contracts as well as lazy payment of bills. If you're big enough (AWS) you can afford to lose customers by being strict about on-time payment for services. Companies that habitually defer payment on bills for a month or two in order to game quarterly results will describe such services as, 'unfriendly to my business'. Companies that expect to get into protracted SLA negotiations will find not nearly enough wiggle room, and the lack of penalties for SLA failures to be contrary to internal best practices. These are abuses that can be levered at startup and mid-size businesses, quite effectively, but not so much at the big public cloud providers.

It really does require a new way of thinking about infrastructure, at all levels. From finance, to SLAs, to application engineering, and to staffing. That's a big hill to climb.

Over the weekend I ran into an article on the Safari Books blog about writing job postings. The big recommendation they put forth was a simple one: the 'requirements' section on every job-posting is where you lose most of the qualified candidates for the job who then elect not to apply. Wouldn't it be nice if you rephrased it to be inclusive? I tweeted about it.

Responses came in two versions:

  1. Favorites.
  2. People telling me I'm senior enough to know better about how the requirements game works.

Yes, I'm pretty senior now. However, in the 15+ years I've been a sysadmin I've job-hunted as one only three times.

Time 1: One app, one offer. It wasn't really a job-hunt so much as a pounce-kill.
Time 2: Was in the middle of a huge recession, I had a stale skill-set, and even in the technology I had skills in I didn't have experience with the hot newness. A startup took a chance on me. That search took better than two years.
Time 3: I was terminated and needed another job RIGHT NOW. It was also a hot market, and I had relevant skills. The firms that gave me offers (I had three) all were applied to in the first week of my search. It only took as long as it did to start working due to the Thanksgiving and Christmas holidays getting in the way of everything. That search took six weeks, of which only three were active search/apply/interview focused weeks.

One of the replies is very good at capturing the spirit of the requirements game as it exists now:

True, that's what the startup that hired me did. They needed a blend of Windows and Linux, and apparently I talked a good enough game they hired me even though I didn't exactly have much being-paid-for-it Linux experience at the time (this was one of the big reasons I left WWU, by the way; they wouldn't let me get out of the Windows box). That job posting? It had a 'Qualifications' section, and didn't mention operating system! This was my in!

I haven't done enough hiring or searching to know how flexible 'requirements' are. If I hit every point, I make sure to mention that in my cover-letter. If I hit all but one, I'm worried but if the one doesn't sound mission-critical I'll still drop an app on it. If I'm missing two... I'll apply only if I really want to work there, or I have some other clue that the 'requirements' are actually a wish-list (big hint: 20 items in the requirements list).

Here's the thing though. If you're suffering impostor syndrome, and I sure as hell was during the Time 2 search, it's extremely easy to talk yourself out of thinking you're good enough for a position.

Do not bother applying unless you have...

  • 6 years of Enterprise Linux administration.
  • 5 years of Python scripting development.
  • 5 years of Postgres administration.
  • 3 years of Chef.

That's what a 'Requirements' section looks like. It takes a sense of entitlement to see that list and add, "...or can talk us into taking you anyway" to the bolded text.

I have one of those bullet-points. However, I do have Ruby, MySQL, and Puppet. Is that enough to take a chance on the position, or are they dead set on not having to train in a new-hire on those things? Can't tell, not going to bother going to all the effort of crafting a resume and coverletter just to be told 'Go away'.

Or maybe I tell my impostor syndrome to go hide in a corner somewhere and trust in my interview skills to win me a chance.

By changing the 'Requirements' away from a checklist of skills, and towards the qualities you need in a new-hire, you remove a big barrier to application. You'll get apps from people in non-traditional career-paths, like the person with a Masters in History but spent the last six years doing statistical analysis automation in AWS on a grant, and kept getting consults from other academic computation people for automating things.

I keep hearing there is a real talent crunch for senior people these days. Doesn't it make sense to encourage applications, rather than discourage them?

Other Blogs

My Other Stuff

Monthly Archives