Recently in opinion Category

Or, a post I'd never thought I'd make seeing as I'm a sysadmin.

But it seems I'm the senior git expert in my team, so I'm making it. So odd.


There are a series of questions you should ask among your team before moving a repo over to git. Git is a hell of a toolbox, and like all toolboxes there are nearly infinite ways of using it. There is no one true way, only ways that are better for you than others. These are a series of questions to help you figure out how you want to use it, so you can be happier down the road.

Q: How do you use the commit-log?

History is awesome. Looking back five years in the code repository to figure out WTF a past developer was thinking about writing that bit of spaghetti code is quite useful if that commit includes something like, "found weird-ass edge case in glib, this is the workaround until they get a fix." That's actionable. Maybe it's even tied to a bug number in the bug tracking system, or a support ticket.

Do you ever look through the history? What are you looking for? Knowing this allows you to learn what you want out of your source-control.

Q: What is the worth of a commit?

A commit in Git is not the same thing as in SVN, Fog, or ClearCase. In some, a commit, or checkin, is a pretty big thing. It takes reviews, or approvals before it can be made.

This question is there to get you thinking about what a commit is. Commits in git are cheap, that changes things. Knowing that you will be facing more of then than you had in the past will help guide you in the later questions.

Q: Is every commit sacred, or you do you value larger, well documented commits more?

Practically everyone I know has made a commit with the message of 'asdf'. If you're grinding on a stupid thing, it may take you 19 commits to come up with the two lines of code that actually work. In five years, when you come back to look at that line of code, the final commit-message on those lines might be '

a1bd0809 maybe this will work

Not exactly informative.

bdc8671a Reformat method calls to handle new version of nokogiri

That is informative.

Most projects value more informative commits over lots of little, iterative ones. But your team may be different. And may change its mind after experience has been had.

Q: Should new features be all in one commit, or in a few modular commits?

Some features are quite large. So large, that rebasing them into a single commit leads to a diff of hundreds of lines. Such a large feature means that the history on those files will be slathered with the same initial-feature-commit with no context for why it is that way.

Is that good enough? Mabe it is, maybe you're more interested in the hotfix commits that are fixing bugs and explain non-intuitive behavior and workaround. Maybe it isn't, and you need each sub-feature in its own. Or maybe you want every non-fixup commit.

This is where your approach to the history really informs your decision. If you know how you deal with the past, you will be better able to put process in place to be happier with your past self.


Once you've thought about these questions and your answers to them, you'll be better able to consider the deeper problem of branching strategy. Git is notoriously lacking in undo features, at least in shared repos, so getting this out of the way early is good.

Public Cloud (AWS, Azure, etc) is a very different thing than on-prem infrastructures. The low orbit view of the sector is that this is entirely intentional: create a new way of doing things to enable businesses to focus on what they're good at. A lot of high executives get that message and embrace it... until it comes time to integrate this new way with the way things have always been done. Then we get some problems.

The view from 1000ft is much different than the one from 250 miles up.

From my point of view, there are two big ways that integrating public cloud will cause culture problems.

  • Black-box infrastructure.
  • Completely different cost-model.

I've already spoken on the second point so I won't spend much time on it here. In brief: AWS costing makes you pay for what you use every month with no way to defer it for a quarter or two, which is completely not the on-prem cost model.

Black-box infrastructure

You don't know how it works.

You don't know for sure that it's being run by competent professionals who have good working habits.

You don't know for sure if they have sufficient controls in place to keep your data absolutely out of the hands of anyone but you or nosy employees. SOC reports help, but still.

You may not get console access to your instances.

You're not big enough to warrant the white glove treatment of a service contract that addresses your specific needs. Or will accept any kind of penalties for non-delivery of service.

They'll turn your account off if you defer payment for a couple of months.

The SLA they offer on the service is all you're going to get. If you need more than that... well, you'll have to figure out how to re-engineer your own software to deal with that kind of failure.

Your monitoring system doesn't know how to handle the public cloud monitoring end-points.


These are all business items that you've taken for granted in running your own datacenter, or contracting for datacenter services with another company. Service levels aren't really negotiable, this throws some enterprises. You can't simply mandate higher redundancies in certain must-always-be-up single-system services, you have to re-engineer them to be multi-system or live with the risk. As any cloud integrator will tell you if asked, public cloud requires some changes to how you think about infrastructure and that includes how you ensure it behaves the way you need it to.

Having worked for a managed services provider and a SaaS site, I've heard of the ways companies try to lever contracts as well as lazy payment of bills. If you're big enough (AWS) you can afford to lose customers by being strict about on-time payment for services. Companies that habitually defer payment on bills for a month or two in order to game quarterly results will describe such services as, 'unfriendly to my business'. Companies that expect to get into protracted SLA negotiations will find not nearly enough wiggle room, and the lack of penalties for SLA failures to be contrary to internal best practices. These are abuses that can be levered at startup and mid-size businesses, quite effectively, but not so much at the big public cloud providers.

It really does require a new way of thinking about infrastructure, at all levels. From finance, to SLAs, to application engineering, and to staffing. That's a big hill to climb.

The No Asshole Rule is a very well known criteria for building a healthy workplace. If you've worked in a newer company (say, incorporated within the last 8 years) you probably saw this or something much like it in the company guidelines or orientation. What you may not have seen were the principles.

Someone is an asshole if:

    1. After encountering the person, do people feel oppressed, humiliated or otherwise worse about themselves?
    2. Does the person target people who are less powerful than them?

If so, they are an asshole and should be gotten rid of.

And if we all did this, the tech industry would be a much happier place. It's a nice principle.

The problem is that the definition of 'asshole' is an entirely local one, determined by the office culture. This has several failure-modes.

Anyone who doesn't agree with the team-leader/boss is an asshole.

Quite a perversion of the intent of the rule, as this definitely harms those less powerful than the oppressor, but this indeed happens. Dealing with it requires a higher order of power to come down on the person. Which doesn't happen if the person is the owner/CEO. Employees take this into their own hands by working somewhere else for someone who probably isn't an asshole.

All dissent to power must be politely phrased and meekly accepting of rejection.

In my professional opinion, sir, I believe the cost-model presented here is overly optimistic in several ways.

I have every confidence in it.

Yes sir.

The thinking here is that reasoned people talk to each other in reasoned ways, and raised voices or interruptions are a key sign of an unreasonable person. Your opinion was tried, and adjudged lacking; lose gracefully.

In my professional opinion, sir, I believe the cost-model presented here is overly optimistic in several ways.

I have every confidence in it.

The premise you've built this on doesn't take into account the ongoing costs for N. Which throws off the whole model.

Mr. Anderson, I don't appreciate your tone.

It's a form of tone policing.

All dissent between team-mates must be polite, reasoned, and accountable to everyone's feelings.

Sounds great. Until you get a tone cop in the mix who uses the asshole-rule as a club to oppress anyone who disagrees with them.

I'm pretty sure this new routing method introduces at least 10% more latency to that API path. If not more.

I worked on that for a week. I'm feeling very intimidated right now.

😝. Sorry, I'm just saying that there are some edge cases we need to explore before we merge it.

😟

Okay, let's merge it into Integration and run the performance suite on it.

This is a classic bit of office-politics judo. Because no one wants to be seen as an asshole, if you can make other people feel like one they're more likely to cave on contentious topics.

Which sucks big-time for people who actually have problems with something. Are they a political creature, or are they owning their pain?


The No Asshole Rule is a great principle in the abstract, but it took the original author 224 pages to communicate the whole thing in the way he intended it. That's 223.5 pages longer than most people have the attention-span for, especially in workplace orientations, and are therefore not going to be clued into the nuances. It is inevitable that a 'no asshole rule' enshrined in a corporate code-of-conduct is going to be defined organically through the culture of the workplace and not by any statement of principles.

That statement of principles may exist, but the operational definition will be defined by the daily actions of everyone in that workplace. It is going to take people at the top using the disciplinary hammer to course-correct people into following the listed principles, or it isn't going to work. And that hammer itself may include any and all of the biases I lined out above.

Like any code of conduct, the or else needs to be defined, and follow-through demonstrated, in order for people to give it appropriate attention. Vague statements of principles like no assholes allowed or be excellent to each other, are not enforceable codes as there is no way to objectively define them without acres of legalese.

Paternity leave and on-call

| No Comments

It all started with this tweet.

Which you need to read (Medium.com). Some pull-quotes of interest:

My manager probably didn't realize that "How was your vacation" was the worst thing to ask me after I came back from paternity leave.

Patriarchy would have us believe that parenting is primarily the concern of the mother. Therefore paternity leave is a few extra days off for dad to chillax with his family and help mom out.

Beyond a recovery time from pregnancy, much of parental leave is learning to be a parent and adjusting to your new family and bonding with the baby. I can and did bond with the baby, but not as much as my female coworkers bonded with their babies.

I should also state, that I don't just want equality, I want a long time to bond with my child. Three months or more sounds nice. Not only can I learn to soothe him when he's upset, put him to sleep without worrying about being paged, but I can be around when he does the amazing things babies do in their first year: learning to sit, crawl, eat, stand and even walk.

At my current employer, I was shocked to learn that new dads get two weeks off.

Two.

At my previous startup, paternal leave was under the jurisdiction of the 'unlimited vacation' policy. Well...

Vacations are important. My friends would joke that the one way to actually be able to take vacations was to keep having children. Here the conflation was in jest, and also a caricature of the reality of vacations at startups.

We had a bit of a baby-boom while I was there. Dads were glared at if they showed up less than two weeks in and told to go home. After that, most of them worked part-time for a few weeks and slowly worked up to full time.

This article caused me to tweet...

The idea here is that IT managers who work for a company like mine with a really small amount of parental leave do have a bit of power to give Dad more time with the new kid: take them off of the call rota for a while. A better corporate policy is ideal, but it's a kind of local fix that just might help. Dad doesn't have to live to the pager and new-kid.

Interesting idea, but not a great one.

Which is a critique of the disaster-resilience of 3-person teams. I was on one, and we had to coordinate Summer Vacation Season to ensure we had two-person coverage for most of it, and if 1-person was unavoidable, keep it to a couple days at best. None of us had kids while I was there (the other two had teenagers, and I wasn't about to start), so we didn't get to live through a paternity-leave sized hole in coverage.

Which is the kind of team I'm on right now, and why I thought of the idea. We have enough people that a person sized hole, even a Sr. Engineer sized hole, can be filled for several to many weeks in the rotation.

That's the ideal route though, and touches on a very human point: if you're in a company where you always check mail or can expect pages off-hours, it doesn't matter if you're not in the official call-rotation. That's a company culture problem independent of the on-call rotation.

My idea can work, but it takes the right culture to pull off. Extended leave would be much better, and is the kind of thing we should be advocating for.

You should still read the article.

This showed up today.

I get that. The little white lie that it's all right, I wasn't offended. The lying silence where the, "check that bullshit," should have been. The desire to belong to the in-group (or an in-group, even if it's an in-group of one) is probably baked into our genetics. Those that arbitrate membership in the in-group set the standards by which membership is granted. So long as there is power there, the little internal betrayals needed to achieve membership, or if that isn't possible, satellite membership, can be justified.

For a while. Until the price starts getting too high.

If the in-group is in all of the positions of both power and employee redress? That's a lot of incentive to shut the fuck up and laugh like you mean it.

And if you keep poking at it, because shuting the fuck up and laughing already is becoming very hard, you lose in-group status.


This is a very human progression, we've been doing it since pre-history. The modern workplace is supposed to be set up to deal with toxic managers and hostile work environments, but cronyism is incredibly corrosive. It takes active push-back to fend off, and of the corruption is deep enough that just costs you your job.

Most corporate severance agreements include something called a non-disparagement clause, which means, in effect:

The severed employee agrees to not say bad things about the Company, or cause material harm to the Company's business through their actions.

And accusing a manager of being a harassing asshole is the kind of thing that could trigger that clause. By telling the world about her experience with this manager, naming names, and calling out the toxic culture of that particular work-unit, she can be considered to be causing 'material harm' and could face serious legal consequences. If Google wants to be assholes about it, of course. But the language is there in the agreement specifically to scare ex-employees out of doing things like this.

The internal system was stacked against her, and the court of public opinion was also stacked against her by the very company that had the bad culture.


I'm guilty of making the same kind of calculations. I didn't seek in-group status as firmly as Kelly did, and it got me fired in the end. It turned out well for me, but was pretty traumatic at the time.

While I was there I did consciously choose to not call out jokes, behavior, or other things that offended me, specifically because I needed to stay on good terms with the in-group. I never got to crying, but the little niggling things did add up. It meant I didn't stay long at company events, didn't follow on after-work outings to bars, and generally stayed quiet a lot of the time. It was noticed.

Smartphone ecosystems have definitely reached the level of complexity where we have to worry about hostile apps. And they're following the pattern shown by the Internet over the years in that there are classes of hostile actions:

  • Known/Allowed, also known as ad/revenue streams. App owners have to pay the bills somehow, and purchase fees only go so far.
  • Known/Disallowed, also known as malware following known exploits. For this we have scanners.
  • Unknown, apps doing things they shouldn't, by ways that aren't in the scanners yet. Evil, evil little beasties.

If there is one lesson about information security that has been true since the beginning, is that it's the victim's fault for getting owned. Really, look at the press following hacks: hacks are entirely the fault of the defending entity for not being good enough. If you just followed accepted security standards, this would never happen. Never mind that transitive trust models in very complex IT infrastructures are nearly impossible to fully secure, especially ones that involve humans, it's still the victim's fault.

Those 'accepted security standards' are somehow lacking in the app-stores, especially Android. It's like the app-owners don't really want you to secure yourself.

What would be very nice in these phone OS security system would be selectable permission filters. Don't want to allow bluetooth-access to any applications except those you whitelist? Don't want to share your contacts with an app that seemingly has no need for it? A limited version of this is in iOS, but as I'll get to in a moment it only goes so far.

There are two methods of denying access to capabilities, and we already have a good example of this two-tier model in the firewall world:

  • Notifies connections of no-connection.
  • Pretends there is nothing there.

The first method is nice for applications since they learn quickly to stop trying. The second is nice for defenders because it means potential attackers have to wait for timeouts before marking a IP:Port tuple as up/down. When it comes to phones, there are two ways to deal with selectable permissions:

  • Notify the app that they don't have rights to that thing. Apps know they're being banned.
  • Lie to the app and provide a stub service that returns nothing or a simple carrier-signal. Apps will have to do tests to see if they're banned.

IOS uses the first model. If you've ever seen a, "turn on bluetooth for an enhanced user experience," modal, that's what happened. I believe that Apple standards say that applications have to honor those settings in that they still run and don't quit in a huff over not getting your identity goodies. You may not be able to do much, but they'll still run.

Android currently doesn't have selectable permissions (out of the box; there are some apps that try to provide it), you decide whether or not an app can be allowed to do it's full list at the time you install it. This can be problematic, especially if circumstances require that you install certain apps, but you want to disable certain capabilities. Such as having only one phone with both work and email on it, and you'd rather they didn't wipe it when they fire you.

That's where things like XPrivacy can come in handy. This only runs on a rooted device, but it provides the stub-services needed to prevent apps from quitting in a huff over not getting the ability to remove accounts on the device, lie about Bluetooth/NFC/Wifi access and state, or falsify 'network' location data. Things like XPrivacy allow us to provide those very 'accepted security standards' that reduce victim-blaming after incidents. It would be awesome if this came stock, but we can't have everything.

Way back when I first got into Group Policies, which was just after Group Policies were released, one of the things we mooted about the BoF den was a simple thing we could do to tell users that they were on a managed station. What we came up with was pretty simple: manage the desktop background.

No, we didn't put an all-seeing-eye on it. That would be creepy, don't be silly. We used a logo of the company.

It made sense! A simple cue, and we'd save RAM (back in those days the desktop background took more than trivial RAM). We were happy.


It turns out, that's not how you build a happy user-base. By doing so, we told people explicitly everything you do can and will be used against you in an HR action. People don't like to be told they're being monitored.

You know who likes to be told they're being monitored like that? No one.

You know who we want to be monitored that way? Prisoners and people likely to become prisoners.

No one wants to be thought of as a prisoner, or likely to be one.

In fact, later GPO guides specifically discouraged doing things like managing the desktop background or theme. It could be done, but... why would you want to? Desktop theme is one very low impact thing on the system and the single biggest thing the user can customize to their preferences. It's a very low challenge to the system to increase user experience by a great amount. Let them customize and don't worry about it.

But still manage their IE zones, certificate enrollment policies, software distribution methods, and event-log reporting.

They can make their jail-cell a pink polka-dot wonder, far better than bare cinder-block! It's still a cell, but without that camera in their face, they're happier about living in it.


It looks like consumer-focused big-data stuff is suffering the same faults as early GPOs did: they're being too obvious about the surveillance.

"Hello, Mister ${mispronounced last name}," said the sales-clerk I'd never met before. I sighed in resignation, vowing to factory reset my cell-phone. Again. One of these days I'm just going go cash only.

Or another one I almost guarantee will happen:

TSA Customer Service
@sysadm1138 We noticed you were in DFW security line for 49 minutes. We would like some feed back about that, https://t.co/...

Er, wait. That's Big Brother. Sorry, dial slipped. Let's try again.

VIctorias Secret
@sysadm1138 We noticed you spent time in our DES MOINES, IA store. If you have time, please take a short survey about your visit. https://t.co/...

You've probably run into this one, but hitting a random website, and then that site haunts your web-ads (for those of you who don't run on AdBlock-Strict) for weeks.

They haven't figured out that a large percentage of us don't like being reminded we live in a panopticon. Give me my false illusion of anonymity and I'm happy!

It's all about the user-factors. What's good for the retailer, is not always good for their consumers. Obviously. But the best kind of thing like that are things that aren't obviously not-good for the consumer.

User-factors, people!

A minimum vacation policy

| No Comments

A, "dude, that's a cool idea," wave has passed through the technology sector in the wake of an article about a minimum vacation policy. This was billed as an evolution of the Unlimited Vacation Policy that is standard at startups these days. The article correctly points out some of the social features of unlimited-vacation-polices that aren't commonly voiced:

  • No one wants to be the person who takes the most vacation.
  • No one wants to take more vacation than others do.
  • Devaluing vacation means people don't actually take them. Instead opting for low-work working days in which they only do 2 hours remotely instead of a normal 10 in the office.

These points mean that people with an unlimited policy end up taking less actual vacation than workplaces with an explicit 15 days a year policy. Some of the social side-effects of a discrete max-vacation policy are not often spelled out, but are:

  • By counting it, you are owed it. If you have a balance when you leave, you're owed the pay for those earned days.
  • By counting it, it has more meaning. When you take a vacation day, you're using a valuable resource and are less likely to cheapen it by checking in at work.
  • There is never any doubt that you can use those days, just on what days you can use them (maintain coverage during the holidays/summer, that kind of thing).

Less stress all around, so long as a reasonable amount is given. To me, this looks like a better policy than unlimited.

But what about minimum-vacation? What's that all about?

The idea seems to be a melding of the best parts of unlimited and max. Employees are required to take a certain number of days off a year, and those days have to be full-disconnect days in which no checking in on work is done. Instead of using scarcity to urge people to take real vacations, it explicitly states you will take these days and you will not do any work on them. For the employer it means you do have to track vacation again, but they're required days, don't create the vacation-cash-out liability that max-vacation policies create, and you only have to track up to the the defined amount. If an employee takes 21 days in a year, you don't care since you stopped tracking one they hit 15.

The social factors here are much healthier than unlimited:

  • Explicit policy is in place saying that vacations are no-work days. People get actual down-time.
  • Explicit policy is in place that N vacation days shall be used, so everyone expects to use at least those days. Which is probably more than they'd use with an unlimited policy.
  • Creates the expectation that when people are on vacation, they're unreachable. Which improves cross-training and disaster resilience.

I still maintain that a max-vacation policy working in-hand with a liberal comp-time policy is best for workers, but I can't have everything. I like min-vacation a lot better than unlimited-vacation. I'm glad to see it begin to take hold.

Categories

| No Comments

Humans are curious critters. We keep trying to pick apart reality to figure out how it works. Part of that is to break reality up into smaller chunks so it makes sense. Abstractions improve understanding and allow further refinements to the model. It's what science is based on.

Biology is a continually vexing problem, though. In many ways, it's a continuous function we keep attempting to turn into a discrete maths problem. Taxonomy, the naming of species, is a great example of this. Early classification methods relied on similar morphology to determine relatedness, and that gave us a nice family tree. Then we figured out how to sequence genomes and we learned how wrong we were; they're now moving whole species/phylum branches around. It turns out nature sometimes solves the same problem the same way through completely unrelated species.

What sparked the rearrangement? A new way to classify. A new method was picked to be more accurate, and changes had to be made.

Topically, take a look at OS classification of non-mobile consumer computing devices (what used to be called desktop-OS). You can see this on any web-visitor analytics platform. Some break it down like:

  • Windows
  • OSX
  • Linux

Others get more specific, breaking it down to versions within OS:

  • Window XP
  • Windows 7
  • Windows 8
  • OSX 10.6
  • OSX 10.7
  • OSX 10.8
  • OSX 10.9
  • OSX 10.10
  • Linux

For some reason they don't break apart the Linux versions. Perhaps because it's such a small segment of the market and highly fragmented at that. Still more detailed charts go down to Windows service-pack levels. OS version is a discrete space, but in order to provide a brief chart some simplifications are made. Each analytics application makes its own classification decisions.

Less topically, lets take a look at a fictional made-up species, the Variegated Civet. Take the physical sex of this critter. The original population study was done in 1906 and an odd sex ratio was observed, 1.3 females to every male. As with all studies of the time, external morphology studies were used to determine sex with a few dissections as a cross-check.

Fast forward a bunch of years and genetic studies become financially doable for an appropriate sample-size of the population. It reveals a funny thing. Some of the females are genetically male. This raises eyebrows and further studies reveal the cause. A significant percentage of males undergo gonadogeneis at puberty, not in-utero, which skewed the original study's sex ratio.

A new classification technique, genetics, reveals an interesting feature in a specific population. It also raises the question of what how to differentiate pre-puberty males with fully formed gonads from those who will do so later. A third sex may need to be created to explain this species.

We're undergoing an attempt to change the cultural classification method for gender in humans. For ages it has been based on physical morphology and came in two types. Nature being nature, there are plenty of ambiguous presentations to make the classifier problem harder (intersex); but not enough to prompt the creation of a third gender. Those weird-cases were assigned into one or the other, which ever was closer, in the opinion of the classifier (sometimes nudging things along with a bit of surgery). For ages gender was a synonym of sex.

That's beginning to change, and it's not been an easy thing to bring about. For one, gender is becoming more widely seen as discrete from sex. For another, gender is at the early stages of redefining its classifier away from external morphology and/or chromosomes and into self description. Self-description brings it away from a discrete function (binary, trinary) and into more of a multi-axis graph.

It takes a long time for a change like this to take hold, and there are fights being had. Vital Records only record one of these, and it's still legally entangling both sex and gender. Maybe in some future time driver's licenses will have both sex and gender fields on it. Or maybe those fields will be left off all together (the better option, in my opinion). Chromosomes are not truth, as nature's continuous function ensures there will always be an exception (complete Androgen Insensitivity Syndrome is the big exception to the XY = Male 'truth').

The work continues.

No, I'm talking about that fancy wristband some of you wear, the one that talks to a smartphone. That's a monitoring system, but for your body.

We're IT. We do monitoring systems, so lets take a look at this one!

Other Blogs

My Other Stuff

Monthly Archives