May 2015 Archives

I've seen this dyamic happen a couple of times now. It goes kind of like this.

October: We're going all in on AWS! It's the future. Embrace it.
November: IT is working very hard on moving us there, thank you for your patience.
December: We're in! Enjoy the future.
January: This AWS bill is intolerable. Turn off everything we don't need.
February: Stop migrating things to AWS, we'll keep these specific systems on-prem for now.
March: Move these systems out of AWS.
April: Nothing gets moved to AWS unless it produces more revenue than it costs to run.

What's prompting this is a shock that is entirely predictable, but manages to penetrate the reality distortion field of upper management because the shock is to the pocketbook. They notice that kind of thing. To illustrate what I'm talking about, here is a made-up graph showing technology spend over a course of several years.

BudgetType-AWS.pngThe AWS line actually results in more money over time, as AWS does a good job of capturing costs that the traditional method generally ignores or assumes is lost in general overhead. But the screaming doesn't happen at the end of four years when they run the numbers, it happens in month four when the ongoing operational spend after build-out is done is w-a-y over what it used to be.

The spikes for traditional on-prem work are for forklifts of machinery showing up. Money is spent, new things show up, and they impact the monthly spend only infrequently. In this case, the base-charge increased only twice over the time-span. Some of those spikes are for things like maintenance-contract renewals, which don't impact base-spend one whit.

The AWS line is much less spikey, as new capabilities are assumed into the base-budget in an ongoing basis. You're no longer dropping $125K in a single go, you're dribbling it out over the course of a year or more. AWS price-drops mean that monthly spend actually goes down a few times.

Pay only for what you use!

Amazon is great at pointing that out, and hilighting the convenience of it. But what they don't mention is that by doing so, you will learn the hard way about what it is you really use. The AWS Calculator is an awesome tool, but if you don't know how your current environment works, it's like throwing darts at a wall for accurately predicting what you'll end up spending. You end up obsessing over small line-item charges you've never had to worry about before (how many IOPs do we do? Crap! I don't know! How many thousands will that cost us?), and missing the big items that nail you (Whoa! They meter bandwidth between AZs? Maybe we shouldn't be running our Hadoop cluster in multi-AZ mode).

There is a reason that third party AWS integrators are a thriving market.

Also, this 'what you use' is not subject to Oops exceptions without a lot of wrangling with Account Management. Have something that downloaded the entire EPEL repo twice a day for a month, and only learned about it when your bandwidth charge was 9x what it should be? Too bad, pay up or we'll turn the account off.

Unlike the forklift model, you pay for it every month without fail. If you have a bad quarter, you can't just not pay the bill for a few months and tru-up later. You're spending it, or they're turning your account off. This takes away some of the cost-shifting flexibility the old style had.

Unlike the forklift model, AWS prices its stuff assuming a three year turnover rate. Many companies have a 5 to 7 years lifetime for IT assets. Three to four years in production, with an afterlife of two to five years in various pre-prod, mirror, staging, and development roles.The cost of those assets therefore amortizes over 5-9 years, not 3.

Predictable spending, at last.

Hah.

Yes, it is predictable over time given accurate understanding of what is under management. But when your initial predictions end up being wildly off, it seems like it isn't predictable. It seems like you're being held over the coals.

And when you get a new system into AWS and the cost forecast is wildly off, it doesn't seem predictable.

And when your system gets the rocket-launch you've been craving and you're scaling like mad; but the scale-costs don't match your cost forecast, it doesn't seem predictable.

It's only predictable if you fully understand the cost items and how your systems interact with it.

Reserved instances will save you money

Yes! They will! Quite a lot of it, in fact. They let a company go back to the forklift-method of cost-accounting, at least for part of it. I need 100 m3.large instances, on a three year up-front model. OK! Monthly charges drop drastically, and the monthly spend chart begins to look like the old model again.

Except.

Reserved instances cost a lot of money up front. That's the point, that's the trade-off for getting a cheaper annual spend. But many companies get into AWS because they see it as cheaper than on-prem. Which means they're sensitive to one-month cost-spikes, which in turn means buying reserved instances doesn't happen and they stay on the high cost on-demand model.

AWS is Elastic!

Elastic in that you can scale up and down at will, without week/month long billing, delivery and integration cycles.

Elastic in that you have choice in your cost accounting methods. On-demand, and various kinds of reserved instances.

It is not elastic in when the bill is due.

It is not elastic with individual asset pricing, no matter how special you are as a company.


All of these things trip up upper, non-technical management. I've seen it happen three times now, and I'm sure I'll see it again at some point.

Maybe this will help you in illuminating the issues with your own management.

The No Asshole Rule is a very well known criteria for building a healthy workplace. If you've worked in a newer company (say, incorporated within the last 8 years) you probably saw this or something much like it in the company guidelines or orientation. What you may not have seen were the principles.

Someone is an asshole if:

    1. After encountering the person, do people feel oppressed, humiliated or otherwise worse about themselves?
    2. Does the person target people who are less powerful than them?

If so, they are an asshole and should be gotten rid of.

And if we all did this, the tech industry would be a much happier place. It's a nice principle.

The problem is that the definition of 'asshole' is an entirely local one, determined by the office culture. This has several failure-modes.

Anyone who doesn't agree with the team-leader/boss is an asshole.

Quite a perversion of the intent of the rule, as this definitely harms those less powerful than the oppressor, but this indeed happens. Dealing with it requires a higher order of power to come down on the person. Which doesn't happen if the person is the owner/CEO. Employees take this into their own hands by working somewhere else for someone who probably isn't an asshole.

All dissent to power must be politely phrased and meekly accepting of rejection.

In my professional opinion, sir, I believe the cost-model presented here is overly optimistic in several ways.

I have every confidence in it.

Yes sir.

The thinking here is that reasoned people talk to each other in reasoned ways, and raised voices or interruptions are a key sign of an unreasonable person. Your opinion was tried, and adjudged lacking; lose gracefully.

In my professional opinion, sir, I believe the cost-model presented here is overly optimistic in several ways.

I have every confidence in it.

The premise you've built this on doesn't take into account the ongoing costs for N. Which throws off the whole model.

Mr. Anderson, I don't appreciate your tone.

It's a form of tone policing.

All dissent between team-mates must be polite, reasoned, and accountable to everyone's feelings.

Sounds great. Until you get a tone cop in the mix who uses the asshole-rule as a club to oppress anyone who disagrees with them.

I'm pretty sure this new routing method introduces at least 10% more latency to that API path. If not more.

I worked on that for a week. I'm feeling very intimidated right now.

😝. Sorry, I'm just saying that there are some edge cases we need to explore before we merge it.

😟

Okay, let's merge it into Integration and run the performance suite on it.

This is a classic bit of office-politics judo. Because no one wants to be seen as an asshole, if you can make other people feel like one they're more likely to cave on contentious topics.

Which sucks big-time for people who actually have problems with something. Are they a political creature, or are they owning their pain?


The No Asshole Rule is a great principle in the abstract, but it took the original author 224 pages to communicate the whole thing in the way he intended it. That's 223.5 pages longer than most people have the attention-span for, especially in workplace orientations, and are therefore not going to be clued into the nuances. It is inevitable that a 'no asshole rule' enshrined in a corporate code-of-conduct is going to be defined organically through the culture of the workplace and not by any statement of principles.

That statement of principles may exist, but the operational definition will be defined by the daily actions of everyone in that workplace. It is going to take people at the top using the disciplinary hammer to course-correct people into following the listed principles, or it isn't going to work. And that hammer itself may include any and all of the biases I lined out above.

Like any code of conduct, the or else needs to be defined, and follow-through demonstrated, in order for people to give it appropriate attention. Vague statements of principles like no assholes allowed or be excellent to each other, are not enforceable codes as there is no way to objectively define them without acres of legalese.

Other Blogs

My Other Stuff

Monthly Archives