Recently in budget Category

For many reasons, discussions at work have turned to recovering/handling layoffs. A coworker asked a good question:

Have you ever seen a company culture recover after things went bad?

I have to say no, I haven't. To unpack this a bit, what my coworker was referring to is something loooong time readers of my blog have seen me talk about before, budget problems. What happens to a culture when the money isn't there, and then management starts cutting jobs?

In the for-profit sector of American companies, you get a heck of a lot of trauma from the sudden-death style layoffs (or to be technical reductions in force because there is no call-back mechanism in the SaaS industry). Sudden loss of coworkers with zero notice, scrambling to cover for their work, scrambling to deal with the flash reorganization that almost always comes with a layoff, scrambling to worry about if you'll be next, it all adds up to a lot of insecurity in the workplace. Or a lack of psychological safety, and I've talked about what happens then.

This coworker was also asking about some of the other side effects of a chronic budget crunch. For publicly traded companies, you can enter this budget crunch well before actual cashflow is a problem due to the forcing effects of the stock market. Big Tech is doing a lot of layoffs right now, but nearly every one of the majors is still profitable, just not profitable enough for the stock market. Living inside one of these companies means dealing with layoffs, but also things like:

  • Travel budget getting hard to approve, as they crank down spend.
  • Back-filling departures takes much longer, if you get it at all.
  • Not getting approval to hire people to fill needs.
  • Innovation pressure: have to hit bolder targets with the same resources, and the same overhead.
  • Adding a new SaaS vendor becomes nearly impossible.
  • Changing the performance review framework to emphasize "business value" over "growth".
  • Reduced support for employee groups like those supporting disabled and racial employees.
  • Reorgs. So many reorgs.
  • Hard turns on long term strategy.

If a company hasn't had to live through this before the culture doesn't yet bear the scars. The employees certainly do, especially those of us who were in the market in 2007-2011. Many of us left companies to get away from this kind of thing. That said, the last major down period in tech was over ten years ago; there are a lot of people in the market right now who've never seen it get actually bad for a long period of time. "Crypto Winter", the fall of many crypto-currency based companies, was as close as we've gotten as an industry.

When the above trail of suck starts happening in a company that hasn't had to live through it before, it leaves scar tissue. The scars are represented by the old-timers, who remember what upper management is like when the financial headwinds are blowing strong enough. The old salts never regain their trust of upper management, because they know first-hand that they're face eating leopards.

Even if upper management turns the milk and honey taps on, brings out the bread and circuses, and undoes all the list-of-suck from above, the old-timers (which means the people in middle management or senior IC roles) will still remember upper management is capable of much worse. That changes a culture. As I talked about before, trust is never full again, it's always conditional.

So no, it'll never go back to the way it was before. New people may think so, but the old timers will know otherwise.

When should you solve a problem by building a solution, and when should you buy a solution? This is an age old problem in the software industry, with no wide consensus beyond a simple aphorism.

Build your core competencies, buy everything else. This allows you to focus your creative efforts on what you do best.

But there is a world of nuance in this seemingly simple statement. At the far end of one side of the argument you have Google: build everything, because no one else operates at our scale. At the far other end you have new startups: make other people do everything except our core business logic, because that gives us the speed to establish ourself in the market.

Lets take a look at the radically different incentives faced by Google and a generic new startup:

What resources do I have to build a new solution?

Google: We're one of the biggest software engineering companies in the world, so have a lot of engineering talent
Startup: We're five people and a big idea. Every engineer is precious.

What resources do I have to buy a new solution?

Google: So much money, but the third parties who can work at our scale are all direct competitors in different ways.
Startup: I have VC money, and may have Board members from SaaS companies who will give us discounts.

What solutions are available on the market for a company my size?

Google: Major foundations like the Cloud Native Computing Foundation and the Apache Foundation may have platforms from companies near our size we can adapt.
Startup: A whole universe of SaaS providers focused on solving my every need.

If I need to buy something from a new vendor, how hard is it?

Google: All new vendors need to go through a Vendor Security Assessment, and once that's complete, Legal needs to ensure certain standard contract clauses are in place before we can start cutting purchase orders. Maybe two quarters before it can be used at all, then another two quarters to production-scale it.
Startup: The CTO has a company credit-card for a reason. Two days to buy, maybe three sprints to get it into production.

If we need to use a new OSS platform, how hard is it?

Google: Throw 30 engineers at it to solve the scaling problems and ship code upstream. Proof of Concept in one quarter, another two to solve the production-scaling issues.
Startup: Buy our cloud vendor's managed version if at all possible, or something off their marketplace if not; one engineer works to build a POC. Maybe three sprints to production.

You begin to see how different the incentives are between one of the biggest software engineering companies on the planet and the new kids. The new kids are all buy buy buy for a lot of good reasons, where Google is build build build for equally valid reasons.

Looking at this from an order of magnitude point of view, a Startup has single to double-digits worth of engineers (1-99). After a few VC rounds they may get into the triple digits (100-999 engineers). A Google has five to six digits of engineers, a radically different company in so many ways. Each engineer at the startup is represented by at least one whole team at Google, more likely a group of teams, and sometimes an entire department.

But what about the middle ground, the companies with four-figures of engineers? Where do they fit on the build/buy continuum?

It turns out that's really freaking hard. A four-digit-engineers company likely has the same barriers to onboarding new vendors Google had above (have to VSA vendors, Legal needs special things in the contract). At the same time, they have a much shallower engineering pool to draw from to build things instead. This is the most awkward spot to be.

  • (buy) Getting new SaaS vendors onboarded enough you can give them money is a lot of painful work.
  • (build) Your engineering depth is pretty great for your core business work, but prone of availability outages for non-core work. You can build it, but once the motivated engineers move on the built system often decays.

It turns out these 1000 - 9999 engineer companies are where another dynamic emerges to push the decision needle towards build before it's a really great idea to do that: the speed of adding headcount versus the speed of paying a vendor.

Adding headcount is a fully pipelined service in a company this size. They're hiring engineers all the time, doing it in multiple legal jurisdictions, and have internal assets dedicated to making this process as frictionless as possible. Engineers are classified on a scale from 1 to 7 (or 8, or 9) which changes how they can be used. If you need to throw engineers at a problem, and have the open requisitions for it, you can have butts in seats in less than three months. Engineers are (an expensive) commodity.

Contrast this to adding a new vendor. Before any contract can be signed the Security group needs to do a Vendor Security Assessment of the vendor to ensure they're up to your company's standards with regards to safe handling of data and a variety of other things. This VSA process can take months and is prone to backlogs if the VSA processors are themselves clogged with work. Once the VSA is done, negotiating the contract comes next. Many companies try to add clauses to contracts to clarify ambiguities or to push for different treatment, which all means legal back-n-forth that adds time. Privacy teams need to asses the vendor for what types of private data the vendor is allowed to touch, which is its own process. Then the contract gets signed. This can take up to a year end to end. Each vendor is a unique snowflake.

But what if we don't have open reqs for headcount?

Even then, there are incentives to build. In this case, you're looking at moving engineering priorities around to make room for the development work on the new thing. That's the nice thing about headcount: you can use it for more than what you initially bought it for, something a vendor solution rarely provides.

Never forget accounting, though. Costs of employees are often tracked rather differently than costs of suppliers, which definitely does mask the full cost of ownership of a homebrew solution versus a vendor-provided solution. Most costing analysis considers engineering capacity to be "affects future roadmap", where vendor solutions come with direct costs measured in currency and profit/loss statements. If you want to be serious about like-to-like comparisons of a homebrew vs vendor solution, factor in the salary/bonus costs of the people in charge of building and then later maintaining the solutions.

If you're in one of these mid-size, four-figures-of-engineers companies, there are a few things you can do to try and de-skew the incentives at play and get closer to the business focused reasons for a build or a buy decision:

  • Adopt a total-cost-of-ownership model. This model should include the salary/benefits costs associated with the people needed to keep the solution running and maintained. By adding people-costs, you fight the speed skew of how fast you can get new employees versus how long it takes to add a new vendor.
  • Consider including reputational enhancements in cost/benefit analysis. If you're considering adopting an open source product, a company of this size has enough engineering talent to contribute back to the project and the industry. This increases your company's influence in overall industry direction, is a useful source of "engineering culture" blog-posts for your recruiters, increases the visibility of your Developer Relations people, and improves your company's visibility in conference talks.
  • Consider engineering onboarding impacts. If you're using an open source project, or a commonly used SaaS vendor, people are far more likely to walk in the door on day 1 already knowing how to use those systems. If you drank deeply from the Google well and built everything yourself, new people will take far longer to work at their full capacity.
  • Resource build efforts for the long haul. If you decide to build something because existing solutions are a poor fit for your needs, commit to resourcing the development and maintenance efforts over several years. Failing to do this leads to iterated build/rebuild cycles, which is bad for business (though, in some companies they're a good source of Staff+ promotion projects).

I've seen this dyamic happen a couple of times now. It goes kind of like this.

October: We're going all in on AWS! It's the future. Embrace it.
November: IT is working very hard on moving us there, thank you for your patience.
December: We're in! Enjoy the future.
January: This AWS bill is intolerable. Turn off everything we don't need.
February: Stop migrating things to AWS, we'll keep these specific systems on-prem for now.
March: Move these systems out of AWS.
April: Nothing gets moved to AWS unless it produces more revenue than it costs to run.

What's prompting this is a shock that is entirely predictable, but manages to penetrate the reality distortion field of upper management because the shock is to the pocketbook. They notice that kind of thing. To illustrate what I'm talking about, here is a made-up graph showing technology spend over a course of several years.

BudgetType-AWS.pngThe AWS line actually results in more money over time, as AWS does a good job of capturing costs that the traditional method generally ignores or assumes is lost in general overhead. But the screaming doesn't happen at the end of four years when they run the numbers, it happens in month four when the ongoing operational spend after build-out is done is w-a-y over what it used to be.

The spikes for traditional on-prem work are for forklifts of machinery showing up. Money is spent, new things show up, and they impact the monthly spend only infrequently. In this case, the base-charge increased only twice over the time-span. Some of those spikes are for things like maintenance-contract renewals, which don't impact base-spend one whit.

The AWS line is much less spikey, as new capabilities are assumed into the base-budget in an ongoing basis. You're no longer dropping $125K in a single go, you're dribbling it out over the course of a year or more. AWS price-drops mean that monthly spend actually goes down a few times.

Pay only for what you use!

Amazon is great at pointing that out, and hilighting the convenience of it. But what they don't mention is that by doing so, you will learn the hard way about what it is you really use. The AWS Calculator is an awesome tool, but if you don't know how your current environment works, it's like throwing darts at a wall for accurately predicting what you'll end up spending. You end up obsessing over small line-item charges you've never had to worry about before (how many IOPs do we do? Crap! I don't know! How many thousands will that cost us?), and missing the big items that nail you (Whoa! They meter bandwidth between AZs? Maybe we shouldn't be running our Hadoop cluster in multi-AZ mode).

There is a reason that third party AWS integrators are a thriving market.

Also, this 'what you use' is not subject to Oops exceptions without a lot of wrangling with Account Management. Have something that downloaded the entire EPEL repo twice a day for a month, and only learned about it when your bandwidth charge was 9x what it should be? Too bad, pay up or we'll turn the account off.

Unlike the forklift model, you pay for it every month without fail. If you have a bad quarter, you can't just not pay the bill for a few months and tru-up later. You're spending it, or they're turning your account off. This takes away some of the cost-shifting flexibility the old style had.

Unlike the forklift model, AWS prices its stuff assuming a three year turnover rate. Many companies have a 5 to 7 years lifetime for IT assets. Three to four years in production, with an afterlife of two to five years in various pre-prod, mirror, staging, and development roles.The cost of those assets therefore amortizes over 5-9 years, not 3.

Predictable spending, at last.


Yes, it is predictable over time given accurate understanding of what is under management. But when your initial predictions end up being wildly off, it seems like it isn't predictable. It seems like you're being held over the coals.

And when you get a new system into AWS and the cost forecast is wildly off, it doesn't seem predictable.

And when your system gets the rocket-launch you've been craving and you're scaling like mad; but the scale-costs don't match your cost forecast, it doesn't seem predictable.

It's only predictable if you fully understand the cost items and how your systems interact with it.

Reserved instances will save you money

Yes! They will! Quite a lot of it, in fact. They let a company go back to the forklift-method of cost-accounting, at least for part of it. I need 100 m3.large instances, on a three year up-front model. OK! Monthly charges drop drastically, and the monthly spend chart begins to look like the old model again.


Reserved instances cost a lot of money up front. That's the point, that's the trade-off for getting a cheaper annual spend. But many companies get into AWS because they see it as cheaper than on-prem. Which means they're sensitive to one-month cost-spikes, which in turn means buying reserved instances doesn't happen and they stay on the high cost on-demand model.

AWS is Elastic!

Elastic in that you can scale up and down at will, without week/month long billing, delivery and integration cycles.

Elastic in that you have choice in your cost accounting methods. On-demand, and various kinds of reserved instances.

It is not elastic in when the bill is due.

It is not elastic with individual asset pricing, no matter how special you are as a company.

All of these things trip up upper, non-technical management. I've seen it happen three times now, and I'm sure I'll see it again at some point.

Maybe this will help you in illuminating the issues with your own management.

End of year spending

It's that time of year for us to get our end of year spending priorities in. Each place I've worked has handled the EOY spending issue differently.

At my first job, civil service, we were on the calendar year for our budgeting year. Because of how the funding worked, we typically started spending heavily in October in order to use up excess IT budget. This was known as "Christmas in October". The reward for thrift, not spending as much as you were given, was to have your savings taken away from you so there was no incentive to do so. So, everything got spent.

At WWU, our budget year ended on June 31st which is smack in the middle of summer. Since Summer is Major Project Season for pretty much any US University, our EOY spending coincided with our annual upgrade-all-the-things spree. Thus, it was kinda hidden.

However, things did change while I was there.

For the first two or so years I was there the CIO gave each of his departments an actual budget for spending on technology. There was some spend-down in April under him, but not as much as that first job. Unlike that job, WWU did allow budget carry-over between years. Then we got a new CIO (retirements, don'tcha know) and he did things differently. He liked to give his departments budget for people and routine expenses, and dolled out funds for technology projects on a project by project basis.

Then 2008 happened and we went into negative-budget land. The concept of 'surplus budget' just didn't exist. There was a brief orgy of "spend it now!" before the budget boom came down, but that ended quickly. Then in 2009/2010 there was a pile of 'legacy' funds that we were told to spend on things to keep our infrastructure up and running until the fiscal environment recovered, since major upgrades were going to be off the table for the foreseeable future. I installed over $200K worth of storage stuff thanks to that.

Here at my current employer I don't yet know how things work. Since I am the entire IT department and don't have a budget, it's mostly just providing suggestions and quotes, with a healthy dose of explaining needs. Or, just like the rest of the year but with a longer planning horizon than usual.
The Sysadmin space is very broad. We do everything IT, and sometimes that unavoidably includes some management stuff as well. Once in a while we're even entrusted with an actual budget that we can spend without asking anyone first. Sometimes, we're even given minions subordinates.

Because of this, I consider certain management functions to be entirely within the rubric of "professional sysadmin".

  • Justifying large purchases.
  • Justifying changes in strategic direction.
  • Justifying changes in end-user IT management.
  • Justifying product upgrades.
  • Justifying staffing changes.
The observant may notice a common word up there, "Justify." This is because the more senior you get as a sysadmin, the more often you have to go before Management to convince them to do something. This is why having high levels in your "speaker to managers" skill is something you want to develop.

So many of the problems we face are both people problems and technology problems. Take email quotas; the technical problem of ever growing mailboxes is can be resolved by putting a mail quota in place, the people problem of such is convincing everyone that such measures are a good idea. Put the technical solution in place before the people solution, and your users will start putting up Mordac, Preventer of Information Services comics up in their cubes.

So, you need to convince management to buy in. There are a number of ways of doing that, but the most effective (in my experience, anyway) are those that tie decisions to dollars.

Take that email problem. It is entirely possible to build a spreadsheet that takes the costs of the mail system as a whole (storage, servers, mail software, AV software, client software) and distils a cost-per-MB of mail storage. By the way, this is why you learned algebra in school. Take that to Management, and point out that the one big user who has the 5.2GB mailbox is costing $x compared to the average user who is costing ($x/20), and they take notice.

You want them to take notice in this case. They may give you more resources to handle it, provide some management pressure on end-users to keep usage down, or a wild-card option (perhaps taking it to the cloud).

When convincing management that you need to spend big-big money for something (perhaps the storage array housing all of the company crown jewels is about to go out of support) phrasing the persuasion in terms of finance is also very effective. You do have to get the risk-management part of it right (risk-management, something else we have to do), but helping to illuminate the costs of inaction is something that's very useful.

It is my experience that non-technical management only ever really see the direct-costs for things. That's items like hardware purchases, ongoing support contract costs, software license maintenance, software upgrade purchases, and staffing. They rarely get a good look at the indirect costs for things. Indirect costs are things like after-hours calls for handling buggy software, backup system outages due to bad tape-drives, and increased support-contract costs for hardware in the 4+ years old range.

All of these things are entirely within the realm of "professional sysadmin". These sorts of finance problems aren't covered in sysadmin classes, and generally are picked up on-the-job. This is why when anyone asks about how to convince management of things, I help.

Budget issues

Yesterday the University President posted a bunch of budget documents that describe how we're handling the 3% cut we need to accommodate. You can start reading them yourselves here (hyperlinked PDFs, not HTML). In and amongst the various line-items was one that caused me a bit of heart failure.

Bring units in leased space to buildings we own. These include our off campus dance facility and all units in the space we currently lease at 32nd Street.

Dude, that's my office! Does this mean we're moving back up to campus? Where in tarnation are they going to fit our datacenter? Moving THAT is not going to be cheap. As in edging on a million bucks worth of not cheap.

But calm prevailed and more information was supplied by the ITS Vice Provost. Human Resources is going to be heading back to campus. The building I'm in is actually owned by WWU.

Whew! Thank goodness, I didn't want to have to move a datacenter. That's a lot of work.

Also, ITS will be taking at least one layoff. I'm pretty sure I know who it is, but I'm not saying here. It isn't in Technical Services though, we've been spared the ax one more time. No idea if our luck will extend to the 10% cut due to take effect 7/1/2011.

And we still have no training budget.

The budget crisis gets deeper

We were told last week that Olympia is requiring WWU to find another 4% to cut from this fiscal year, and another 10% for next fiscal. Fortunately (?) this is within our own internal budget forecasting so we at least have a plan for dealing with it, mostly. The hard part will be the 4% right now.

This is leading to creative thinking. We've done a lot of that over the last two years but now we're scraping the bottom of the barrel. We got told late last week that Technical Services will no longer be able to use the ADMCS supply closet for office supplies and we have to make our own. There are all of 7 of us in this department, we don't go through a lot of Post-Its, pens, and DVD blanks. What we do go through is paper and toner, because one or two of us still prints off 200-600 page manuals once in a while (the rest of us just keep the PDFs around).

And yet, I just turned in a pair of hardware quotes that came to low six figures. A lot of us are confused as to why we're even bothering if money is that tight, but apparently The Powers That Be are confident that there really is money. I do know that there are different flavors of money out there; Capital Funds can't be swept to fix operational budget holes, for instance. Apparently the money for these quotes is coming out of a similarly protected fund, but I don't know what it is or how it works. It'll be nice to get that hardware as it'll keep me busy for the better part of a month.

And of course, the 10% for next fiscal is causing everyone sweat. Technical Services hasn't had to take a layoff yet in the two rounds we've had so far, and it just might be our turn. A 10% cut to our budget is either a person, or a handful of Furlough Days.

Furlough bill

WWU has delivered its guidance to the Office of Financial Management. You can read it here (pdf). In short, we're looking at a choice between one furlough day, or leaving tenure track faculty positions open for the next 18 months. WWU management like the second option, since furloughs are problematical due to the different ways they'd be handled in the different classes of employees.

Nice to know. I really didn't want to muck with 11 such days.

Mandatory time off

The Governor signed the furlough bill, which I had been expecting for some time. The only reason she hadn't signed it yet is because she was waiting on analysis from the budget office. As the bill reads, all state agencies have to come up with 10 days in which to either close operations, or find some other way to save an equivalent chunk of salary money.

How does this apply to me? Well, we're not sure. The University President sent a message out to all staff last week describing what this bill would likely mean for WWU. It means we'll have to come up with $1,172,000 in salary savings one way or another, and if we can't do that we'll have to come up with whole days in which we can shut down operations.

I say 'close operations' even though the bill exempts anyone in a direct teaching function, so in theory we could still teach. However, that's teach without any support staff what so ever, Some can do it, others can't, and if things break in any way they'll stay broken until the next day. Suffice it to say, we can't plan on teaching on the furlough days.

Can we even find 10 days to shut down? Our biggest target is the summer/fall intersession where we have four weeks of no teaching, followed by the fall/winter intersession that's three vacation-heavy weeks as it is. Winter/spring, and spring/summer are only single weeks and... we can't afford to take a mandatory day off that week; there is simply too much changeover going on.

However, as the President said, we may not have to find 10 days. Maybe only five. Or if we're lucky, none. Agencies have to come up with a dollar figure cut in personnel expenses, which will come in the form of furloughs if the agency can't find any other way to reach it. Lay-offs are an option. As are leaving positions open through retirement open for longer, eliminating already open positions, and work-hour reductions.

So while there was much complaining in the office this morning about this bill, the exact nature of the impact we perceive to our jobs is solely in the hands of the WWU budget process. And we just don't know what that looks like yet.

2010-11 budget

The WA legislature finally, finally, got around to passing an amended budget for the second half of the biennium. They had to fill budget holes, and have spent the last three and a half weeks in Special Session arm-wrestling between the House and Senate versions. The main sticking point was over revenue-enhancers (sometimes referred to as 'new taxes'). Anyway, they reached an agreement, and the Governor should sign it real soon now. This means that we (WWU) now know what our budget cut is going to be (5.2%).

WWU's Budget Planning office has a nice chart up describing how our cut has evolved as the legislative session progressed: link.

In a letter to all staff this morning, the President said:

What remain possibilities at this point could affect 39 positions.  Of these, 10 that are currently occupied would either be eliminated or have the FTE reduced.  An additional 7 would be continued but funding sources would be changed.  The remainder, 22, are currently vacant or will become so through retirement. 
I'm safe. Technical Services staff has been told that any one of us will cause grievous pain in the event of a departure, so the cuts would have to be very bad for us to get passed an ax.

On the down side, this does mean that normal expansion-of-business upgrades will be much harder to fund. Exciting times.