Recently in budget Category

I've seen this dyamic happen a couple of times now. It goes kind of like this.

October: We're going all in on AWS! It's the future. Embrace it.
November: IT is working very hard on moving us there, thank you for your patience.
December: We're in! Enjoy the future.
January: This AWS bill is intolerable. Turn off everything we don't need.
February: Stop migrating things to AWS, we'll keep these specific systems on-prem for now.
March: Move these systems out of AWS.
April: Nothing gets moved to AWS unless it produces more revenue than it costs to run.

What's prompting this is a shock that is entirely predictable, but manages to penetrate the reality distortion field of upper management because the shock is to the pocketbook. They notice that kind of thing. To illustrate what I'm talking about, here is a made-up graph showing technology spend over a course of several years.

BudgetType-AWS.pngThe AWS line actually results in more money over time, as AWS does a good job of capturing costs that the traditional method generally ignores or assumes is lost in general overhead. But the screaming doesn't happen at the end of four years when they run the numbers, it happens in month four when the ongoing operational spend after build-out is done is w-a-y over what it used to be.

The spikes for traditional on-prem work are for forklifts of machinery showing up. Money is spent, new things show up, and they impact the monthly spend only infrequently. In this case, the base-charge increased only twice over the time-span. Some of those spikes are for things like maintenance-contract renewals, which don't impact base-spend one whit.

The AWS line is much less spikey, as new capabilities are assumed into the base-budget in an ongoing basis. You're no longer dropping $125K in a single go, you're dribbling it out over the course of a year or more. AWS price-drops mean that monthly spend actually goes down a few times.

Pay only for what you use!

Amazon is great at pointing that out, and hilighting the convenience of it. But what they don't mention is that by doing so, you will learn the hard way about what it is you really use. The AWS Calculator is an awesome tool, but if you don't know how your current environment works, it's like throwing darts at a wall for accurately predicting what you'll end up spending. You end up obsessing over small line-item charges you've never had to worry about before (how many IOPs do we do? Crap! I don't know! How many thousands will that cost us?), and missing the big items that nail you (Whoa! They meter bandwidth between AZs? Maybe we shouldn't be running our Hadoop cluster in multi-AZ mode).

There is a reason that third party AWS integrators are a thriving market.

Also, this 'what you use' is not subject to Oops exceptions without a lot of wrangling with Account Management. Have something that downloaded the entire EPEL repo twice a day for a month, and only learned about it when your bandwidth charge was 9x what it should be? Too bad, pay up or we'll turn the account off.

Unlike the forklift model, you pay for it every month without fail. If you have a bad quarter, you can't just not pay the bill for a few months and tru-up later. You're spending it, or they're turning your account off. This takes away some of the cost-shifting flexibility the old style had.

Unlike the forklift model, AWS prices its stuff assuming a three year turnover rate. Many companies have a 5 to 7 years lifetime for IT assets. Three to four years in production, with an afterlife of two to five years in various pre-prod, mirror, staging, and development roles.The cost of those assets therefore amortizes over 5-9 years, not 3.

Predictable spending, at last.


Yes, it is predictable over time given accurate understanding of what is under management. But when your initial predictions end up being wildly off, it seems like it isn't predictable. It seems like you're being held over the coals.

And when you get a new system into AWS and the cost forecast is wildly off, it doesn't seem predictable.

And when your system gets the rocket-launch you've been craving and you're scaling like mad; but the scale-costs don't match your cost forecast, it doesn't seem predictable.

It's only predictable if you fully understand the cost items and how your systems interact with it.

Reserved instances will save you money

Yes! They will! Quite a lot of it, in fact. They let a company go back to the forklift-method of cost-accounting, at least for part of it. I need 100 m3.large instances, on a three year up-front model. OK! Monthly charges drop drastically, and the monthly spend chart begins to look like the old model again.


Reserved instances cost a lot of money up front. That's the point, that's the trade-off for getting a cheaper annual spend. But many companies get into AWS because they see it as cheaper than on-prem. Which means they're sensitive to one-month cost-spikes, which in turn means buying reserved instances doesn't happen and they stay on the high cost on-demand model.

AWS is Elastic!

Elastic in that you can scale up and down at will, without week/month long billing, delivery and integration cycles.

Elastic in that you have choice in your cost accounting methods. On-demand, and various kinds of reserved instances.

It is not elastic in when the bill is due.

It is not elastic with individual asset pricing, no matter how special you are as a company.

All of these things trip up upper, non-technical management. I've seen it happen three times now, and I'm sure I'll see it again at some point.

Maybe this will help you in illuminating the issues with your own management.

End of year spending

| No Comments
It's that time of year for us to get our end of year spending priorities in. Each place I've worked has handled the EOY spending issue differently.

At my first job, civil service, we were on the calendar year for our budgeting year. Because of how the funding worked, we typically started spending heavily in October in order to use up excess IT budget. This was known as "Christmas in October". The reward for thrift, not spending as much as you were given, was to have your savings taken away from you so there was no incentive to do so. So, everything got spent.

At WWU, our budget year ended on June 31st which is smack in the middle of summer. Since Summer is Major Project Season for pretty much any US University, our EOY spending coincided with our annual upgrade-all-the-things spree. Thus, it was kinda hidden.

However, things did change while I was there.

For the first two or so years I was there the CIO gave each of his departments an actual budget for spending on technology. There was some spend-down in April under him, but not as much as that first job. Unlike that job, WWU did allow budget carry-over between years. Then we got a new CIO (retirements, don'tcha know) and he did things differently. He liked to give his departments budget for people and routine expenses, and dolled out funds for technology projects on a project by project basis.

Then 2008 happened and we went into negative-budget land. The concept of 'surplus budget' just didn't exist. There was a brief orgy of "spend it now!" before the budget boom came down, but that ended quickly. Then in 2009/2010 there was a pile of 'legacy' funds that we were told to spend on things to keep our infrastructure up and running until the fiscal environment recovered, since major upgrades were going to be off the table for the foreseeable future. I installed over $200K worth of storage stuff thanks to that.

Here at my current employer I don't yet know how things work. Since I am the entire IT department and don't have a budget, it's mostly just providing suggestions and quotes, with a healthy dose of explaining needs. Or, just like the rest of the year but with a longer planning horizon than usual.
The Sysadmin space is very broad. We do everything IT, and sometimes that unavoidably includes some management stuff as well. Once in a while we're even entrusted with an actual budget that we can spend without asking anyone first. Sometimes, we're even given minions subordinates.

Because of this, I consider certain management functions to be entirely within the rubric of "professional sysadmin".

  • Justifying large purchases.
  • Justifying changes in strategic direction.
  • Justifying changes in end-user IT management.
  • Justifying product upgrades.
  • Justifying staffing changes.
The observant may notice a common word up there, "Justify." This is because the more senior you get as a sysadmin, the more often you have to go before Management to convince them to do something. This is why having high levels in your "speaker to managers" skill is something you want to develop.

So many of the problems we face are both people problems and technology problems. Take email quotas; the technical problem of ever growing mailboxes is can be resolved by putting a mail quota in place, the people problem of such is convincing everyone that such measures are a good idea. Put the technical solution in place before the people solution, and your users will start putting up Mordac, Preventer of Information Services comics up in their cubes.

So, you need to convince management to buy in. There are a number of ways of doing that, but the most effective (in my experience, anyway) are those that tie decisions to dollars.

Take that email problem. It is entirely possible to build a spreadsheet that takes the costs of the mail system as a whole (storage, servers, mail software, AV software, client software) and distils a cost-per-MB of mail storage. By the way, this is why you learned algebra in school. Take that to Management, and point out that the one big user who has the 5.2GB mailbox is costing $x compared to the average user who is costing ($x/20), and they take notice.

You want them to take notice in this case. They may give you more resources to handle it, provide some management pressure on end-users to keep usage down, or a wild-card option (perhaps taking it to the cloud).

When convincing management that you need to spend big-big money for something (perhaps the storage array housing all of the company crown jewels is about to go out of support) phrasing the persuasion in terms of finance is also very effective. You do have to get the risk-management part of it right (risk-management, something else we have to do), but helping to illuminate the costs of inaction is something that's very useful.

It is my experience that non-technical management only ever really see the direct-costs for things. That's items like hardware purchases, ongoing support contract costs, software license maintenance, software upgrade purchases, and staffing. They rarely get a good look at the indirect costs for things. Indirect costs are things like after-hours calls for handling buggy software, backup system outages due to bad tape-drives, and increased support-contract costs for hardware in the 4+ years old range.

All of these things are entirely within the realm of "professional sysadmin". These sorts of finance problems aren't covered in sysadmin classes, and generally are picked up on-the-job. This is why when anyone asks about how to convince management of things, I help.

Budget issues

| No Comments
Yesterday the University President posted a bunch of budget documents that describe how we're handling the 3% cut we need to accommodate. You can start reading them yourselves here (hyperlinked PDFs, not HTML). In and amongst the various line-items was one that caused me a bit of heart failure.

Bring units in leased space to buildings we own. These include our off campus dance facility and all units in the space we currently lease at 32nd Street.

Dude, that's my office! Does this mean we're moving back up to campus? Where in tarnation are they going to fit our datacenter? Moving THAT is not going to be cheap. As in edging on a million bucks worth of not cheap.

But calm prevailed and more information was supplied by the ITS Vice Provost. Human Resources is going to be heading back to campus. The building I'm in is actually owned by WWU.

Whew! Thank goodness, I didn't want to have to move a datacenter. That's a lot of work.

Also, ITS will be taking at least one layoff. I'm pretty sure I know who it is, but I'm not saying here. It isn't in Technical Services though, we've been spared the ax one more time. No idea if our luck will extend to the 10% cut due to take effect 7/1/2011.

And we still have no training budget.

The budget crisis gets deeper

| No Comments
We were told last week that Olympia is requiring WWU to find another 4% to cut from this fiscal year, and another 10% for next fiscal. Fortunately (?) this is within our own internal budget forecasting so we at least have a plan for dealing with it, mostly. The hard part will be the 4% right now.

This is leading to creative thinking. We've done a lot of that over the last two years but now we're scraping the bottom of the barrel. We got told late last week that Technical Services will no longer be able to use the ADMCS supply closet for office supplies and we have to make our own. There are all of 7 of us in this department, we don't go through a lot of Post-Its, pens, and DVD blanks. What we do go through is paper and toner, because one or two of us still prints off 200-600 page manuals once in a while (the rest of us just keep the PDFs around).

And yet, I just turned in a pair of hardware quotes that came to low six figures. A lot of us are confused as to why we're even bothering if money is that tight, but apparently The Powers That Be are confident that there really is money. I do know that there are different flavors of money out there; Capital Funds can't be swept to fix operational budget holes, for instance. Apparently the money for these quotes is coming out of a similarly protected fund, but I don't know what it is or how it works. It'll be nice to get that hardware as it'll keep me busy for the better part of a month.

And of course, the 10% for next fiscal is causing everyone sweat. Technical Services hasn't had to take a layoff yet in the two rounds we've had so far, and it just might be our turn. A 10% cut to our budget is either a person, or a handful of Furlough Days.

Furlough bill

| No Comments
WWU has delivered its guidance to the Office of Financial Management. You can read it here (pdf). In short, we're looking at a choice between one furlough day, or leaving tenure track faculty positions open for the next 18 months. WWU management like the second option, since furloughs are problematical due to the different ways they'd be handled in the different classes of employees.

Nice to know. I really didn't want to muck with 11 such days.

Mandatory time off

| No Comments
The Governor signed the furlough bill, which I had been expecting for some time. The only reason she hadn't signed it yet is because she was waiting on analysis from the budget office. As the bill reads, all state agencies have to come up with 10 days in which to either close operations, or find some other way to save an equivalent chunk of salary money.

How does this apply to me? Well, we're not sure. The University President sent a message out to all staff last week describing what this bill would likely mean for WWU. It means we'll have to come up with $1,172,000 in salary savings one way or another, and if we can't do that we'll have to come up with whole days in which we can shut down operations.

I say 'close operations' even though the bill exempts anyone in a direct teaching function, so in theory we could still teach. However, that's teach without any support staff what so ever, Some can do it, others can't, and if things break in any way they'll stay broken until the next day. Suffice it to say, we can't plan on teaching on the furlough days.

Can we even find 10 days to shut down? Our biggest target is the summer/fall intersession where we have four weeks of no teaching, followed by the fall/winter intersession that's three vacation-heavy weeks as it is. Winter/spring, and spring/summer are only single weeks and... we can't afford to take a mandatory day off that week; there is simply too much changeover going on.

However, as the President said, we may not have to find 10 days. Maybe only five. Or if we're lucky, none. Agencies have to come up with a dollar figure cut in personnel expenses, which will come in the form of furloughs if the agency can't find any other way to reach it. Lay-offs are an option. As are leaving positions open through retirement open for longer, eliminating already open positions, and work-hour reductions.

So while there was much complaining in the office this morning about this bill, the exact nature of the impact we perceive to our jobs is solely in the hands of the WWU budget process. And we just don't know what that looks like yet.

2010-11 budget

| No Comments
The WA legislature finally, finally, got around to passing an amended budget for the second half of the biennium. They had to fill budget holes, and have spent the last three and a half weeks in Special Session arm-wrestling between the House and Senate versions. The main sticking point was over revenue-enhancers (sometimes referred to as 'new taxes'). Anyway, they reached an agreement, and the Governor should sign it real soon now. This means that we (WWU) now know what our budget cut is going to be (5.2%).

WWU's Budget Planning office has a nice chart up describing how our cut has evolved as the legislative session progressed: link.

In a letter to all staff this morning, the President said:

What remain possibilities at this point could affect 39 positions.  Of these, 10 that are currently occupied would either be eliminated or have the FTE reduced.  An additional 7 would be continued but funding sources would be changed.  The remainder, 22, are currently vacant or will become so through retirement. 
I'm safe. Technical Services staff has been told that any one of us will cause grievous pain in the event of a departure, so the cuts would have to be very bad for us to get passed an ax.

On the down side, this does mean that normal expansion-of-business upgrades will be much harder to fund. Exciting times.
When we got warning that the Governor would be putting a draconian spending freeze into place, our supreme masters informed us we had to spend a certain amount of money now or we would lose it. HP-Boxes.pngAdditionally, we were told that funds in the next 12-24 months would be downright scarce, so order now while we still could.  I've talked about this in a few previous posts, but the orders have started to arrive.

We have a nice pile of HP boxes in the data-center right now, and they haven't all arrived yet. Most of the boxes in this picture are dedicated to storage in one way or another.

We haven't gotten the box with 200 LTO4 tapes in it, which should be a nice, big box. We did get the box with the labels for the tapes, though; that's that little one on the foreground. That box contained two folders of tape bar-codes, that box was w-a-y overkill. It also looks likely that HP managed to not ship us a monster box with 20+ individually boxed hard-drives! Talk about over-packaging, Batman.

We're not touching these boxes until they're all here, and we're done with the Spring Break madness. So once quarter starts (3/30) we'll have time to do things like install the new tape library, add a few shelves to our EVA4400. And figure out what we're doing with a storage server we're building (OpenNAS is a strong contender). As well as integrating one or two new servers into our ESX cluster while we're at it.

And then... we wait. Perhaps until 2012.

Solving budget problems

| No Comments
Two days ago we got to meet with my great-grand-boss, the Provost for Academic Affairs. She's a fresh transplant from Michigan who has been in public higher-ed for over 20 years. It was a good talk, and I am encouraged.

One of the take-away quotes from that meeting was, "This is one of the most micro-managing legislatures I've ever seen." And then went on into details. One of the things she mentioned is a bill I was aware of but haven't mentioned yet, a plan to force furlough days on state employees such as myself. SB6503 is the bill.

One of the loonier provisions is a list of days to be considered by appropriate institutions for full closure:

For those agencies and institutions of higher education that do not have an approved compensation reduction plan by June 1, 2010, the agency or institution shall be closed on the following dates in addition to the legal holidays specified in RCW 1.16.050:

(a) Monday, June 14, 2010;
(b) Tuesday, July 6, 2010;
(c) Friday, August 6, 2010;
(d) Tuesday, September 7, 2010;
(e) Monday, October 11, 2010;
(f) Friday, November 12, 2010;
(g) Monday, December 27, 2010;
(h) Friday, January 14, 2011;
(i) Friday, February 18, 2011;
(j) Friday, March 11, 2011;
(k) Friday, April 15, 2011;
(l) Friday, May 27, 2011; and
(m) Friday, June 10, 2011.

In the immortal words of Bill Cosby, "Riiiiight." As it happens certain parts of higher ed are exempted from this, not affected by this is, "classroom instruction, operations not funded from state funds or tuition, campus police and security, emergency  management and response, and student health care." So I would be furloughed, but not the teaching staff. And woe unto the faculty member with a problem logging in to Blackboard that day, for they will be alone and all the staff pointedly ignoring their phones that day.

As it happens, this is perhaps the stick to get people to develop an 'approved compensation reduction plan.' This would allow WWU to create its own ways of reducing payroll, be it through head-count reduction, hours reduction, or interspersed furlough days arranged so as to minimally impact function of the University.

What's a furlough? "voluntary and mandatory temporary layoffs," according to this bill. So if I'm on a furlough day, you can guarantee I'm pretending I'm unemployed and will not be responding to anything work-related. The one thing that could keep me in the money during such a 'shutdown' is clause S under the exemptions: "The minimal use of state employees on the specified closure dates as necessary to protect public  assets, information technology systems, and maintain public safety." Right now that's unworkably vague in meaning, but it could mean that a small selection of tech staff could be present to help the teaching function work.

This is another bill we're keeping a close eye on.

Other Blogs

My Other Stuff

Monthly Archives