Recently in budget Category

End of year spending

| No Comments
It's that time of year for us to get our end of year spending priorities in. Each place I've worked has handled the EOY spending issue differently.

At my first job, civil service, we were on the calendar year for our budgeting year. Because of how the funding worked, we typically started spending heavily in October in order to use up excess IT budget. This was known as "Christmas in October". The reward for thrift, not spending as much as you were given, was to have your savings taken away from you so there was no incentive to do so. So, everything got spent.

At WWU, our budget year ended on June 31st which is smack in the middle of summer. Since Summer is Major Project Season for pretty much any US University, our EOY spending coincided with our annual upgrade-all-the-things spree. Thus, it was kinda hidden.

However, things did change while I was there.

For the first two or so years I was there the CIO gave each of his departments an actual budget for spending on technology. There was some spend-down in April under him, but not as much as that first job. Unlike that job, WWU did allow budget carry-over between years. Then we got a new CIO (retirements, don'tcha know) and he did things differently. He liked to give his departments budget for people and routine expenses, and dolled out funds for technology projects on a project by project basis.

Then 2008 happened and we went into negative-budget land. The concept of 'surplus budget' just didn't exist. There was a brief orgy of "spend it now!" before the budget boom came down, but that ended quickly. Then in 2009/2010 there was a pile of 'legacy' funds that we were told to spend on things to keep our infrastructure up and running until the fiscal environment recovered, since major upgrades were going to be off the table for the foreseeable future. I installed over $200K worth of storage stuff thanks to that.

Here at my current employer I don't yet know how things work. Since I am the entire IT department and don't have a budget, it's mostly just providing suggestions and quotes, with a healthy dose of explaining needs. Or, just like the rest of the year but with a longer planning horizon than usual.
The Sysadmin space is very broad. We do everything IT, and sometimes that unavoidably includes some management stuff as well. Once in a while we're even entrusted with an actual budget that we can spend without asking anyone first. Sometimes, we're even given minions subordinates.

Because of this, I consider certain management functions to be entirely within the rubric of "professional sysadmin".

  • Justifying large purchases.
  • Justifying changes in strategic direction.
  • Justifying changes in end-user IT management.
  • Justifying product upgrades.
  • Justifying staffing changes.
The observant may notice a common word up there, "Justify." This is because the more senior you get as a sysadmin, the more often you have to go before Management to convince them to do something. This is why having high levels in your "speaker to managers" skill is something you want to develop.

So many of the problems we face are both people problems and technology problems. Take email quotas; the technical problem of ever growing mailboxes is can be resolved by putting a mail quota in place, the people problem of such is convincing everyone that such measures are a good idea. Put the technical solution in place before the people solution, and your users will start putting up Mordac, Preventer of Information Services comics up in their cubes.

Dilbert.com

So, you need to convince management to buy in. There are a number of ways of doing that, but the most effective (in my experience, anyway) are those that tie decisions to dollars.

Take that email problem. It is entirely possible to build a spreadsheet that takes the costs of the mail system as a whole (storage, servers, mail software, AV software, client software) and distils a cost-per-MB of mail storage. By the way, this is why you learned algebra in school. Take that to Management, and point out that the one big user who has the 5.2GB mailbox is costing $x compared to the average user who is costing ($x/20), and they take notice.

You want them to take notice in this case. They may give you more resources to handle it, provide some management pressure on end-users to keep usage down, or a wild-card option (perhaps taking it to the cloud).

When convincing management that you need to spend big-big money for something (perhaps the storage array housing all of the company crown jewels is about to go out of support) phrasing the persuasion in terms of finance is also very effective. You do have to get the risk-management part of it right (risk-management, something else we have to do), but helping to illuminate the costs of inaction is something that's very useful.

It is my experience that non-technical management only ever really see the direct-costs for things. That's items like hardware purchases, ongoing support contract costs, software license maintenance, software upgrade purchases, and staffing. They rarely get a good look at the indirect costs for things. Indirect costs are things like after-hours calls for handling buggy software, backup system outages due to bad tape-drives, and increased support-contract costs for hardware in the 4+ years old range.

All of these things are entirely within the realm of "professional sysadmin". These sorts of finance problems aren't covered in sysadmin classes, and generally are picked up on-the-job. This is why when anyone asks about how to convince management of things, I help.

Budget issues

| No Comments
Yesterday the University President posted a bunch of budget documents that describe how we're handling the 3% cut we need to accommodate. You can start reading them yourselves here (hyperlinked PDFs, not HTML). In and amongst the various line-items was one that caused me a bit of heart failure.

Bring units in leased space to buildings we own. These include our off campus dance facility and all units in the space we currently lease at 32nd Street.

Dude, that's my office! Does this mean we're moving back up to campus? Where in tarnation are they going to fit our datacenter? Moving THAT is not going to be cheap. As in edging on a million bucks worth of not cheap.

But calm prevailed and more information was supplied by the ITS Vice Provost. Human Resources is going to be heading back to campus. The building I'm in is actually owned by WWU.

Whew! Thank goodness, I didn't want to have to move a datacenter. That's a lot of work.

Also, ITS will be taking at least one layoff. I'm pretty sure I know who it is, but I'm not saying here. It isn't in Technical Services though, we've been spared the ax one more time. No idea if our luck will extend to the 10% cut due to take effect 7/1/2011.

And we still have no training budget.

The budget crisis gets deeper

| No Comments
We were told last week that Olympia is requiring WWU to find another 4% to cut from this fiscal year, and another 10% for next fiscal. Fortunately (?) this is within our own internal budget forecasting so we at least have a plan for dealing with it, mostly. The hard part will be the 4% right now.

This is leading to creative thinking. We've done a lot of that over the last two years but now we're scraping the bottom of the barrel. We got told late last week that Technical Services will no longer be able to use the ADMCS supply closet for office supplies and we have to make our own. There are all of 7 of us in this department, we don't go through a lot of Post-Its, pens, and DVD blanks. What we do go through is paper and toner, because one or two of us still prints off 200-600 page manuals once in a while (the rest of us just keep the PDFs around).

And yet, I just turned in a pair of hardware quotes that came to low six figures. A lot of us are confused as to why we're even bothering if money is that tight, but apparently The Powers That Be are confident that there really is money. I do know that there are different flavors of money out there; Capital Funds can't be swept to fix operational budget holes, for instance. Apparently the money for these quotes is coming out of a similarly protected fund, but I don't know what it is or how it works. It'll be nice to get that hardware as it'll keep me busy for the better part of a month.

And of course, the 10% for next fiscal is causing everyone sweat. Technical Services hasn't had to take a layoff yet in the two rounds we've had so far, and it just might be our turn. A 10% cut to our budget is either a person, or a handful of Furlough Days.

Furlough bill

| No Comments
WWU has delivered its guidance to the Office of Financial Management. You can read it here (pdf). In short, we're looking at a choice between one furlough day, or leaving tenure track faculty positions open for the next 18 months. WWU management like the second option, since furloughs are problematical due to the different ways they'd be handled in the different classes of employees.

Nice to know. I really didn't want to muck with 11 such days.

Mandatory time off

| No Comments
The Governor signed the furlough bill, which I had been expecting for some time. The only reason she hadn't signed it yet is because she was waiting on analysis from the budget office. As the bill reads, all state agencies have to come up with 10 days in which to either close operations, or find some other way to save an equivalent chunk of salary money.

How does this apply to me? Well, we're not sure. The University President sent a message out to all staff last week describing what this bill would likely mean for WWU. It means we'll have to come up with $1,172,000 in salary savings one way or another, and if we can't do that we'll have to come up with whole days in which we can shut down operations.

I say 'close operations' even though the bill exempts anyone in a direct teaching function, so in theory we could still teach. However, that's teach without any support staff what so ever, Some can do it, others can't, and if things break in any way they'll stay broken until the next day. Suffice it to say, we can't plan on teaching on the furlough days.

Can we even find 10 days to shut down? Our biggest target is the summer/fall intersession where we have four weeks of no teaching, followed by the fall/winter intersession that's three vacation-heavy weeks as it is. Winter/spring, and spring/summer are only single weeks and... we can't afford to take a mandatory day off that week; there is simply too much changeover going on.

However, as the President said, we may not have to find 10 days. Maybe only five. Or if we're lucky, none. Agencies have to come up with a dollar figure cut in personnel expenses, which will come in the form of furloughs if the agency can't find any other way to reach it. Lay-offs are an option. As are leaving positions open through retirement open for longer, eliminating already open positions, and work-hour reductions.

So while there was much complaining in the office this morning about this bill, the exact nature of the impact we perceive to our jobs is solely in the hands of the WWU budget process. And we just don't know what that looks like yet.

2010-11 budget

| No Comments
The WA legislature finally, finally, got around to passing an amended budget for the second half of the biennium. They had to fill budget holes, and have spent the last three and a half weeks in Special Session arm-wrestling between the House and Senate versions. The main sticking point was over revenue-enhancers (sometimes referred to as 'new taxes'). Anyway, they reached an agreement, and the Governor should sign it real soon now. This means that we (WWU) now know what our budget cut is going to be (5.2%).

WWU's Budget Planning office has a nice chart up describing how our cut has evolved as the legislative session progressed: link.

In a letter to all staff this morning, the President said:

What remain possibilities at this point could affect 39 positions.  Of these, 10 that are currently occupied would either be eliminated or have the FTE reduced.  An additional 7 would be continued but funding sources would be changed.  The remainder, 22, are currently vacant or will become so through retirement. 
I'm safe. Technical Services staff has been told that any one of us will cause grievous pain in the event of a departure, so the cuts would have to be very bad for us to get passed an ax.

On the down side, this does mean that normal expansion-of-business upgrades will be much harder to fund. Exciting times.
When we got warning that the Governor would be putting a draconian spending freeze into place, our supreme masters informed us we had to spend a certain amount of money now or we would lose it. HP-Boxes.pngAdditionally, we were told that funds in the next 12-24 months would be downright scarce, so order now while we still could.  I've talked about this in a few previous posts, but the orders have started to arrive.

We have a nice pile of HP boxes in the data-center right now, and they haven't all arrived yet. Most of the boxes in this picture are dedicated to storage in one way or another.

We haven't gotten the box with 200 LTO4 tapes in it, which should be a nice, big box. We did get the box with the labels for the tapes, though; that's that little one on the foreground. That box contained two folders of tape bar-codes, that box was w-a-y overkill. It also looks likely that HP managed to not ship us a monster box with 20+ individually boxed hard-drives! Talk about over-packaging, Batman.

We're not touching these boxes until they're all here, and we're done with the Spring Break madness. So once quarter starts (3/30) we'll have time to do things like install the new tape library, add a few shelves to our EVA4400. And figure out what we're doing with a storage server we're building (OpenNAS is a strong contender). As well as integrating one or two new servers into our ESX cluster while we're at it.

And then... we wait. Perhaps until 2012.

Solving budget problems

| No Comments
Two days ago we got to meet with my great-grand-boss, the Provost for Academic Affairs. She's a fresh transplant from Michigan who has been in public higher-ed for over 20 years. It was a good talk, and I am encouraged.

One of the take-away quotes from that meeting was, "This is one of the most micro-managing legislatures I've ever seen." And then went on into details. One of the things she mentioned is a bill I was aware of but haven't mentioned yet, a plan to force furlough days on state employees such as myself. SB6503 is the bill.

One of the loonier provisions is a list of days to be considered by appropriate institutions for full closure:

For those agencies and institutions of higher education that do not have an approved compensation reduction plan by June 1, 2010, the agency or institution shall be closed on the following dates in addition to the legal holidays specified in RCW 1.16.050:

(a) Monday, June 14, 2010;
(b) Tuesday, July 6, 2010;
(c) Friday, August 6, 2010;
(d) Tuesday, September 7, 2010;
(e) Monday, October 11, 2010;
(f) Friday, November 12, 2010;
(g) Monday, December 27, 2010;
(h) Friday, January 14, 2011;
(i) Friday, February 18, 2011;
(j) Friday, March 11, 2011;
(k) Friday, April 15, 2011;
(l) Friday, May 27, 2011; and
(m) Friday, June 10, 2011.

In the immortal words of Bill Cosby, "Riiiiight." As it happens certain parts of higher ed are exempted from this, not affected by this is, "classroom instruction, operations not funded from state funds or tuition, campus police and security, emergency  management and response, and student health care." So I would be furloughed, but not the teaching staff. And woe unto the faculty member with a problem logging in to Blackboard that day, for they will be alone and all the staff pointedly ignoring their phones that day.

As it happens, this is perhaps the stick to get people to develop an 'approved compensation reduction plan.' This would allow WWU to create its own ways of reducing payroll, be it through head-count reduction, hours reduction, or interspersed furlough days arranged so as to minimally impact function of the University.

What's a furlough? "voluntary and mandatory temporary layoffs," according to this bill. So if I'm on a furlough day, you can guarantee I'm pretending I'm unemployed and will not be responding to anything work-related. The one thing that could keep me in the money during such a 'shutdown' is clause S under the exemptions: "The minimal use of state employees on the specified closure dates as necessary to protect public  assets, information technology systems, and maintain public safety." Right now that's unworkably vague in meaning, but it could mean that a small selection of tech staff could be present to help the teaching function work.

This is another bill we're keeping a close eye on.

Spent money

| 2 Comments
It has been a week plus since we spent a lot of money and the question has been raised, what are we doing with that storage? Exactly?

It isn't fancy storage. In fact, it is the cheapest performant storage we could budget for. It's not SATA, but it is 7.2K RPM SAS. And there are 35TB of it. It's a server, with direct attached storage. Not a dedicated storage unit. Not fibre channel. An off the shelf server, a high quality RAID card, a bunch of storage shelves, and a pair of network ports.

A final decision hasn't been made yet for how we're presenting this storage to consumers, but iSCSI of some kind is the 90% likely choice. Whether that's Linux (a.k.a. the free option) or something else (a.k.a. the pay option) remains to be seen. The whole point of this storage is to be cheap per GB.

We're also adding a pair of Fibre Channel Drive enclosures to our EVA4400 to provide true high-speed low-latency service at a much reduced cost versus the EVA6100. Yes FC drives are EOL in a very short while, but the EVA4400 doesn't support SAS (yet). This is where our ESX cluster is likely to expand into when the time comes, that kind of stuff.

And a new tape library. It's an HP StorageWorks MSL4048. It is LTO4, fibre-attached, and has lots of slots. The native capacity of this guy is 37.5TB which is a whole lot larger than the 7.9TB of our current SDLT320 unit. It only has two drives for now which will limit flexibility somewhat, but it is upgradeable to four drives later when the money tree starts producing again. If we really need to, we can stack another MSL4048 on top of it for even more storage. Because it is drive-limited, we kind of have to stage all backups to disk and then copy from disk to tape; we won't be doing any backups directly to tape.

When it'll get here is anyone's guess. Purchasing is currently digging out of an avalanche of last-minute orders just like ours, so they're w-a-y backed up down there.

Other Blogs

My Other Stuff

Monthly Archives