Centralized IT

I've had quite a bit of experience with the process of centralizing IT. At my last job I was at ground zero as I was on the committee that was charged with rationalizing an IT job family structure that was grounded in the early 1980's (key clue, the phrase, "electronic data processing" was slathered across many job titles, a phrase not at all in vogue in the 1990's). This particular consolidation event was driven from a directive from on high, above the CIO. So, as it were, it happened in spite of the grumbling.

WWU has gone through some of its own consolidations, but there are natural barriers to complete consolidation in the Higher Ed market. I'll get to those in a bit. The one thing acting as a serious barrier to consolidation in any organization are departments that are large enough to support their own multi-person IT departments. Departments with one or two people effectively doing the full IT stack (stand-alone sysadmins who also do desktop support, database maintenance, to-the-desk network wiring, and maybe a bit of app-dev along the side) are most vulnerable to being consolidated into the central Borg.

Some departments are all too happy to join the central IT infrastructure, as they see it as a way to shed costs onto another business unit. Others are happy because their own IT people are so overworked, the idea of getting them help is seen as a cost-free mercy; or put another way agreeing to consolidation is seen as a cost-free way to increase IT investment. Still others are happy to join because they want some nifty new technology and their stick-in-the-mud IT people keep saying, "no," and view the central Borg as a way to get that thing.

The big reason departments don't want their IT people consolidated away from them is personalized service. These are people who know the business intimately, something those central-office folk don't. The cost of maintaining an independent IT infrastructure is seen as a perfectly valid business investment in operational efficiency. Any centralization initiative will have to deal with this concern.

The other big reason shows up less often, but is very hard to overcome without marching orders delivered from On High: distrust of central IT in specific. If the business unit that contains central IT is seen to be less competent as compared to the local IT people, that business unit will not consent to centralization. If the people in central IT are collectively viewed as a bunch of idiots, or run by idiots, the only way that unit is centralizing is if a metaphorical gun is held to their heads.

My last job handled the all of the above and eventually came to an agreement. First and foremost, it was a fiat from On High that IT centralization would happen. All IT job titles started being paid out of the same budget. We then spent the next four years hammering out the management structure, which meant that for a long time a whole bunch of people had their salary paid by people with 0% influence on their work direction.

Many departments gleefully joined the central infrastructure, driven in large part by their own IT people. They'd been overworked, you see, and the idea of gaining access to a much wider talent pool, and a significantly deeper one as well, was hard to not take advantage of. These were the departments with 1-3 IT people. In almost every case the local IT people stayed in their areas as the local IT contact, which maintained the local knowledge they'd developed over the years.

There was one small department that was a holdout until the very end. An attempt to merge some 5 years earlier had gone horribly wrong, and institutional memory remembered that very clearly. It wasn't until that department got a new director that an agreement was reached. The one IT guy up there stayed up there after the merger and stopped doing server and desktop support in favor of department-specific app-dev work, what he was hired to do in the first place as it happened.

Then the arm-wrestling over the bigger departments took place. For the most part they kept near complete control over their own IT staffs, but their top level IT managers were regularly hauled back to the home IT office for 'management team meetings'. This ended up being a good move, since it reduced the barriers for communication at the very top level, and ultimately lead to some better efficiencies overall; especially in the helpdesk area as staff started to move between stacks after a while. Also, the departments that had been deeply skeptical of this whole centralized IT thing started working with other IT managers and getting their concerns heard, which reduced some of the inherent distrust.

With Higher Ed, there is an additional factor or two that my previous job didn't face. First of all, the historic independence of specific Colleges. Second, Universities are generally a lot less command-and-control than their .com or even .gov brethren. This means that centralization relies far more on direct diplomacy between IT business units than it does on direct commands from on high. Distrust in this environment is much more hard to overcome as coercion is not a readily available option.

Back in the day, WWU had 7 separate NDS trees. 7. That's a lot. Obviously, there wasn't much in the way of cross-departmental access of data. Over the course of around 5 years we consolidated down to a single 'WWU' NDS tree. Some departments happily stopped spending IT time on account maintenance tasks and let central IT do it all. Some departments gave up their servers all together. Time passed and still more areas decided they really didn't need to bother keeping local replicas, and let central IT handle that problem.

In the end, handling IT in Higher Ed means dealing with a more heterogeneous environment than is otherwise cost-effective. I've mentioned before how network management on Higher Ed networks resembles ISPs more than it does corporate networks, and that unfortunately applies to things like server and storage purchases. Now that we're in the process of migrating off of NetWare and onto Windows, it means we're now in the process of wrangling over rules governing Active Directory management.

We wrangled NDS control back in the 90's and early 00's, and now it's Microsoft's turn. As with the last round of NDS wrangling, some departments have gleefully turned over control (GPOs and file-server management specifically) of their department over to us in ITS. Others, specifically one with a large local IT presence, is really holding out for complete control of their area. They're clearly angling to just use us as an authentication provider and they'll do the rest, something that... well... negotiations are ongoing.

My crystal ball says we have somewhere between 5 to 10 years before the next wave of 'directory' upgrade forces another consolidation. That consolidation just might involve consolidating with a State agency of some kind. Perhaps the State will force us to use a directory rooted in the wa.gov DNS domain (wwu.univ.wa.gov perhaps), and our Auth servers will be based in Olympia rather than on our local network. Don't know. What is true, is that we'll be going through this again, probably within the next decade.


I'm sure there's a better blog entry to ask this question on, but can you post a direct comparison detailing what you used to have with Novell and what you now have with MS? Server count, are you still clustering, downtime, patches, etc.?