September 2010 Archives

We go through paper

We're a University, and you'd expect that in this modern era of ipads replacing textbooks and suchlike that our paper costs would be going down. You'd be wrong. We go through a heck of a lot of paper in a quarter. Spring quarter earlier this year generated 1,899,865 pages of printing, which is actually a bit up from what we did last year. Ouch.

For a nice visual clue to what we go through in a day, here is Monday of this week:
Pages-per-hour for September 27, 2010
41,375 pages is the total for Monday. Monday is also our heaviest printing day. That spike you see between 11am and Noon is regular. We've had the 11am printing peak for years. There is a smaller spike between 1 and 2pm. This time of quarter we don't have any printing going on at 5am, though the closer we get to Finals Week the more dark-o-night printing goes on.

Budget issues

Yesterday the University President posted a bunch of budget documents that describe how we're handling the 3% cut we need to accommodate. You can start reading them yourselves here (hyperlinked PDFs, not HTML). In and amongst the various line-items was one that caused me a bit of heart failure.

Bring units in leased space to buildings we own. These include our off campus dance facility and all units in the space we currently lease at 32nd Street.

Dude, that's my office! Does this mean we're moving back up to campus? Where in tarnation are they going to fit our datacenter? Moving THAT is not going to be cheap. As in edging on a million bucks worth of not cheap.

But calm prevailed and more information was supplied by the ITS Vice Provost. Human Resources is going to be heading back to campus. The building I'm in is actually owned by WWU.

Whew! Thank goodness, I didn't want to have to move a datacenter. That's a lot of work.

Also, ITS will be taking at least one layoff. I'm pretty sure I know who it is, but I'm not saying here. It isn't in Technical Services though, we've been spared the ax one more time. No idea if our luck will extend to the 10% cut due to take effect 7/1/2011.

And we still have no training budget.

Paying interest on hacks

| 1 Comment
I didn't know about this blog-post until today, but...

It's a good read, so I suggest going there and reading it. And that's a reaction to another, longer post:

The tl;dr summation:
Any IT system develops Technical Debt. Things that require attention as a result of past decisions. For sysadminly stuff, systems that require hand-holding. As time progresses, the amount of technical debt being serviced can outweigh any new stuff being done. Possibly by quite a lot. Therefore, it pays to reduce technical debt in order to become more overall efficient.
Or as Jeff put it, they had to put a hold on the StackOverflow new-feature development in order to service some of the debt incurred by earlier decisions. That liberated dev-time later on to make new stuff. For a dynamically growing site like StackOverflow, that's important.

This applies to not just code writing, but even SysAdminly things. Something that starts as an occasional manual process (say, reindexing a database table when helpdesk call volumes get above a certain level), ends up being a daily one, then an hourly one. That's enough to motivate the most blunt of thinkers to just script up a way to index the database as a way to get more free hours in a day. Though the real fix is to put pressure on the application doing that to fix their index problem. IT is an ecosystem after all.

This also ties into IT politics right nicely as well. If most of the debt-servicing is being done by, say, the Sysadmin or helpdesk staff, then the need to pay it down in a determined way in order to reduce it may be hidden from the decision-makers who need to push that kind of thing. For self-built applications, debt-servicing requiring dev-time tends to be a lot more visible than helpdesk time. So having the various IT functions talk to each other is a very good thing when making a paying-down decision.

It's a nice way of putting it.

Start of fall quarter

Wednesday marks the official start of fall quarter. This weekend was move-in for the dorms. Students are here and settling in already, and our labs are bumping as students grab course materials and handle all the online-only pre-class bureaucracy.

This year's major new IT stuff is actually a short list, but it is important:
  • A majority of our Computer Labs are now running Windows 7.
  • Blackboard has been upgraded from 7.something to 9.1, a very significant upgrade
We spent most of last year figuring out how to get Win7 into our labs and it is now trial by fire time. Most of our lab managers have opted into Group Policy based printer distribution. We're cheering this. According to our old timers (heh) the Win7 upgrade has gone smoother than the WinXP upgrade did back in the day. Good to hear.

Printing is always a pain at the start of the quarter, that isn't new. Last year we had one student who managed to blow through their entire free-printing quota in the space of 5 hours, as they printed off what looked to be whole chapters of a textbook. Some students still dilligently print out instructor-provided PDFs. The good thing about that this year is that we now have a way for students to refill their quota with a paid quota; that system was still broken at the start of the quarter last year. Hooray!

This week is always fun.

Un-dreading IPv6, addressing

| 1 Comment
IPv6 addressing is scary looking. Anything that calls "2608:af09:ce24:342a:00a1:3def:ff31:0122" an IP address is going to cause some fear in anyone who is supposed to support that environment. While they can get that complex, they don't have to be.

IPv6 addresses operate the same as IPv4 addresses do. They have a network part and a node part, delineated by the subnet mask. Just as denotes a subnet in IPv4, 2608:af09/32 denotes one in IPv6. Unlike IPv4, IPv6 subnets are always defined by bit-count; the IPv4 equivalent of is /24, but you'll never note an ipv6 subnet mask as ffff:ffff::. The bit-count in IPv6 increments by four for each digit from the start of the address.


And so on.

The other thing to keep in mind is that is perfectly acceptable to have large strings of zeros in IPv6 addresses. You can replace one such string with "::" in the address. Thus an address like:


can be represented by


Also, you may have noticed that leading zeros in an address can be trimmed. The above address assumes ":30" is ":0030" and ":0:" is ":0000:". That also helps keep things short.

This also means that those IP numbers you just have to memorize, DNS and gateway addresses, can be kept simple if your network administration staff don't actively hate you. Your DNS server may be located at 2608:af09:134::134. Not scary at all.

Now for some of the differences with IPv4. They're not scary, just different. Below the fold.

IPv6: How 6to4 works, a visual guide

I used my gimpy drawing skills to make a diagram describing how 6to4 works. This is what that 2002:[something] address you may have noticed on your Windows Vista/7 machine. Those of you at WWU probably all have this because we're on publicly routeable IP addresses. 6to4 uses AnyCast to accomplish what it does. It can be tricky to wrap your head around since it is also a stateless protocol. Anyway, here is a picture worth a couple hundred words:

For those that don't load images, here is the traffic path between a client at talking over 6to4 to

  1. Client starts a TCP session with the 6to4 AnyCast address, with an encapsulated IPv6 packet destined for 2001:4860:8005::6a.
  2. The nearest 6to4 AnyCast router strips off the IPv4 wrapper and forwards the formerly encapsulated packet to 2001:4860:8005::6a
  3. The server at 2001:4860:8005::6a sees an incoming connection from 2002:8ca0:7b2d::8ca0:7b2d, replies with ACK packet.
  4. As the ACK packet has 2002 prefix, it is forwarded to the 6to4 router nearest to the source which encapsulates the packet within an IPv4 packet, with a Source address of and sends it on to
  5. Client receives ACK packet from, sends SYN ACK to
As you can see, a firewall between the source and the first 6to4 AnyCast router only sees a regular IPv4 session. At the same time, a firewall sitting in front of only sees a pure IPv6 session. Unfortunately for the IPv4 firewall, unless it is smart enough to decode the 6to4 session it has no idea where that packet is ultimately going and therefore can't apply firewall rules to it. The IPv6 firewall has no such limitation.

It has been 10 years

| 1 Comment
10 years now since Windows 2000 came out, and I still run into SysAdmins who refer to an AD Domain Controller as a "PDC". Or worse, seem to think additional DCs past the first are "BDCs." As in, "I added a new BDC," when talking about a Server 2008 AD network.

I realize that "The PDC" is a kind of short-hand for "fully featured domain controller in a Windows Domain," but it is a reference that increasingly few of us will get. 10 years is an age in the IT industry. We have a whole new generation of Microsoft SysAdmins out there who've never clamped eyes on Windows NT, and therefore have to be reminded of the difference between P and B on your DCs since it doesn't matter on AD. Or worse, get trained by the old-guard and call AD domain-controllers PDCs since that's what everyone in the office does.

I also still see "WinNT" as short-hand for "Windows Server OS" in everything from internal documentation (I'm changing that where I find it and can fix it), to job postings ("10+ years WinNT experience"), to resumes, to job-descriptions. Happily, it has disappeared from marketing glossies. Job-descriptions I can kind of understand since those can be anywhere from three to fifteen years out of date. But job postings and resumes baffle me. Do they really mean WinNT the product, or WinNT the short-cut for "Windows Server"? I feel pity for the job applicant who has been subjected to supporting a desperately out of date operating system for the last eight years, and scorn for a business still using it.

As with all Microsoft operating-systems, there are some special cases that demand using it. We have scientific equipment on campus that costs Quite A Lot Of Money and therefore only get replaced every 10-20 years, but still have to attach to a computer. This is why we still have some Windows 3.11 in back corners, since drivers/protocols/whatnot for the very expensive gadget only exist on that platform. The same can be said for WinNT. So these edge cases exist.

However, for business? Yeah, it can take a decade to rewrite a bank's financial platform. That's where the pity comes in.

It has been 10 years since Windows NT was superseded by Windows 2000 and Active Directory. It's time to put the last nail in that coffin and bury it already.

IPv6 backwards and forwards

I've been trying to get my head around IPv6 addressing lately. This isn't a result of any projects I'm doing, more of a 'get ready for the future' thing. One of the problems I'm having to deal with is the scary-looking nature of IPv6 addresses. For those of us brought up in IPv4, IP addresses with hexadecimals in them are intimidating. In fact, this specific issue has been brought up in meetings when v6 has come up. As I've mentioned before, our telecom group isn't going v6 until they get pushed. Having a /16 address block to play with means regulatory or technical obsolecence will be the drivers to move to v6, not IP exhaustion.

Anyway, IPv6 addresses are not as intimidating as you might think. Yes, they do have 8, 4-digit numbers in them and there is no way on this earth that your average helpdesk technician (or sysadmin) is going to memorize more than a very few 32-digit hexadecimal numbers. Happily, thanks to how v6 works, there are built in short-cuts.

First and foremost, what constitutes the 'network' portion of the address. The entire IPv4 address space can be contained in two of the 4-digit numbers. We care about this, since this is what will drive IPv6 address-space exhaustion. You can look up which regional registrars have been assigned what v6 prefixes on this link:

2600:0000::/12 given to ARIN means that the network-number is "260"

Which means that it is entirely likely that when WWU gets its IPv6 allocation, it'll probably get something similar to:


Which gives us a nigh unimaginable amount of addresses. With 'nigh unimaginable' space to work with we can leverage the power of inefficiency in the pursuit of human understanding. It is entirely allowable for us to configure local addresses that look like this: = 2608:AA11:243:0:0:0:0:16 = 2608:AA11:243::16

In essence, treat our hypothetical 2608:AA11 prefix the same way we're treating 140.160 right now. It's a bit different to remember, but at least once that prefix is memorized the rest just falls into place. Heck, you can even have up to 9999 hosts in a subnet without ever using a letter or an extra colon!

On the back-end turning 2608:AA11:243::16 into a hostname is uglier. The reverse lookup for that address would be found:

Only the DNS admin needs to worry about entering monstrosity into the named tables. Oh wait...

Once the transition to v6 is completed we can start doing funny things with our subnets, or adding colon block to addresses depending on what we need to do with them. And since v6 isn't limited to 256 in each block we can have a lot more subnets. However, we have to get there first.

Times change, alas

Right now we're giving serious consideration to using folder mount-points in Windows in order to solve a specific storage problem. The one thing that make me go, "oh, please, no," is the fact that the disk-space monitoring script I've been using for years, the one that also monitors NetWare, Windows and ESX, can't handle folder-mounts. Why? Because the Windows SNMP agent doesn't give any information about folder-mounts, just drive-letter mounts.

SNMP was very nice since I didn't have to use Windows to get the information I needed. However, Microsoft hasn't been really paying attention to SNMP in recent versions so I am not at all surprised to learn that this hasn't been put in place. Or if it is, they're using a MIB I don't know about.

I suspect I'll have to carve my script up in twain, into Windows and non-Windows variants. That way I can continue to keep data in this particular database (with data that goes back to 2004!).

But still, the core engineering of this guy was done back in 2001, with efforts later on to shim in  Windows and ESX support. I looked into Linux a couple years ago and determined that I could add support for that pretty simply, but never did as we didn't have a call for it yet. 9 years is a long life for a script like this. I suppose it's time.

Or maybe we can not use folder-mounts.