October 2008 Archives

Printer stats

It has been a while since I last talked about printer statistics. Being a higher ed place of learning, we have student labs. These labs have printers. And like pretty much every place of learning I've ever spoken to, we do whatever we can to restrict printing to reasonable levels. In our case, we do this with PCounter.

For October, the sum total of pages printed (as of this morning, so it isn't ALL of October) from student printers we audit, was...

585,000 pages

That's a lot of paper.

Our busiest hour was 11:00-11:59, where 70,000 (12%) of those pages were printed.
Our second busiest hour was 13:00-13:59, where 63,000 (10%) of those pages were printed.
Our lightest hour was 4:00-4:59, where 360 of those pages were printed (the large majority from the HH154 printers).
The busiest single printer was one of the Library printers, with 48,000 pages.
The busiest lab (multiple printers) was the HH154 lab, with a combined total of 71,000 pages.
The busiest dorm printer was in the Fairhaven complex, with a total of 32,000 pages.
Of the around 22,000 active students we have, 10,509 (47.8%) of them printed at least one page from one of the audited printers.

The above comes to about 43 pages per full-time-equivalent student, or around 26 pages per active student for the month of October. The busiest user printed 892 pages in October, though only three users were over the 500 standard page-quota. Paper and toner are a significant cost of doing business.

The gift of security

Last Christmas my parents bought me a 4GB IronKey. This is a nifty little device! And really, the gift of data security is rather thoughtful. And yesterday, it finally got the new firmware that enables Linux support!

Between then and now I haven't really been able to use it at work. And since transporting files between work and home is one of the nicer features of it, it has largely sat unused. But right this moment it is mounted to my openSUSE 10.3 workstation. This beats a floppy disk for transporting pgp/gpg keys.
Earlier today, Bugtraq saw a couple of messages with links to actual exploit code for this patch. Now anyone can play!

On the up side, stuff built with this code will in all probability be detectable with IPS technologies. But that doesn't help devices in places that lack IPS, such as your local Starbucks.

An old theme made new

Yesterday on Slashdot was a link to an article that sounds a lot like one I published two years ago tomorrow. The main point in the article is that due to the unrecoverable-read-error rate in your standard SATA drive (10^14 bits, or 12.5TB), and the ever increasing sizes of SATA drives means that Raid 5 arrays can get to 12.5TB pretty quickly. Heck, high-end home media servers chock full of HD content can get there very fast.

While it doesn't say this in the specs page for that new Seagate drive, if you look on page 18 of the accompanying manual you can see the "Nonrecoverable read error" rate of the same 10^14 as I talked about two years ago. So, no improvement in reliability. However.... For their enterprise-class "Savvio" drives, they list a "Nonrecoverable Read Error" rate of 10^16 (1 in 1.25PB), which is better than the 10^15 (125TB) they were doing two years ago on their FC disks. So clearly, enterprise users are juuuust fine for large RAID5 arrays.

As I said before, the people who are going to be bitten by this will be home media servers. Also, whiteboxed homebrew servers for small/medium businesses will be at risk. So those of you who have to justify buying the really expensive disks, when there are el-cheepo 1.5TB drives out there? You can use this!

Dorm printing

| 2 Comments
On my post about finally running vista patrickbuller asked:
So you have printers that students in the dorms can print to? Wow. Do you audit all those and charge the numbers of pages against the student?
The answer to that is that we make big use of AND Technology's PCounter product. When paired with their PrintStations, it makes a very nice way to put a lid on unrestricted 'free' printing in the dorms. The PrintStations also make sure that only jobs people want to pick up get printed, which saves a serious amount of paper.

PCounter is core to our student printing. We'll only move our NDPS/iPrint infrastructure over to OES2-linux when Pcounter is supported on that platform, not before. We'll keep a 2 node NetWare cluster around just for printing if we have to. Since accounting support is one of the features that's supposed to be in OES2-SP1, it is my hope that PCounter will support OES2-Linux within a year after SP1's release. But I haven't heard any specifics.

Cool things with powershell

| 1 Comment
Now that we're on Exchange 2007,we've had to figure out PowerShell. When I went to the Exch2007 class, it was pretty clear that Microsoft had redesigned their GUI tools under the 80/20 rule. 80 percent of the functionality that'll get used on a daily basis is in the GUI, and the 20 that gets used rarely or only by automation is on the command-line.

Which means that the once a year you go do something oddball you're hitting google to try and figure out the ruddy command-line options.

Any way, I digress. I've been writing a pair of powershell scripts to do some internal tasks (one of which is to create Resources the way we want them created), and have run into a few snags. The first snag is that a script that looks like this:
new-distributiongroup -Name $groupName -Type security -Yadda True
Add-ADpermission -Identity $resourceName -user $groupname -ExtendedRights "Send-as"


Won't work. That's because "new-distributiongroup" returns before the new distribution group can be acted upon by PowerShell. So I had to introduce a loop to make sure it was getable before I tried setting the permission. This is what vexed me. The loop I came up with is cludgy, but it does what I need it to.
$groupExists="False"
new-distributiongroup -Name $groupName -Type security -Yadda True
do {
sleep -seconds 1
$groupExists = get-ADpermission -Identity $groupName -blah blah |fw Isvalid
} while (!$groupExists)
Add-ADpermission -Identity $resourceName -user $groupname -ExtendedRights

While it works, when the script runs that loop creates a sea of StdErr output I don't care to know about. I'm waiting until it stops returning an error. Sometimes it takes only two seconds for the group to exist, other times it can take as long as 10. I still need to trap for it.

Today I finally figured out how to quash stderr so I don't see it. A very simple modification. It's in the test. Instead of "|fw IsValid", I use "2>1 |fs IsValid". This quashes StdErr, and still populates $groupExists. The script run looks a lot cleaner too.

The other thing I learned the hard way is that if you're doing multiple sets of mailbox or AD permissions, doing them too fast can cause the updates to collide. So I've taken to putting the above loop in to verify the previous permission mod has taken effect before I throw another one in. Annoying, but can be worked around.

Finally running Vista

| 2 Comments
Why? So I can upload Vista32 drivers to the student NDPS/iPrint broker. No other reason.

It would seem that an amazingly high percentage of dorm-residents this year are running vista. Something close to 50%. So it has become rather urgent that we finally get Vista drivers into the iPrint enabled printers in their area.

Getting Vista64 drivers will be more challenging, as I don't have a V64 VM yet. :P

OES2 SP1 (public beta) has been posted

The public beta of OES2 SP1 has been posted.

I believe the NDA has lifted, but I'm not 100% on that. Will check. But, some of the new stuff in SP1:
  • An AFP stack that doesn't suck. Or more specifically, an AFP stack that scales beyond 100 users and is eDirectory integrated.
  • A new CIFS stack written by Novell, so it can scale well past the Samba limit.
  • A migration toolkit in one UI, rather than a cluster of scripts.
  • A new version of iFolder
  • EDirectory integrated DNS/DHCP. But no eDir integrated SLP yet, open-source politics you know.
  • IIRC a beta of eDir 8.8 SP4.
  • The ability to put iPrint-for-Linux on NSS volumes (handy for Clustering).
  • And lots more I can't remember off the top of my head.
Go forth, and have fun. There is a beta-feedback box on the beta page I linked to above in case you find a bug and want to tell Novell about it.

One thing I think it is safe to say, is that even though it says "Beta4" on it, it's really a release-candidate. Only major bugs are getting quashed right now. UI freeze was a month or more ago, and strange, annoying behaviors may get "fixed in doc" rather than getting true fixes which will have to wait for SP2. Still report them anyway, since it'll go on the list to fix in the next SP.

Office

| 1 Comment
OpenOffice has released version 3.0 in the last few days.

This naturally brought me to think about how uptake is going. WWU does offer OpenOffice as an option on our ATUS Lab machines, along side MS Office, but I have zero idea how often it is used by the student body. I have OO installed on my primary work machine, which is no surprise considering that's a Linux machine.

What is the future of OO? Will it ever supplant MS Office?

It seems to me that Microsoft is really pushing for SharePoint to take over from plain old file-serving. This is evident in the extent that SharePoint is integrated into Office 2007. I have to expect that the next version of Office will be even more tightly coupled to some web-based platform.

This is a problem for OO. While it may be possible to shim SharePoint integration into OO, perhaps through cunning uses of Mono, it means that OO will of necessity be one to three versions behind Microsoft in terms of features. Alternate platforms like Novell Teaming and Conferences are SharePoint-like, but they're not, you know, SharePoint.

Unfortunately, it looks like you can only go so far being able to read and create 100% MSO-compatible files. There's this other stuff that needs to be able to be done. WordPerfect learned this lesson in the years between MSO's ascension and the coming of age of OpenOffice as the, "Office package that is not Microsoft's," first choice. WordPerfect had the ability to save to PDF for years, and from what I hear is still the default choice in Law offices. However, WordPerfect is now the #3 behind OO and MSO.

OpenOffice has a difficult road ahead.

More on the budget

The budget crunch I mentioned the other day has made the local paper. This is not that surprising, since we're one of the bigger employers in the area, and 20K students make a noticeable economic impact to the area.

Erm, about the budget

From an email sent to all points from the U President this afternoon:

In the OFM spreadsheets received today, we were stunned to find that targets had been set for higher education. Western, today, is now expected, from the sorts of measures outlined in the August 4 memorandum, to "save" $1,827,000 in the current fiscal year. (This major reduction applies across all budgets, including instructional budgets.)

Add that to the earlier number, and our total budget reduction is NOT the $176,000 representing 1% of non-instructional budgets. It is $2,003,682.

Aaaaaand...

Further, we have been advised to expect these reductions to be permanent; that is, to also be a part of our 2009-11 budget.

Pardon me whilst I mutter things.

This means that it is nearly certain that we will NOT be getting any new hardware for the Novell cluster next summer. We'll have to do it on hardware we already own right now. This means I won't be able to partake of that lovely 64-bit goodness. Drat drat drat.

We're already under-funded for where we need to be, this won't help. Even with the storage arrays we just bought, in terms of total disk-space we've managed to fully commit all of it. There is no excess capacity. What's more, there is no easy way to ADD new capacity since any significant amounts will require purchasing new storage shelves.

In the intermediate term, this means that WWU will now descend into bureaucratic charge-back warfare. As service-providing departments like ours try to find ways to finance the needed growth, we'll start being hard-ass about charging for exceptional services. And they'll do it to us too. So if the College of Arts and Sciences comes to us and asks us for space to host 2TB of, say, NASA data, we'll have to bill them for it. And that cost will be a 'total cost' which will by necessity include the backup costs. In return, if we need 16 ethernet jacks added to the AC datacenter, Telecom may start billing us.

And I get a new boss Thursday. Happily, since there is overlap between outgoing and incoming they've been briefing a lot. This is to prepare the new guy for the challenges he'll face in his first few weeks flying solo. There may even be the odd phone-call for advice, we'll see.

Gonna get real interesting around here.

MSA performance in the new config

Today I reconfigured the MSA1500 to run in Active/Active mode. While there, I also rearranged our disk arrays. We have 41, 500GB, 7.2K RPM drives in there. I created two, 20 disk Arrays, and filled each array with Raid 0+1 LUNs. This yielded 9TB of useful space. That extra drive will stay extra until we get an odd number of new drives.

Yes, a profligate waste of space but at least it'll be fast. It also had the added advantage of not needing to stripe in like Raid5 or Raid6 would have. This alone saved us close to two weeks flow time to get it back into service.

Another benefit to not using a parity RAID is that the MSA is no longer controller-CPU bound for I/O speeds. Right now I have a pair of writes, each effectively going to a separate controller, and the combined I/O is on the order of 100Mbs while controller CPU loads are under 80%. Also, more importantly, Average Command Latency is still in the 20-30ms range.

The limiting factor here appears to be how fast the controllers can commit I/O to the physical drives, rather than how fast the controllers can do parity-calcs. CPU not being saturated suggests this, but a "show perf physical" on the CLI shows the queue depth on individual drives:
Queue depth chart
The drives with a zero are associated with LUNs being served by the other controller, and thus not listed here. But a high queue depth is a good sign of I/O saturation on the actual drives themselves. This is encouraging to me, since it means we're finally, finally, after two years, getting the performance we need out of this device. We had to go to an active/active config with a non-parity RAID to do it, but we got it.