August 2004 Archives


Well, it seems I was right in regards to CPQACPI vs CPQMPK. I rebooted one of the STUSRV machines after changing it to use CPQMPK and whamo! Zippy. The TSA600 backup the previously ran at the sort of zippy 250 megs/minute ran at a much nicer 840 megs/minute, coming just shy of the old TSAFS number. The TSAFS number after switching over is 2553 megs/minute for the same STU1: data I ran with in the first tests.

And when I run three (three!) TSATESTs at the same time I get an aggregate backup speed of arounf 4800 megs/minute out of the same disk-pack. I start a fourth stream and my aggregate swells to 5800 megs/minute. All the data being played with here resides in the same disk-group on the EVA, so we're getting a good stressing of the performance of the disks themselves. I think we can squeeze even more out, since as I look at the drives on the EVA with all four streams running and they aren't completely pegged yet.

We have plenty of throughput to play with.

Speeeeeed with data!

All values Megs/Minute:
[Using TSAFS]
FacSrv1/Share1 FacSrv2/User3 FacSrv2/Share3 StuSrv2/Class1
908 2106 1565 2492 846
930 2105 1536 2447 885
925 2070 1582 2519

[Using TSA600]
StuSrv1/Stu1 FacSrv1/Share1 FacSrv2/User3 FacSrv2/Share3 StuSrv2/Class1
238 875 625 1160 221
265 904 656 1160 218
272 895 670 1184

The format of the test is nodename/volume. Each dataset had more data than known buffers in the data path in an effort to minimize the accelleration gained by such. In most tests, subsequent runs of the same dataset were faster than previous runs due to the caching on the server itself and on the EVA back-end SAN. The data were gathered using the TSATEST shipping in the NW6SP4 service-pack. The FacSrv2 tests are unique in that first User3 was run at 5GB then Share3 was run at 5GB in order to bust buffers where possible. The only variable between datasets was the TSA loaded and any itty bitty file changes that may have happened during normal usage.

What this test clearly shows is that TSAFS is faster than TSA600. In all cases tested the speed-up was north of 200%.

Other observations

One thing I did observe is that the FacSrv servers are faster than the StuSrv servers when it comes to TSA backups. This did merit investigation and I have come to the conclusing that the PSM file used has impacts on speed. Spurious interupts (displayable by "display interupts" from console) increment dramtatically when CPQACPI.PSK (v1.06 5/12/03) is loaded, as it is on the StuSrv-series of servers. The FacSrv series of servers are old enough that they load CPQMPK.PSM instead, and the spurious interupts on those servers are worlds lower than the numbers reported on the StuSrv series. I predict that backup speed improvements are to be had by changing which PSM we load.

It also looked like data characterization played a role in how fast things backed up. User volumes went slower than 'shared' volumes (Class1 is the student version of Share1). This may have something to do with the average file size being smaller on User volumes, or perhaps the churn-rate on the user-volumes leads to noticable fragmentation.

Another test I ran once just because I was curious was to back up Share1 from FacSrv1 and Share3 from FacSrv2 at the same time with TSAFS and monitor the results. Both servers backed up 5GB of data and when they hit the 5GB mark I recorded the rate. When the first server hit that mark I let the backup continue in order to provide the same contentious I/O path for the slower server. The totals were:

FacSrv1/Share1 @ 1862 MB/Min
FacSrv2/Share3 @ 2065 MB/Bin
Aggregate throughput of SAN I/O channel: 3927 MB/Min

This tells me a number of things:
  • Parallel backups won't drop absolute performance below the rated tape-drive performance spec
  • The SAN storage really is fast
  • A third stream (Share2?) could be added and still probably maintain real-world network speeds.

We have some work to do. For one, we need to either get Compaq to address the spurious interupt issue, or drop back to CPQMPK.PSM in order to get good results out of the StuSrv series. Once this is done, I hope that the StuSrv series will be able to provide performance matching or besting that of the older FacSrv line.

The networking infrastructure needs attention. The maximim theoretical throughput for 100Mb ethernet is 750 megs/minute, and each one of the TSAFS backups went faster then that. The maximum speeds observed are fast enough to dent even Gig Ethernet, though at those speeds tape-drive latency comes into play. The current 100Mb connections are not adequate.

The backup server needs to be robust. With the possibility of multiple high-rate streams coming to the server, its own Gig Ethernet connection may become saturated if all four tape-drives are streaming from a fast source. With speeds up that high, PCI-bus contention actually becomes a factor here so the server has to be built with high I/O in mind. No "PC on steroids" here. Best case would be multiple PCI busses, or at the very minimum 64-bit PCI; neither option shows up on sub $2000 hardware.

Our storage back-end is robust. The EVA on the back end can pitch data at amazing speeds. It is very nice to see rates like that, even on volumes that have had 12 months of heavy usage to fragment the bejebers out of them. We can scale this system out a lot further before running into caps.

We need to verify if our backup software solution is compatible with TSAFS. We use BackupExec for Windows, and that answer is not quite clear yet. BackupExec for NetWare is already there, so at least part of the product line knows how to handle it. We need a decision on how to handle open files before progressing on that front. But TSAFS versus BEREMOTE is something I can't test easilly.

TSAFS is the way to go, but it is only one piece of the puzzle.


Backing Up Vol:   SHARE1:

Read Count: 90253
Last Read Size: 17566
Total Bytes Read: 4664767995
Raw Data MB/min: 2231.01
Backup Sets: 26424
Ave. Open Time: 000uss
Ave. Close Time: 000us
Effective MB/min: 2183.85
Holy fiber channel, Batman! I've never seen a stream go that fast. This is a TSATEST backup using TSAFS (TSA600 was giving me still-impressive numbers around 900 MB/min for this dataset), with an EVA300 back-end.

Updates galore

In order to get Exchange 2003 System Manager installed on my system, I had to put a few things on. Rather than have an SMTP server on my machine just for exchange (WTF?) I put on XP-SP2 to prevent that need from being there. Plus NAI came out with VirusScan Enterprise v8i so I threw that in there too.

Something doesn't like OpenSSH for Windows. It just crashes every time the sshd-daemon sees an incoming connection. Something about a bad fork. I harken back to nmap's problem with SP2 and wonder if they're related at all. Ah well. No mo' po' man VPN, I'll have to use the real thing.

Plus today, I found and installed Win2K-Server Resource Kit. Lots of fun stuff in there, but then reskits usually are fun that way. I hadn't thought to look it up when I got the new machine. There isn't a kill.exe in there, but there are some other nice utils. Like a tail.exe. THAT is nifty. And linkd.exe has helped me exile one of my recurrant disk-space hogs to the big partition and off of System.

And I also learned that my system directory (C:, I shove everything else onto D: where possible) has about 3gb of stuff just for the OS. Once upon a time I'd call that horribly crufty, but since then I've seen Linux distros match that.

And it lives!

Exchange is now up! Once the cables were in and all the servers were seeing their SAN bits, the rest was simple. We now have storage servers with storage groups and stuff, theoretically able to start receiving users. Not to shabby! And we were worried about missing deadlines.

Still no word on the NWLIB issue.

New cables

We got replacement fibre cables darned fast! This is good since now the Exchange project can proceed as planned. We had two cables go bad on us from our earlier shipment. Murphy has not been kind to this project so far, which makes me nervous for when we start moving mailboxes over.

Exchange Project

Delayed on account of bad cables. Two of the three short-wave fibre cables connecting the future storage servers to the fibre-channel SAN ended up bad. On one I can see light, so the return-leg is broken. The other gave me the amber light on the switch, which usually means either one channel is working or the pairs are swapped; as this is a prebuilt cable swapped-pairs is unlikely.

We had a spare cable available which greatly helped in proving the broken cable problem. But it also means that only two of the three future storage servers is able to be clusterable.

I have to assume that installation killed the cables. And we didn't do the install. Aie. Fragile stuff, fibre.

New ported tool: MRTG on netware

| 1 Comment
MRTG running on Netware

That's right, the MRTG monitoring tool actually running on Netware. I've been monitoring Netware from MRTG for years, but this is a document to get your MRTG running on netware itself. The 'rateup' utility was ported to NLM. Right now there are some scaling issues, but they're being dealt with. I'm not in a position to port my stuff over to Netware, but this is just too nifty to let pass up.

Java consoles

The management utility for the Liebert remote-shutdown agent is Java.

I do not like java when it is anywhere other than an applet on a web-page. Developers make all sorts of insane assumptions about the operating environment when they're running out of the sand-box as it were. Chief among them, as in the prime number one biggest problemo, is assuming that there isn't any java already installed.

No, sorry. You get to assume that I, as a consumer of management consoles for IT widgets, have lots of JVMs cluttering up my machine. And a-l-l the stability fun that entails.

Widget A requires 1.3.1_03
Widget B requires 1.3.1_05, and aggressively detects its version
Widget C requires, because it is brand new, 1.4.2_02
Widget D requires, because it is really old, 1.2.1

Even ConsoleOne gets cranky when OtherJava is running at the same time it is, and ConsoleOne is almost a compiled program. Some environments, such as NetWare, do not have the ability to run multiple JVM versions so what you are using is critical to know. Windows theoretically does, but I've had too many problems with it to trust it to anything critical.

In case you hadn't figured it out already, the Liebert console gave me grief. It doesn't require a JVM install, per-se, it just loads one it ships with. This is sub-optimal if there are other JVM versions lurking around as they can cross contaminate memory. I couldn't get the Netware console to load on my machine. It hung it so hard I needed a reboot to clear the JAVA32.EXE process from memory (unkillable modules are fun that way, and it even resisted kill.exe and a few other ungraceful nukers out there). I had to set up a VPC session with a base OS, Client 32 and nothing else to get it to run appropriately. Unacceptable.

Wahoo! Liebert is talking!

After w-a-y too much time involving another admin around here who has been doing serial comm since the begining of time (relative to me), and some additional quality time with a good multi-meter I FINALLY have an adapter to allow serial comm with the Liebert UPS. The catch was that three and only three wires are to be connected, and any signal on any other wire will hork it up. This is what was causing my earlier "I can see it, but it doesn't respond to my keystrokes" problem.

Anyway, the thing is now on the network and I've gotten multilink to talk with it. AND a test of the shutdown procdure. Works on Win2K so far.

But I must say I'm not impressed with the shutdown/management software. For one, for some reason I can't bring up the config screen without having the monitoring service shut off. This isn't how it is supposed to work, I know this because I can see the java32.exe process launch then hang.

I'm really worried about the netware shutdown. If that's in Java as well, then we MAY have conflicts with our Zen software. Won't know until we try, alas.

Bugfix it ain't

Didn't work for us. According to the developer of mod_edir, it worked for him. Now we get to figure out why we're special! I did some serious work on it Saturday (I was here for SummerStart) and was able to determine that the way it is breaking is identical to how it was breaking before; specifically the remote server is passing a -601 (Not Found) error when the web-server attempts to authenticate to the remote server.

I have network sniffs of this in action, so I can prove it and stuff. Our incident isn't closed yet, and I intend on working this one pretty hard. Our upgrade window for the cluster starts this Saturday, so time is really begining to press.

August Info Security ad-counts

I got my "Information Security" magazine today, and ran a count of the ads in it:

Security Service Management
in-house ads

Total ads

A bit weirdly formatted, but there you are. It seems that FUD has fallen from first place and convenience has taken its place. It is worth noting that the 'before the table of contents' ads are both security management firms, and the back-cover is a spam widget.

From this it would seem that the Big Money in security these days is in management firms. The message is, "Why do this hideously complex task yourself, when you can buy our expert services?" The widget-makers still have a high FUD quotient, but are coming around to the ease-of-management line. Even though this issue is "wonder appliances", there were very few wonder appliances advertised like that. I'm somewhat surprised that regulatory-compliance isn't a big mover yet, so far only the one ad stressed that.

How does this impact us? Not a whit! We're a University. Information shall be free, remember? Plus, we don't get a lot in the way of budget for restrictive technologies. Organizationally speaking, we're only now coming around to the idea that denying outbound traffic might be a good idea in some cases (individually we've been saying this for years). The concept of an actual honest-to-god firewall at 'the border' is one that causes reflexive quivering on a number of points, so we've been making do with router rules all throughout the infrastructure (which can do way more than I thought they could, having come from a firewalled OldJob).


Novell released a new Libc! This should fix the problem I've been having with mod_edir and our cluster! Woot!

Now to go test.


We learned yesterday that the install disk we have of Windows 2003 Enterprise Server is actually the MSDN install. This is not good. The MSDN version can not be legally used in a production environment.

On the good side, we caught it fairly early.

On the bad side, we will have to rebuild five servers, which happens to include all of our AD domain controllers. Eep.

Lice-ridden backups

There is a reason I hate them.

I hate them because if they fail, we have a hole that we hope no one notices.

I hate them because they can always be a little faster than they have been going.

I hate them because they take too many tapes.

I hate them because the backup agents never perform the way I NEED them to.

I hate them because two otherwise identical servers can back-up at completely different speeds.

I hate them because we never have enough time to get them all in.

I hate them because we can never leverage the resources we need to get the damned things to work well.

I hate them because our backup server is a homebrew frankenstien that probably can't keep up with the brand spanking new tape library.

And most of all, I hate them because we get to manage the backup infrastructure with no budget of our own.

More disk-space options

Today I learned that Caminosoft has an HSM solution for Netware. Nifty, huh? No idea of pricing, through I fully expect that we're about half the size where it'd be a good idea.

We do have a couple of other options to reduce costs.

Veritas BackupExec Working Set

BackupExec has a newer feature in it called "Working Set". A "working set" backup is a backup of files modified in the last X days, where X is defined at job-creation time. For example, if X is set to 365, you only need full backups a couple times a year. That kind of thing.

This particular feature would allow us to cut down how much data we throw onto tape, and thus provide significant cost-savings. It won't cut down on how much disk is actually used, but it will save us on our backup costs. Usefull.

Novell Archive and Version Services

Documentation for this puppy:

This is a separate server that runs a versioning service against specified volumes. This is similar to Salvage, but both more and less robust. Unlike Salvage, versioning doesn't depend on how much free-space is left on the volume. Unlike Salvage, all versions of the files are not kept. Unlike Salvage, you can restore files from NetStorage.

This particular server would require a LOT of disk-space, since it would be providing version-control to most of our cluster. Since this space is by definition off-line, it doesn't have to be high-performance space; U160SCSI RAID would be just peachy. It would need another server.

The savings here would be that we wouldn't have to take daily backups in order to provide "oops" coverage to our users. This version control thing would handle that part of it. The backups go back to being a disaster-recovery tool.

In Memorium

One year ago today, my other half at OldJob died suddenly in her sleep of congestive heart failure. She was in her mid forties with two children, aged 15 and 6. She was my other half in the sysadmin job, in that she handled the user environment. That's password resets, standardizing user-object field contents, managing name changes, moving objects between containers, some GroupWise foo, organizing the helpdesk so they did tape-changes, and keeping track of printer additions. Among other things.

I took care of "server stuff", such as login-scripts, patch-levels, and most aggrivatingly, tape backup. We worked very closely on server migrations as that is a major task, and one we got really good at over the years. We had it down to a science. And we both were peeved when the word came down from on high that the user environment would be migrated to Microsoft (no word from OldJob on if there is any progress on that). Our setup was unique at OldJob since we managed by a factor of four the largest NetWare environment than the next largest group of users, the two of us were kept quite busy.

Oddly, it was a year ago on the 3rd that I submitted my application to this job. I gave serious thought to calling WWU and telling them nevermind since we needed to deal with this sudden removal of a 27 year veteran who had been with Information Services since the seventies. As burocracies turn, by the time I DID leave OldJob her old job-functions had been fully subsumed into other functions.

As always, time does blur things. We worked really well together, and she did a very good job of insulating me from the more mundane day-to-day tasks of administering an environment with anywhere between 300-600 users (depends on how you counted noses, and who did the counting). We had that well oiled team thing down good, something that was very noticably skipping in her absence.

She was one of those people who archived every e-mail she sent. At one point her archive at 1.6gb was larger than the whole IS post-office (she was my #1 disk-space user on our server, no surprises there). This keep-it habit held to her paper files, as she did keep every set of minutes she wrote and received. And knew how to find things in there too. She WAS our organizational memory, and a close second was our boss who had been with OldJob for only slightly less time than this gal.

The day after, the 7th, I did a bit of forensics on her work machine looking for passwords and a will. It seemed she had been updating it, and the family wasn't finding it anywhere obvious. I didn't find the will, but I did find a palm-db file with passwords to certain financial sites. THOSE did prove useful.

Penny Kohler was the second funeral at OldJob. But it was the most personally impactful.

Tracking disk-space usage

One of the themes management is singing these days is better management of our disk-space resources. One of the areas that needs improvement is better tracking of usage trends. In the past disk-quotas have been restrictive enough that we've had the luxury of not having to be day-to-day obsessive over how much space is left on the various volumes.

At OldJob, we didn't have disk-quotas, so we had day-to-day obsession over how much we had. One of my better ideas (I'm proud of this one) was to come up with a perl script that checks the available space of the various volumes, and dumps the data into a database table. This table then has reports run against it to make pretty pictures. Get enough time into the table (like say... 4 years) and you can get some real trending information. This was a VERY valuable tool when it came time for budgets as I was able to say things like:
See this line here? If this trend continues [draws line] we should hit the roof about 9 months from now. Now, we know that when we start hitting our heads on the roof, the users start to complain; but we can resist that for a few months. So we add a few months here. So we have 12 months until we hit the point where our good rep goes down the tubes. From the looks of things, another 125gb should keep us for 2-3 years. Perhaps we should put that in the budget?
Very nice to have that data, and we're not tracking it yet. There are plans to have Zen for Servers do that for us, but that's months away. So yesterday I took out my old script and hacked at it. I had to remove the db-dump module as I don't have a database I can dump so, and replaced it with dumping to a CSV file instead. The conversion took about four hours. Also added was an edit to track only specific volumes, which is needed to track resources in the cluster. So the old script is currently monitoring our Netware disk-usage through this script (which uses SNMP to query, by the way).

I have all of two days of stuff in it, so it's hard to look at. But the next step is writing reports. I have one version that looks to do good, but the real test will be running them three months from now when we have much denser data.

Backup fun, again

No crashes last night! I have hopes that the newer code is doing its job. Some of the incrementals are taking way too long, so I changed one of them to Full since its taking that long anyway. We'll see tonight how things go.

So far, so good!

Backup fun

The backup fun continues. Had a few more crashes last night, and this has now reached a big problem.

It turns out that there are a few others folk out there who are experiencing crashes on their clusters when TSAFS is loaded. Interesting, thought I. In my pre-incident checkout, I found that Novell has some newer TSA's than what I have loaded. Rather than burn an incident to get told, "update, you idiot," I updated. We'll see how things run tonight.

The TID doesn't have the list of issues, but here is an extract of some of the neat stuff this update does:
7. TSAFS was limited to 1024 bytes for the path names. This limitation has been
10. Fixed a problem with remote selective restores. An error would be returned:
"Could not connect to the TSA. Check the TSA is loaded correctly
SMDR : The required entry for protocol or service is not found.
Register appropriate protocol or service."
11.Fixed some pagefault processor abends.
13. Fixed a problem where ACCESS DENIED errors were returned backing up certain
directories. This happend because the zOpen used by SMS is requesting read access to
any object and the starting point for the backup is a location that the
specified user has visibility in but no read access.
16. fixes the invalid name parameters that caused abends. This was due to invalid
data structures that were being passed into tsafs.nlm. They were in the wrong
17. Fixed abends that would result if the server was in an out of memory condition.
This was caused by the tsafs.nlm read ahead cache pointer that was getting
messed up. Check the server to see if it's getting low on ram.
19. Fixed abends in tsafs doing restores

2. Fixes a problem with the archive bit not being cleared. This would result
in too much data being backed up.
3. Fixed a problem with the tsa600.nlm leaking memory on restores and copies
because RestoreFullPaths was not calling Free.
7. Fixed abend problems with the tsas amd smdr
10. Fixed problems with slow incremental backups. The full backups would be fast.
Only the incrementals would be slow.

1. IPX dependency of SMDR removed. [YAY!]
13. Smdr has been enhanced to resolve the server/cluster poolname to an ip
address thru the sys:etc\hosts file.
14. IP address mgmt support added.
17. SMDR has been modified so that it does not need to parse the SYS:ETC\HOSTS file on it own.
It uses the gethostbyname() function to do that. In addition, this method
expands its name resolution to use DNS as well.

Very exciting weekend

I wake up Saturday morning to find 25 text pages on my cell-phone. It seems that at 5:17am the one service on the cluster I have monitoring set up for crashed. No biggie, usually that means just migrating the service to a different node, kicking the web-server and migrating it back. Not today.


As in, everything was done. Red lights galore. I don't like that.

I also don't like that I can't get into the iLO cards for these servers and reboot at least one of 'em remotely. So I have to go in. Which I do.

I find that all three servers abended during some part of the backup process. Each abend had a TSA error, such as TSA SCAN, or TSA READ. Further investigation shows that the servers went down at 11:19pm, 1:28am, and 5:17am. Some services were completely knockec out at the 1:28am crash and the rest came to a halt at 5:17am. This is the first all-down event since 10/18/03.

I do some updating to see if I can patch it for Monday. I notice this morning when I was going through the abend.logs to see if I can find anything, I see that the ArcServe agent was loaded. This is interesting since it was Veritas BackupExec doing the backup at the time. I don't know if this was a case of unholy reverberations, or just plain luck. At any rate, NWAGENT is not loaded anymore, and I have hopes that the backups tonight won't cause EVIL.

Needless to say, I'll be watching backups like a hawk tonight.