January 2004 Archives

This morning's excitement has to do with the darned worm again. It seems we got enough hits that the virus logs on the exchange servers are, in a word, huge. Paring them down to a sane size is taking some time, and we've a backlog of mail to process at the same time. Needless to say, things are slow.

We had another case of, "Chief MuckityMuck JoeBob of Smurf-corp mailed me a cranky note saying that I need to stop sending him viruses. I've been on vacation all week, and I didn't do it. Fix it," happen. As the knowledgeable reader is already aware, this latest virus forges the From: header. It happens. And our own Grand High SubMuckityMuck leaned on us to Fix It, as his good name is on the line.

*sigh* User education. E-mail is easily forged. If I remember right, every single e-mail worm since Klez that has reached any sort of notoriety has forged the From: header. It only makes sense from a virus-writer's point of view. Most AV systems are set to notify the 'sender' if a virus is detected. That way, stuff that doesn't forge the From: header gets fixed pretty quick. How do you get around it? Forge the From:. SMTP isn't hard to code, it was written in 1982 afterall.

So we had another conversation with our own managers locally who are the ones being leaned on. He wanted us to tell him, "we can do nothing," so he can go on up the management tree with, "we can do nothing, our experts said so," legitimately. So we did. And in the process vented a lot about the state of user education these days.

The group that hacked us a week ago did another SQL scan yesterday. I don't think they found anything, as thats the same scan they used when they cracked the first server. And we haven't put anything new in yet. But I could be wrong.

In other news, I did find a couple of MSDE installations mission a particular MS patch, but that isn't one that is remotely exploitable if you don't already have a login. Still, it needs to get patched.

Today's project has come to a successful conclusion. The TSBUNT server now has a working video driver. It wasn't that the machine couldn't detect the NVidia driver correctly. Nor was it that the internals were all using VIA drivers while the clone machine was using Standard drivers. It was the fact that the card was actually a Diamond card. Oops. Once the Diamond drivers were in, it all worked juuuuust fine. Didn't find that out until I took the machine out of the rack to rip the case off. Some things you just have to do for yourself.

All is quiet on the Western front.

In the past two days I've found and fallen in love with an ISO-Linux distro called Phlak. It is security oriented, and has a number of really nice tools in it. Next time we get hacked, this will be a very nice thing to have handy when it comes time to figure out what the heck the barsterds did to the server.

Due to the wonders of Virtual PC, I've set up a WinXP station that I can monitor from boot to shutoff. This is being useful, as I'm monitoring the scanning attempts on it from non WWU networks. So far the most interesting thing to happen was about 30 minutes ago, the same folks who compromised that server last week made another scan of our network. Useful to know.

The MyDoom worm has made its presence felt here already. Last night we started blocking .zip files as a temporary measure until NAI could get a datfile out. We still got hit in many places, partially due to mail that got through before the ban, and partly due to folk who've set up their outlook to fetch their private mail from work. So far, things seem to be holding steady. The mailserver is keeping up with load. Our helpdesk doesn't seem too overworked. Even if our mail admin is Cran-key with a capital CRAN. People keep calling him. Not a good idea.

The MRTG project may get some life. It depends. All I need is the RO password, not the Exec password. I think. Once that's done, it'll be 2 days (max) before I know the OIDs I need and can hand-code up a config file. 2 weeks for me to code up a perl-script to generate that config file dynamically. They just need to give me the info.

Woohoo! Pet Theory #1 was right! I changed the load order of the modules so the ones responsible for NetStorage load last and ka-boom! Working. I love it when a plan comes together. There may be a TID on this one once Novell gets back to me.

Pet Theory #1: The load order of modules may be significant. Ensure that XSVR gets loaded after MODHDIRS.

Turns out that NetStorage hasn't been working since the 9th. And that is due to the config that allows this very web-page to exist! Modhdirs.nlm seems to make the Tomcat servlet engine not work. Most strange. Novell will be testing this out on Monday to see if it is config thing, or a modhdirs thing.

NetStorage seems to have forgotten how to talk to its servlet engine. This has prompted me to call Novell to Fix It. Strangely, it seems to have hit the entire cluster.

Got it. There are a couple of methods of doing this.

Easy way: http://server/users/username
Hard way: http://server/username

Both are doable, but the 'hard' way involves pairing mod_rewrite and mod_proxy, where the easy way just substitutes 'users/' for the '~'. Guess which I prefer?

WRONG! I wanna leave it alone. But the management hath spoken.

Next project: Get the ~ out of the URL.

Did you know that CD-image downloads take a while? Not for the faint of heart.

Now I'm attempting, once again, to get an install of SUSE-linux running. Novell bought these folk a while back, and it is about time I look into getting my hands dirty in it. Also, it's a good chance to build up a forensic toolkit.

Set up the process to keep the web-logs of the new myweb servers from taking over the world. Nice to have. That way when a certain heiress video lands on our servers by a student, we can go through the logs and figure out who to have a chat with.

There is no evil conspiracy to track every web hit and charge students for their bandwidth usage.

A fairly not-so-bad day. No emergencies. No real surprises. Spent some time trying to figure out what our backup rotation is like, as I'll eventually get to figure out how to implement that in Veritas BackupExec. Good thing I spent that near two years doing exactly that at OldJob. Though, our rotation there wasn't as media-intensive here.

They struck again yesterday. Another server was exploited with an unpatched buffer-overflow and they managed to get very deep into the system. Definate evidence of a password-hash grabber, and a likely key-stroke logger as well. With that info, we ended up changing our admin passwords on all Windows machines. A large undertaking.

In other news, Microsoft has released v1.2 of their Baseline Security Analyzer.

Yesterday was MLK-day, and as a state employee, I got it off. Yea.

This morning has been sedate. The admin who was off Friday is back, and has been briefed on the exciting events of that day. We also had a talk this morning with ADIC about tape libraries. Specifically, the model I installed at OldJob. HP/Compaq is the other serious contender I know about.

Busy day. A test machine got compromised by a bug. Set up a FTP server and attempted to become a Warez site. I miss my old restrictive firewall environment. In this environment, everything is visible to the world, darned near. Hard to set up brand new machines that way, eh?


Trying to figure out how to make apache give a 'nice' url for NetStorage that satisfies Windows WebDAV hooks.

Normal URL: https://server.serverdom.domain.dom/oneNet/NetStorage
Nice URL: https://myfiles.domain.com/

The problem is, that the "oneNet" bit is hidden in a <Location> section, and I can't figure out how to make it all work right. You can't use a redirect because WebDAV pukes on it. Mod ReWrite might be able to to it, but I haven't figured out how to do an invisible rewrite. Is it worth the effort to start hitting up mod_proxy, or do we just write up detailed instructions for users and live with the people with a 7-character memory buffer?

Once more into the breach. The specter of tape backup has risen again, and we are contemplating what to replace the current system with. Several vendors are possibilities. One handicap is that the budget, as it sits right now, is not enough to get what we really need. But... the disaster recovery project, which I didn't even know we had, may be able to pick up the remainder and allow us to get what we need. That would be nice. At OldJob I had the pleasure of installing a rebranded ADIC Scalar 100 so this is familiar ground for me. It looks like we're setting up a bunch of conference calls to speak with vendors.

And for my next trick...

Hi. I have a brain. No really. You know the NetWare portal? That wonderful thing you can get to by going to https://server:8009/ ? And how you get an SSL error (usually) if you go there with https://server:8009/ instead of https://ip-of-server:8009/ ? Well, if you change your load line:

load httpstk.nlm /SSL /keyfile:"SSL CertificateIP"


load httpstk.nlm /SSL /keyfile:"SSL CertificateDNS"

That goes away. Nifty, huh? Only took me, oh, FOUR YEARS to figure that out. Aie.

Just a quick reference for tape-speeds. We're looking for our next backup technology.

Raw Capacity200100160110
File access time49ms54ms70ms70ms

Turns out there was a dependancy that we didn't know about that was mucking up the works. Part of the printing sub-system was (surprise!) residing on a volume we weren't expecting. So once we migrated that volume to the server running print-services, things started working great. Now we're trying to get rid of the dependancy.

The reboots last night did not go well it seems. Several things are broken, among them student printing. Stoopid P-counter. And Netware, actually.

Hahahahhaaa! Og munch obits. Ook.

Good vintage too. These seven were hanging around since August!

The NICI update we need to get eDirectory 8.7 is going in on a bunch of servers tonight. Now we get to convince the rest of the admins that they need to get this on their own servers before we can proceed.

Hurrah for code re-use! It turns out we do keep student web-server logs for a time. I needed a way to keep that log directory from getting too huge, so I checked the archives. It seems I had an early copy of a perl-script I used at OldJob to ensure files of a certain age or newer were in a directory tree. A few simple modifications, and it'll serve my purpose just peachy. Now to figure out the best way to invoke it every day, and I'll be set. Whee!

I learned something today. Excessive DOS accesses in NetWare can cause a cluster poison-pill! I'm guessing this is a manifestation of the Realtime 'bug'. If so, this is an aspect of my favorite rant against Netware.

Real Mode Operations
NetWare is one of the few operating systems out there that still insists on putting the CPU into Real Mode for operations. There are a few things that really kick it off:
  • Extended SCSI operations, such as those that command an auto-loader robot arm, or on some baddly designed tape-drives, the eject/insert mechanism. In this case, the server drops into Real Mode, gives the SCSI command, and waits in real mode until the command finishes. For things like 'Go To Slot 5' this can take whole seconds to complete. For things like "Eject tape in drive & put in slot 5" it can take up to five MINUTES.
  • DOS-partition access You can get around this one by loading DOSFAT.NSS, but almost noone does that. Access to the DOS partition is done in Real Mode because DOS-FAT can't understand multiple accesses, so NetWare has to make sure that access to it are done sequentially.
The problem with Real Mode access is that when it goes into real mode, all other I/O operations on the server STOP UNTIL THE OPERATION IS COMPLETED. That's everything. Disk I/O (how I ran into it the first time). Network I/O. You name it. All halted until the *#$!~ tape can finish rewinding and go back to slot 2. Highly annoying.

Yes, I know there is a TID on this, but we apparently haven't used it here yet.

The problem is that on a cluster, real-mode can make you miss your heartbeat. And that makes you go split-brain. Not Good. I managed to crash two of the three servers this way! At essentially the same time! Good thing that third server was around, I tell yah.

Got NICI 2.6 installed to a pair of servers, but it seemed to have caused silliness in the timesync. The server rebooted, and suddenly four servers are screaming that they can't find their timesync servers. *sigh. No amount of TIMESYNC RESET fixed it. The server being rebooted was an SLP server, though. So these servers may have gotten horribly confused as to where to find their time. An SLP RESET on each of 'em fixed it right up. Silly things. They had another DA RIGHT OVER THERE that was identically configured, so they should have failed over, rather than obsess over the missing data. Arg.

Given the go-ahead to plot the upgrade to eDirectory 8.7. Due to how our environment is configured, the first steps of this won't involve after-hours work! I approve. Lots.

...aaaand it seems to work from the outside, too. Nifty. All is good.

The 'myweb' sites are now set up with dns entries. Lo the fun begins. Lets see how long it takes folks to figure this one out on thier own before we announce it.

Modified the SSL setup for the myfiles system to reduce (but not eliminate) the number of SSL prompts people receive. Apache is Nifty. Still not king.

The exchange thing yesterday...

After confusing Microsoft support, and futzing with a memory dump utility, it was eventually traced to a problem with our virus scanner. At which point we learned that we don't have phone support for our virus products, probably having something to do with the fact that this is only the second time in four years we've had a need to call 'em. There is a fix out there, and we may have found the file ourselves. But the AV company hasn't gotten back to us yet. The problem was traced, we think, to a bad e-mail somewhere. And the problem hasn't reocurred after putting in the fix we found. We cross fingers collectively.