July 2005 Archives

iPrint musings

The big question is how do you allow people to install iPrinters without hosting the web-page on the iPrint server itself. We're working on that. More testing needs to be done, but this is the latest version of how we're getting it to work.

The Maptool that comes with iPrint creates html pages. They have a background image used as a map, and printers are placed on that map to create a sense of floor-plan. No biggie. The URL that actually performs the install is pretty simple, if a bit big:


Clearly, if the "isinstf.htm" page isn't on the same server that's hosting the document, it'll fail. No worries. Grab a copy of it, or just reference the iPrinter server directly:


That'll work to a point. But the "isinstf.htm" file itself has relative pathing in it that'll trip you up. Probably more so in a cluster. As it turns out, that file, located in SYS:Apache2/htdocs/ippdocs/isinstf.htm, also contains relative pathing that'll screw things up. So prefix "https://iprintserver.yada.yada/" to all of the links, and it seems to work.

Quiet time

Most of our major projects are on hold thanks to vacations, and getting a lot done already. Some of the stuff we can't start on until 8/22, the first Monday after the last Summer Session lets out. Others require folk who are out of state at the moment.

In other news:
  • BI-Query is making us crazier than usual
  • Blackboard will probably get a rebuild during break
  • Our SAN will need replacing a year earlier than we had planned, thanks to greater than expected usage. 06-07 timeframe
  • ResTek really really likes iPrint
  • The libC problem is still a problem, even with nwlib6d. Yep, that incident is still open.
So we're just performing station-keeping for the time being. And whacking BI-Query nearly daily. But other then that, quiet.

In local news

| 1 Comment
A tunnel across the border up near the Aldergrove/Lynden border crossing has a local connection. It turns out that a coworker of mine toured the US house in question when he was house-hunting several years ago.

Cooling followup

Now that it has been over 24 hours since I jiggered things around the blade-rack, things have improved markedly. The exhaust coming off the back of the blade-rack is very noticably cooler than it was before. This makes me a lot happier. We might, might be able to stuff one or two more blade chassis into that rack now.

All I did was move a couple of vent-tiles to right in front of the rack, and replace the front door with a fully perforated door. The previous one was only about 20% perforated. With the old door I noticed from back to front circulation over the top of the existing chassis in the rack. Before the exhaust was dragon-like. Now it is merely rather warm.

Rack cooling

I just spent some time trying to improve the air-flow around and through our blade-rack. In the process I learned that our vent-tile lifter has wandered away somewhere along the line. I think I managed to improve things a touch.

Our datacenter's air-flow is a little hap-hazard, but our cooling capacity is far enough ahead of heat generation that we don't have hot-spots yet. The blade-rack is by far the #1 heat producer in the room. The SAN disk-packs do not pump out the heat you'd think they would with the power they suck, all that energy gets converted to actual work spinning disks rather than being resisted off in the process of number-crunching. The rise in temp from front to back for the blade rack has to be pushing 25 degrees F.

I moved a couple of vent tiles to right in front of the blade-rack, and that helped some. Then some air-flow testing (thanks to a couple of hairs around 4" in length, they make good free flow testers) inside the rack itself showed that we're getting a bit of back-to-front circulation over the top of the topmost blade chassis. That's not good. Opening the rack door caused the circulation to maintain front-to-back up there, so that told me the front door needed attention. We found another fully perforated front door and replaced it. The circulation went a bit neutral, but at least it wasn't back-to-front like before.

All in all, the work reduced the temp of the back of the rack to 'only' about +15 to +20 degrees. I don't have a thermometer I can use for this, so I'm not certain. But it is a bit cooler back there now.



Looks like there has been a real uptick in scanning for VNC servers. I wonder what prompted that?

New phish

I got a new kind of eBay phish today. Apparently I can become a 'titanium power seller', even though I don't even have an eBay account! I haven't looked at it in rendered HTML form, but the mail includes some items that are interesting.
With this mark, you can be confident that you are transacting
with an experienced eBay user who has proven that they're committed
to customer satisfaction.
And so it goes. Along with this gem.
Joining this grup is a simple two- step process.
Hardly and eBay account. Considering that the image they use is located somewhere else:
Anyway, this is a deviation from the standard, 'please verify your information or we'll cut you off,' phish.

eBay Help - which goes to http://mail.yahoo.com/config/login?/xxxxxxxxxxxxxxxx instead. Riiiiight.

NetWare and the 4GB limit

There is a TID on this one. In short, NW65 can address up to 64GB of memory, and NW6 is limited to 4GB. NW65 can't do as much as you'd think with that access.

Think of it like EMS back in the DOS days. In order to address memory over the limit, the requested info has to be paged below the limit before it can be handled. Just like a page-file, but with faster access. This can be done invisibly to the user, but it is an expensive operation all things considered.

NW65 considers memory above 4GB as virtual memory. NSS can use this memory as well, though due to the speed issue it only caches files larger than 128K. Protected memory spaces can use this memory since they can access VM. If you have whonking huge applications that need lots of RAM, you'd better run them in a protected memory space to gain the most benefit.

When the 64-bit version gets released, the memory limits go into the stratosphere again.

But since the reason we need to add RAM is to make the NSS cache larger, we're just FINE with this. We're going to end up with 5gb in the student-side cluster nodes due to how things work. This'll give us p-l-e-n-t-y of room to grow in the next year.

New MBSA released

MBSA 2.0 came out. I hadn't heard. Looks like mostly improved UI.

More migrations

| 1 Comment
I just finished migrating the first node of the cluster over to a blade-server. That went surprisingly well! I'm very happy.

The first step was the same as it was with the NDS servers earlier this year. Install the new server into its own tree, take a config.txt from the old server for reference, run the Migration Wizard, fiddle things to get the identity settled down. With the NDS servers this was pretty much it, since all they ran was NDS (iManager was on the one server that didn't get migrated) and didn't need any additional work.

The second step was installing the apps. NetStorage needed to be added in and configured. And that required all the supporting Apache and Tomcat bits. Thrown in OpenSSH and FTP, with a bit of iManager, and a dash of apps that the config.txt document said were installed on the old machine, and I had it all in. NetStorage got configured right.

The third step was to get all the cluster scripts we use into their correct places, and all the supporting config files where they needed to go. Since we don't put NetStorage into httpd.conf, rather into its own file, that had to be copied. As did the config file for the myweb service. Then the ncf-files that kick off both web-servers (myfiles and myweb), as well as the trio of FTP/SFTP services. That went pretty well.

Then came the Very Scary process of allowing the new server to see the cluster volumes on the SAN. That took a bit of work. The Fibre switches are zoned by port rather than WWN, which made that step easier. The EVA then needed to be told to permit the new WWN to see the Novell cluster volumes. That took a bit of poking, but in the end it was pretty simple. I hit the commit button on the EVA interface, scanned for new devices, and then loaded NCS. It saw volumes! So I rebooted, and it entered the cluster just peachy.

From begining to end, the whole process took maybe five hours. And that includes the initial imaging of the blade server, and the install of NetWare to the blade. I'm pumped.

AND Novell released nwlib6d today. They promised me that it fixes the nxcreatepathcontext problem I've been having for just over a year now with the myweb and sftp services. I've heard that one before. It's in testing on the student side of the cluster. Early indications are good, but we'll have to see how it lasts over a couple of weeks.

Cleaning servers

We had another round of hacked servers in the past few weeks. Strangely enough, the rootkit used was very similar to the one used around the time of this cryptic post. Since I cleaned that round the last time, I was able to figure out this lated round pretty quickly.

What hit us was a variant of the Hacker Defender toolkit, but localized into French (I think). I was able to manually remove the puppy before other folks around here were able to locate a remover program. Just because I like sharing, this is what I discovered.

The toolkit creates three services:
  • dcrssdrv, hidden, launches %windowsdir%\system32\spool\tracking\in\dcrssdrv.sys
  • mmsm, visible, launches %windowsdir%\system32\spool\tracking\in\mmsm.exe
    • Description: Optimize transfert between CPU&RAM
    • Display Name: Memory Manager
  • tcp-ip_port_analyzer, hidden, launches %windowsdir\system32\spool\tracking\in\dcrs.exe dcrs.ini
    • Description: Manage all Netbios Connections
    • Display Name: Tcp-ip Netbios Monitoring
The MMSM service launches a Serv-U FTP daemon. The control file for that is mmsm.sys in the same directory as the executable. Investigating this file will tell you where they were planning on stashing files. In my case, they attempted to dump them into C:\Recycler\tracking, but no such directory was created. The really funny bit is that the typo in the mmsm service, and the odd caps in the Tcp-ip_port_analyzer service were identical to that hack back in November. This is the same kit, though clearly used with a different entry-point.

They also created another folder in "C:\System Volume Information\tracking\", which you normally can't get at from explorer. You have to set permissions on the System Volume Information folder in order to get into it. In that folder was various recon and spreading tools, as well as pwdump4 for dumping the local SAM database. They didn't get to anything with any real data in it, so our passwords are safe. Plus, our local admin passwords are longer than 16 characters, so rainbow-tables won't work.

In order to get things back, I set the SYSTEM user to Deny Access for the following directories:
  • %windowsdir%\system32\spool\tracking\
  • c:\System Volume Information\tracking\
This has the effect of blocking access to these directories for the SYSTEM account. Since these were created by the kit, I had no doubts that setting things that way would cause problems with the OS. Once those were set, I rebooted. As I expected, I got "service unable to start" errors on login, and the Event Log showed the services that couldn't start due to Access Denied errors.

As a result of this, the rootkit wasn't able to initialize and the hidden registry keys were visible again.
  • HKLM\System\CurrentControlSet\Services\dcrssdrv
  • HKLM\System\CurrentControlSet\Services\mmsm
  • HKLM\System\CurrentControlSet\Services\tcp-ip_port_analyzer
  • HKLM\System\ControlSet002\Services\dcrssdrv
  • HKLM\System\ControlSet002\Services\mmsm
  • HKLM\System\ControlSet002\Services\tcp-ip_port_analyzer
  • HKLM\System\CurrentControlSet\Control\SafeBoot\Minimal\Tcp-ip_port_analyzer
  • HKLM\System\CurrentControlSet\Control\SafeBoot\Network\Tcp-ip_port_analyzer
  • HKLM\System\ControlSet002\Control\SafeBoot\Minimal\Tcp-ip_port_analyzer
  • HKLM\System\ControlSet002\Control\SafeBoot\Network\Tcp-ip_port_analyzer
The last four entries tell the Registry that the bad process is to be launched in safe-mode. Thus, safe-mode is not a good spot to clean things. Once those services are manually removed, it should all be good and you can delete the directories. You can't delete 'em before, since the kit prevents that sort of white-hat activity; but setting the permissions for System-deny-access gets around that little tweak.

None of the above stuff is googleable that I've found, but I've seen it before. So now when other people get hacked by this crew, they'll have a resource.


I'm back!

Things did not melt down in my absence. No swarms of e-mail to thumb through. No voice-mail waiting for me to fix problems two weeks old.

But things continue to change. The AD controllers, all but one, have been migrated to blades. From the spew of boxes in the computer room, it looks like the new Titan machine is here. There is a swank new PowerEdge 2850 sitting on a table for a department's off-the-shelf something-or-other; it'll get racked in a week or so when we make space for it. The Blackboard server needs a nuke-from-orbit rebuild, which will probably get done in the dead-weeks between the end of summer session and begining of fall session.

The next big project is migrating the Faculty/Staff side of the cluster to blades. Shouldn't take all that much time, but running the fibre will be the big part.