November 2004 Archives

I found a beta release of SP3 for NW6.5 and started scrolling through the documentation. I found a couple of interesting features.

New OS features

• Formatting local FAT volumes from console
• Scrolling the System Console screen with a SERVER switch (-cs)
• Suppressing unwanted screens from console screen-cycle (alt-esc, not ctrl-esc)
  

BASH for Netware

Apparently it's included in the release! This has been on the forge site for some time.

More IP6 features

Still not fully featured IP6 quite yet, but it shows the folks in Provo are thinking about it.

Enhanced management of core-dumps from the Remote Manager

SDebug has been around for a while, but they're improving how easy it is to deal with it. Though I'm not sure, it also looks like they're improving the 'who can do what' features in NRM. Nifty.

Some of this stuff looks like back-ports from the work to get OES 1.0 off the ground. Nice to see. There isn't a beta for NW6.0 out that I can find quite yet so no idea if this'll be in there. It should imagine some of it would be since the codebase between NW6.0 and NW6.5 is pretty close.

http://homestarrunner.com/sbemail118.html

StrongBad. In case you hadn't seen it recently, he recently covered the topic of spyware/adware/virus-mails. Downright funny, it is.

http://www.novell.com/coolsolutions/nnlsmag/features/a_pitching_the_boss_nls.html

The grand prize ($1000) was awarded to Duane Fish, the creator and master of the www.iwantnetware.com website. You should take a look at his site: it's packed with great links and information, and we've seen several new additions over the past weeks.

For those of you who get the dubious task of attempting to explain to Upper Management why this Netware/Novell thing doesn't need to be ripped out and replaced by Microsoft stuff, IWantNetware is a good site to find resources. I've linked to it for a few weeks now. We're lucky here that our Management really likes what they're getting out of Novell; but then we're education, one of the last bastions of Novell presence.

Stuff inbound to the Exchange cluster had some delays recently. Saturday night, one of the two front-end servers that accept traffic from the internet got plugged with logfiles. All in-bound e-mail then arrived on the server and sat there. As in, not delivered.

We discovered it yesterday afternoon. There wasn't enough space on the volume to commit the transation logs, so we had to take steps to get things back into working. Around 10:30pm last night, the second node came back online and the backlog started hitting queue.

Users these days assume that e-mail should transfer more or less instantly. Or failing that, within a very few minutes. SMTP wasn't designed for that. It's a best-effort thing. And in this case, mail that arrived early Saturday finally got here late Monday. It happens. Yes, the US Postal Service could have gotten some of this here earlier, but 80% of it was spam anyway.

I figured I'd post what kinds of things are in my 'rip it apart and see what's going on' toolkit. For Windows. The Unix/Linux stuff is Someone Elses Problem so I don't have much experience with it. Likewise, NetWare doesn't need it.

Foundstone tools:
Fport: One of several tools that performs open-port / PID identification. I've had trouble with it on Win2003, but for Win2K and under it works just peachy.
BinText: Finds human-readable strings in files. Useful for scanning weird EXE files to see what's hidden in 'em.
SQLScan: Scans for MS-SQL and MSDE instances suseptable to things like Slammer. Yes, Slammer is still around. And in our environment, we're vulnerable to the thing if we leave vulnerable instances up.

SysInternals Tools:
FileMon: Monitors filesystem activity in real time. It associates the access to a PID and program, so I've found it useful in ferreting out hidden processes, and where on a system files are being secreted. It also allows filtering so you can exclude (spammy) expected traffic and see what else is being done. Some rootkits may specifically kill this executable if they detect it running.
PsTools: A toolkit itself, it contains a variety of very useful things. Including: psservice, list and control services remotely from command-line; psinfo, dump information about local and remote machines; pslist, dump info about running processes (looks a lot like "ps" output); pskill, kills a process. A very NICE toolkit.
RegMon: Like FileMon, this monitors registry accesses. Like filemon, it can be filtered to exclude certain (spammy) expected traffic and help isolate the traffic you want to look at.
TCPView: This tracks TCP connections and associates them to PIDs. Very nice for figuring out what's running on a system that shouldn't be. As with FileMon, some rootkits know about this utility and will shut it down. It comes with a command-line version too (tcpvcon).
Autoruns: This handy thing will tell you what programs are set to auto-run on your system. Including obscure registry places, as well as services. Includes file-paths too, so incase that spyware/perp dropped things in a weird spot it'll stick out.
ProcessExplorer: Another very hand thing, this was designed to ferret out hidden processes. It uses a number of things to do this, including TCPView, FileMon, and RegMon type accesses. I haven't used it much since I just discovered it recently.

Sniffing tools:
Ethereal: In my opinion, no toolkit should be without this tool. Darnit. It's a sniffer. It's great.
WinPCap: Windows version of the PacketCapture library for Unix, needed for Ethereal to run.

Microsoft tools:
Yes, Microsoft has some nifty things to play around with when trying to rip apart a system. They're hidden most of the time, but they DO exist!
Debugging tools for Windows: I was pointed to this one on a service call for an Exchange problem, but it turned out to contain some very nice features. It has several tools that are replicated in the PsTools archive, such as a process lister (tlist) and a process killer (kill). A very complete kit I haven't had the chance to fully delve into.
Port Reporter: Fairly new, this particular tool tracks and logs all TCP connections made to the server. For XP and Win2003 machines the log contains, Port -> PID mapping, modules loaded as part of the process, and whether the process is a service or not. As you can see, this has the potential to be very useful as well.

Other tools:
nmap: The classic port-mapper. Now with new application-scanning ability! This new bit is quite nice. Say an attacker loads a ftp server on a well-known but seldom really used port like the Echo port. This new app-scan ability will tell you that the port there on tcp/7 isn't Echo it's "Serv-U FTP"! Handy.
OpenPorts: Yet another open-port <-> PID mapper. You'd think there was a problem identifying running processes on Windows or something. I've had good luck with this in the past, but Win2003 seems to make it a bit weird. Still plenty of Win2000 and WinNT around to use it on, though.

Apparently you too can invest in my blog. Sort of like Fantasy Football, but without all the fun stats. Weird stuff.

Turning SNMP off on all the Printer Agents helped a little bit, but it hasn't stopped the problem. Just had an abend to that affect right now, so I took a core-dump. It is my suspicion that printer-status does play a roll in whatever it is, but I'm lacking solid proof.

It has been almost a year since I left OldJob. In that time I've gotten a pretty good handle on how things work here at NewJob. While the office politics that I need for day to day getting along is done, I've still to learn even the majority of the politics related to deciding what we will be doing in 6+ months.

My position at OldJob was both more and less responsible than what I have here. There, I was working for a unit that losely compares with ATUS/ADMCS, though with broader powers of enforcement. There, there were two people under 'Jerry'-equivalent, and they headed up ATUS-equivalent and ADMCS-equivalent. In addition, Jerry-equivalent also had the CAS-equivalent and CBE-equivalent tech-heads reporting to him directly. As the astute reader will notice, this gives the Jerry-equivalent (dare I say, CIO?) much broader powers than Jerry actually has here. Since my office was down the hall from Jerry-equivalent, and my boss right next door to Jerry-equivalent, we saw a lot of eachother.

When it came to strategic planning of the future, I was one of the first people management came to for technical advice (ATUS/ADMCS stuff, the CBE/CAS stuff was largely informal for all but the biggest of projects). I spent four years proving the case for a Storage Area Network, and got budget approval in time to deploy it. When budget-season rolled around, I was one of the people expected to submit budget requests into the process for aggregation into the whole Information Services request. Since I was working in the 'core services' area, a lot of the stuff we worked on were all-enterprise projects. While I was involved at most steps, I wasn't a true decision maker.

I was in the process of training up a pair of minions. These people would (and eventually did) ostensibly backfill me in the (all too soon) case I fell off the face of the planet. These weren't people I was Supervising, I was more of a team lead. I wasn't involved in the Job Review process, though I suspect I would have if I had stuck around long enough. It was quite clear that Management had me marked up real good in their succession planning (boomers, gotta love 'em).

Here things are different. Now, I'm one of the minions; though technically the three of us are supposed to be interchangable on matters of technology. The department I work for (Technical Services) is a small branch next to the main trunks of ADMCS and ATUS. The Division I work for had less control over the enterprise than I had at OldJob. The silos of tech out there big enough to support their own IT departments still have them, and getting them centralized will involve arm-wrestling of epic proportions at the Dean-level.

Then there is the money. The budget process is an every-two-years thing instead of every year, and I've yet to see it in action. The department I work for has no effective budget beyond salary/office-supplies, so all of the stuff we spend money on has to come from someone else (usually ADMCS and/or ATUS). I've yet to penetrate how spending money is accomplished around here, something that took only three years to pick up at OldJob.

At OldJob there was a mindset on the helpdesk (and in certain other IT areas) that if it is broken, ask me as I might have a clue as to why it broke or where to look next. This meant I got a lot of oddball questions about products I didn't know a lot about, but sometimes the issues brought forward were truely interesting. That hasn't taken hold here generally, but when it comes to things Netware it seems I'm the new goto person. I'm not sad at this at all, as I've come to learn that my Netware knowledge is right up there with some of the techs AT Novell (first-line support at least in most areas), which does save us service-requests. I hold that CNE for a reason darnit.

On the technology front, I'm doing a lot less Windows stuff these days. At OldJob I was the primary Windows admin for our area, and that included the gamut between Oracle database servers, to IIS webservers, to application-servers of many kinds. Here, I do some Exchange, a little SQL-server and generic troubleshooting when the other admins aren't around.

And most importantly (hehhe) no backups. OldJob, I was He Who Manages The Backups. Here, I just back up the guy who does it. Hee. I'm not sad about this at all since tape-backups are the bane of my existance. Tapes fail. Fact of life. Users expect to be able to undo Oops! mistakes. We expect to be able to resurect-from-the-dead servers that smoke. The attitude at OldJob was that backups are mission critical and any failure in the backup process must be troubleshoot as data is at risk, and most importantly they funded backups like the critical systems they were. Here, that attitude is just now begining to take hold, and that is something I'm having trouble adjusting to. Rant for another day.

Probably most telling is that the two admins who replaced me at OldJob mentioned to me when I visited recently that they had real trouble keeping up with my duties in my absence. Two people had trouble. Here I actually have time to thumb-twiddle once in a while, which is a luxury I didn't have at OldJob. I realize that workloads like that are years in the making, as the maintenance of systems is assumed piecemeal and in aggregation over years gets to be quite sizable. So, ask me in four years what my workload is like, and I'm sure I'll tell you "full".

I've come to suspect that running DSRepairs on the database on our replica servers sets them up for memory-fragmentation and cache-allocator problems later. Not enough data for a real theory yet, but it is something I'm... concerned about. We're at eDir 8.7.3.2 at the moment.

Yep. Exciting alright. I've had opportunity to stock my forensics toolkit with a few more tools out there, and I learned that filemon doesn't like Windows 2003 for some reason (or at least the flavor of it we run; it crashed repeatedly on multiple servers). Yep.

Circumstances have forced us to completely turn off one of the two Cluster services. One of the cluster nodes will not take the Service, and the other node it is associated to has gone inoperable. We're rebuilding Node3 and talking with Microsoft on how to get Node2 to take the service. This could be an outage up to three hours, perhaps longer.

Today I learned of a splendid little tool called "w32tm". Which is sort of similar to the ntpq command for NTP installations. It allows you to query the sync-status of time in domains and on various computers. Very nice. Especially when I was trying to figure out why the AD structure was at +19s to the eDirectory time.

Now both AD and eDir get time from the same ntp-source. I'm a happy time-geek.

We've been fighting an NDPS problem since 9/18. This one has gone so far that our incident has involved developers at Novell. This is my second such incident so far this year, the first one involved mod_hdir and mod_rdir for Apache.

This is a real doozy. Somewhere along the line, we get a CPU-hog abend in "NDPSGW LPR Check Process". According to the engineer at Novell, this is because a linked-list is going circular somewhere in the NDPSGW memory space. What is causing the circularization is the $10,000 question. This is especially interesting since we're not the only one with the problem and beta support packs are begining to show up.

So far I've submitted nine coredumps for analysis on three different NDPSGW builds.

I went and saw The Incredibles last night. Good flick! As a point of history, Toy Story came out right at the same time I was in my Computer Graphics class in College. Our teacher took us out to the local theater and we watched it as a class then dissected it the next day. How neat is that?

Skills that came in handy, I tell you. Though not for anything Pixar did. You see, we saw this movie in a Regal Cinema venue. Not surprising since every single first run theater in town is a Regal Cinema. Ahem. In the place where The Twenty usually was, they ran lots (oh god, lots) of ads for a piece called "Bigg's Adventure". The ads were coming so thick, that when the last one ran before the previews even some kids were complaining about it.

But the animation quality of this flick is abysmal. When compared with what Pixar put up after the previews its like night and day. Twice I caught hidden-surface-removal errors when a character opened his mouth and I saw trees behind him. A couple of other times I saw physics or frame-of-reference errors where the actor wasn't moving quite in sync with the background. Movements were far from natural, far enough that they even looked like animatronic pupets rendered by computer. Shading, always a rough patch in CG, was haphazard at best; in fact it looked like they skipped lighting in favor of making *everything* bright. Where Pixar has skin react to the environment by deforming, this film ignored that completely. I've seen better stuff come out of a Playstation II.

It has been quiet lately. Not a lot is going on that I'm involved with. The Exchange migration is almost complete, all active users are moved but the disabled accounts are still on the old system. On the cluster front, we're waiting on upgrade licenses before we proceed there. There is a new ticketing system going in that we have real reservations regarding reliability and our normal dislike of being called by panicing users at 7:23pm.

We had a DoS attack this morning. I'm not sure yet if we were receiving, or if we were merely supplying the zombie. But we killed the puppy. The odd thing was that at least some of the traffic was on UDP/22.