January 2007 Archives

Filefinder feedback

| 1 Comment
Two weeks ago I blogged about Novell getting rid of FileFinder in favor of putting everything on download.novell.com. As you can expect, there has been some pushback. This morning I found a Wiki entry about that, over on Cool Solutions. Don't care for the differences? Want to see what other people are griping about? It's a good read.

That's odd

HP is advertising their blade systems. I saw an ad for one this weekend on SciFi, and now a banner ad on the front page of economist.com.

This is a system that has zero presence in the home, entirely in datacenters. This is clearly an effort to influence decision makers outside of industry rags like ComputerWorld or NetworkWorld. Yeah, a lot of fellow IT workers I know were sad when Farscape died, so I guess we have a lot of SciFi buffs around.


An incompatibility

I've been working on Zen Asset Management 7.5 the past few days. In the process I discovered a rather significant incompatibility with the client. Well, significant for me since it'll make client testing harder.

When run on a Windows XP virtual machine running on Xen 3.0.3 that comes with openSUSE 10.2, it causes the clock in the VM to slow w-a-y down. On the order of 1 tick per 30 ticks on the host machine slow. This makes it unusable in a rather significant way.

It also is completely unfixable! Running Windows in a full VM in Xen on openSUSE 10.2 is an unsupported operation. I have the CPU for it, and it runs pretty well in every other way. But something the inventory process does causes some Xen emulation to go 'poink' in a bad way. It is so bad that even after the VM is powered off and the BIOS is putting the virtual machine to rest, it STILL takes a very long time for the VM to unload. No idea where to report this one.

In general, the product looks interesting. Getting it rolled out everywhere will take a lot of work. Plus, for some reason it isn't accepting my license code. But that's something that can be fixed with a call in to Novell.
A conversation I read recently reminded me of this one. One of the things that caused me to grind teeth at my old job was the use of the word 'dsrepair' as a catch-all for anything run inside of dsrepair. This extended to some TIDs, which made the problem more wide-spread than my little corner of NetWare-land. The problem only started abating when it became possible to do some of the dsrepair tasks without actually loading DSREPAIR.NLM.

I'd see documentation like this one (generated internally, though my treacherous memory is telling me that a very early version of the NDS Health Check TID also had the bad phrasing):
  1. Rconsole to the console
  2. Run dsrepair until there are no errors
  3. In NWAdmin...
What am I doing in DSREPAIR? Unattended health check? Timesync-check and sych-status check? Local trustee check? Check external references? Full database repair? WHAT? DSREPAIR does a lot of things. It is not VREPAIR, where it either works or it doesn't.

The phrase, "run dsrepair until there are no errors," showed up a lot for a few years there. The conversation I read showed a clear case of an Old Tyme NetWare Admin maintaining the sense of dark mystery around dsrepair, which caused their young apprentice to go out into the world with an incomplete understanding of what this tool really does. This is a verbal short-cut that I'm glad has died out.


The decision to tell the appliances to delete Spam was made yesterday. Anything coming in flagged as Spam, not Suspect Spam, will be dropped. This is 99% of the stuff flagged as spam, as 'suspect' is a really small category. This does reduce the load on the Exchange front-end servers as they have to do much less spam checking and handle a lot fewer messages. Though, as I'll show below, only a little less data.

And now, fun stats for Yesterday!

Total messages processed: 193,242
Percentage flagged as Spam: 49%
Percentage flagged as Suspect Spam: less than 1%
Virus mails: 731 messages
Top virus: Trojan.Peacomm (45% of viruses)
Top non-WWU inbound mailer:
Top spam sender: service@watermarkcu.org, 4% of spam (go phish!)

The mail flow goes something like this:

[inbound] -> BigIP -> Appliance -> BigIP -> Exchange FrontEnd -> Exchange

The BigIP is used to load-balance between the exchange front-ends for SMTP traffic. As it flows through the BigIP, I get stats on data volume over those ports

Mail volume to Appliances: 1.7G
Mail volume to Exchange: 1.4G

So data volume isn't greatly affected by dropping 49% of incoming mail. What is affected is the number of messages being processed. The front-end servers weren't terribly loaded as it was, this just means that Outlook Web Access is more responsive than it was.

DST change, pointless.

Bellingham, Washington, where WWU lives, is about 30 miles south of the Canadian border. Or, about 30 miles south of the 49th parallel. This means we get some nice extremes when it comes to daylight hours. Here is what the differences are between the solstices (times converted to Standard for better comparison):

June 21st, 2006:
  • Sunrise 4:05am
  • Sunset 8:17pm
  • Total daylight hours: 16:12
December 21, 2006:
  • Sunrise 8:00am
  • Sunset 4:16pm
  • Total daylight hours: 8:16
So yeah, we don't get a lot of sun in the winter. This also means that doing Daylight Savings Time three weeks earlier makes it darker in the mornings. Here is a handy chart comparing the sunrise-times for 2007 on the DST dates. As above, without DST for comparison's sake.

March 11, 2007April 1, 2007October 28, 2007November 4, 2007

Yep. Lots o' change there. Same chart, with DST calculations this time.

March 11, 2007April 1, 2007October 28, 2007November 4, 2007

The DST change on 3/11/2007 will bring dawn to where it was on 2/06/2007. Oh. Goodie.

Just for comparason's sake, here is the same DST chart for Washington, DC.

March 11, 2007April 1, 2007October 28, 2007November 4, 2007

DST this year will bring Washington DC's sunrise to the same time it was on January 15th (a few days ago!). On Jan 15 up here next to Canada, sunrise was at 7:58am.

What does this buy us, exactly?

Busy week

Last week was interesting. I spent most of it hacking on the new spam appliances. Then we had another virus outbreak. A bad one. For a first-hand view of what happened read this. Due to a vacation on Friday, I wasn't directly involved in this one to any significant degree. Normally, ferreting out root-kits is right up my alley. Instead, I was have routine maintenance performed on certain small animals in my house.

As this is the second incident of a Symantec worm, it makes it especially galling. The machines that got infected were all ones that got missed during the last post-worm remediation. Unfortunately, certain computer labs did not have the new AV software updated on their images so they'll be vulnerable for a while; revising the lab image during session is apparently a major undertaking. I have been tasked with assessing with an assumption of eventually installing Zen Asset Management in an effort to allow us to identify 'non-compliant' machines when something like this happens in the future.

The go-ahead decision rests with the Vice Provost, which itself should provide an interesting insight into the new guy. This isn't the first time an inventory solution has been proposed. The previous Vice Provost had a low tolerance for back-talk from the Deans, so if enough Deans complained about the new whatever-it-is he usually backed down from supporting it. From the sounds of it, the new Vice Provost wants this capability quite badly, so we shall see how the cards fall.

Brainshare session catalog is up!


TUT202 Migrating a NetWare Cluster to an Open Enterprise Server Linux Cluster

TUT211 NetWare Virtualization

TUT326 Virtual Machines and Storage Foundation

TUT212 Novell Storage Services

TUT205 Dynamic Storage Technology: Reducing the Cost of Storage

IO101 Open Enterprise Server 2 Introduction, Overview and Futures

TUT101 Open Source Stack vs Open Enterprise Server (OES)

TUT204 Configuring Samba on Open Enterprise Server

TUT210 Open Enterprise Server: An Architectural Overview

TUT246 ZENworks: Design and Best Practice

TUT247 ZENworks: Designing “Pulsar” to Scale to Your Environment

TUT104 Choosing the Right File System for Open Enterprise Server

TUT129 Troubleshooting a SUSE Linux Enterprise 10 System

BOF120 All Things Samba

BOF100 Interoperability with Microsoft Windows and Active Directory

TUT215 Data Protection Solutions on Linux

TUT218 Learning to Live With Microsoft Without Turning Blue

TUT106 Distributed File Services

My interests so far. And there are much more sessions to be posted. The Laura Chappel sessions aren't even up yet.

Changing times: Novell patches

Weird. I saw this this morning because I'm slow:
No new patches will be put on ftp.novell.com (ftp://ftp.novell.com/) or FileFinder (http://support.novell.com/filefinder) after January 12, 2007 because patches are being moved to the Novell Downloads website.

On the evening of January 16th, all new patches and all patches currently on FileFinder will be made available on the Novell Downloads web site http://download.
novell.com/ instead of through FileFinder and ftp.novell.com.

To allow transition time, patches that are currently on ftp.novell.com will remain there until February 12th at 5:00 PM Mountain Time. At that point in time, all patches in the /pub directory will be taken off ftp.novell.com and will need to be accessed on the Novell Downloads web site.
So, in other words, downloading patches will require a novell.com login from now on. Not that I mind, I have that. But I find such requirements annoying from other vendors.

Huh, HTML and Outlook 2007

| 1 Comment
So I see on Slashdot that Microsoft has announced that Outlook 2007 will use Word as the HTML rendering engine. This makes sense, because they were planning on making Word the default e-mail editor.

I STILL think this is abomination, but then I've thought that about HTML in e-mail for a while now. The best thing that happened to Outlook was Plain Text mode, which if I remember right was a reaction to the HTML-virus mails of a few years ago. Only rarely do I take my Outlook out of plain-text mode to read a mail, generally because some Helpdesk person sent me a mail with an embedded screen-shot in it that can't be viewed any other way.

According to Slashdot, 'email designers' are up in arms because they lose things like CSS. Yes, it takes mail layout back 1998, but I can't view this as a bad thing. That is a personal view. And a slightly professional one, as I've spent the last three days pouring over spam and all the false-positives generated by a certain famous lingerie company having a sale; a mail that contained about every single bad behavior that folk concerned about privacy worry about. Yeah. E-mail based 'newsletters' that look like a web-page... I can see the attraction, but they make my life harder so I avoid them where possible.

So yeah, Microsoft taking CSS support out of email is something I have faintly good ideas over. It'll break the formatting of existing mails, but I very, very rarely see those anyway.


On days like this, I really think I should pick up this T-Shirt. I've been tempted by it for a while. Just sayin'.

That said, now that the thingy has been in place for more than 24 hours I have some interesting data to play with. Unlike previous estimates, the appliance has handled 'only' 230,000 emails in the 24 hours period defined as 9am to 9am today. This is about a fifth of previous estimates, which makes me wonder what we were counting.

What's also interesting is how few viruses have been detected. It looks like the era of the mass mailer worm is largely over. Of that 230K odd mails, only 240 viruses were found. Most of them were mass-mailers, of course, but this is not the way things were even 3 years ago. This appliance is an anti-spam appliance that also does anti-virus, not the other way around like some other appliances I can think of.

New anti-spam appliance

| 1 Comment
The new anti-spam appliance finally has a license file, so I can start dorking around with it.

Happily, this appliance DOES catch picture-spam! YAY!

Unfortunately it also classifies the following as pic-spam:
To: <Everyone>
From: "The Bowler Family" <redacted>
Subject: In need of a serious laugh?

The Purina Diet

I was in Wal-Mart buying a large bag of Purina for my dogs and was in line to check out.

A woman behind me asked if I had a dog........ Duh!

I was feeling a bit crabby so on impulse, I told her no, I was starting The Purina Diet again, although I probably shouldn't because I'd ended up in the hospital last time, but that I'd lost 50 pounds before I awakened in an intensive care unit with tubes coming out of most of my orifices and IV's in both arms.


[attachments: "dadshirt Bkgrd.gif"]

Perhaps the spam/ham threshold was a bit low. Most pic-spam I know of is one line of text and an attached image. Which also makes it hard to differentiate between that stuff and stuff like this:
To: You
From: Me
Subject: Too damned cute

Dickens was sleeping upside down again. This time, I got a picture.

[attachment: UpsidedownHedgehog.JPG]
It's the pic-spam that is causing the powers that be to start mumbling about finding money, somewhere, anywhere, to just stop it. We've had these appliances sitting on the floor for a few months now, waiting for priorities to shift to the point where we can work with them. Now they have, and now I have.

I must say, it does a pretty good job. It scores on a 0-100 scale, which it sadly doesn't expose, and is hardcoded to toss anything that scores in the 90-100 range. And... it makes good decisions. You can tune the 'suspected spam' threshold lower then that, which is what I've been tweaking. Happily, it's in 'monitor and record' mode, so I can watch message flow without actually DOING anything with the messages; letting the antispam software actually on the Exchange boxes handle the load. This allows me to set the 'suspect' threshold to various spots and look to see what it tags.

Set it low enough, and I saw one message from a student to Financial Aid, asking about canceling a loan for the quarter, got picked up. Yep, raised the threshold a few ticks after that one. Apparently The Economist sends out bulletins, and that gets picked up around the 65 range. A group of students was chatting in e-mail about a class that got canceled yesterday (ice and snow), which got tagged due to the number of people on the To: line (also at about 65). One googlegroups message discussing in a scholarly way a subject that appears in spam a lot, which was tagged when the filter was set to 70.

All in all, less than 1% of the messages tagged as SPAM are tagged 'suspect'. This thing does a good job.

Brainshare, oops

| 1 Comment
So. BrainShare is during Finals Week this year, not during Spring Break like most years. I had been telling everyone that I would be gone over Spring Break. Oopsie!

This is actually better. During finals week we don't to much besides work on test/dev system and firewatch on the production systems. Good week to not be here. Spring Break on the other hand is a prime time to roll out new production systems, and being in Salt Lake City during it makes that process harder. Spring Break is not a vacation for us in IT.

If we go to NetWare SP6, it'll be during Spring Break. I'll actually be here for that.

Index spam

We had a humorus event happen today which underlined a problem some people face with the scourge known as 'index spam'. As anyone who has used Google Desktop or let Mac Spotlight troll through our Shared volume knows, just because someone thinks you should be able to see it doesn't mean you care to know about it. One of the biggest draws of these tools is that it searches YOUR STUFF for things you're looking for, by dragging in a bunch of things that you can see but don't care for... it dilutes the usefullness of search-results in these tools.

We have several volumes that have gobally-readable data in them. Some of them are system directories we need everyone to be able to see, others are folders where the managers for those folders figured that there was no need for privacy here. Whatever the case, the amount of 'everyone can read it' data is not what you'd call trivial.

This is the sort of thing that security managers cringe at. But then, we're a governmental agency that:
  1. Is subject to Freedom of Information Requests
  2. Does not handle classified data
  3. Over the years has had several unflattering stories in the local paper using data obtained by FOIA requests.
So people tend to be a bit blase about data security. Why bother, since the paper will find out about it anyway? That said, we do have some data that is subject to other standards (PCI, HIPPA, etc) which is locked down.

That said, when people look at the results of their index spam and try and 'fix it', things can get... messy. Some applications, and I think Google Desktop is one of them, allow you to set indexing blacklists. Others just assume that if you can see it, you'll need it sometime. But when an end-user doesn't know about those features, and has just had to page through four pages of results before the document they were actually looking for was present, then goes about deleting the files that are 'in the way', the presence of 'write' privs suddenly becomes much more important.

Updated cool-tool

SEG has been updated!

New version, new look.
Screen-cap of the new SEG main-page

Dethroning Exchange

| 1 Comment
A lot of talk has gone into how to overthrow the Windows lock on the Desktop market. The server market is more fluid, but it STILL dominates that space. Linux and OSX are both making real strides in that space, though Apple's ad campaign focusing on, "Windows is for Work, Mac is for Fun," doesn't exactly improve Mac adoption in the workplace.

There aren't any clear threats to Exchange. The other two big players in the arena, GroupWise, and Lotus Notes, have both been there a long time. Both benefited from what I call, 'the Melissa years defections.' I know for a fact that OldJob stayed with GroupWise precicely because we were still up when Melissa and company nuked most of the Exchange shops in the area.

Melissa introduced the era of the mass-mail worm. The clean up efforts from those worms drove billions of dollars of investment into Exchange recovery tools, Exchange anti-virus tools, and other related technologies. Thanks to that burst of innovation, this is a largely solved problem (given a sufficient investment in 3rd party defensive tools). WWU hasn't had a mass-mail-worm-related Exchange outage since I started here three years ago.

What's also helping is that the mass-mail worm is slowly dying by the side of the road in favor of much more lucrative mails. The current SPAM problem is turning into a sort of global denial-of-service attack against SMTP in general, not just Exchange. Trojan emails that contain images that exploit Windows image handling, not just Outlook's, affect even Pegasus users.

The best defence against the current crudware infecting e-mail these days is to use a non-Windows desktop. If that's not in the cards (it isn't for WWU) then the field opens up much more dramatically. Most larger shops are looking seriously into anti-spam appliances as a load-shedding technology to help their mail-transfer-agent (whatever it is) keep up with legitimate load. Some very minority players in the MTA market only can use appliances, and don't have the option of hooked-in anti-spam software.

The days of viruses and other crud scaring people off of Exchange are long gone. Now the fight has to be taken up on, unfortunately, features and mind-share. In the absence of a scare like Melissa provided, migrations from Exchange to something else will be driven by migration events. Microsoft may be providing just that threshold in the future, as they've said that they will be integrating Exchange in with SharePoint to create the End All Be All of groupware applications. Companies that aren't comfortable with that, or haven't deployed SharePoint for whatever reason may see that as an excuse to jump the Microsoft ship for something else. Unfortunately, it'll be executives looking for an excuse rather than executives seeing much better features in, say, GroupWise.

Exchange isn't as dominant as Windows-on-Desktop is, but its market-share isn't exactly declining the way Windows desktop ownership is (really! It is declining! Minuscule amounts, but it is there!). New deployments of Notes or GroupWise, which is different from migrations, are due largely to geeks or management familiar with either technology requesting it specifically. The default is still Exchange when it comes to a big-boy groupware application. That'll take real time to change.

So, Exchange will be with us a long time. What'll start making the throne wobble is if non-Windows desktops start showing up in great numbers in the workplace. THEN we could see some non-MS groupware application threaten Exchange the way that Mac (and Linux) are threatening the desktop.