January 2009 Archives

Link grabbed on Slashdot:


A project is now in the works to bring true Exchange compatibility to a non-Windows desktop. The Exchange Connector for Evolution and others of that ilk have done so using the WebDAV or RPC-over-HTTP interface. This project is different in that it is trying to emulate how Outlook interacts with the Exchange system; using RPC over SMB among other protocols.

This would bring the full power of the Exchange groupware environment to anything that can run KDE from the sounds of it. And that's really nifty. It uses Samba 4 code to a great extent, which makes sense as there is a lot of neat new things in that codebase. This would be a big step for Linux in the enterprise! I also saw mention of Novell input to the project, which is nice to see.

Spam stats

It has been a while. Some current spam stats from our border appliance:

Processed Spam Suspected Spam Attacks Zombie Blocked Allowed Viruses Suspected Virus Worms Unscannable Malware Passworded .zip file
Last 30 days
9,489,662 4,328,334 (46%) 8,895 (<> 221,926 (2%) 4,163,253 (44%) 1 (<> 10,248 (<> 1,265 (<> 1,233 (<> 2,590 (<> 26,508 (<> 0 (0%) 13 (<> 0 (0%)

That is a lot of spam. The 'Zombie' class are connections rejected due to IP reputation. Between zombies and outright spam detections, 95% of all incoming mail has been rejected over the last 30 days. Shortly after the McColo shutdown that percentage dropped a lot. We're now back to where we were before the shutdown.

The paradox of legitimate mass-mailings

Spam has been with us for over a decade now. Sad fact, but true. Also sad is the need for legitimate mass mailings. Here at WWU, these can look like budget updates from the University President, something we're all deeply interested in. Due to the cat and mouse games between spammers and the anti-spam vendors, it is an ever shifting game trying to figure out what'll get through the various spam checkers. At this moment we have three layers:

  1. Border appliance spam checker, which does IP reputation of incoming mail connections, resulting in a 93% bounce rate (as of this morning, just shy of the pre-McColo-shutdown high of 95%).
  2. Software on the Exchange servers themselves, which don't catch much, but do catch the few connections that leak around the border appliance.
  3. The Outlook Junk Mail filter
As it happens, it is #3 that has been causing us the most grief. In general, the rule of getting past the filters is simple: don't look like spam. In specific, this is really hard. Especially since our border appliance is good enough that most users don't see what a truly unfiltered email stream looks like.

Last week the President's office sent out a notice about a Mid Year Report to Campus. Unfortunately, they sent the mail from a mailbox that we haven't taken great pains to whitelist, and the mail was just an embedded image. In other words, they sent a mail that functionally looked exactly like the pump-n-dump stock scams of 2 years ago, and did so from a mailbox we haven't white-listed. Of course Outlook junked it, as its the dumbest filter of the bunch.

We're not the only place this this problem. All-hands emails are something most organizations have to do at some point, as Intranet sites with posted announcements get vastly fewer eyeballs. Perhaps mandated RSS feeds in Outlook is the future of this... huh. Anyway, we're constantly working with the internal mass-mail sources to help them tune their mail to actually arrive.
In this modern era, it is becoming more an more common for vendors in the Windows world to either require, or strongly suggest that the vendor perform a software install on a server. In the past this required either sending a physical body out to the location, or using something like PC-Anywhere to do the install. Now, a wide variety of web-based remote-control packages are on the market that greatly simplify getting the knowledgeable install-geek onto the server in question.

More and more often, vendors are offering maintenance and update contracts contingent on console access. While these greatly simplify maintaining a package for small offices who don't have the IT oomph to really do it themselves, these are a great pain for those of us who manage the servers themselves. What's really bad is when web software (typically IIS and .NET based) is subject to these sorts of contracts.

We have a small number of IIS-based web-servers that are shared with a variety of departments. ATUS and ADMCS are the biggest consumers, of course, but other departments have their own stuff on there. This also includes several 3rd party apps we've put in over the years. These servers have a lot on them.

What happens when we get more than one software package with this sort of contract attempting to run on these IIS servers? It means that, at least in theory, multiple vendors have nearly unrestricted access to these web-servers. As these servers are general-purpose servers and not dedicated to this one application, this is a pretty major data-security issue.

This isn't quite as big a problem when the application is a more traditional client/server app, or the app resides on its own dedicated server. We don't like that kind of app-server running with AD credentials on console, but we can work around that. Web-servers, though, run a lot of apps.

In the UNIX world, I have heard of vendors requesting the ability to SSH into a server in order to do installs. The context of this was humor, as in, "look at the stupid vendor." In general, if a vendor asks for local root to a server for a simple install, the answer will be a resounding no way. So what makes the Windows world different, that they'll permit a third party root-access to their own servers? Perhaps because it takes a lot less skill to do Windows administration at least half-way right, so vendors have to compensate for less comprehensively trained system administrators. Unfortunately, it makes them less nimble when they do run into shops with strict controls on what runs on the web servers, and who is allowed console access to them.

SeaMonkey 2.0 and SSL

Yesterday I downloaded a nightly-build of SeaMonkey so I could see how things are going. It's functional, I think most of the updates are on the mail side which I haven't tried. I like it as a browser anyway.

Looking at the what's new list there are a few things that stand out:
  • Making the Extension environment more Firefox like
  • Making the rendering engine more Firefox like
  • Migration from Thunderbird
  • Add-On notification, like Firefox
And something they didn't point out...
  • Complain about un-chained SSL certificates like Firefox.
I could have sworn I've griped about this before, but I can't find the post. Here in the land of IT, vendors of all kinds ship web interfaces with self-signed SSL certificates. Generally speaking this is just fine, since these are appliances/applications/interfaces that a VERY small group of people have access to, and any SSL is better than none. Chaining to a trusted CA isn't as important since I very likely manage the box/widget/app myself and trust it. But Firefox (and now SeaMonkey 2.0) gripes about it, since it is, technically, unsafe.

Yes, I can add exceptions. This works for things with a static file. But for certain other things, such as HP Integrated Lights Out boards, regenerate their SSL certificates every time they power-cycle, forcing you to re-add the exception every time that server reboots. For these, adding exceptions doesn't work. In my line of work, I must have umpty hundreds of little SSL-enabled web-pages all over the place.

NetWare, at least last time I checked, defaulted to using the IP-certificate instead of the DNS-certificate for the Novell Remote Monitor. Since I never access that with an IP address, this will trigger a gripe from Firefox as the Subject doesn't match what's in the URL bar. For this reason, and others, one of my post-install tasks is to change that to load the DNS certificate (and disable the Xserver, nfs, and afp servers).

When I'm doing a lot of work on systems with funky certificates, this can get downright aggravating. When I was doing work in the Novell Beta for OES2-SP1, I had a lot of test trees set up, with their own Certificate Authorities, and PKI environment. If I had been using Firefox for all of that, by now I'd have had A LOT of "Organizational CA" certificates in my browser root-cert store. Instead, I was lazy, used SeaMonkey, and just clicked past the gripes.

Since I have legitimate reason to be regularly hitting web-sites with bad SSL certificates, it would be really nice if there was some way to turn off the hard-stop warning FF (and now SM 2.0) come with, and go back to an earlier mode.

Also on my List? Vendors who don't allow anyway to update their pre-shipped SSL certificates. Grrrr.

NetStorage and IE7

| 1 Comment
Looks like there is a bug in NetStorage (NW65SP7, not sure if SP8 fixes it or not) and IE7. When you're browsing along, select a file for download, then go to File -> Download, you get a login screen. No matter what you enter, it won't let you download the file. Access forbidden!

You can get around it by double-clicking on the file you want to download.

However, this also breaks upload and there is no workaround for that.

Works just fine in non IE7 browsers. I understand this issue is known by Novell.

On the server side, I can see a few signs of this in the log-files. There is a line like this for a failed download attempt: - - [22/Jan/2009:11:15:40 -0800] "GET /oneNet/netstorage/Home@WWU/ac228.tgz HTTP/1.1" 401 1370 "-" "Java/1.4.2_13"

That IP is the server's IP address, not the clients. The user agent is Java. Clearly (to me anyway) Tomcat is proxying the download request and thus creating the new user-agent string. The rest of this session is with a normal IE7/WinXP user-agent.

Now a successful download (firefox): - username [22/Jan/2009:11:16:39 -0800] "GET /oneNet/NetStorage/Home@WWU/cert.txt HTTP/1.1" 200 3329 "-" "Java/1.4.2_13"

The observant may notice some case differences there. I thought of the same thing, and did some poking around in IE to get this: - - [22/Jan/2009:11:18:14 -0800] "GET /oneNet/NetStorage/Home@WWU/cert.txt HTTP/1.1" 401 3329 "-" "Java/1.4.2_13"

Same case as the Firefox access, but still failed. I don't know why this is doing this, but clearly something inside Tomcat isn't happy with how IE7 is handling the POST request that requests the download.

Website stats

We purchased Urchin's web stats package before Google bought them. We're still using that creaky old software, even though its ability to interpret user agent string is diminishing. I'm not in the webmaster crew for this university, I'm just a client. But I do track the MyWeb stats through Urchin.

I also track our MyFiles (NetStorage) stats. This became more interesting the other day when I noticed that just over 40% of the bandwidth used by the Fac/Staff NetStorage server was transferred to user agents that are obviously WebDav. This is of note because WebDav clients are not javascript enabled, and thus will not register to things like Google Analytics. If I had been relying on Google Analytics for stats for NetStorage, I'd have missed the largest single agent-string.

Even though Google bought Urchin, it makes sense that they dropped the logfile-parsing technology in favor of a javascript enabled one. Google is an advertising firm with added services to attract people to their advertising, and it's hard to embed advertising in a WebDav connection. It used to be the case that RSS feeds were similar, but that's now a solved problem (as anyone who has looked at slashdot's feed knows).

In my specific case I want to know what the top pages are, which files are getting a lot of attention, as well as the usual gamut of browser/OS statistics (student myweb has a higher percentage of Mac hits, for one). One of the things I regularly look for are .MP3 files on the Student MyWeb service getting a lot of attention. For a while there the prime user-agents hitting those files were flash-based media player/browsers embedded on web pages, just the sort of thing that only logfile-parsing would catch.

One thing that the NetStorage logs give me is a good idea as to how popular the service is. Since apache logs will show the username if the user is authenticated, I can count how many unique user ID's used the service over a specific period. That can tell me how strong uptake is. I may be wrong, but I believe Google Analytics wouldn't do that.

The Urchin we have is still providing the data I need, but it's still getting stale. It's OS detection is falling apart in this era of Vista and 64-bit anything. But, it's still way better than Analytics for what I need.

Inept phishers

Over the weekend (Saturday, in fact) we had a phish attempted against us. As this is still a relatively new experience for us, it got the notice of the higher-ups. When I got in, I got the task of grepping logs to see if anyone replied to it.

While doing that I noticed something about the email. It had no clickable links in it, and the From: address (there was no Reply-To: address) was @wwu.edu, and was an illegal address.

In short, anyone replying to it would get a bounce message, and there was no way for the phishers to get the data they wanted.

More broadly, we've noticed a decided increase in phishing attempts against .edu looking for username/password combinations. The phishers then use that information to log in to webmail portals to send spam messages the hard way, copy-paste into new emails. This has the added benefit (for them) of coming from our legitimate mailers, from a legitimate address, and thus bypasses spam reputation checks, SPF records, and other blacklists. It doesn't have the volume of botnet-spam, but its much more likely to get past spam-checkers. At last check, about 50% of incoming mail connections to @wwu.edu are terminated due to IP-reputation failures.

Network stability

There is a certain, shall we say, major event today that was estimated to clog wireless networks and presumably netcasters as well. As it happened, I listened to it on old fashioned broadcast media (FM radio) so I didn't have a problem with latency.

From what I can see on our bandwidth chart, we didn't have much of a bump from faculty/staff watching the event online. Either the netcasters all melted, there wasn't as much interest on campus as I thought, or it went without a hitch and was just lost in the background noise. Either way, nice to see.

W-a-y back in 2001 there was another event that generated a massive amount of news interest. I managed to hear about it early enough that cnn.com actually resolved for me, which it shortly stopped doing.

Things have changed!

NTP on NetWare

A while back I did some work setting up an ntp peer-group on a pair of SLES servers (SLES9 and SLES10). That worked pretty good, and I managed to get autokey security working, which I thought was nifty. Then my thoughts turned to the OES environment.

If/when we get off of NetWare and move to the Linux kernel, NTP becomes the only way to do timesync. So I figured I'd see how amenable NetWare's xntpd was to secure configuration. Turns out it can do it, but there are some caveats.

First of all, it seems that the NTP for NetWare is based on NTPv3, not NTPv4, which means it doesn't support autokey and only supports symmetric keys. This also means that some other items on the ntp.conf file on the sles servers couldn't be carried over.

As it happens, the following sys:/etc/ntp.conf file works pretty well:
server ntpserver1
server ntpserver2 minpoll 6 maxpoll 13
peer ntppeer1 key 1

enable auth monitor
keys sys:\etc\ntp.keys
trustedkey 1
requestkey 1

restrict default ignore
restrict mask nomodify nopeer
restrict [ip of ntpserver1]
restrict [ip of ntpserver2]
restrict [ip of ntppeer1]

Populating the ntp.keys file couldn't be done from NetWare directly, I had to do that on a SLES server and copy it over. But once I did that, the ntppeer1 server and the NetWare server correctly authenticated to each other.

Interestingly, when I pointed an NTPv4 linux machine at the NetWare NTP setup I got complaints on the NetWare server about the incoming timehost not having the correct key and not being able to sync time. This is interesting because this linux machine was NOT one of the specified time-hosts. When I put in the 'restrict' line above with the 'nopeer' flag on it, those messages stopped.

The above configuration was successful in enabling a peer relationship between the two timehosts. This is loosely analogous to a PRIMARY group in traditional NetWare TIMESYNC setup. Should one or both of these hosts lose connection to the non-WWU time-servers (which are in essence equivalent to REFERENCE servers in Timesync, but unlike Timesync you can have more than one), they can negotiate time between themselves. This is important, as it prevents them from going out of sync, which would have dire consequences if allowed to happen more than a few minutes.

Legislation-watch begins

Today is the first day of the Washington State legislative session. 105 days for a budget bill to be passed and presented to the governor. You can bet that this will be closely watched by us.

Learning new tricks

This morning we got a standard form for an account status change, moving from Employee to Student. We do these all the time, though usually it's the other direction. We processed it like we always do, and moved it on.

Three hours pass, and the user goes to the ATUS helpdesk. They can't get into email, and would like to know why.

The reason is because we remove the Exchange mailbox when moving to Student. The Student email system is different. So far, perfectly normal.

It turns out that what they really wanted was an account on the shell-server, which they didn't already have. They were a staff member taking some classes, and needed the account. Not an ex-employee now taking classes. Someone, somewhere, filled out the wrong form and sent it along to us. So we needed to put their accounts back where they were.

We moved the account back to the Employee side of the virtual fence, but couldn't get the mailbox back. The mailbox wasn't showing in the Disconnected Mailboxes list like it should. Under Exchange 2003 the fix for that was pretty simple, right-click on the Mail-store and select, "Run Clean up process." That would force any recently deleted mailboxes to show up in the disconnected list.

Thing is, that isn't there in Exchange 2007. I went on up to the Mailstores in Exchange Console, and there was no option to force the clean-up process to run. I suspected this was one of those things consigned to the command-line thanks to Microsoft's 80/20 rule for Exchange management utilities (the 80% of the things you do every day and don't think about are in the GUI, the 20% you do once a month and need RIGHT NOW has to be googled and run through the powershell cmd).

I was right. It took some googling, but I found it.

Clean-MailboxDatabase -Identity "exchmailboxsrv1\First Storage Group\TwinklyVampireDatabase"
(Not real names)

That will force the clean-up process. After running this, the missing mailbox showed up, and we were able to reconnect it to their user object with no loss of mail.

DataProtector 6.00 vs 6.10

A new version of HP DataProtector is out. One of the nicest new features is that they've greatly optimized the object/session copy speeds.

No matter what you do for a copy, DataProtector will have to read all of one Disk Media (50GB by default) to do the copy. So if you multiplex 6 backups into one Disk Writer device, it'll have to look through the entire media for the slices it needs. If you're doing a session copy, it'll copy the whole session. But object copies have to be demuxed.

DP6.00 did not handle this well. Consistently, each Data Reader device consumed 100% of one CPU for a speed of about 300 MB/Minute. This blows serious chunks, and is completely unworkable for any data-migration policy framework that takes the initial backup to disk, then spools the backup to tape during daytime hours.

DP6.10 does this a lot better. CPU usage is a lot lower, it no longer pegs one CPU at 100%. Also, network speeds vary between 10-40% of GigE speeds (750 to 3000 MB/Minute), which is vastly more reasonable. DP6.10, unlike DP6.00, can actually be used for data migration policies.