Recently in exchange Category

It all began with a bit of Twitter snark:


SmallLAMPStack.png

Utilities follow a progression. They begin as a small shell script that does exactly what I need it to do in this one instance. Then someone else wants to use it, so I open source it. 10 years of feature-creep pass, and then you can't use my admin suite without a database server, a web front end, and just maybe a worker-node or two. Sometimes bash just isn't enough you know? It happens.

Anyway...

Back when Microsoft was just pushing out their 2007 iteration of all of their enterprise software, they added PowerShell support to  most things. This was loudly hailed by some of us, as it finally gave us easy scriptability into what had always been a black box with funny screws on it to prevent user tampering. One of the design principles they baked in was that they didn't bother building UI elements for things you'd only do a few times, or would do once a year.

That was a nice time to be a script-friendly Microsoft administrator since most of the tools would give you their PowerShell equivalents on one of the Wizard pages, so you could learn-by-practical-example a lot easier than you could otherwise. That was a real nice way to learn some of the 'how to do a complex thing in powershell' bits. Of course, you still had to learn variable passing, control loops, and other basic programming stuff but you could see right there what the one-liner was for that next -> next -> next -> finish wizard was.

SmallLAMPStack-2.png

One thing that a GUI gives you is a much shallower on-ramp to functionality. You don't have to spend an hour or two feeling your way around a new syntax in order to do one simple thing, you just visually assemble your bits, hit next, then finish, then done. You usually have the advantage of a documented UI explaining what each bit means, a list of fields you have to fill out, syntax checking on those fields, which gives you a lot of information about what kinds of data a task requires. If it spits out a blob of scripting at the end, even better.

An IDE, tab-completion, and other such syntactic magic help scripters build what they need; but it all relies upon on the fly programatic interpretation of syntax in a script-builder. It's the CLI version of a GUI, so doesn't have the sigma of 'graphical' ("if it can't be done through bash, I won't use it," said the Linux admin).

Neat GUIs and scriptability do not need to be diametrically opposed things, ideally a system should have both. A GUI to aid discoverability and teach a bit of scripting, and scripting for site-specific custom workflows. The two interface paradigms come from different places, but as Microsoft has shown you can definitely make one tool support the other. More things should take their example.

Email is not IM

| 3 Comments
Anyone who has worked with email knows this, but that doesn't stop mail users from calling us and asking why a message is late. But hey, what's yet another example? This describes the mail routing a specific message followed.

Received: from [140.160.12.34] by web120819.mail.ne1.yahoo.com via HTTP; Sun, 20 Feb 2011 17:48:33 PST
Received: (qmail 81576 invoked by uid 60001); 21 Feb 2011 01:48:33 -0000
Received: from [127.0.0.1] by omp1039.mail.ne1.yahoo.com with NNFMP; 21 Feb 2011 01:48:33 -0000
Received: from [98.138.88.239] by tm4.bullet.mail.ne1.yahoo.com with NNFMP; 21 Feb 2011 01:48:33 -0000
Received: from [98.138.90.51] by nm24.bullet.mail.ne1.yahoo.com with NNFMP; 21 Feb 2011 01:48:34 -0000
Received: from nm24.bullet.mail.ne1.yahoo.com ([98.138.90.87]) by BAY0-PAMC1-F6.Bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675);Sun, 20 Feb 2011 19:37:38 -0800
Received: from mail pickup service by SNT0-XMR-002.phx.gbl with Microsoft SMTPSVC; Sun, 20 Feb 2011 19:37:48 -0800
Received: from SNT0-XMR-002.phx.gbl (10.13.104.140) by BL2PRD0103HT012.prod.exchangelabs.com (10.6.4.137) with Microsoft SMTP Server id 14.0.650.68; Mon, 21 Feb 2011 03:37:54 +0000

Normally these headers are read from bottom up, but this is ordered top down. Translated, this means:

  1. A message was sent from a computer on campus via a Yahoo mailer (98.138.88.239) at 17:48:33 PST.
  2. This was processed by another Yahoo mailer (98.138.90.51) at 17:48:33 PST
  3. And again (98.138.90.87), at 17:48:33 PST
  4. It was picked up by the Microsoft mailer (10.13.104.140) at 19:37:48
  5. And forwarded on to an ExchangeLabs server (10.6.4.137) where it came to rest.
Note the nearly two hour delay from start to finish. And yet, the mail shows a timestamp of 17:48 in the mailbox in question even though it arrived much later then that. That's because the Date: field was set by the sending mailer and not modified throughout its travels. In this case, the exact source of the delay is uncertain, it could be a queuing delay on the Yahoo side when talking to Microsoft, or it could be a queuing delay on the Microsoft receiving side. Can't tell from here.

This does show that mail consumers tend to assume two key things:

  1. Mail transit is very fast (where 'fast' can be defined as anything under a minute to up to five minutes depending on the user).
  2. The Date is when it is received.
The fact that mail can legitimately take hours to get to where it is going is not good enough for these users. The fact that it takes less than 5 minutes nearly all the time is really gravy, but it does set the service expectation of the entire email service. So we get grief when the expectation runs against the reality of it.

Guaranteed delivery of email

Simply put, can't be done.

Even in the pre-spam Internet, SMTP was designed around the fact that email delivery is fundamentally best-effort. Back then it was more about handling network outages, but the fact remains that by design email is best-effort not guaranteed.

That doesn't stop people from demanding it, though.

More recently, with the Virginia Tech shootings a lot (I'd even go so far as to say most) of Higher Ed has taken a really close look at emergency alerting systems. WWU is not immune to this. We have an outside entity that handles this, and WWU upper administration has asked that such emails NOT end up in the Junk Mail folder (we also have an SMS alerting system to go along side this). This is harder than they think, which just makes my life less pleasant every time such a notice does end up in a Junk Mail folder.

With spam making up anywhere from 92-98% of all incoming email, email is fundamentally lossy these days. We LIKE it that way. The hard part is picking the good stuff out of the sea of bad stuff. And unfortunately, there is no one way to guarantee email WILL be seen by the recipient.

The most recent major junking event was because our outside mailing entity changed what servers they do their mailing from, which meant they weren't getting the benefit of the IP-whitelist. Either they didn't notify us, or the people who communicate with them didn't realize that was important and therefore didn't send the change notice to me. The fact that the message in question was a simple web-page copy with a lot of hyper-linked images just made it extra-spammy.

The WWU marketing department has been having their "WWU News" messages, emails with lots of links including mentions of WWU and links to WWU events, end up in Junk Mail about 80% of the time, even though the service they send through is ALSO whitelisted.

The one thing that makes my life all too interesting when attempting to guarantee email delivery? Outlook's junkmail filters. We can't do a thing with them, and Microsoft purposely makes them hard to predict. Nearly all of the junkings I end up troubleshooting end up being Outlook independently deciding it was crap and binning it. I can guarantee delivery right up to the point where Outlook analyzes new messages for spam-factors, but once it gets there all bets are off.

Unfortunately.

I can't guarantee email delivery. I never could, but it's harder now.

3rd party application headaches

A while back we managed to push through some new purchasing rules that required IT review of any IT technology purchases. This is needed, since end-user departments haven't the first clue what'll work with our existing infrastructure, and it helps us advise them of complications. For instance, if a product requires PHP on IIS for some reason, we really want to be able to let them know before they purchase that doing so will require a server purchase as well since we don't support that environment currently.

Unfortunately, a small number of things still slip through. Perhaps we didn't read the manuals enough. Perhaps a high enough manager expended sufficient political capital to Make It So. But complications can arise when we go to make the new thingy work.

A case in point:

For the last two weeks I've been attempting to get a certain package up and running that has email capabilities. This has to fit within our Exchange system, which is a rather common environment. What isn't so common, it seems, is our insistence on secure protocols for authentication. While Exchange 2007 is perfectly willing to support naked POP3 and even naked SMTP-Auth, we, on the other hand, are not so forgiving. We wisely have a security standard in place that says that all authentication traffic must be encrypted, and this prevents us from running POP3 and SMTP in a way that allows passwords in the clear.

This package has support for one SSLed service: POP3-SSL. We don't support POP3 since our users were forever screwing themselves thanks to the default of "Delete on retrieval" in most mailer clients, which kind of pissed them off when they got to the office the next morning and their mailbox was empty.

Thanks to the use of stunnel I was able to tunnel unencrypted IMAP to Exchange's IMAP-SSL port at least, so that channel got working.

Right now I'm trying to convince stunnel and the application to work together to get SMTP-TLS working. Sadly for me, I have to wait a couple of hours before the app attempts an SMTP check for me to see if it works.

On the 'up' side, we're charging this department by the hour to get this set up. So the labor bill on this will be fairly high.

It's the little things

| 1 Comment
My attention was drawn to something yesterday that I just hadn't registered before. Perhaps because I see it so often I didn't twig to it being special in just that place.

Here are the Received: headers of a bugzilla message I got yesterday. It's just a sample. I've bolded the header names for readability:
Received: from ExchEdge2.cms.wwu.edu (140.160.248.208) by ExchHubCA1.univ.dir.wwu.edu (140.160.248.102) with Microsoft SMTP Server (TLS) id 8.1.393.1; Tue, 15 Sep 2009 13:58:10 -0700
Received: from mail97-va3-R.bigfish.com (216.32.180.112) by
ExchEdge2.cms.wwu.edu (140.160.248.208) with Microsoft SMTP Server (TLS) id 8.1.393.1; Tue, 15 Sep 2009 13:58:09 -0700
Received: from mail97-va3 (localhost.localdomain [127.0.0.1]) by mail97-va3-R.bigfish.com (Postfix) with ESMTP id 6EFC9AA0138 for me; Tue, 15 Sep 2009 20:58:09 +0000 (UTC)
Received: by mail97-va3 (MessageSwitch) id 12530482889694_15241; Tue, 15 Sep 2009 20:58:08 +0000 (UCT)
Received: from monroe.provo.novell.com (monroe.provo.novell.com [137.65.250.171]) by mail97-va3.bigfish.com (Postfix) with ESMTP id 5F7101A58056 for me; Tue, 15 Sep 2009 20:58:07 +0000 (UTC)
Received: from soval.provo.novell.com ([137.65.250.5]) by
monroe.provo.novell.com with ESMTP; Tue, 15 Sep 2009 14:57:58 -0600
Received: from bugzilla.novell.com (localhost [127.0.0.1]) by soval.provo.novell.com (Postfix) with ESMTP id A56EECC7CE for me; Tue, 15 Sep 2009 14:57:58 -0600 (MDT)
For those who haven't read these kinds of headers before, read from the bottom up. The mail flow is:
  1. Originating server was Bugzilla.novell.com, which mailed to...
  2. soval.provo.novell.com running Postfix, who forwarded it on to Novell's outbound mailer...
  3. monroe.provo.novell.com, who attempted to send to us and sent to the server listed in our MX record...
  4. mail97-va3.bigfish.com running Postfix, who forwarded it on to another mailer on the same machine...
  5. mail97-ca3-r running something called MessageSwitch, who sent it on to the internal server we set up...
  6. exchedge2.cms.wwu.edu running Exchange 2007, who send it on to the Client Access Server...
  7. exchhubca1.univ.dir.wwu.edu for 'terminal delivery'. Actually it went on to one of the Mailbox servers, but that doesn't leave a record in the SMTP headers.
Why is this unusual? Because steps 4 and 5 are at Microsoft's Hosted ForeFront mail security service. The perceptive will notice that step 4 indicates that the server is running Postfix.

Postfix. On a Microsoft server. Hur hur hur.

Keep in mind that Microsoft purchased the ForeFront product line lock stock and barrel. If that company had been using non-MS products as part of their primary message flow, then Microsoft probably kept that up. Next versions just might move to more explicitly MS-branded servers. Or not, you never know. Microsoft has been making placating notes towards Open Source lately. They may keep it.

Exchange Transport Rules, update

| 2 Comments
Remember this from a month ago? As threatened in that post I did go ahead and call Microsoft. To my great pleasure, they were able to reproduce this problem on their side. I've been getting periodic updates from them as they work through the problem. I went through a few cycles of this during the month:

MS Tech: Ahah! We have found the correct regex recipe. This is what it is.
Me: Let's try it out shall we?
MS Tech: Absolutely! Do you mind if we open up an Easy Assist session?
Me: Sure. [does so. Opens sends a few messages through, finds an edge case that the supplied regex doesn't handle]. Looks like we're not there yet in this edge case.
MS Tech: Indeed. Let me try some more things out in the lab and get back to you.

They've finally come up with a set of rules to match this text definition: "Match any X-SpamScore header with a signed integer value between 15 and 30".

Reading the KB article on this you'd think these ORed patterns would match:
^1(5|6|7|8|9)$
^2\d$
^30$
But you'd be wrong. The rule that actually works is:
(^1(5$|6$|7$|8$|9$))|(^2(\d$))|(^3(0$))
Except if ^-
Yes, that 'except if' is actually needed, even though the first rule should never match a negative value. You really need to have the $ inside the parens for the first statement, or it doesn't match right; this won't work: ^1(5|6|7|8|9)$. The same goes for the second statement with the \d$ constructor. The last statement doesn't need the 0$ in parens, but is there to match the pattern of the previous two statements of having the $ in the paren.

Riiiiiight.

In the end, regexes in Exchange 2007 Transport Rules are still broken, but they can be made to work if you pound on them enough. We will not be using them because they are broken, and when Microsoft gets around to fixing them the hack-ass recipes we cook up will probably break at that time as well. A simple value list is what we're using right now, and it works well for 16-30. It doesn't scale as well for 31+, but there does seem to be a ceiling on what X-SpamScore can be set to.

Exchange transport-rules

| 1 Comment
Exchange 2007 supports a limited set of Regular Expressions in its transport-rules. The Microsoft technet page describing them is here. Unfortunately, I believe I've stumbled into a bug. We've recently migrated our AntiSpam to ForeFront. And part what ForeFront does is header markup. There is a Spamminess number in the header:
X-SpamScore: 66
That ranges from deeply negative to over a hundred. With this we can structure transport-rules to handle spammy email. In theory, the following trio of regexes should catch anything with a score of 15 or higher:
^1(5|6|7|8|9)$
^(2|3|4|5|6|7|8|9)\d$
^\d\d\d$
Those of you that speak Unix regex are quirking an eyebrow at that, I know. Like I said, Microsoft didn't do the full Unix regex treatment. The "\d" flag, "matches any single numeric digit." The parenthetical portion, "Parentheses act as grouping delimiters," and, "The pipe ( | ) character performs an OR function."

Unfortunately, for reasons that do not match the documentation the above trio of regexes is returning true on this:
X-SpamScore: 5
It's the second recipe that's doing it, and it looks to be the combination of paren and \d that's the problem. For instance, the following rule:
^\d(6|7)$
returns true for any single numeric value, but returns false for "56". Where this rule:
^5(6|7)
only returns true for 56 and 57. To me this says there is some kind of interaction going on between the \d and the () constructors that's causing it to change behavior. I'll be calling Microsoft to see if this is working as designed and just not documented correctly, or a true bug.
We missed a step or something in decommissioning our Exchange2003 servers. As a result, we have a whole lot of... stuff going 'unresolvable' due to how Outlook and Exchange work. There is an attribute on users and group called LegacyExchangeDN. Several processes store this value as the DN of the object. If that object was created in Exchange 2003 (or earlier) it's set to a location that no longer exists.

The fix is to add an X500 address to the object. That way, when the resolver attempts to resolve that DN it'll turn up the real object. So how do you add an X500 address to over 5000 objects? Powershell!

$TargetList = get-distributiongroup grp.*

foreach ($target in $TargetList) {
$DN=$target.SamAccountName
$Leg=$target.LegacyExchangeDN
$Email=$target.EmailAddresses
$Has500=0

if ($leg -eq "/O=WWU/OU=WWU/cn=Recipients/cn=$DN") {
foreach ($addy in $Email) {
if ($addy -eq [Microsoft.Exchange.Data.CustomProxyAddress]("X500:" + $Leg)) {
$Has500=1
}
}
if ($Has500 -eq 0) {
$Email += [Microsoft.Exchange.Data.CustomProxyAddress]("X500:" + $Leg)
$target.EmailAddresses = $Email
set-distributiongroup -instance $target
write-host "$DN had X500 added" -foregroundcolor cyan
} else {
write-host "$DN already had X500 address" -foregroundcolor red
}
} else {
write-host "$DN has correct LegacyExchangeDN" -foregroundcolor yellow
}
}


It's customized for our environment, but it should be obvious where you'd need to change things to get it working for you. When doing users, use "get-mailbox" and "set-mailbox" instead of "get-distributiongroup" and "set-distributiongroup". It's surprisingly fast.

My contribution to the community!

ForeFront and spam

They have an option to set a custom X-header for indicating spam. The other options are subject-line markup and quarantine on the ForeFront servers. What they never document is what they set the header to. As it happens, if the message is spam it gets set like this:
X-WWU-JunkIt: This message appears to be spam.
Very basic. And not documented. Now that we know what it looks like we can create a Transport Rule that'll direct such mail to the junk folder in Outlook. Handy!

Outlook for everything

| 2 Comments
Back when I worked in a GroupWise shop, we'd get the occasional request from end users to see if Outlook could be used against the GroupWise back-end. Back in those days there was a MAPI plug-in for Outlook that allowed it to talk natively to GroupWise and it went rather well. Then Microsoft made some changes, Novell made some changes, and the plug-in broke. GW Admins still remember the plug-in, because it allowed a GroupWise back end to look exactly like and Exchange back end to the end users.

Through the grapevine I've heard tales of Exchange to GroupWise migrations almost coming to a halt when it came time to pry Outlook out of the hands of certain highly placed noisy executives. The MAPI plugin was very useful in quieting them down. Also, some PDA sync software (this WAS a while ago) only worked with Outlook, and using the MAPI plugin was a way to allow a sync between your PDA and GroupWise. I haven't checked if there is something equivalent in the modern GW8 era.

It seems like Google has figured this out. A plugin for Outlook that'll talk to Google Apps directly. It'll allow the die-hard Outlook users to still keep using the product they love.