April 2011 Archives

In the beginning, there was Slackware. And lo, was it good. The kernels did compile, and there were software packages aplenty. For the rest, there was 'make'.

And then there came RedHat with support contract in hand, promising both stability and accountability. Business distrust of 'free' was assuaged by accountability, and the quiet uptake began. To better manage their profit-generating systems, RedHat begat RPM.

And so did professional geekdom learn to use RPM, for with it came pay-checks. And with the engines of commerce running RedHat, the major software vendors did release their software packaged with RPM. RPM being a robust solution, other Linux projects did adopt RPM as their own package manager.

But quiet as a mouse Canonical did release a distribution that users found easy and simple. It just worked, they cried. Their DEB package manager did match RPM in both flexibility and scalability. Professional geekdom did install it at home, or ignored it to their peril.

Time passed, and more private users were using Ubuntu than any other distribution. Professional geekdom did stop paying RedHat support contracts for large installs, preferring instead to use the free CentOS. 'Patches-only!' being all that professional geekdom deemed needed for most scalable web infrastructures.

Quietly beneath senior professional geekdom a whole generation of new geeks had come of age under Ubuntu and DEB and not Fedora and RPM. Canonical did release a Long Term Support version of their distribution, designed specifically to provide code stability for several years and did also provide the option of a support contract which did supply accountability.

The frothy churn of startups saw Ubuntu, not CentOS in use far more often than even some years before. So much so, that professional geekdom, in their inter-career travels, ran into it more and more. What worked for RPM did not work for DEB, though they did solve the same problem. As with all startup churn, some few did make it big with their Ubuntu infrastructure supporting them.

And so did DEB come of age in the ranks of professional geekdom.

Some time in the last 15 or so years the closer on a "help me please" email moved from "any ideas?" to "any thoughts?". I have no idea what has driven this shift in usage, but it is definitely there. And for whatever reason, it drives me crazy. I was never much of a fan of "any ideas?" but "any thoughts?" is the same but worse.

When the BOFH in me gets ascendent I answer such emails in the way they're written, an entreaty to participate in brain-storming, rather than how they're meant, an plea to come up with action-items that will fix the problem. The BOFH is not safe for public consumption and always gives IT people a bad name, which is why I try to repress it whenever possible. I'm not always successful, but I try.

"Any ideas?", for whatever reason, means more of a plea for resolution than "Any thoughts?" does to me. Both are frequently used as the closing sentence of a trouble report, and are frequently superfluous; the act of submitting a help request is an implicit, you know, cry for help. I see this a lot on ServerFault, but it also shows up in internal correspondence with me at my workplace.

Good:

My remote-desktop sessions keep crashing. They never last more than 2 minutes for some reason. This is new as of yesterday.

This needs to get fixed.

Bad:

My remote-desktop sessions keep crashing. They never last more than 2 minutes for some reason. This is new as of yesterday.

Any ideas?

ARRRG:

Hi,

My remote-desktop sessions keep crashing. They never last more than 2 minutes for some reason. This is new as of yesterday.

Any thoughts?

Thanks in advance.

All three need more information before supplying a probable answer. The first is most likely to get me digging into a problem directly. The second is more likely to cause me to reply with a fault-tree for following to the ultimate cause. The last is more likely to cause me to throw an off the top of my head answer ("network problems can cause resets") rather than actually fix the problem.

Some people have a hard time with declaratives and feel much more comfortable ending with a question. Still others, like many on ServerFault, feel the grammatical need to wrap their trouble report in a "Hello/Thanks" block. And then there are those who don't like to presume upon my time for their silly little problem and try to blunt the force of the need for help by just asking for some tips.

But still, for me "any thoughts?" is the dripping faucet in the night of my problem-fixing day. Knowing why people use it helps a little, but when the veneer of civilization rubs off and the server troll peeks through I become... less helpful.

Yesterday we were caught up in the Media Temple DNS server outage. Details about what happened are found here. We were out name resolution for two and a half hours. Paying customers couldn't get at our hosting, and our email stopped flowing.

This is not an outage I'd dealt with before, as both WWU and my earlier job self-hosted their DNS servers. WWU had one off campus, but the first two records were on campus.

This was a case of, "a lot of fire, nothing I can do about it," which is one of the hardest major events to endure as one of the prime fire-fighters. Happily, people figured out fairly quickly that this was an external event we had no control over. Until that time, the statusing was pretty intense.

And unfortunately, there is nothing much you can do about a DNS domain outage. The 24-48 hour DNS propagation time pretty much shoots down any short term quick-fixes. I went several rounds with a couple people here about ways to work around the problem and there was much educating going on as a result. We did give some end-users the IP address of the servers in question. And then the SSL based services stopped working, of course.

As with any major failure of a system you never thought to think about before, there is discussion about moving our DNS to somewhere else. We'll see if this holds up.

A nice article over at The Economist covers email disclaimers.

The long and short of it? No case anyone can remember ever hinged on them being there. And in Europe at least, such unilateral contracts are completely unenforceable. They're useless.

In my time in the computer industry, or heck anyone who does email professionally, I've seen some doozies. One memorable disclaimer had six paragraphs and lined out in detail the exact constraints placed on how that email shall be handled, the relationship established between me and the company, and expectations set by receiving such. All for an email that said, "I won't be there tonight. Maybe next month."

Clearly the person sending the email had never read their own disclaimer, since this was on a mailing list for non-work activity and that was clearly excluded by the disclaimer.

Which just goes to show, no one reads the damned things.

Today I ran into a limitation of robocopy that just might have bitten me in the past and I never knew it.

I got pulled aside to ask my opinion of how a robocopy of a large filesystem yielded no errors in the robocopy log, but differed in file-count by 6 files. The user was able to identify a directory with a file that got missed, which helped. The files in the directory were named similar to this:

1. ITWA-National Introduction.doc
2. ITWA-National Overview.doc
3. ITWANA~2.DOC

The destination directory had only the first two files, but most interestingly, the second file had the same size as the third file on the source. Hmmmm.

Looking at the short-names of the two directories I noticed something else. In the source directory...

ITWA-National Introduction.doc = ITWANA~1.DOC
ITWA-National Overview.doc = ITWANA~3.DOC
ITWANA~2.DOC

And on the destination directory...

ITWA-National Introduction.doc = ITWANA~1.DOC
ITWA-National Overview.doc = ITWANA~2.DOC

AHA! Robocopy isn't preserving the short-names! When it copied the second document to the destination it allowed the system to auto-generate the short-name, and as that was the second ITWANA document, it got ~2. So when it copied ITWANA~2.DOC in, it overwrote the already existing ITWANA~2.DOC file.

We're now looking for a tool, preferably scriptable, that'll preserve short-names. I found a couple of 'em, but they're !free. Currently we're laundering things through the Windows backup facility since that preserves ALL attributes. If you know of such a tool, drop a comment.

This will be old hat to some of you, but hey, I'm running into it for the first time.

Having spent 14-odd years in civil service and the career-type people that tends to attract (i.e. ones that stay and grow roots, deep roots), coming to what's effectively a startup is a major culture shift. One of the more distinctive changes is in the hours people work. We work on what I'm calling Startup Standard Time, which is in at 9am or so, out by 6pm or so, mostly.

Also, I'm one of the oldest people in the company, which is a shock. At WWU I was the youngest for my entire career there. At my older job the only people younger than me were some of our helpdesk techs.

At both prior jobs there were people who were at the office by 7am. At WWU that was my office-mate. At the earlier job, it was a couple of people, some of whom were rarely arrived later than 6:30am.

At WWU, my arrival at 8:10 or so made the the late-arriver on the Windows side, though it ties me for the earliest arriver on the Linux side (2 of the 3 on that side of the office generally got there about noon and stayed until 8pm).

Here, I don't think anyone is there at 8am other than the person who answers the official phone. The one time I got there at 8:45 I was the third one in the office, and one of the other two had clearly just walked in. And yet, by 9:30 everyone is there.

I suspect some of the early arrivals has to do with the impact of children on the sleep cycle. Once kids reach the age where they have to be at school by 7:30am (if not earlier for extra-curriculars or "zero-hour" classes), it makes parents get up earlier as well. As it happens, the transition from High School (up at ungodly early hours) to college (no longer in the house) means that, well, you're still getting up early even if there is no reason for it, so may as well go to work and get home early.

Also, arriving at 7am means you can leave at 3:30pm, which should be in enough time to catch the kid's sporting events without having to take vacation time.

Some of my co-workers have kids, or are working on having them. None of them have had 'em long enough to get to High School age, or even sporting age, yet. At WWU, three of my six co-workers had kids, two of whom had at least one child in the process of hunting up college or going there already; unsurprisingly, they were both early-arrivers. The third had just spawned child number two, and bossman may have had grandkids already so we ran the whole range.

The late departure also means we're eating dinner later, much later than normal. Still getting used to that.

Today I did something I never did at WWU, interviewed a potential intern. Yes, Logik is looking for a pair of interns this summer. If you're in the DC area, or know a CompSci student who is, we're looking. Click the link for details.

This particular intern was focused more on the "IT" side of the operation, rather than the dev side. His resume and associated materials were pretty clear in that he was focusing on eventually entering the job market as a system administrator of some type (there are many types). The place of education he was at was focused on producing people like that, which is encouraging, and from what he was describing of his education it was actually pretty close to where I'd like it to be.

Since this person, or one a lot like him, would be following me around this summer, it got me thinking about what I'd like to see in such a person. Logik is small enough and our product focused enough that 'helpdesk grunt' is not a job title we have any use for. Any person coming in on the IT side will be doing hard labor as well as helping make decisions. I don't think there is enough work for another me around here, but there will be plenty of work for me and a minion I'm mentoring.

For that's what it would be, mentoring. There is a mindset I look for in sysadmin-type people that's hard to teach, though this particular school sounds like they're doing a solid job of attempting to train it in. Most of us get it on-the-job, but a good amount comes with the person. He had a couple of prior internships that didn't challenge him in the right ways, and the description of those jobs and what he hopes to get out of this possibility were more clues that he had the beginnings of the right stuff.

I've worked with student workers at WWU who have the beginnings of this mindset and just need some years of polishing to be right in the sweet spot. For those student workers, rubbing elbows and working along side the grizzled veterans (at 12+ years in the industry I merit some minor grizzle now) is highly valuable experience. A summer internship with pretty close one-on-one interaction with said GV, in our startup-like environment, will be a really big leg up for a guy like the one we interviewed today.

And finally, it was an interesting continuity to interview a student for an internship, since I just spent the last 7 years working for a university. Heh.

Years ago an old co-worker quipped,

The day you wear a skirt and pantyhose to work is the day you end up crawling around under the raised floor.

The same applies to menfolk, just with the easy-to-scuff shoes and nice pants. There is truth to this.

However, with my new job I'm now for the first time working for some place that doesn't have their own dedicated datacenter and uses a professional colocation facility for all the really nifty stuff. The side effect of this is that I no longer  have to deal with a raised floor. I do still have to deal with a ladder (for some reason the main office network switch is mounted above the door in the wiring closet), but running things under the floor is no longer something I have/get to do.

Also, that was my first visit to a professional colo facility and I am deeply impressed. The WWU datacenter I used to work with could fit into 2-3 of those cages, and that's a small amount of that vast, vast facility. So. Cool.

Back to clothing... this being a startup-style company, dress-code tends to the 'allowable to get dirty' end of the spectrum. My title is such that I just might be called on to meet with potential big clients to reassure them that we can meet their needs, but that's almost always going to be over the phone. We'll see how the office dress-code shifts when the heat index starts creeping into the high 90's and higher. I doubt I'll need to keep a pair of Emergency Pants in the office just in case some high-spending person drops by.

Yes, I left WWU. And yes, I've been cagey about where I'm going. I didn't know if the new place wanted the scoop on announcing where I went. Well, they didn't, so I get to do it.

I'm now working for Logik, the eDiscovery people. WWU regularly hands over Outlook archives and other files to people filing Public Records Requests. Logik is the kind of firm that those people give WWU's data to for indexing, deduplication, and isolating the interesting bits. This is a job that legions of law clerks used to do, and now automation is starting to do that work as well.

This is an interesting place. They hired me because they wanted a professional IT point of view in their operations, and that's me in spades. It's a much smaller operation than I'm used to, but unlike WWU it's revenue generating and fairly homogeneous. I expect to exercise my scale-out skills in a way that WWU never really asked of me.