January 2012 Archives

Judicial rubber-hoses

The other day a Colorado court ordered a defendant to produce the unencrypted contents of their own laptop. This is what I called "rubber hose cryptography", and previously we've heard of efforts in the UK to compel decryption. It has now happened here, and not at the US border. Unlike the UK, this decryption demand in Colorado is not based on a law that specifically says that courts can demand this.

Wired article

The counter-argument is quite clearly the 5th amendment right guaranteeing the ability to not self-incriminate. If that decryption key only exists in your head, and disclosing it would incriminate you, then you don't have to yield the key.

This judge disagreed. I'm not a lawyer, so I can't tell what legal hairs were split to come to this decision. But the fact remains that this judgment stands. The only concession he appears to have made for the defendant is to preclude the prosecution from using the act of disclosure as a 'confession', but the data yielded by the disclosure is still admissible.

Moderator endorsements

As promised, my endorsements.

for Iain at Server Fault, Q&A for system administrators and
desktop support professionals
Meta Activity
Not the most active meta-denizen, but Iain's contributions are always well thought out and with an eye to the community. Importantly, I agree with most of what he says that he'll keep the moderating consistent. Everyone likes an even hand, especially those of us who have to defend what we do.

Flag Activity
Iain's is ServerFault's only person with a Marshal badge, and I handled a large chunk of the flags needed to get there (our schedules overlapped, it was a rather hectic September and October). His flags nearly always get actioned. I have very high confidence that Iain has a sound understanding of what does and does not need moderator attention, and should have an easy time getting on board.

Conflict Management
Iain has participated in several of the conflagrations we've had in the past year, and he has always been on the side of sane. He has been baited, and failed to take the bait. I am completely satisfied that he can handle the job calmly and professionally, and

Iain has earned the gold Copy Editor, Sportsmanship, and Yearling (twice!). This is a person who is engaged and active.

I look forward to working with Iain, and think he is a most excellent fit to the moderator team.

for Ben Pilbrow at Server Fault, Q&A for system
administrators and desktop support professionals
Meta Activity
Ben has always been an active member of Meta, and I've found his contributions well worth the read. His contributions are very frequently in line with the emerging community standard, which shows both that he has a solid sense of the community and knows it intuitively. Very valuable for a moderator charged with enforcing those standards.

Flag Activity
When flag-rank was introduced, Ben already had one of the highest ranks on the site. Additionally, most of the flags of his I've handled have been actionable. Ben's flagging, and his close voting back in my pre-mod days, show a strong understanding of the culture of ServerFault.

Conflict Management
Ben doesn't usually get involved in the messy stuff, but the few times he has his voice has been calm and reasoned. Watching his discussions over the years shows he is level headed, and can take debate in many forms.

Ben has also earned the gold Copy Editor badge, and Yearling.

Ben was my #3 choice in last year's election, even though he had just broken 3K, and I think he has had some of the rough edges worn off since then. In the face of a strong candidate field, I believe he is worthy of the diamond.

My third choice presented me with problems. After viewing the candidates nomination statements, Town Hall Q/A, and the other information available to me, I find myself deeply conflicted. The two candidates it came down to were one who is very well versed in the culture of ServerFault, has good judgement, and is stable in conflict, and another one who is not well known on ServerFault but is well known to StackExchange moderators. One candidate who knows the culture cold and would be a valuable addition to the team, and another candidate who knows how StackExchange moderators are supposed to moderate and is a member of our community albeit somewhat in the background.

Because of this, I'm going to present both candidates. Unlike Ben and Iain, I won't be doing a full breakdown. I'll be describing why they're worthy of your vote.

profile for freiheit at Server Fault, Q&A for
system administrators and desktop support professionals
Freiheit is known to me as a moderator on some other StackExchange site. Bicycles. I know of him because he's frequently in the Teachers Room chat when I am. This is a person who knows how Stack Exchange moderators are supposed to moderate.

His ServerFault record shows participation, though not to the extent of many of the other candidates. His flagging history is light, but he's yet to flag incorrectly. His meta-participation is similarly light. His main site participation is spot on though. While he may be big in bicycles, he knows of what he's talking about when it comes to professional systems administration.

In spite of his light participation in ServerFault (well, light meta-participation anyway) I believe he is worthy of my #3 spot because his demonstrated theory of moderation and how he handles conflict would make him a very low friction addition to the ServerFault moderation team.

I do not hold his mod-bit on another site against him. We've done that before and benefited from it. ServerFault can be quite insular at times, and bringing in someone more widely involved in StackExchange would be very good for us.
profile for Chris S at Server Fault, Q&A for
system administrators and desktop support professionals
Chris S has been involved in ServerFault since practically the beginning. What's more, he has been active in meta-activities since then as well. His flagging is high quality. He is not a big editor, but he is big on tag organization. And he shows a level head when tempers start to flair.

Unlike my first two picks, Chris's main active times overlap with my own. This will provide another valued pair of eyes on the flag-queue during what can be busy times.

Chris's vision of what ServerFault is about aligns closely with my own. The discussions he's participated in on meta.sf have been well thought out. His reasoned approach should mean fewer hotly contested mod-actions, something all mods can appreciate after they've been in the seat a while.

Chris is worthy of the ServerFault moderation team.
One of the tasks I have is to build some separation between the production environment and test/integration environments. As we approach that Very Big Deadline I've talked about, this is a pretty significant task. Dev processes that we've been using for a really long time now need to change in order to provide the separation we need, and that's some tricky stuff.

One of the ways I plan on accomplishing this is through the use of Environments in Puppet. This looks like a great way to allow automation while delivering different code to different environments. Nifty keen stuff.

But it requires that I re-engineer our current Puppet environment, which is a single monolithic all-or-nothing repo to accommodate this. This is fairly straight-forward, as suggested by the Puppet Docs:

If the values of any settings in puppet.conf reference the $environment variable (like modulepath = $confdir/environments/$environment/modules:$confdir/modules, for example), the agent's environment will be interpolated into them.

So if I need to deliver a different set of files to production, such as a different authorized_keys file, I can drop that into $confdir/environments/production/modules and all will be swell.

So I tried to set that up. I made the environment directory, changed the puppet.conf file to reflect that, made an empty module directory for the module that'd get the changed files (but not put the files in there yet), and ran a client against it.

It didn't work so well.

The error I was getting on the puppetmaster was err: Could not find class custommod for clunod-wk0130.sub.example.local at /etc/puppet/manifests/site.pp:84 on node clunod-wk0130.sub.example.local

I checked eight ways from Sunday for spelling mistakes, bracket errors, and other such syntax problems but could not find out why it was having trouble locating the custommod directory. I posted on ServerFault for some added help, but didn't get much beyond, "Huh, it SHOULD work that way," which wasn't that helpful.

I decided that I needed to debug where it was searching for modules since it manifestly wasn't seeing the one staring it in the face. For this I used strace, which produced an immense logfile. Grepping out the 'stat' calls as it checked the import statements, I noticed something distinctly odd.


It was attempting to read in the init.pp file for the custommod, even though there wasn't a file there. And what's more, it wasn't then re-trying under /etc/puppet/modules/custommod/manifests/init.pp. Clearly, the modulepath statement does not work like the $PATH variable in bash and every other shell I've used. In those, if it fails to find a command in (for example) /usr/local/bin, it'll then try /usr/bin, and then ~/bin until it finds the file.

The modulepath statement in puppet.conf is focused on modules, not individual files.

This is a false cognate. By having that empty module in the production environment, I was in effect telling Puppet that the Production environment doesn't have that module. Also, my grand plan to provide a different authorized_keys file in the Production environment requires me to have a complete module for the Production environment directory, not just the one or two files I want changed.

Good to know.

What I look for in a moderator

Since ServerFault is holding community-moderator elections, I figured I'd weigh in with my opinion of what I look for in a mod-candidate. I've been at this a little under a year now, and have a vastly better understanding of the challenges a mod faces.

Some of what I look for can be found in the numbers, flag-rank, meta-activity, chat activity, what badges they've earned, but others can only be learned by watching them.

Meta Activity
As a moderator I have to keep an eye on meta. So much so that any time someone posts a question there, I get notified in my system inbox. What this means is that I read just about every Meta question that comes up. A good mod-candidate will be familiar to me from those questions and answers, since it shows they are actively interested in how the site runs and how to make it better.

Flag Activity
Since a very large part of my job is handling mod-flags, a good mod-candidate should be both familiar to me, and known to be trustworthy with the flagging. A high flag-rank is a fair proxy for this, though there are some exceptions. A better but nigh impossible to produce statistic is the flags to actioned-flags ratio, since a higher 'actioned' number suggests the flagger has a very good grasp of what really needs action. It is quite possible to have a high flag rank and have only a small number of them actioned because the flags are on the wrong side of the edge-case, but not off enough to incur the Nose-bop of Unhelpful.

Good at Conflict Management
Since the moderators get involved whenever there is a temper flare-up there are some users that I routinely see in those. Some of them are there urging the combatants on, if not starting the fight. Others are de-escalating. A good mod candidate will be in there de-escalating, or otherwise attempting to turn the tempers down.

This is especially important since it is guaranteed that some decision you make as a mod will be second-guessed by someone (why was my question closed is a popular one). And sometimes, the person doing the second-guessing is someone you consider to be barely worthy of being on the site, or is edging towards that indignant stridency that encourages a heated response. A good mod-candidate will have proven to me that they can take criticism from any quarter without responding with heat.

There are certain badges that are interesting to me, since they touch on engagement, interest in the community, and judgment:
  • Strunk & White / Copy Editor -- Because editing content is important, and I was big on that before I got the diamond.
  • Yearling -- Because they've been around long enough absorb the culture.
  • Sportsmanship -- Because they vote up good answers, even if they're competing with their own.
  • Badges on Meta -- Since you can earn badges there too.

A lot of the above are things that I'm well positioned to see. There is a reason that the Deputy and Marshal badges aren't on the list, and that's because I'm the one handling the flags needed to get those badges. Conflict management is something I'm in the middle of, so I see the action more than most. Meta-monitoring is an ongoing thing, and hard to really get a feel for when browsing the archives after the fact.

I do plan on endorsing candidates, since I do have a vested interest in who I work with. However, I won't post that until after the primary. If we stay at 10 candidates we'll go straight to the general, if so I'll post my endorsements at that time.

And now a word about timeliness

I mentioned in the last post:

...since we're at a part of the product deployment schedule where hours count.
Yes, we are.

We're not yet at the WORKING EVERY WAKING MOMENT AND WHEN WE SLEEP WE'RE DREAMING ABOUT BEING GUILTY FOR NOT WORKING part of the cycle, but it's coming. And this, fair reader, is where you come in. Blogging, and ServerFault answering, are a category of activity that'll be greatly reduced in the coming weeks until that big dev milestone is behind us.

There may be some humorous posts where I blow some steam, but big thinky posts are not likely. I'll still be moderating on SF, since I was elected for that and I do have a duty to the community to keep up with it. I apologize in advance for that, but it's the business cycle.

Carry on.

When there is no clean back-out

| 1 Comment
Today I ended up banging my head on a particularly truculent wall. A certain clustering solution just wasn't doing the deed. No matter how I cut, sliced, sawed or hacked it wouldn't come to life. I wanted my two headed beast! Most vexing.

Then I hit that point where I said unto myself (I hope), "!#^#^ it, I'm reformatting these two and starting from scratch."

So I did.

It worked first time.


Ok, I'll take it.

Mind, all the cutting and slicing and sawing and hacking was with me knowing full well that iterative changes like that can stack unknowingly, and I was backing out the changes when I determined that they weren't working. Clearly, my back-out methods weren't clean enough. Okay, food for thought; dirty uninstallers are hardly an unusual thing.

Of course I kicked myself for not trying that earlier, but during the course of aforementioned reanimatory activity I did learn a fair amount about this particular clustering technology. Not a complete loss. I do wish I had those 4 hours back, since we're at a part of the product deployment schedule where hours count.

A year of stats

2011 was a busy year. I changed jobs and became a moderator on ServerFault, both of which impacted my blogging activity. The former more than the later due to greatly reduced incidences of boredom. What kind of traffic did I get last year? Not a lot, but enough.

  • 40K pageviews over the year.
  • 20K separate visitors, clearly I get a lot of search traffic.
  • 1.1M feed-hits.
  • 412K feed-article views. Clearly most of my readers are using, er, readers.
  • Busiest day: January 27 (I made reddit)
  • Max feed-subscribers: 450
  • Max feed-hits on a day: 1,295

Top Content, Web

  1. The Linux Boot Process, a chart
  2. LIO-Target on OpenSUSE
  3. Powershell and ODBC
  4. Reverting LVM Snapshots
  5. Sysadmin Best Practices

Top Referring Sites

  1. Reddit (nearly all of it for the boot-process chart)
  2. Google (thank you search)
  3. Stumbleupon (more boot-process chart, but a few others as well)
  4. Planet Sysadmin (an RSS aggregator of a bunch of sysadmin blogs)
  5. ServerFault (some from questions, more from links in the chat-room. Hi guys).

Is this the big-time? Hardly. Making reddit that one time added about 40 subscribers. Leaving WWU lost me about 50 subscribers, which I've since gained back with interest for various reasons.

This does represent growth over 2010, which I'm quite OK with. Blogging for a for-profit company that has secret-sauce to protect and I'm also working on has constrained what I can talk about here, which is why ServerFault content has been more prominent of late. However, 2012 is the year where some of what I'm working on will be let out the door to soar (or crater), so hopefully I'll be able to talk more about that stuff once it's out.

Meta-commentary on sysadminly things in general will continue, though. 

Happy New Year!