January 2011 Archives

The growth of StackExchange

StackExchange is a project put on by the same people who brought you StackOverflow. It has been no secret that they've been trying to make the StackOverflow lighting strike twice, and they've made some big strides since their first attempts with ServerFault and SuperUser. They've instituted a community-driven process for proposing new sites and eventually running them.

They've been around long enough that there is a good list of new permanent sites out there. A selection:

  • askubuntu.com : Q/A for Ubuntu. StackOverflow inc worked out a deal with Canonical to do this.
  • gaming : Q/A for Games, be they console or PC. There is heavy competition for this space, so we'll see how it survives.
  • webapplications : Q/A for users of web-applications like Google Apps, BPOS, twitter, greasemonkey scripts for browsers, and facebook.
  • apple : Q/A for the Apple platform. Most of this is handled on SuperUser these days, I expect that traffic to start migrating over soon.
Interestingly, they do have some topical sites of interest to academic types!

  • cstheory : Q/A surrounding theoretical CompSci. 
  • mathematics : Q/A for people who do math for a living. We're not talking helping solve algebra homework, we're talking the kind of things people who teach undergrads would be interested in.
  • english : Q/A for linguists in the English language. For the serious grammar nerd.
Lurking in the beta process are a few more items of interest to academic types.

  • gis : Q/A for geospatial engineers. We have some of those 'round these parts.
  • physics : Q/A for professional physics people and interested parties.

RDP from Linux to Windows

One of the biggest frustrations with attempting to use RDP to talk to Windows is that you can't use Network Level Authentication for RDP with Vista/Win7/Server2008. Since that comes turned on by default, you have to de-secure your Windows machine to allow connections from Linux. The 'rdesktop' project appears dead.

There is hope! FreeRDP is a branch of rdesktop that is actively being developed. In fact, three days ago they announced that they had cracked the NLA problem and are testing builds. This is devel, but it's progress!

The Linux boot process, a chart

| 1 Comment
The Linux boot process, a branching chart. For the most part, the levels do not depend on column. LILO does not require EFI, nor does GRUB require BIOS. EFI and BIOS support are now included in both bootloaders. Yes, there is some simplification here. EFI is very featured, but hasn't been commonly exercised to its full extent. Eventually EFI could possibly replace GRUB/LILO, but that day hasn't come yet.

Pre-Boot Enviroinment
  • POST
  • Read bootable media
  • Load Master Boot Record
  • Execute MBR
  • POST
  • Read bootable media
  • Load the GPT table
  • Mount the EFI system-partition
  • Load EFI-drivers for the system
  • Execute MBR
Bootstrap Environment
  • Stage 1 loaded into MBR/EFI and gets executed by BIOS/EFI
  • Stage 1.5 loaded by Stage 1, including critical drivers
  • Stage 2, in the boot filesystem, executes
  • Stage 2 loads the kernel
  • Stage 1 loaded into MBR/EFI and gets executed by BIOS/EFI
  • Load first sector of core.img
  • Continues loading core.img
  • Loads GRUB config
  • Loads the kernel
  • Stage1 loaded into MBR (or EFI by ELILO) and executed by BIOS/EFI
  • Stage 1 loads the first cluster of Stage 2, and executes.
  • Loads LILO information.
  • Loads the kernel
Kernel Load
  • The kernel uncompresses into memory
  • If configured, the kernel mounts the Initial Ramdisk, which contains needed modules to load the rest of the OS
  • Mounts the root filesystem, loading any needed modules from initrd
  • Swaps / from initrd to the actual root filesystem
  • Executes the specified init process
UID 1 process
  • Checks /etc/inittab for loading procedures
  • Runs scripts specified by inittab
    • Mounts needed filesystems
    • Loads needed modules
    • Starts needed services based on runlevel
    • Finishes setting up userspace
  • Reads /etc/system.conf
  • Mounts needed filesystems
  • Loads needed modules
  • Starts services as needed
  • Runs startup events listed in /etc/events.d based on runlevel.
  • Loads needed modules
  • Mounts needed filesystems
  • Starts needed services
  • Reads /etc/launchd.conf for config details
  • Reads /etc/launchd.plist for per-driver/service details

I did this because Things have Changed from the last time I really studied this. Back when I started it was BIOS, LILO, and Initd. I never did bother to wrap my head around Grub, mostly because the automatic tools have gotten a lot better so knowing it just to install isn't needed, and I haven't had the right kind of boot problems where I'd learn it through troubleshooting. I've also yet to run into EFI in the wild (I think...). Now that OpenSUSE is actively considering moving over to SystemD and Ubuntu having thrown initd over the side some time ago in favor of Upstart, it's time to revisit.

I'm still fuzzy on SystemD so that's probably wrong.

Changing student storage habits

I had to do some maintenance on my script that gathers disk-space usage, so the stats database has been on my mind lately. It's been a while since I posted any graphs. This particular graph is a unified chart of the student home-directory volumes over time. I merged the NetWare and Windows volumes into a single space-used chart.

This is a very noisy chart.The discontinuities are mostly student-account-purge events that happen once a quarter, but the fall purge is by far the largest.

Note the downward tail at the end! The same chart for staff is a pretty smooth line straight up at a pretty steady slope. This? Clearly usage-habits are changing. I don't know if this is reflected by habitual USB-drive use or if they're using the cloud in some way to store their files, but clearly student-driven storage demand (at least for home-directories) is falling.

One area where it is clearly increasing is the Blackboard Content volume.
This data is noisy in that we purge old courses, but we've also changed how many quarters of courses we keep in the system. Looking at this growth chart, it's pretty clear to me that the downtick in student home-directory and class-volume consumption is made up for in increased Blackboard usage. Each quarter more and more professors sign on, other professors increase their usage, and the average size of the files being passed into the system increases.

SysAdmin best-practices

| 1 Comment
There are best-practices you just pick up after a while, but who sets them? Ahm, I don't know. And that is part of the problem. There are area-specific best-practices makers, such as the SANS institute for IT Security issues and Microsoft TechNet for Microsoft, but there isn't anything out there for overall best-practices setting.

Over the years I've been asked to identify best-practices and evaluate how close we are to meeting them. Coming up with that list involves a lot of web-searching, learning where the standard-makers are, and not a little of distilling of best-practice out of mailing-list posts.

Do we need a single repository for sysadmin best-practice?

As I mention in a previous post System Administration has a very broad definition, and varies dramatically from organization to organization. What's more, the industry is fractured on vendor lines, as what works for Cisco may not be entirely valid on Juniper, and the same goes for Linux/Windows, AIX/Solaris, Oracle/Postgresql, Government/Private-Industry, EMC/NetApp, SAP/PeopleSoft, and many, many more.

One best-practice that has broad recognition is the idea that Administrative users like me should have two accounts; one for regular work, and another for admin tasks. I should not be running Outlook and hitting refresh on cnn.com while logged in as Domain Admin, that should be in as a normal user. And yet even this has differences between Linux and Windows.

On Linux, sudo is a great way to do one-time tasks as root and if you need to you can create a specific terminal window that's in as root. On Windows, UAC means you're running as normal-user for most of your tasks but can promote certain tasks to admin rights by opening them the right way or opening certain tools at all. And yet... on Windows, even with UAC a domain-admin logged in to a workstation will still be able to change NTFS rights everywhere, as icacls isn't subject to UAC. Windows does have 'run as' functionality now, which allows sudo-like workflow, but it isn't as easy to work with as sudo is.

Is Microsoft's attempt at account separation good enough, or does an admin need the ability to have two discrete login sessions going at the same time? Pre-UAC, this is exactly what Microsoft was recommending.

In my past best-practices work I know that many managers would prefer their best-practices be accompanied with step-by-step guides for following the best-practice so they can be certain they're doing it right. Best-practice-by-rote. So finding a document that describes privilege separation for admin users in generic terms isn't as useful as the one that comes with a step-by-step guide.

Providing best-practice guides for the breadth of Systems Administration is fantastically hard.

Creating a list of general principles is relatively easy. An engaged sysadmin group such as LOPSA or SAGE could generate such a list given a few months and a lot of word-smithing.

Creating platform-specific best-practices guides to go with the statements of general principle will take an orders of magnitude more effort. It means identifying enough subject-matter experts to create guides for everything, which greatly increases the number of people that need to be involved. Worse, the step-by-step guides become stale far faster than the general statements do, so regular maintenance will need to be done.

Because of this difficulty we have the status-quo. Best-practices guides are found in area-specific domains, even the ones that cover a lot of the system-administration space. LOPSA is heavy in Linux/Unix admins and light in Windows, so may not do as good a job generating guides for the Windows platform as they would be for Linux. The MSRC would be great at putting out guides for Windows, but useless for anything else. SANS is a great source of general guides for security-topics, and even has some step-by-steps, but does not cover things like backup rotations.

By my read of it outside of Security we have very few non-vendor organizations publishing best-practices guides for system administration topics. The few that are tend to be concentrated in one domain, *nix administration. Creating any kind of broadly-applicable best-practices documents will require participation across the entire breadth of the System Administration space, and no organization even approaches that yet. LOPSA has hopes in that direction, but before they can really attack the problem they need to attract a broader membership. That takes time. Until then, things stay fractured.

The changing end-user environment

The end-user environment is becoming heterogeneous at a time when our central computing environment is becoming more homogeneous due to cost constraints. This has impacts to how we do business. The latest features may not be able to be deployed until the lagging platforms get sufficient support. Formerly ignored platforms will need central resources to effectively manage them.

IT scale-out

Keying on my last post, I give you an infographic.


From One-Gun-IT to siloed departments after their first billion dollar merger. Not all companies break out IT this way. Some of these never emerge. Some divide on completely other areas. I just like the picture.

Defining "System Administrator"

It's bloody hard. When asked what I do for a living by people who I don't suspect are terribly computer literate my answer is usually, "I work with computers at Western." The crux of the problem is that what a System Administrator does varies hugely across the spectrum. Take the one-gun-sysadmin:
They do everything.They may not even know that what they're doing is everything in quite this way, but they're it. Desktops, laptops, wireless access-points, firewalls, software installs, windows deployments, script-writing, web-page authoring, hardware orders of all stripes. The consummate jack-of-all-trades.

Contrast this to the IT department of an example very large corporation.
With discrete silos for each area of responsibility. The System Administrator group has nothing, at all, to do with configuring and maintaining the network hardware. Or the firewalls. Or the desktops. They do servers. They probably do a LOT of servers, enough that automatic deployment and provisioning of servers becomes a key skill.

What do these two system-administrators have in common? Um... they both work with computers...

This is the hard part. They both keep the enterprise logic core running. They both work with servers. They have frequent contact with, if not directly manage, such infrastructure things like the datacenter network and security policy implementation. These are details that are hard to explain to people.

One side-effect of this disparity in duties is that if the mega-corp SA gets laid off for some reason, they'll have trouble getting a job at a one-gun shop for the simple reason that they won't have enough enough of the job-requirements on their resumé to seem like they'd do it well. At the same time, the one-gun looking for a new situation would be hard pressed to work at the mega-corp because their scale-out skills don't exist on paper.

Me? I'm somewhere in the middle.
Most of 'security' is in Telecom, but that may be changing soon. I kiss on networking because Storage-Area-Networking is my thing and I now am the proud owner of a completely isolated iSCSI LAN! Development is... way over there. I've been doing a fair amount of desktop lately, but that's an exceptional circumstance, not a usual one. Siloed enough I'll have a hard time doing one-gun-IT, but not quite scaled enough to go to the mega-corp. In between.

The three of us would still find plenty to talk about though. Not the least of which is how we describe what we do to other people.

A local sysadmin conference

I had heard of it before, but it looks like they managed to get it off the ground. The Cascadia IT Conference is a LOPSA-sponsored event, focusing on System Administration. It being within driving-range of my house and having a conference fee of under $1000 means that my ability to go is not zero. This is exciting. I haven't been to a conference since my last BrainShare in 2008, people are going to forget I exist.

The training schedule is pretty light, but they're in their first year so attracting both talent and people confident enough to present is tricky at this stage.

Organic networks

A friend of mine recent tweeted the following:
STILL trying to rebuild my productivity after losing my work hard drive. I miss my mapped drives!
Because we were so recently in the NetWare paradigm of networks, we still have a login-script that gives everyone the drive-mappings they need. Organic mappings are beginning to crop up, but because we give users nearly everything they need for data-access right out of the box our users haven't gotten in the habit of mapping drives to every little thing in their own way. The P: drive is universal, everyone has one, and our Helpdesk gets calls like "I can't get access to the P: drive."

Pure Windows networks, I gather, are much more organic affairs. The worst offenders are small offices without a central filer of some kind, so everyone just shares directories off of their own workstations. Even if they do have a NAS, there are probably lots of shared directories on workstations. I've seen them, it made me cringe.

Larger pure-Windows offices... I don't have much experience in, but I suspect undisciplined server growth leads to multiple file-servers with data scattered on all of them. This makes new users have to do detective work to figure out where all of their needed data is and then map accordingly. Disciplined server growth, on the other hand, is beyond my experiences.

Because we're so drive-letter centric our helpdesk occasionally gets calls like, "My M: drive went away." In this case, the M: drive is not one we routinely map so the user set it up themselves (or worse, had it set up for them by someone else), so the helpdesk tech gets to ask the questions to figure out where their M: drive was pointing and assist them in re-mapping it. In the few pure-Windows environments I've touched such questions seem to be of the "I can't get to the Faculty share" type, which still requires a bit of helpdesk detective work to figure out which server the 'faculty' share is on.

For those of you who've spent a lot of time in pure Windows environments, how are your users handling drive-mappings, and how are shares administered? We're pretty tightly controlled around here (hello, legacy speaking) but I know there is a lot of variability out there.

Migrating off of NetWare

It has been around a year since we did the heavy lifting of migrating off of NetWare and retiring our eDirectory tree. By this point last year we had our procedures in place, we just needed to pull the trigger and start moving data around. I was asked to provide some hints about it, but the mail bounced with a 550-mailbox-not-found error *ahem*.

Because it's such a narrowly focused topic, and the WWU people who read me lived through it and therefore already know this stuff, I'm putting the meat of the post under the fold.

You're welcome.