February 2012 Archives

One of the questions that SysAdmins frequently get asked is:

I have a web application based on $Platform. It needs to support $NumUsers concurrent connections. How much server do I need?
$Platform can be anything from 'php' to 'tomcat' to the incredibly unhelpful 'linux'. $NumUsers can be anything from a reasonable number to completely unreasonable numbers representing the anticipated worst-case (or maybe that's 'best case') scenario (50,000 users! Two meeeeelion Users!).

The answer they're looking for is:

Two AWS large instances.

The answer they'll get is:

It depends on the application code.
They're laboring under the misconception that $Platform and $NumUsers are the only variables in the Grand Equation of Scaling. HAHahahahaahaha.  There actually is a GEoS, but I'm getting ahead of myself.

Today while running through my feeds, I ran across an article describing how nifty Windows 8 Server will be for storage. And tripped when I got to the following line:

The new OS will support Hyper-V and SQL Server running over the Samba file sharing protocol Server Message Block (SMB) v2.2, including remote direct memory access (RDMA) over Ethernet and InfiniBand.


The Samba file-sharing protocol? Since when?

Perhaps this is what happens when a linux-person writes an article on a Windows technology. Who knows.
Since I had some time this weekend and actually had my work laptop home, I decided to take the plunge and get OpenSUSE 12.1 onto it. I"ve been having some instability issues that trace to the kernel, so getting the newer kernel seemed like a good idea.


The upgrade failed to take for two key reasons:

  1. VMWare Workstation doesn't yet work with the 3.1 kernel.
  2. Gnome 3 and my video card don't get along in a multi-monitor environment.

The first is solvable with a community patch that patches the VMware kernel modules to work with 3.1. The down side is that every time I launch a VM I get the "This is running a newer kernel than is known to work" message, and that gets annoying. Also, rather fragile since I'd have to re-patch every time a kernel update comes down the pipe (not to mention breaking any time VMWare releases a new version).

The second is where I spent most of my trouble-shooting time. I've been hacking on X since the days when you had to hand-roll your own config files, so I do know what I'm looking at in there. What I found:

  • When X was correctly configured to handle multi-monitors the way I want, the Gnome shell won't load.
  • When the Gnome shell was told to work right, GDM won't work.

Given those two, and how badly I need more screen real-estate at work, I'm now putting 11.4 back on the laptop.

Looking to the future person

It is becoming increasingly clear that we'll be hiring a second full-time IT-type person some time this year. What's more, they're likely to be directly reporting to me. That would turn me into a straight up Manager, something I haven't been before. This is somewhat scary! The closest I've been was two jobs ago when I was providing work direction to two other people, but the actual time-card signoffs and vacation approvals was handled by someone else.

Which means I'm giving thought on what we'll need in terms of skillset for this nebulous person. As I see it, there are five primary knowledge domains that we're interested in:

The "buying things" one is new to this chart, in the past I've always had either a purchasing department to handle things like chasing down late orders, or have had single-source contracts that take a lot of the choice out of what we can buy. When you get to the senior levels and get 'recommend' powers (if not straight up purchasing authority) this kind of thing is actually pretty key. In fact, right now I have a specific order that I need in my hands by next Thursday OR ELSE, and I'm having to apply mallet to a supplier to get it. I never had to do this kind of thing before.


The tricky part is, what weight do we assign to each knowledge domain? Based on my workload right now, I'd have to say "automation coding" is primary. However, come August the pain we're feeling may be somewhere else entirely. Come August, our product should have been released for several months and we'll then have a lot of operational experience, and what we may need most of all is someone to share the midnight callout duties more than anything else.

And then there is the whole, "Now that I'm a manager-type person, what work is most suited to that?" question.

For me, of those five domains the Automation Coding portion is the domain with the highest interrupt costs; it takes me a good while to get back on track after someone asks me something random. Considering it's my job to be interrupted (even though that's a bad thing for systems administrators), this suggests having Someone Else do the automation coding part is a good idea. On the other hand, for large web-scale and highly homogeneous infrastructures automation coding is the majority of the systems engineering work needed to make it all function.

Right this moment, it's looking like my #2 will have to be someone with some years under their belt, which runs against our company's proven track-record of hiring promising people right out of college. Automation coding is that curious mix of a lot of Software Engineering crossed with the domain specific knowledges of OS lore, management frameworks, and application-specific functioning. Of these, it's the OS lore that's the hardest to train for.


If I've done my job right, by the time we release the automation framework should be largely completed and should be good enough to handle at least half a year's worth of scale. If we release, and the industry loves us like a loving thing, we'll be scaling out madly. At that point, having another set of hands well versed in the Network Configuration and Server Hardware parts will be more important than a systems programmer. THAT is someone who could be a fresh-from-college person, much like our summer intern IT last year.

Won't know until we get there.