July 2012 Archives

The difference a year makes

I already wrote about the main reason I left WWU. Specifically, the paltry excuse for summer I had out there. Well, it's been a year since that article and I figure it's time for a weather update.

Anecdotal reports held the summer of 2011 to be one of the hottest the area has seen in a while.

I didn't notice.

In fact, when it was a bajilionty degrees outside, I'd occasionally step out of my A/C bubble just to stand in it. Revel in its glorious sunny warmosity. The walk from the Metro stop to the office was pleasant, even if there was sweat involved.

It was, in short, the summer I'd been waiting for several years to have. And I got it. Boy did I. It was awesome. Also? Shorts! In April! April! How cool is that.

The calendar turns, and 2012 looms, overtakes, and drenches.

Apparently I've gone native, because I'm sweltering. It took a year to bake the cold out of my bones, but they've baked out now. I now admit that I live in a drained swamp, because I didn't really feel it last year. I can probably consider vacationing some place cooler and not wince at the very idea now.

Changing cultures

| 1 Comment
It's now about a year and a half into my tenure here at newjob. They hired me as their first full time IT hire. Previously they'd been making do with part-time consultants in the sysadmin role.

Consider, also, that this company is definitely in startup mode, so 40 hour weeks are something of a minimum. So yeah, that part time sysadmin had his hands full. He was definitely involved in the classic sysadminly things like hardware installs, network expansions, remote access management, and random troubleshooting. Though the later was restricted to things that could wait until he was on the clock.

Now that they have a full-timer, the list has expanded. It now includes items closer to the development side of the house. Things like:

  • Automation engineering. 
    • System Deployment automation
    • Software Deployment automation.
    • OS Update automation
  • Configuration Management (previously this was an entirely Dev thing)
  • Consulting on strategic planning for future growth
  • Performance analysis
  • Monitoring System deployment

In fact, the automation-engineering part is probably what I've spent 50% of my time on in the last 6 months and is an almost 100% writing-code task. The traditional sysadminly things I itemized above remain a small but significant part of what I do.

But what I wanted to talk about was Configuration Management.

When I got here they already had a Puppet installation. It was designed to be git-cloned into /etc/puppet on every deployed machine, and to be run exactly once when the machine was being installed. They weren't using puppet as a Configuration Management utility, they were using it as a Deployment Automation tool. Which it works as, but is definitely leaving a lot of functionality on the table.

There were many modules in there that were written to run every time puppet was run, regardless of the need to run it. One egregious example was a tar-ball that was downloaded from the Internet and then compiled. If the local compile-directory was still there and hadn't been cleaned, this actually ran fairly quickly on re-applys. But once I made /tmp ephemeral across reboots? Applys now took 15 minutes regardless of what actually needed application.

There were several more examples of this. I've spent time making it so it can be run multiple times with a minimum of redundant processing.

I've also put in a Puppetmaster server, since /etc/puppet on everything is a waste of space and a challenge to keep updated.

The repo as it stands right now can be run repeatedly without side-effects. It has taken me a long time to get here. Now I'm trying to convince the rest of Development that when a package needs to be added to all of a certain class of machine the best way to do this is NOT to go to each once and install it, rather add it to Puppet and then just do an Apply on all of 'em; it'll get done the same way on each, as it should.

Unfortunately, somewhere along the line most of the devs stopped paying attention to Puppet. Importantly, one of the release-engineers only uses it when it shows up on checklists and is definitely of the "apply the package now, get it into the automation later" mindset. Clearly, I have more work to do.

Just one more step in my ongoing battle to minimize the harm caused by organically grown systems. It's an ongoing process, and at the end of it we'll either be a straight up DevOps team or the change-demon will be running rampant causing everyone grief.

Fixing the other home network

| 1 Comment
Part of the blog-silence the past few weeks is because I was on vacation.

Nice, nice vacation.

However, it was at my parent's place and as with any technically savvy sprog who comes home, there be questions. In my case, it was an ongoing problem of slowness with the network. The first few days I didn't have time to delve, but I did eventually get into it.

The symptoms:

  • Wifi download speeds were about 75KB/s, and upload about 3x that. Yes, upload was faster. Yes, my 4G phone had faster downloads.
  • Wired speeds were at the rated speeds for the broadband connection.

Clearly, something was wrong with the wireless.

My little spectrum analyzer showed remarkably clean airwaves for a residential area. There was some congestion on their frequency, but moving it didn't help much.

And then I noticed something when I ran iwconfig on my laptop:

wlan0     IEEE 802.11abgn  ESSID:"dadhome"  
          Mode:Managed  Frequency:2.462 GHz  Access Point: 06:1D:61:FF:BB:00  
          Bit Rate=54 Mb/s   Tx-Power=15 dBm  
          Retry  long limit:7   RTS thr:off   Fragment thr:off
          Encryption key:on
          Power Management:off
          Link Quality=70/70  Signal level=-31 dBm 
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:2942  Invalid misc:1892   Missed beacon:0

Specifically, that "Tx excessive retired" statistic was incrementing, and I'd never seen that other than zero before. Most odd.

The router was a Cisco/Linksys, and a pretty new one at that. The firmware was latest, so that wasn't it. After a bit of poking about I found out that I could get vastly better throughput by setting the Wifi to G-Only, instead of B/G/N. In fact, setting it to N-only made the problem worse! Clearly, this router's N implementation is a bit off.

Wifi wasn't at the broadband speed yet, but it was still a vast improvement. That's where I left it, and they're happy.