Recently in opinion Category

At both this job and my last one I have ended up becoming the WTF person. The WTF person is the person people go to when things are acting strangely, they can't figure it out, and need another set of eyes. Preferably a set of eyes with a reputation for pulling rabbits out of hats.

WTF people are the kind of people that end up on level 2 or 3 tech support, because that's who you want to have at that level. People who solve weird stuff.

At a place like ours where the support relationships are largely informal, at least among people who dink around with servers, the concept of L2 or L3 support doesn't really exist. It manifests as phone-calls or emails from people with strange questions, looking for leads in their own inquiries. Or in the case of my immediate co-workers, a head poked around the door, and, "I'm lost, can you take a look?"

As I alluded to before, becoming the WTF person takes time. You have to make some awesome saves so people notice, and then continue to crack weird, hard to describe problems. It helps a lot to have a deep understanding of the technology you work with. I suspect being ebullient about how you found the problem and describing the problem once it was resolved helps in this.

Once you get there, though, you do get passed some strange, strange things. I've been asked advice on figuring out how something broke in that specific way when the symptoms described... have no causal relationship I can think of. I also get passed weird questions in areas I don't know much about (MS Office for one), but at least those can be deflected.

Honest to goodness bugs are perhaps the hardest to figure out. These are problems that take a few conditions to set up, and it isn't always clear that those conditions are in place. This skill got a lot of work back when I was working on the OES2 SP1 beta. On software that's already been through a beta-test and perhaps a service-pack or two, the bug conditions can be very arcane.

One-man IT shops tend to attract WTF people, simply due to the breadth and complexity of the environment. People who thrive in such environments definitely are. They do a little bit of everything, which sets them up to make connections that other people miss.

At the other end of the IT spectrum, highly specialized IT people in large organizations, you still find WTF people. They're perhaps not as common, but they do exist. And strange but awesome synchronicities can occur if WTF people from different specialties start hammering on a problem together. This kind of thing sometimes happens when I talk to L2/3 vendor-support.

I'm proud to see this happen, even if in the moment I'm also going WTF?? in my head.

The death of the desktop computer has been predicted for years, and yeah. I can see why. At home, we have one desktop. I'm not counting the servers I run in headless mode for this, otherwise the count would be higher and I'm never on the keyboard for those anyway. The desktop gets used for a very few things:

• That's where the budget is kept
• PC Gaming
• The few applications for which a huge screen and a mouse are a real benefit.

We use it on average 1.5 times a month. Unless someone bought a new game at which usage will be fairly constant for about two weeks, at which point interest will be lost and we'll be back to 1.5x month. Our laptops do darned near everything we need, as fast as we needed, in a way that is comfortable. Plus, we can take them places.

At work I'm not giving one up because I consciously made the mobility/performance tradeoff in favor of performance. I've got over 36" of linear monitor, and I use it. I have quite a lot of memory in there, as well as a quad core CPU, that gets well used. As well as two hard-drives because I have needs. This is not a desktop it's a workstation.

While the desktop may be mostly dead in the home, the only serious niche keeping it going is PC gaming and even that's changing thanks to consoles, it's going strong in the workplace.

A lot of the squee I've heard about the sequencing of the human genome and the ever dropping cost to completely sequence a single genome has been in the nature of "we're figuring out how nature programs biology!" This is true to a point, but the reality of programming new life or new functions of life is far, far in the future. Yes, we've already created artificial life, but it wasn't done with full understanding of the source code we used; we took the code that governs functions we wanted and fitted them together.

Biology is in some ways like computers in that there is a (presumably) deterministic process that governs the rules of how it works. It exists in a fundamentally chaotic environment, which makes extracting that determinism pretty hard. But we're sure there is a causal chain for most anything, if only we look hard enough. For computers we know it all end to end, we wrote the things so we should, but are only now getting to the levels of complexity in these systems where they can mimic non-deterministic behavior. But if we dig down into the failure analysis we can isolate root and associated causes of the failure chain. We want to do that with biology.

We are far, far away from doing that.

Biology up until the genomics 'revolution' has been in large part describing the function of things. Our ability to stick probes in places has improved over time, which in turn has increased our understanding of how biology interacts with the environment at large. We've even done large scale changes to organisms to see how they behave under faulty conditions, just so we can better figure out how they work. Classic reverse-engineering, in other words. You'd think having access to the source code would make it go much easier. But... not really.

Lets take an example, a hand-held GPS unit. This relatively simple device should be easy to reverse engineer. It has a simple function, provide precise location. It has some ancillary functions such as provide accurate time, and give a map of the surroundings. Ok.

After detailed analysis of this device we can derive many things:

• It uses radio waves of a specific wavelength set to receive signals.
• Those signals are broadcast by a constellation of satellites, and it has to receive signal from no less than three of them before it can do so.
• The time provided is very stable, though if it doesn't receive signals from the satellites it will drift at a mostly predictable rate.
• The bits that receive the satellite signal, since it doesn't work if they're removed.
• Where the maps are stored, since removal of that bit causes it to not have any.
• A whole variety of ways to electrically break the gizmo.
• How it seems to work electrically.

Additionally, we can infer a few more things

• The probable orbits of the satellites themselves.
• The math used to generate position.
• The existence of an authoritative time-source.

Nifty stuff. What does the equivalent of 'genomics' give us? It gives us the raw machine code that runs the device itself. Keep in mind that we also don't know what each instruction does, and don't yet have high confidence in our ability to discriminate between instructions. And most importantly, we don't know the features of the instruction-set architecture. There is a LOT more work to do before we can make the top-level functional analysis meet up with the bottom-level instructional analysis. Once we do join up, we should be able to understand how it fundamentally works.

But in the mean time we have to reverse-engineer the ISA, the processor architecture itself, the signal processing algorithms (which may be very different than we inferred with the functional analysis), how the device tolerates transient variability in the environment, how it uses data storage, and other such interesting things. There is a LOT of work ahead.

Biology is a lot harder, in no small part because it has built up over billions of years and the same kinds of problems have been solved any number of ways. What's more, there is enough error tolerance in the system that you have to do a lot of correlational work before you can identify what's signal and what's noise. Environment also plays a very key role, which is most vexing as environment is fundamentally chaotic and can not be 100% controlled for.

We're learning that a significant part of our genome is dedicated to surviving faulty instructions in our genetic code, and we hadn't realized they were there before. We're learning ever more interesting ways that faults can change the effect of code. We're learning that the mechanics we had presumed existed for code implementation are in fact wrong in small but significant ways. The work continues.

We may have the machine-code of life, but it is not broken down into handy functions like CreateRetina(). Something like that would be source-code, and is far more useful to us systemizing hominids. We may get there, but we're not even close yet.

My thoughts on this quote:

Theoretical risks and real risks are generally the same thing when you're talking about IT security.

In large part, this is correct. Especially when getting audited. We have regular audits here, both internal and external. We have servers that handle credit-card data, so we have to deal with PCI compliance as well. So yeah, we know about this. We're also familiar with the debate.

In order to get our PCI stuff certified we have to have security scans performed against our credit-card processing servers. In order to do this, we grant a specified IP address full and unrestricted access to an internal IP list. The third party then scans that from wherever they are, and sends us the report full of red Xes.

The internal debate goes like this. I'm not naming names for obvious reasons. I like my job.

Tech: Why do we have to let them in to scan? That's, like, completely bypassing the security provided by our firewall. Both firewalls. It's not like a regular hacker has that kind of access. These servers can not normally be reached from the internet at all! They should be scanning THAT!

Manager: Because that's what the PCI standard says they have to do.

Tech:  It makes no sense!

The reason for this is because they're testing how vulnerable we are if our other servers get hacked and they have enhanced access to that subnet. That's also very unlikely in our case (see also: two firewalls), but the fact remains that it still has to be checked. Because we've never been attacked that way (that we know of), that kind of attack is seen as theoretical rather than real.

All it takes is one attacker, or a group of attackers, to REALLY WANT SOMETHING for theoretical attacks to become real. The concerted attacker, as opposed to the casual attacker, is the one that'll employ novel methods of getting what they want. Door-rattlers looking for phat pipes for their warez repos are looking for any fat pipes they can find and the resources they expend per target are pretty small. Someone looking to break in for a specific reason is targeting us specifically, and the resources they'll expend to get it is a LOT higher.

It is the concerted attacker that'll spend the time to worm their way from internet-facing systems, to intranet-facing systems, to get to secure-net facing systems. It is this kind of attacker that'll do targeted phishing against user most likely to have inner-firewall access of some kind and then attempt to create VPN sessions with those credentials to do scanning from a far more advantageous network position. It is the concerted attacker that'll do targeted DNS hijacks in order to get better information. These are not the kinds of things that Joe Warezer or Ben BotHerder are going to bother with.

It is also true that the concerted attacker can be vastly more damaging than their younger cousin who is just looking to leech resources or reputation. So yeah, it's a very low likelihood of running into that kind of threat, but the risks of not doing something about it are pretty high. That's what makes the theoretical real. 

Using multiple web-browsers is kind of a power-user thing. Most people just stick with one, and only vary if they need to access a certain site that is doggedly IE only, such as Outlook Web Access, so they have to leave Firefox to use it. Since 51% of you, my intrepid readers, are Firefox users, someone you know well what I'm talking about.

A long time in Mozilla's past, it was possible to run multiple Mozilla instances running out of different profiles. This was handy if you desired process separation for browsing activity, or, ahem, didn't want to pollute your work profile with certain, ahm, sites. For that we now have Porn Mode Privacy Mode. Mozilla removed multiple concurrent profile support with Firefox, IIRC.

However, having multiple browsers is still useful. The reasons I do it:

• I can stay logged in to multiple GMail accounts this way.
• I can log in to Google in one browser, and do all of my other searching, browsing, whatnot in another browser and not have all of those searches directly associated with that one Google ID.
• Sites that are browser specific (OWA is a major one).
• The Opera email client is really very good.
• I can have a different plugin-setup, which may help diagnose problems with sites.
• Browsers on different operating systems behave differently. This can be useful.

The main problem getting in the way of sticking with a single browser is Firefox's insistence that only one profile can be active at any given time. Thus, the need for more than one. I've been using SeaMonkey, the Mozilla descendant and also a Gecko-based browser, for a lot of this.

The browser I use for most general purpose surfing I don't leave logged into Google, Facebook, Twitter, or anything else like that. It minimizes what these social networks track of my browsing habits, especially with Facebook Like and Twitter badges appearing everywhere these days. Ad-networks grab this stuff too, but at least I don't have a login with them that explicitly links me with my browsing habits; it's implicit for them. If WWU ever goes GoogleApps for whatever reason, this will be doubly useful.

The down side is I have to have enough RAM to support two browsers. I'm lucky enough that I do. Useful though!

I've talked about this before, and I'm sure I'll do it again. We do need to reduce some of the excessive packaging on the things we get. I can completely understand the need to swaddle a $57,000 storage controller in enough packaging to survive a 3 meter drop. What I don't understand is shipping the 24 hard drives that go with that storage controller in individual boxes. It wouldn't take much engineering to come up with a 6-pack foam holder for hard-drives. It would seriously reduce bulk, which makes it easier and cheaper to ship, and there is less material used in the whole process. But I guess that extra SKU is too much effort.

Today I turned this:


Into this:

This past weekend I got into a pretty long discussion about privacy, governmental, corporate, and criminal tracking of everything you do (Big/Little/Silent brother), and such related topics. It was good debate. One of 'em was an actual lawyer versed in these issues who works for a library-related non-profit. How cool is that? Working as I do for a liberal institution of higher ed we do value our individuality and right to express same.

Big Brother. We know this one, Orwell told us all about it in his book 1984. Governmental tracking of people for their own safety.

LIttle Brother. A more recent development, but private-sector tracking of people for reasons relating to profit. Your browsing habits are being tracked by the ad agencies. That kind of thing.

Silent Brother. A term I came up with, but it's obvious enough I wouldn't be surprised to learn someone else came up with it too. Criminal tracking of everything you do for reasons of illicit profit. Russian crime gangs specializing in identity theft.

Now thats out of the way, some nitty gritty. Under the fold.

XKCD made an observation last week:



I find it impressive because I can sit on my couch and watch well detailed images, not upright in my dining room chair at the laptop, or staring down at the iphone on the table or in my lap. Apparent pixel size makes a big difference here.

Let's take a look at the 24" Apple Cinema display. It has a 1920x1200 native resolution. If you put that at the arms-length recommended by ergonomicists, it's 28" away for me. What does that mean? It means an apparent horizontal pixel width of 0.022274 degrees, that's what. And the math:

Actual horizontal width per pixel = 1920 / 20.9" (the actual width of the screen) = 0.010885"
Distance to that pixel = 28"
Angle = tan^-1 ( 0.010885 / 28 ) = 0.022274 degrees

Lets say I have a 42" HD-TV at home that sits 8 feet from the couch. That's a 1920 horizontal resolution at 96 inches. Giving an apparent horizontal pixel width of 0.011376 degrees, markedly smaller than the Apple display at 28".

An iPhone at 12 inches has an apparent pixel size of 0.034817 degrees. Just so you know.

Generally speaking, smaller apparent pixel sizes allow you to cram more detail into a given viewing angle. However, there are limits here. As human eyeballs age, their ability to distinguish very fine detail fades; a 16yo may be perfectly happy with a 1920x1200 monitor and 9pt type, but their 60-something grand-parents most definitely won't be and their parents would have to squint hard. 

When I was hunting for an HD-TV I found a few articles describing how far away from the TV you had to be to tell the difference between 720p and 1080p. What that distance was depended on two things; how old the viewer was, and how wide the screen was. For a 42" TV at 96 inches, only 5 year olds can tell the difference between 720p and 1080p. If they're on the couch that is, and not parked on the floor 3 feet from the TV.

For comparison, a 720p 42" TV at 96" gives an apparent pixel size of 0.017066 degrees. A 1080p panel with the same apparent pixel size would have to be 54.9 inches wide (a 63" panel).

I've read reports of display makers showing off HD+ TV displays with in excess of 2000 vertical pixels. These aren't really commercially available in no small part due to the lack of media available at that resolution, but also the fact that the panels would have to be very large indeed for the average consumer to notice a difference from 1080p in their actual living rooms. Eyeballs are so limiting.

So I am impressed with HD TV, even though I've been a daily user of display tech capable of more detail for years before I got one. Scaling displays up to that size takes work, and getting them affordable to the likes of me takes very high manufacturing quality. The fact they've done it is woot-worthy.

But don't get me started on 3D.

Science Fiction author and armchair tech industry analyst Charles Stross has written an article on what the 'Apple/Adobe letter' signals for the future.

Go read it. It's good.

He sees the letter as a clear signal that Apple is actively trying to ensure that the Apple brand is relevant in a future in which computing is even more commoditized/cloudy than it is now. He sees PC hardware sales becoming even more of a loss-leader than it is now, and both Apple and HP (hello Palm purchase) have identified what the (profitable side of the) future looks like:

• Wireless broadband everywhere
• Very little local storage, basing everything on the cloud
• Tight vendor controls on the software ecosystem, for safety. "Cross-platform" is for skeevy hackers.
  

There will still be traditional PC environments around, Microsoft won't be able to allow a monopolitic stack like Apple i$Device to develop for legal and technical reasons, but they won't be where all the money is being made. The real money in PC-land will be made in software, not hardware and OS. *I* suspect it'll get more annoying to rip apart a new machine to get linux on it. Because of naughtiness in the 90's, Microsoft won't be allowed to produce a vertically integrated Hardware/OS/Software stack like Apple is actively doing with iPhone/iPad.

The future has lots of mobile bandwidth, enough mobile bandwidth that having your primary data-storage be a few network hops away is not annoying; especially if there is a local, and large, cache available. The future is a lot more paternal, software will auto-update in the background without notifying you and will be hard to get around; you better hope updates don't trash your other software. The future has software cops preventing bad stuff from getting on your gear, and the software cops will be the device vendor (Apple, HP, Google).

At least, at the consumer level. How this all will interact with workplace environments is an open question. There are some tasks for which a full sized keyboard is really required, as well as 22" displays and high-volume printers. I strongly suspect there will be large computer-environment differences between home-computing and work-computing. We shall see how it develops.

That last series of articles might suggest I've been doing storage administration for a while. And I have. But every so often I run across an article that just reminds me that I'm still in the shallow end.

Like this article from The Register, going over Quantum's new mega-library, the i6000. I have a buddy who has an i2000 and I've petted it. Lovingly. *sigh* This new baby can store 8PB. Petabyes, baaybee. LTO5. Mmmm. Sexy.

Storage is a major concern just now. One of the main reasons that there are still IT stacks on campus that aren't centralized is storage. We have researchers, generally in the College of Science and Technology, that use departmental, rather than central, resources for storing their data. Departmental means servers, so CST represents the biggest non-ITS concentration of IT at WWU. They don't have any shared storage arrays over there, so they make do with large direct-attach-storage servers over there. A quick back-of-envelope calculation says that they have about as much storage in DAS as we have on our fastest SAN-attached storage array. Combine that with the chronic storage shortages central IT has had for the past, oh, 15 years and you have an entrenched set of servers over there.

If they were to join us in the borg ITS my area just might crack 100TB in disk space. Ooo. An i2000 with LTO4 still would be overkill for a storage network that large. And the i2000 can expand to several cabinets.

Yeeeah. WWU is still strictly small time when it comes to storage. In a lot of ways I'm a Stand Alone Storage Administrator.