Recently in virtualization Category

Like many olde tyme sysadmins, I look at 'cloud' and shake my head. It's just virtualization the way we've always been doing it, but with yet another abstraction layer on top to automate deploying certain kinds of instances really fast.

However... it's still new to a lot of entities. The concept of an outsourced virtualization plant is very new. For entities that use compliance audits for certain kinds of vendors it is most definitely causing something of a quandary. How much data-assurance do you mandate for such suppliers? What kind of 3rd party audits do you mandate they pass? Lots of questions.

Over on 3 Geeks and a Law Blog, they recently covered this dynamic in a post titled The Inevitable Cloud as it relates to the legal field. In many ways, the Law field shares information handling requirements similar to the Health-Care field, though we don't have HIPPA. We handle highly sensitive information, and who had access to what, when, and what they did with it can be extremely relevant details (it's called spoliation). Because of this, certain firms are very reluctant to go for cloud solutions.

Some of their concerns:

• Who at the outsourcer has access to the data?
  
• What controls exist to document what such people did with the data?
• What guarantees are in place to ensure that any modification is both detectable and auditable?

For an entity like Amazon AWS (a.k.a. Faceless Megacorp) the answer to the first may not be answerable without lots of NDAs being signed. The answers to the second may not even be given by Amazon unless the contract is really big. The answers to the third? How about this nice third-party audit report we have...

The pet disaster for such compliance officers is a user with elevated access deciding to get curious and exploiting a maintenance-only access method to directly access data files or network streams. The ability of an entity to respond to such fears to satisfaction means they can win some big contracts.

However, the costs of such systems are rather high; and as the 3 Geeks point out, not all revenue is profit-making. Firms that insist on end-to-end transport-mode IPSec and universally encrypted local storage all with end-user-only key storage are going to find fewer and fewer entities willing to play ball. A compromise will be made.

---

However, at the other end of the spectrum you have the 3 person law offices of the world and there are a lot more of them out there. These are offices who don't have enough people to bother with a Compliance Officer. They may very well be using dropbox to share files with each other (though possibly TrueCrypted), and are practically guaranteed to be using outsourced email of some kind. These are the firms that are going into the cloud first, pretty much by default. The rest of the market will follow along, though at a remove of some years.

Exciting times.

Question: When a (IaaS) cloud provider charges per hour for a machine, what's it an hour of? Do I get charged when it's doing nothing? If so, why is that fair?

All the IaaS cloud providers I've run into (which isn't all of them by any stretch) charge by the running hour. If that micro-mini instance is doing nothing but emailing the contents of a single file once a day, it'll still get charged for 24 hours of activity if left on. The same goes for a gargantuGPU instance doing the same work, it'll just cost more to do nothing.

Why is that fair?

Because of resources.

The host machine running all of these virtual machines has many resources. CPU, memory, disk, network, the usual suspects. These resources have to be shared between all the virtual machines. Lets take a look at each and see how easy that is.

CPU

To share CPU between VMs the host has to be able to share execution between them. Much like we do... well, practically everywhere now. We've been doing multiprocess operating systems for a while now. Sharing CPU cycles is dead easy. If a process needs a lot it gets what's available. If it needs none, it gets none. A thousand processes all doing nothing causes... nothing to happen! It's perhaps the easiest thing to share. But, we'll see.

Memory

We've been sharing RAM between processes, with good isolation even, for some time now. Even Apple has joined that game to great effect. Unlike CPU, processes sit on RAM the entire time they're running. It may be swapped out by the OS, but it's still accounted for.

Disk

Disk? Disk is easy. It's just files. Each file gets so much, and more if needed up until you run out. At which point you run into problems. Each VM uses disk to store its files, as you'd expect.

Network

To share network, a host machine has to proxy network connections from a VM. Which... it kinda already does for normal OS processes, like, say, Apache, or MySQL. If a process doesn't need any network resources, none gets used. If it needs some, it uses up to what's available. A thousand processes all doing nothing uses no network resources. Same for VMs really. Its right up there with CPU for ease of sharing.

Now ask yourself. Of these four major resources, which of them are always consumed when a VM (or if you rather, a process) is running?

If you said "memory and disk" you've been paying attention.

If you said "all but network, and maybe even that too", you've been auditing this answer for technical accuracy and probably noticed a few (gross) simplifications so far. Please bear with me!

Now of the two always-consumed resources, memory and disk, which is going to be the more constrained one?

If you look at it from the old memory hierarchy chart based on "how long does the CPU have to wait if it needs to get data from a specific location", you can begin to see a glimmer of the answer here. This is usually measured in CPU cycles spent waiting for data. The lower down the chart you get (faster) the more expensive the storage. A 2.5GHz CPU will have 2.5 billion cycles in a second. Remember that number.

A 7.2K RPM hard-drive, the type you can get in 1TB sizes for cheap, has a retrieval latency of 8.9 miliseconds. Which means that best-case the 2.5GHz CPU will wait  22,250,000 cycles before it gets the data it needs. Thats... really slow, actually.

The RAM in that 2.5GHz server can be fetched in 10 nanoseconds. Which means that best-case the 2.5GHz CPU will wait only... 25 cycles.

Biiiiiiig differences there! RAM is vastly faster. Which means its also vastly more expensive^[1]. Which in turn means that RAM is going to be the more constrained resource.

---

So we have determined that of the four resource types RAM is the most expensive, always-on resource. Because of that, RAM amount is the biggest driver of cost for cloud-computing providers. It's not CPU. This is why that 64MB RAM VM is so much cheaper per-hour than something with 1.6GB in it, even if they get the same CPU resources.

Because RAM amount used is the cost-center, and a 1.6GB VM is using that 1.6GB of RAM all the time, the cloud providers charge by hour of run-time. And this is fair. Now you know.

---

[1]: How much more expensive? A 1TB disk can be had for $90. 1 TB of RAM requires a special machine (higher end servers), and will run you a bit under $12,000 at today's prices.

The lazyCoder is someone sees a need to write code, but doesn't because it's too much work. This describes a lot of sysadmins, as it happens. It also describes software engineers looking at an unfamiliar language. Part of the lazy_coder is definitely a disinclination to write something in a language they're not that familiar with, part of it is a disinclination to work.

It has been said in DevOps circles (though I can't hunt up the reference):

A good sysadmin can probably earn a living as a software engineer, though they choose not to.

A sentiment close to my heart as that definitely applies to me. I have that CompSci degree (before software engineering degrees were common, CSci was the degree-of-choice for the enterprising dot-com boom programmer) that says I know for code. And yet, when I hit the workplace I tacked as close to systems administration as I could. And I did. And like many sysadmins of my age cohort or older, I managed to avoid writing code for a very large part of my career.

I could do it as needed, as proven by a few rather complex scripts I cobbled together over that time. But I didn't go into full time code-writing because of the side-effects on my quality of life. In my regular day to day life problems came and went generally on the same day or with in a couple days of introduction. When I was heads down in front of an IDE the problem took weeks to smash, and I was angry most of the time. I didn't like being cranky that long, so I avoided long coding projects.

Problems are supposed to be resolved quickly, damnit.

Sysadmins also tend to be rather short of attention-span because there is always something new going on. Variety. It's what keeps some of us going. But being heads down in front of a wall of text? The only thing that changes is what aggravating bit of code is aggravating me right now^[1]. Not variety.

So you take someone with that particular background and throw them into a modern-age scaled system. Such a system has a few characteristics:

• It's likely cloud-based^[2], so hardware engineering is no longer on the table.
• It's likely cloud-based^[2], so deploying new machines can be done from a GUI, or an API. And probably won't involve actual OS install tasks, just OS config tasks.
  
• There are likely to be a lot of the same kind of machine about.

And they have a problem. This problem becomes glaringly obvious when they're told to apply one specific change to triple-digits of virtual machines. Even the laziest of LAZY_CODER will think to themselves:

Guh, there has got to be a better way than just doing it all by hand. There's only one of me.

If they're a Windows admin and the class of machines are all in AD as it should, they'll cheer and reach for a Group Policy Object. All done!

But if whatever needs changing isn't really doable via GPO, or requires a reboot to apply? Then... powershell starts looming^[3].

If they're a *nix admin, the problem will definitely involve rolling some custom scripting.

Or maybe, instead, a configuration management engine like Puppet, CFEngine, Chef or the like. Maybe the environment already has something like that but the admin hasn't gone there since it's new to them and they didn't have time to learn the domain-specific-langage used by the management engine. Well, with triple digits of machines to update learning that DSL is starting to look like a good idea.

Code-writing is getting hard to avoid, even for sysadmin hold-outs. Especially now that Microsoft is starting to Strongly Encourage systems engineers to use automation tools to manage their infrastructures.

This changing environment is forcing the lazy coder to overcome the migration threshold needed to actually bother learning a new programming language (or better learning one they already kinda-sorta know). Sysadmins who really don't like to write code will move elsewhere, to jobs where hardware and OS install/config are still a large part of the job.

One of the key things that changes once the idea of a programmable environment starts really setting in is the workflow of applying a fix. For smaller infrastructures that do have some automation, I frequently see this cascade:

1. Apply the fix.
2. Automate the fix.

Figure out what you need to do, apply it to a few production systems to make sure it works, then put it into the automation once you're sure of the steps. Or worse, apply the fix everywhere by hand, and automate it so that new systems have it. However, for a fully programmable environment, this is backwards. It really should be:

1. Automate the fix
2. Apply the fix.

Because you'll get a much more consistent application of the fix this way. The older way will leave a few systems with slight differences of application; maybe config-files are ordered differently, or maybe the case used in a config file is different from the others. Small differences, but they can really add up. This transition is a very good thing to have happen.

The nice thing about Lazy Coders is that once they've learned the new thing they've been avoiding, they tend to stop being lazy about it. Once that DSL for Puppet has been learned, the idea of amending an existing module to fix a problem becomes something you just do. They've passed the migration threshold, and are now in a new state.

This workflow-transition is beginning to happen in my workplace, and it cheers me.

---

[1]: As Obi-Wan said, It all depends on your point of view. To an actual Software Engineer, this is not the same problem coming back to thwart me, it's all different problems. Variety! It's what keeps them going.
[2]: Or if you're like that, a heavily virtualized environment that may or may not belong to the company you're working for. So there just might be some hardware engineering going on, but not as much as there used to be. Sixteen big boxes with a half TB of RAM each is a much easier to maintain physical fleet than the old infrastructure with 80 phsysical boxes of mostly different spec.
[3]: Though if they're a certain kind of Windows admin who has had to reach for programming in the past, they'll reach instead for VBScript; Powershell being too new, they haven't bothered to learn it yet.

AMD has released their server-version of the Bulldozer CPU class they released over a month ago, called Interlagos.

Bulldozer/Interlagos is AMD's attempt to grab more of the market from Intel. Currently, it's competing in the value sector but not on performance. The days when AMD CPUs were the virtualization kings have been gone for a couple years now. AMD would like that crown back, thank you, and they're driving to go there.

That said, comparing performance between equivalently clocked AMD and Intel CPUs is hard. They're optimized for different tasks, which means that the smart Systems Engineer looking for the next CPU to base their environment on should pay attention. Workload matters! Those AMD CPUs may be damned cheap compared to Intel, but if you're doing the wrong things with them you'd be better off buying previous-gen Intel chips.

The most controversial thing AMD has done is to make two cores share a Floating Point Unit. They've also done quite a bit of optimization in their Arithmatic Logic Unit, where Integer math is handled. The reasoning behind this is that most server usage these days is integer heavy, highly parallelizeable workloads; most database and simple web-serving workloads are entirely Integer and parallel-friendly, and that's a large part of the webapp stack right there. The likes of Google Plus, StackExchange, and Reddit do far more Integer work than floating-point, so something like Interlagos should be a good fit.

And the early benchmarks show that AMD does indeed have an edge on integer-heavy workloads over equivalent generation Intel parts. Intel still has an edge on compute-performance-per-watt, but AMD holds the edge on compute-performance-per-GHz. Pick which is more important to you.

Specialist workloads like render farms are edge cases, if big consumers, so engineering to handle those workloads is not worth the time. By staking out the middle of the market, AMD can drive innovation in the marketplace by forcing Intel to get creative in the middle. It's good for everyone.

---

Yes, but what about me, you cry.

Or something. News of the new vSphere 5 pricing guide has leaked out. Kind of like the NetFlix announcement, it has raised a lot of ire on the part of their customers. As would be expected when your preferred vendor announces you'll be paying a lot more.

The key problem has to do with how they're changing the licensing model for vSphere. We knew they'd change it, we just didn't know if they were going to put DRS and HA into a new Enterprise Plus Ultra tier, or do something else. They did something else.

With vSphere 4 the licensing tiers were based on the processor socket, number of cores, and desired features. If you had over 6 cores on that processor, you needed Enterprise Plus to use them all. If you had 6 or fewer, you could go with one of the three cheaper options.

With vSphere 5 the licensing tiers are now based on a combination of processor socket and RAM (as well as features). A 2-core socket counts as much as a 12-core socket in this scheme (yay). Unfortunately, if that dual-socket 12-core server has 256GB of RAM in it, you'll be paying for 6 Enterprise Plus licenses and not the 2 you were paying under vSphere 4. Also? The prices for Enterprise Plus haven't changed, so you just tripled your licensing costs.

vSphere 4's licensing model encouraged cramming as much RAM into a single server as possible. 12-core CPUs and buckets and buckets of RAM. And this happened, since cheaper is always good, and most VM environments are more RAM constrained than CPU constrained. With pricing per socket and not per core, you could maintain efficient RAM-to-Core ratios with licensing efficiency to boot.

vSphere 5's licensing model encourages servers with much fewer cores and a lot less RAM. Keeping a good RAM-to-Core ratio will involve a lot more physical hosts if you wish to maintain licensing efficiency. And you simply won't be able to reach the heights of efficiency you could with vSphere 4.

This is going to be expensive. We'll see if the industry moves as a whole to something else, I'm sure Citrix is salivating at the thought of upgraders upgrading to XenServer and not vSphere, or lumps it and just starts resenting the hell out of VMware the way they already resent (but still use) Oracle.

Way back in college, when I was earning my Computer Science degree, the latencies of computer storage were taught like so:

1. On CPU register
2. CPU L1/L2 cache (this was before L3 existed)
3. Main Memory
4. Disk
5. Network

This question came up today, so I thought I'd explore it.

The answer is complicated. The advent of Storage Area Networking was made possible because a mass of shared disk is faster, even over a network, than a few local disks. Nearly all of our I/O operations here at WWU are over a fibre-channel fabric, which is disk-over-the-network no matter how you dice it. With iSCSI and FC over Ethernet this domain is getting even busier.

That said, there are some constraints. "Network" in this case is still subject to distance limitations. A storage array 40km from the processing node will still see more storage latencies than the same type of over-the-network I/O 100m away. Our accesses are fast enough these days that the speed-of-light round-trip time for 40km is measurable versus 100m.

A very key difference here is that the 'network' component is handled by the operating system and not application code. For SAN an application requests certain portions of a file, the OS translates that into block requests, which are then translated into storage bus requests; the application doesn't know that the request was served over a network.

For application development the above tiers of storage are generally well represented.

1. Registers, unless the programming is in assembly, most programmers just trust the compiler and OS to handles these right.
2. L1/2/3 Cache, as above, although well tuned code can maximize the benefit this storage tier can provide.
3. Main memory, this is directly handled through code. One might argue that at a low level memory handling constitutes a majority of what code does.
4. Disk, This is represented by file-access or sometimes file-as-memory API calls. These tend to be discrete calls from main memory.
5. Network, This is yet another completely separate call structure, which means using it requires explicit programming.
   

Storage Area Networking is parked in step 4 up there. Network can include things like making NFS connections and then using file-level calls to access data, or actual Layer 7 stuff like passing SQL over the network.

For massively scaled out applications, the network has even crept into step 3 thanks to things like memcached and single-system-image frameworks.

Network is now competitive with disk, though so far the best use-cases let the OS handle the network part instead of the application doing it.

For the most part, I don't deal with software licensing. I like it this way. WWU has one person who has dealing with software licensing as a core part of her job. She does a great job at it! And like a tax-accountant, the fact that this is what she does day in and day out means that I rarely end up applying head to appropriate walls over licensing.

Licensing is a hard, hard problem, especially when you throw in that heady mix of virtualization and discounts into the mix. The software industry as a whole is still figuring out how to license stuff in a virtual environment, and we work in an industry that typically gets discounts above and beyond what others get. Add in a healthy dash of smaller software vendors that have simpler licensing regimes, and you have enough for a full time job.

Our particular place in the software ecosystem is defined by these characteristics:

• Higher Education
• State Government (we are a public university that received state support)
  
• ~4,200 staff
• ~14,000 full-time-equivalent students
• ~23,000 users (+/- 1500 depending on our exact percentage of part-timer students and where we are in the year)
• ~1,800 computer-lab seats
• ~3,000 computers

Any given software licensing regime will take one or more of the above. Most will take 'Higher Ed' into account into their licensing, which we appreciate since it makes it cheaper. Being State means we can leverage master-contracts negotiated by the State, but frequently the higher-ed discount makes independent contracts cheaper than going through Olympia.

Then comes the rest of it.

For stuff that everyone gets, like anti-virus software, the per-seat charge is where we pay the closest attention. The last time the AV contract went out for bid, the entity that won did so in large part because they applied their per-seat charge against the staff count (4,200), where the others applied it to the FTE-student + Staff number (18,200); even with a higher per-seat charge that still came in markedly cheaper.

For stuff that goes on every lab-seat, we have a mix of software that has to be licensed for every seat (1800) or a concurrency arrangement that requires a license server. We like the concurrency arrangement, but it means we need a license server.

Which brings me to license servers. We do have a FlexLM license-server, but not everything uses FlexLM. Also, FlexLM version support may end up forcing us to have more than one licensing server (which incurs OS licensing costs). It's working, but a trial.

---

That's end-user software licensing, but IT-software can be worse. We solved our Microsoft problem by caving and getting a Select agreement. While our client-access-licenses are covered, we still need to purchase Server licenses. Rolling out Exchange 2010 required picking up OS and Exchange server licenses, but we didn't have to worry about umpteen thousand CALs for our end-users. Simple, in its way. However, knowing, or know how to find out, what exact products are covered by our Select agreement is part of why our licensing person does this full time.

I have yet to meet the system administrator of any experience who hasn't had to shake their fist at ala-carte pricing for IT software. We ended up changing our backup vendor over a gross misunderstanding of the differences in licensing models between our old vender and the prospective new one. The new one wasn't any cheaper, the biggest benefit there is that we had to buy new licenses a lot less often than we had to with the old vendor.

IT software pricing competence is the biggest value-add that Value Added Resellers give, at least for us (see also, Higher Ed discounts). We don't do enough volume with IT software for our in-house software person to gain any real experience with it, so we have to trust outsiders. We trust them to know WTF they're talking about when it comes to reselling IT software. Which means that when they fail us, we get very angry. The backup software debacle ended up costing us about 120% more than we expected to pay once we got truly fully licensed for what we wanted to do, and getting to that step took nearly two years before we got it all ironed out. Since then, however, they've caught things we never even knew existed.

Like tax accountants handling very complex returns, getting a definitive answer for what is exactly the amount of licensing needed varies with who you ask. You'd think the rules would be nicely deterministic, but that presupposes complete knowledge of the whole system and uniform understanding of definitions. In the past when attempting to get an idea as to what licensing was required for a specific thing I'm thinking about we've gotten different answers from different people at the vendor itself, as well as a couple of VARs. Four people, four answers, two of whom actually work for who we were trying to buy from, fairly large price spread (though the higher-ed discount percentage was uniform across all of them).

Who do you trust? Whichever one sounds reasonable, and hope. If you get in trouble with a vendor, at least you have someone complicit in your unintended perfidity. In the case of the backup software we learned the hard way that how the software enforces licenses was different from how the VAR understood licensing to work (imperfect understanding of definitions). They did help make things right, but the sheer magnitude of the definition misunderstanding still made it very expensive.

---

For entities smaller than us, licensing is just as much of a head-ache, especially for end-user software. They may not have a Select agreement so may be purchasing off of the Microsoft/Adobe/Apple rolling cart. Virtual desktops, which we've avoided so far, make that hard to pin down since the definition of 'machine' is variable. I'm not in that space, but I hear things. It's hard.

It's hard everywhere.

I would not be at all surprised if someone could found a business on advising companies on software licensing issues.

I have a server attempting to talk SMTP to our internal smart-host. But it seems our hardware load-balancer is getting in the way. When sniffing the switch-port the server is on, the  conversation goes like this:

Server -> Mailer [SYN]
Mailer -> Server [SYN, ACK]
Server -> Mailer [Ack]
Mailer -> Server [RST, ACK]
[3 seconds pass]
Mailer -> Server [SYN, ACK]
Server -> Mailer [RST]
[6 seconds pass]
Mailer -> Server [SYN, ACK]
Server -> Mailer [RST]

What's going on here?

Well, the first three packets are the classic TCP 3-step handshake. The Mailer then issues a Acknowledge-Reset packet, which shuts down the conversation. Then things get weird. Three seconds pass, and the mailer retransmits the second packet. The Server, having shut down the TCP conversation normally like it was told to in the 4th packet, just issues a RESET packet telling the sender there is no connection to ACK and to stop trying. This repeats 6 seconds later.

So how did the Mailer forget it had torn down the TCP connection? That is the mystery. I haven't had a chance to get a sniffer on the Mailer side of things yet, so I'm not certain what it's seeing. It could be the load-balancer is throwing a fit, and the follow-on packets at 3 and 6 seconds are from the Mailer server itself somehow.

Strange things.

Virtualizing the desktop is something of a rage lately. Last year when we were still wondering how the Stimulus Fairy would bless us, we worked up a few proposals to do just that. Specifically, what would it take to convert all of our labs to a VM-based environment?

The executive summary of our findings: It costs about the same amount of money as the normal regular PC and imaging cycle, but saves some labor compared to the existing environment.

Verdict: No cost savings, so not worth it. Labor savings not sufficient to commit.

Every dollar we saved in hardware in the labs was spent in the VM environment. Replacing $900 PCs with $400 thin clients (not their real prices) looks cheap, but when you're spending $500/seat on ESX licensing/Storage/Servers, it isn't actually cheaper. The price realities may have changed from 12 months ago, but the simple fact remains that the stimulus fairy bequeathed her bounty upon the salary budget to prevent layoffs rather than spending on swank new IT infrastructure.

The labor savings came in the form of a unified hardware environment minimizing the number of 'images' needing to be worked up. This minimized the amount of time spent changing all the images in order to install a new version of SPSS for instance. Or, in our case, integrating the needed changes to cut over from Novell printing to Microsoft printing.

This is fairly standard for us. WWU finds it far easier to commit people resources to a project than financial ones. I've joked in the past that $5 in salary is equivalent to $1 cash outlay when doing cost comparisons. Our time management practices generally don't allow hour by hour level accounting for changed business practices.

Two days ago (but it seems longer) the drive that holds my VM images started vomiting bad sectors. Even more unfortunately, one of the bad sectors took out the MFT clusters on my main Win XP management VM. So far that's the only data-loss, but it's a doozy. I said unto my manager, "Help, for I have no VM drive any more, and am woe." Meanwhile I evacuated what data I could. Being what passes for a Storage Administrator around here, finding the space was dead easy.

Yesterday bossman gave me a 500GB Western Digital drive and I got to work restoring service. This drive has Native Command Queueing, unlike the now-dead 320GB drive. I didn't expect that to make much of a difference, but it has. My Vista VMs (undamaged) run noticibly faster now. "iostat -x" shows await times markedly lower than they were before when running multiple VMs.

NCQ isn't the kind of feature that generally speeds up desktop performance, but in this case it does. Perhaps lots of VM's are a 'server' type load afterall.