November 2006 Archives


Novell Open Audio had a podcast last week about Open Enterprise Server 2 (it's official, that's the name). It was quite long, and full of nice information. Probably the best bit was about Shadow Volumes, which I've mentioned before. That just keeps getting better and better! I highly recommend listening to the pod-cast.

I've known for a while that it allows policy based migration of data to older/slower/cheaper media. Unlike traditional HSM technologies, Shadow Volumes are based on the last-modified date rather than the last-accessed date. Also, policies can include file types as well, so you can migrate your large multi-media files to media that handles long contiguous reads better. Or just migrate files larger than 50MB to that faster media. Whatever.

One scenario mentioned in the pod-cast was about data migrations of extremely large data. One Novell cluster mentioned in the pod-cast had 420TB of data in it. Ooo! Migrating THAT to a new SAN would take weeks. How it works is this:
  1. Set up the new server
  2. Configure the volume on the new server to use the old SAN as the migrate (i.e. slow) media
  3. Do the server migration itself
  4. As users modify data on the old SAN, it gets tagged for migration to the new SAN. In a week/month/whatever most of the active data is on the new SAN, and the older SAN gets less and less data.
  5. When the time comes to decomission the old SAN (assuming that's what you want to do) the total data migration is a lot easier.
Freaking cool.

Unfortunately, this is an OES2-Linux product only. It can use NetWare volumes as migration targets, but NetWare won't do the policy based decisions. Darn.

Also mentioned is that OES2 will include SP7 for NetWare 6.5, which will introduce Xen paravirtualization to NetWare. IMHO, this is spiffy if your hardware vendor has stopped significant NetWare support (*cough*dell*cough*) and you still need to use it. For us we'll probably stick with 'bare metal' installs for the time being, at least until we get proof that running a Xen-virtualized NetWare instance on a 64-bit server runs faster than the same NetWare running bare-metal on a 64-bit server (in 32-bit emulation mode).

It also sounds like they've spend serious time getting NSS and NCP faster. This is very needed, as I showed earlier. As file I/O is much more CPU bound on Linux than on NetWare, any improvements they can make will be appreciated.

Also, they hope (but are not promising) to give out a public beta of OES2 to all BrainShare attendees. I predict another round of benchmarking come early April.

OES2 is currently slated for relase in the late-May early-June timeframe. This is nifty, as that's the start of Summer for us. Though, we're not migrating right away unless we are blown away by the differences.


To the best of my knowledge WWU is the only place of higher education open north of Snohomish county. There is one public school open in the county, and that's in the back country where they get snow more often due to elevation. Classes here started at 9am, only the 8am classes got canceled.

The drive into work was quite interesting. They don't believe in salt out here, nor do they get this kind of snow every year. This means that the bare-and-dry formula for roads of:
  1. Plow to get the top layer off
  2. Salt to melt the compacted stuff
  3. Plow again to get the now-slushy compacted stuff off the road
  4. Sand to make refreeze less dangerous and provide targets for solar melting
Isn't followed here. In town they just plow and sand. Happily, on I-5 WSDOT handles that and they know about deicers (not salt, though). I-5 is pretty clear right now, even though downtown Bellingham has no clear roads. The major arterials have bare pavement poking out in spots, but it isn't what you'd call clean. Or safe.

I put chains on my car to get out of where I live. It's the only way. I also kept them on for the whole drive in to work, because that's safest and it's hard to unchain on demand. Happily, I don't have to use I-5 for any of my drive to work.

There are a lot of people in the office today, but I know of a few snow birds who are iced in and not coming to work.

What's worse is that we're expecing snow, then rain overnight. Which means tomorrow Bellingham may be a skating rink!

Snow day

Due to the weather system we're having, WWU has canceled classes Monday. This is good, because the only way I'm getting in is with a 1.5 mile hike up and down some nasty hills and taking the bus. Total commute time: 3 hours. Experience tells me that I'll be out Tuesday, since the right weather conditions will not be present to make the roads up here passible.


We had a SATA drive go out in the MSA. It was supposed to arrive Monday, but clearly that's not going to happen. UPS should re-deliver Tuesday. Happily, we have a UPS Store on the other side of the shopping mall from us.

Changing pains

Well! In my attempt to re-tag old posts, it turns out blogger is also updating the RSS/Atom files as I publish them. That's why you have a boat load of new/old posts. Sorry about that, didn't know they were doing that. Bit of a pain, really.

Changing templates

Right. I've migrated to the new blogger system, and hope that everything works out. It looks like they didn't make significant template changes, but we'll see about that.

Update: It took bloody forever for the blog to convert. I guess my 679 posts took a while to chew over. Also, it looks like they're ALSO republishing each post in its own label area. If you click on a label, you get a page with ALL THE POSTS that have that label. If I reindexed everything and you clicked on the 'novell' link you'd probably download an unusably large file. Not sure what I think of that yet.
Several of the sessions I attended at BrainShare this year were on AppArmor. The project lead for that product presented several times, and several times he repeated this mantra. A Virtual Machine is not a security barrier. This is true for full-virtualization products such as VMWare, and paravirtualization such as Xen.

Yesterday's SANS diary had an entry about VM detection on the part of malware. As you can well imagine, spyware and adware researchers make great use of products like VMWare to analyze their prey. VMWare has handy snapshoting abilities, which makes reverting your VM to pre-infection state easy. Unfortunately for them, "3 out of 12 malware specimens recently captured in our honeypot refused to run in VMware." The bad-ware authors are also aware of this and program their stuff to not run.

What's more insidious is that there are cases where the malware doesn't use the VMware detection to not run, but to infect the HOST machine instead. While this may not affect something like ESX Server which is a custom OS, other products like Xen in full virtualization mode or VMWare Server running on Windows or Linux would be exposed this way. Figuring out that your malware process is running in a virtual machine is easy and scriptable, and breaking out of the VM is just as scriptable.

Virtual Machines are not a security barrier, nor do they make you significantly safer. They're just different.

Tags: , ,

New PC update


I managed to get SLES10 installed in the end. Getting it there required installing WinXP on one drive, fiddling with that, then creating a FAT32 volume of about 5GB. Then copying the contents of the DVD to that itty-bitty partition (when did 5GB become itty bitty? Hmm). Then booting from the SLES10 DVD and pointing the install at the hard-drive. That got it in there alright! Woo!

But... SLES10 isn't quite what I'm looking for as it turns out. So now I'm repeating the same trick with openSUSE 10.2b2 as I type this. I don't have a DVD burner so I couldn't install from DVD. On the other hand, during the boot messages I definitely saw it notice the JMicron chipset for PATA. I'm pretty sure this WILL see my optical drives!

Maybe whatever version of Xensource that comes with openSUSE will work a bit better with WinXP as a guest. Either way, I'm nuking this build when the Release comes.

Blogging quiet

The hardware for our new computers arrived yesterday. This means that I have the functioning wreckage of two PCs strewn across my office as I get the data transferred. It also means that I haven't had much time for blogging, or much to blog about.

Perhaps the most significant thing is that at least two of us will be using virtualization for our desktops. Base OS, and the real production stuff sitting on top of that ready for VMWare snapshotting. Going in with both feet you might say.

I tried really hard to get SLES10 (we don't have any SLED10 licenses) onto the new system, but I ran into two big road-blocks. First and foremost, the Intel P965 chipset has no built in PATA support so that had to be provided third party. All of my optical drives are PATA, and the third party, JMicron, just plain don't work with Linux (though it will with the 2.6.18 kernel, which SLES doesn't have, but openSUSE 10.2 will). Secondly, the Marvell NICs need a driver that isn't in SLES making a network install challenging.

The first made a DVD install impossible, though it could boot-from-DVD. The second required me to put an E100 card into the machine to attempt a NFS install. Unfortunately, I ran into problems with the network install dying periodically about packages failing MD5 checks... even though the DVD iso MD5 checked out just peachy. Three times, three different packages. That's when I threw in the towel.

So I'm keeping one of the two hard-drives in this new machine blank until either openSUSE with Xen releases, or a SLES version with 2.6.18 (or newer) comes out. Whichever. I REALLY wanted that 64-bit support, as this machine has 4GB of RAM and I'd really like to use it all at full speed. With WinXP, I'm stuck for the time being. Happily, VMWare machines are transportable between Windows and Linux.

Maybe I'll look at Server 2003-64 bit tomorrow. Hmm.

Getting new hardware

It's time to refresh what we're using for desktop machines here in Technical Services. We'll be whiteboxing this for a couple of reasons:
  1. The guy who is doing the whiteboxing did it for a living before he got out of the business due to low margins.
  2. At no time will ATUS be servicing these machines, we'll be doing all the support ourselves.
  3. Part-reuse, such as optical devices, brings the price down noticably.
These are workstations for people who routinely use various God-level passwords and accounts. Our rank-n-file desktop support people aren't responsible for managing these stations.

That said, it'll be a Core2 Duo config. Several of us are actively considering using virtualization on these stations as mainline, rather than development. One is going to be using a bare install of Windows as the Host OS and doing everything productiony in a guest on VMWare. The guest snap-shot abilities of VMWare are the prime reason he wants to do that, as that beats the pants off of Microsoft's "system restore" process.

Me? I just spent about an hour reading up on Xen in SLES10. So far only SLES10 is supported as a guest OS, which is dissapointing. WinXP is in 'technical preview, don't use it,' mode. One thing that Xen doesn't support very well is CD-Burners, which will be tricky to engineer around. I had higher hopes for Xen. Perhaps SLES10 SP1 will have improvements? Hard to say.

NW65SP6 is out

SP6 for NetWare 6.5 is out. It had a rather short beta period, though, so I'm not inclined to trust it quite yet.

Tags: ,

Vendor lock-in, my reality

| 1 Comment
Freedom from vendor lock-in is one of the louder themes in the criticism in this post Novell/Microsoft-deal world. Free and Open Source Software (FOSS, learned that one this week) is all about avoiding vendor lock-in. Linux comes in the wide flavors it does so that vendor lock-in doesn't have to happen.

Here is how I view vendor lock-in as a part of my job. This is coming from a system administrator who manages in a largish regional enterprise. Both this job and OldJob didn't have significant WAN links, but we did have well over 1000 users to play with.

Both my old job and my new job saw significant savings by sole-sourcing server hardware. OldJob was a Dell shop pretty completely, with a very, very few Compaq hold-outs. Here in Technical Services we're a very solid HP shop, with some Dell thrown in as we inherit systems from other departments. Both old and new jobs had some Solaris around for various things. At OldJob we didn't use OpenManage much at all. Here, we're not really using OpenView.

Both companies realized that whiteboxing, building servers from component parts and supporting things internally, was a losing idea at the scales both jobs built out at. Having a single vendor support us for server hardware eased support costs and made the internal knowledge-base required to support our hardware much less complex.

OldJob used to be a Compaq shop, but dropped it in favor of Dell due to and I quote directly, "all of that proprietary crap." The fact that Dell was a lot cheaper at the time also helped significantly. As I understand it, Technical Services was that rarity, a HP shop. The Compaq merger was viewed with some dismay in certain parts. This is why we have some HP LH3's stacked against one wall.

As for operating systems, OldJob and NewJob had different ideas. One of the reasons I left OldJob was a decision from the top of IT management to unify on a single Directory; Microsoft's. How that would interact with GroupWise, which was VERY firmly established in the enterprise and not being moved out, was a complete mystery to us techs who would eventually be asked to make it work. Accompanying this decision was the directive to start migrating the NetWare servers out in favor of Microsoft Windows. They were reducing support costs by reducing the number of OSs in the datacenter.

WWU works differently. They've had a 'best of breed' philosophy for services for some time, and NetWare is still king for File Serving performance. Solaris was king for Oracle for a long time, though that throne is in a bit of question at the moment. Linux has started creeping in as an application server a few places, so far just web-servers (the web-server that drives the time-card web front-end is Linux, with a Solaris/Oracle back end).

One of the differences between my old environment and this environment is budgetary. OldJob had a strict three year replacement cycle for servers, that they were thinking of extending to 3.5 or 4 yeaers to better handle server swaps (3-6 month stage up and migration, 3-6 month demigration and stage down). WWU plans on keeping servers in some form of production for 5 years, though the last one to two years may be as a dev server rather than mainline production. The same sort of pressures happen on the software we use on the servers, as we're more likely to use OSS than we were at my old job.

That said, when it comes to package management on the Solaris and Linux servers we rarely go outside of the vendor for packages. I know we compile Samba from source on Solaris, but then we're using an old Solaris version that doesn't do what we need out of the box very well. If it isn't on the Solaris or SLES9 repositories (we don't have any SLES10 in yet), we probably won't deploy on that.

Our OS vendors are Microsoft, Novell, and Solaris. Other departments use Gentoo, Slackware, and I'm sure some Ubuntu and Red Hat.

All that said, we do deal with, and embrace, vendor lock-in. We do it in hardware. We do it in software, as our Help Desk software only runs on a Windows/IIS stack (though can do MSSQL or Oracle as a DB source). We do it in file serving OS, as it would be a royal pain to use anything but an NCP server from Novell on NetWare (even OES-Linux presents some problems). We do it in small-office databases, which are all housed on an MS-SQL server (we can do oracle, but it is a pain and the processes are not well established or documented).

We also embrace diversity. Our HelpDesk app requires Windows/IIS, but our student email web-access app required Apache/PHP. Web-serving is perhaps our most diversified environment, as we are widely deployed in both Apache and IIS based technologies. Several main line applications are java driven, others are driven by ASP, still others use PHP.

When it comes to an OS we'll use on mainline applications, we have to have enterprise support. This means buying our OS from a firm that can provide 24x7x365.25 support. This reduces the number of firms we can purchase Linux support from. Additionally, we need some form of patch management system. NetWare is the least well supported when it comes to patch management, as it is a 100% manual process, happily it doesn't need a lot of attention. We won't do NetWare-style patch management on Linux, as that method doesn't scale. In effect we're limited to Red Hat or Novell for Linux, and as we already have a contract with Novell for NetWare support that's where we went. Furthering our lock-in with Novell.

So vendor lock in is really a fact of life here. I appreciate the fact that I can toss a Slackware server in amongst the servers and have a fair chance of making it work if I really need it to. But that'll be emergency processing, not standard procedure.

I'll leave what the Novell/Microsoft deal means in terms of the viability of Linux in the enterprise for another post.

Tags: ,

The Novell/Microsoft compact

| 1 Comment
I just don't have enough background in the Open Source Software environment to comment knowledgably about how this new deal impacts OSS as a whole. I know there are severe misgivings that Novell has sold out, Novell will go the way of a lot of past Microsoft partners, or Novell will by proxy poison Linux with Microsoft's 'adopt and extend' tendrils. It is not without reason that the OSS community views Microsoft as the poster-company for closed-source development.

Now I'm just speaking for me. Speaking as a person who takes vendor lock-in in my datacenter as a cost of doing business, the prospect of only being able to get Linux from Novell doesn't strike fear into my heart. Novell is already a provider of Operating Systems in my datacenter, so this doesn't change anything. It just means we're a bit less exposed to patent-based lawsuits, and I wasn't terribly concerned about that anyway.

The prospect of increased interoperability between SLES and Windows, and hopefully by proxy eDir on NetWare, is a good one. We have a lot of call to replicate identity information between AD and eDir, which is a role handled through custom software. We can't afford pay-for software like IDM. The possibility of federating identity between the two will make some things a lot easier. GroupWise on AD? Exchange on eDir? Could happen! Wouldn't that be a kick in the pants?

In terms of open source in general, what we do use is generally provided by Solaris not Novell. Part of that is an artifact of the fact that most of the xNIX we're running around here is Solaris, not Linux. We have a couple of SLES servers in production, but again generally speaking the only OSS we're running on those is the OS itself.

Virtualization is another area where we'd see some gains. If those two can work together to make Windows more paravirtualization-friendly, or SLES more windows-in-VM friendly we might actually use it. Right now ESX Server is the only real choice, and that's the way we're going to dance for the time being.

And now we all wait with bated breath for what the heck Red Hat is going to do after all of this.

Tags: ,

An interesting request

| 1 Comment
The other day I got asked a question that I hadn't considered before.

"Can you set up rights so that a certain group of student workers can only access this directory when they're at work, and not anywhere else?"

Clearly they don't trust the students all that much, but the question is still interesting. How to do that? We've spent a lot of effort promoting 'mobility' in terms of getting at your files from anywhere. Novell NetStorage is designed for exactly this. Yet how do you restrict access to a specific directory to:

(userMemberOf specificgroup.groups.wwu) AND (workstationMemberOf othergroup.wsgroups.wwu)

In NetStorage perhaps the easiest way is to make sure the drive that directory is in is not contained in any login-scripts the user has. That means it won't show up in NetStorage. On the other hand, if they come in on SFTP the files are still there for the taking. The problem with this is that the volume in question is in their login script already.

Another way to handle that is to create a second set of accounts for the students. These accounts would be workstation-login-restricted to just the workstations the department designates. Because of this, they won't be able to use those logins for NetStorage or SFTP as the servers (what shows up in the 'Network Address' field when you log in via NetStorage or SFTP) isn't in the approved list. The problem with this is that we have a strong 'one account' policy, even super-users like myself don't have a second low-priv account for routine use.

The crux of it is that this is the first time I've been asked to build location-awareness into an ACL. I wonder how other companies are handling this?

Tags: ,

A disturbance in the force

There was a press conference and now the Novell Press Release is out.

In short, Novell and Microsoft are teaming up! They'll be not stomp on eachothers toes in a few key areas, and are infact planning on working together to make some things work better industry wide. Chief among them is virtualization.

Not that surprising, but I think this means that Xen may get a lot better support as a result of this. I've seen tech demos, but haven't seen any productions sytems yet. This could be some serious mojo.
Web services for managing physical and virtual servers. Web services and service-oriented architectures continue to be one of the defining ways software companies can deliver greater value to customers. Microsoft and Novell will undertake work to make it easier for customers to manage mixed Windows and SUSE Linux Enterprise environments and to make it easier for customers to federate Microsoft Active Directory® with Novell eDirectory.
Federating AD and eDir? That's a heckova thing. Somehow I don't think they'll let you run Exchange with eDir, but a guy can dream.

Could this mean... a Microsoft booth at BrainShare??

This is a major announcement. Wow.