Monday, February 22, 2010
That's it for blogger
The new location?
See you there.
Thursday, February 11, 2010
We cut purchase orders today and rushed them through Purchasing. A Purchasing who was immensely snowed under, as can be well expected. I think final signatures get signed tomorrow.
What are we getting? Three big things:
- A new LTO4 tape library. I try not to gush lovingly at the thought, but keep in mind I've been dealing with SDLT320 and old tapes. I'm trying not to let all that space go to my head. 2 drives, 40-50 slots, fibre attached.
Made of love. No gushing, no gushing...
- Fast, cheap storage. Our EVA6100 is just too expensive to keep feeding. So we're getting 8TB of 15K fast storage. We needs it, precious.
- Really cheap storage. Since the storage area networking options all came in above our stated price-point, we're ending up with direct-attached. Depending on how we slice it, between 30-35TB of it. Probably software ISCSI and all the faults inherent in the setup. We still need to dicker over software.
The last thing we have is an email archiving system. We already know what we want, but we're waiting on determination of whether or not we can spend that already ear-marked money.
Unfortunately, I'll be finding out a week from Monday. I'll be out of the office all next week. Bad timing for it, but can't be avoided.
Wednesday, February 10, 2010
Budget crisis: a new bill
(4) For the 2009-2011 biennium, the following limitations are established upon information technology procurement:
(a) State agencies are not permitted to purchase or implement new information technology projects without securing prior authorization from the office of financial management. The office of financial management may only approve information technology projects that contribute towards an enterprise strategy or meet a critical, localized need of the requesting agency.
(b) State agencies are not permitted to purchase servers, virtualization, data storage, or related software through their operational funds or through a separate information technology budget item without securing prior authorization from the office of financial management. The office of financial management shall grant approval only if the purchase is consistent with the state's overall migration strategy to the state data center and critical to the operation of the agency.
(c) State agencies are not permitted to upgrade existing software without securing prior approval from the office of financial management. In reviewing requests from state agencies to upgrade software, the office of financial management shall grant approval only if the agency can demonstrate that upgrade of the software is critical to the operation of the agency.
In case your eyes glazed over at that, here are the bullet points:
- No software upgrades without approval from Olympia (what about service-packs? Is that an 'upgrade'?).
- No server or storage purchases without approval from Olympia.
- The State will be greatly incentivising (stick-style) usage of the State's central storage services.
- A centralized State PC Replacement process with, "at a minimum, a replacement cycle of at least five years," with a master contract containing no more than three providers of PCs with no more than four models on each contract. Which means no more than 12 PC models available at any given time.
- All mobile phone contracts are to be centralized in OFM. Presumably this includes things like Blackberry Enterprise Server, though that's not stated in the bill yet.
- The State shall develop a comprehensive data retention policy. That's OK, we probably need one anyway.
- Establish a centralized tiered storage service for use by State agencies, and all storage purchases have to be approved by OFM.
- Establish technology project standards for all K-12 school districts, mandated and overseen by OFM.
We will be keeping a close eye on this bill, yes sir.
Monday, February 08, 2010
Friday, February 05, 2010
Dealing with User 2.0
User 2.0 has different expectations of their work environment. Social and work activities are blurred, different means of communications are used. Email is dated, IM, twitter, facebook, myspace, etc are the tools to use to communicate. There is also an expectation/desire to use own equipment. Own phone, own laptop, own applications. I can hear the cries of "over my dead body" from security person 0.1 through to 1.9 all the way over here in AU. But really, why not? when is the last time you told your plumber to only use the tools you provide? We already allow some of this to happen anyway. We hire consultants, who often bring their own tools and equipment, it generally makes them more productive. Likewise for User 2.0, if using Windows is their desire, then why force them to use a Mac? if they prefer Openoffice to Word, why should't they use it? if it makes them more productive the business will benefit.
Here in the office several of us have upgraded to User 2.0 from previous versions. Happily, our office is somewhat accommodating for this, and this is good. I may be an 80% Windows Administrator these days, but that isn't stopping me running Linux as the primary OS on my desktop. A couple of us have Macs, though they both manage non-Windows operating systems so that's to be expected ;). I have seen more than one iPod touch used to manage servers. Self-owned laptops are present in every meeting we have. See us use our own tools for increased productivity.
The SANS Diary entry closed with this challenge:
So here is you homework for the weekend. How will you deal with User 2.0? How are you going to protect your corporate data without saying "Nay" to things like facebook, IM, own equipment, own applications, own …….? How will you sort data leakage, remote access, licensing issues, malware in an environment where you maybe have no control or access over the endpoint? Do you treat everyone with their own equipment as strangers and place them of the "special" VLAN? How do you deal with the Mac users that insist their machines cannot be infected? Enjoy thinking about User 2.0, if you send in your suggestions I'll collate them and update the diary.
Being a University we've always had a culture that was supportive of the individual, that Academic Freedom thing rearing its head again. So we've had to be accommodating to this kind of user for quite some time. What's more, we put a Default-Deny firewall between us and the internet really late in the game. When I got here in 2003 I was shocked and appalled to learn that the only thing standing between my workstation and the Internet were a few router rules blocking key ports; two months later I was amazed at just how survivable that ended up being. What all this means is that end-user factors have been trumping or modifying security decisions for a very long time, so we have experience with these kinds of "2.0" users.
When it comes to end-user internet access? Anything goes. If we get a DMCA notice, we'll handle that when it arrives. What we don't do is block any sites of any kind. Want to surf hard-core porn on the job? Go ahead, we'll deal with it when we get the complaints.
Inbound is another story entirely, and we've finally got religion about that. Our externally facing firewall only allows access to specific servers on specific ports. While we may have a Class B IP block and therefore every device on our network has a 'routable' address, that does not mean you can get there from the outside.
As for Faculty/Staff computer config, there are some limits there. The simple expedient of budget pressure forces a certain homogeneity in hardware config, but software config is another matter and depends very largely on the department in question. We do not enforce central software there beyond anti-virus. End users can still use Netscape 4.71 if they really, really, really want to.
Our network controls are evolving. We've been using port-level security for some time, which eliminates the ability of students to unplug the ethernet cable connected to a lab machine and plug it into their laptop. That doesn't stop conference rooms where such multi-access is expected. And we only allow one MAC address per end-port, which eliminates the usage of hubs and switches to multiply a port (and also annoy VMWare users). We have a 'Network Access Control' client installed, but all we're doing with it so far is monitor; efforts to do something with it have hit a wall. Our WLAN requires a WWU login for use, and nodes there can't get everywhere on the wired side. Our Telecom group has worked up a LimboVLAN for exiling 'bad' devices, but it is not in use because of a disagreement over what constitutes a 'bad' device.
However, if given the choice I can guarantee certain office managers would simply love to slam the bar down on non-work related internet access. What's preventing them from doing so are professors and Academic Freedom. We could have people doing legitimate research that involves viewing hard core porn, so that has to be allowed. So the 'restrict everything' reflex is still alive and strong around here, it has just been waylaid by historic traditions of free access.
And finally, student workers. They are a second class citizen around here, there is no denying that. However, they are the very definition of 'User 2.0' and they're in our offices providing yet another counter-weight to 'restrict-everything'. Our Helpdesk has a lot of student workers, so we end up with a fair amount of that attitude in IT itself which helps even more.
Universities. We're the future, man.
Wednesday, February 03, 2010
Free information, followup
- Academic freedom. This has been a tradition for longer than 'information wants to be free' has been a catch-phrase. While I'm on the business side rather than the academic side, some of that liberalism splashes over. Which means I can talk about what I do every day.
- I work for the state. In theory everything I do in any given day can be published by way of a Freedom of Information Act request, or as they're called here in Washington State a Public Records Request. Which means that even if I wanted to hide what I was doing, any inquisitive citizen could find it out anyway. So why bother hiding things?
I passed my 6 year blogaversary earlier last month, and if it is one thing I've learned is that people appreciate examples. It's one thing to describe how to fix a problem, and quite another (more useful) thing to provide the context in which a problem arose. It's the examples that are hard to provide when you have to protect trade secrets.
So, yes. I'm creating free information, in significant part because I work somewhere that values free information.
Part of his point is that it is one thing to enable information to be free, and quite another to create free information. Information creation is the ultimate validation of this credo. In his case, he can work with his publishers to release novels in a non-DRMed format; something he has done once and will do again soon.
But he closes with a question:
What have you created and released lately?That's a very good question. The quick answer to that is this blog. My experiences wrestling with technology have proven useful to others. The search key-words that drive people here have evolved over time, but give a nice snapshot for what issues people are having and are looking for answers about. For a long time that was news about the Novell client for Vista. Right this moment the top trending keywords all include two of the following terms 'cifs', 'Windows 7', 'Netware', and 'OES', strongly suggests people looking for how to connect Vista/Win7 to NetWare/OES. Comments I've received have also proven that what I've posted here has been useful to others.
But what about beyond that? I've written a couple of AppNotes for Novell over the years covering topics that the NetWare-using community didn't have adequate coverage over. Novell has always had a stake in 'community', which fosters this sort of information sharing.
I've also been active on ServerFault, a sort of peer-support community for system administrators. I don't get as good data about what my contributions there are being used for, but I do still get comments on accepted answers months after their original posting. I'm in the top 25 for reputation there, so that's something.
It doesn't look like a lot, but it is free information out there. In both senses of the word.
Tuesday, February 02, 2010
As far as WWU is concerned, we know we'll be passed some kind of cut. We don't know the size, nor do we know what other strings may be attached to the money we do get. So we're planning for various sizes of cuts.
One thing that is definitely getting bandied about is the idea of 'sweeping' unused funds at end-of-year in order to reduce the deficits. As anyone who has ever worked in a department subject to a budget knows, the idea of having your money taken away from you for being good with your money runs counter to every bureaucratic instinct. I have yet to meet the IT department that considers themselves fully funded. My old job did that; our Fiscal year ended 12/15, which meant that we bought a lot of stuff in October and November with the funds we'd otherwise have to give back (a.k.a. "Christmas in October"). Since WWU's fiscal year starts 7/1, this means that April and May will become 'use it or lose it' time.
Sweeping funds is a great way to reduce fiscal efficiency.
In the end, what this means is that the money tree is actually producing at the moment. We have a couple of crying needs that may actually get addressed this year. It's enough to completely fix our backup environment, OR do some other things. We still have to dicker over what exactly we'll fix. The backup environment needs to be made better at least somewhat, that much I know. We have a raft of servers that fall off of cheap maintenance in May (i.e. they turn 5). We have a need for storage that costs under $5/GB but is still fast enough for 'online' storage (i.e. not SATA). As always, the needs are many, and the resources few.
At least we HAVE resources at the moment. It's a bad sign when you have to commiserate with your end-users over not being able to do cool stuff, or tell researchers they can't do that particular research since we have no where to store their data. Baaaaaad. We haven't quite gotten there yet, but we can see it from where we are.
Thursday, January 28, 2010
You see, since we've been a NetWare shop since, uh, I don't know when, we have certain habits ingrained into our thinking. I've already commented on some of it, but that thinking will haunt us for some time to come.
The first item I've touched on already, and that's how you set permissions at the top of a share/volume. In the Land of NetWare, practically no one has any rights to the very top level of the volume. This runs contrary to both Microsoft and Posix/Unix ways of doing it, since both environments require a user to have at least read rights to that top level for anything to work at all. NetWare got around this problem by creating traverse rights based on rights granted lower down the directory structure. Therefore, giving a right 4 directories deep gave an inplicit 'read' to the top of the volume. Microsoft and Posix both don't do this weirdo 'implicit' thing.
The second item is the fact that Microsoft Windows allows you to declare a share pretty much anywhere, and NetWare was limited to the 'share' being the volume. This changed a bit when Novell introduced CIFS to NetWare, as they introduced the ability to declare a share anywhere; however, NCP networking still required root-of-volume only. At the same time, Novell also allowed the 'map root' to pretend there is a share anywhere but it isn't conceptually the same. The side-effect of being able to declare a share anywhere is that if you're not careful, Windows networks have share-proliferation to a very great extent.
In our case, past-practice has been to restrict who gets access to top-level directories, greatly limit who can create top-level directories, and generally grow more permissive/specific rights-wise the deeper you get in a directory tree. Top level is zilch, first tier of directories is probably read-only, second tier is read/write. Also, we have one (1) shared volume upon which everyone resides for ease of sharing.
Now, common-practice among Microsoft networks is something I'm not that familiar with. What I do know is that shares proliferate, and many, perhaps most, networks have the shares as the logical equivalent of what we use top-level directories for. Where we may have a structure like this, \\cluster-facshare\facshare\HumRes, Microsoft networks tend to develop structures like \\cluster-facshare\humres instead. Microsoft networks rely a lot on browsing to find resources. It is common for people to browse to \\cluster-facshare\ and look at the list of shares to get what they want. We don't do that.
One thing that really gets in the way of this model is Apple OSX. You see, the Samba version on OSX machines can't browse cluster-shares. If we had 'real' servers instead of virtual servers this sort of browse-to-the-resource trick would work. But since we have a non-trivial amount of Macs all over the place, we have to pay attention to the fact that all a Mac sees when they browse to \\cluster-facshare\ is a whole lot of nothing. We're already running into this, and we only have our user-directories migrated so far. We have to train our Mac users to enter the share as well. For this reason, we really need to stick to the top-level-directory model as much as possible, instead of the more commonly encountered MS-model of shares. Maybe a future Mac-Samba version will fix this. But 10.6 hasn't fixed it, so we're stuck for another year or two. Or maybe until Apple shoves Samba 4 into OSX.
Since we're on a fundamentally new architecture, and can't use common-practice, our sense of best-practice is still evolving. We come up with ideas. We're trying them out. Time will tell just how far up our heads are up our butts, since we can't tell from here just yet. So far we're making extensive use of advanced NTFS permissions (those permissions beyond just read, modify, full-control) in order to do what we need to do. Since this is a deviation from how the Windows industry does things, it is pretty easy for someone who is not completely familiar with how we do things to mess things up out of ignorance. We're doing it this way due to past-practice and all those Macs.
In 10 years I'm pretty sure we'll look a lot more like a classic Windows network than we do now. 10 years is long enough for even end-users to change how they think, and is long enough for industry-practice to erode our sense of specialness more into a compliant shape.
In the mean time, as the phone ringing off the hook today foretold, there is a LOT of learning, decision-making, and mind-changing to go through.
Monday, January 25, 2010
HP: What kind disk are you thinking of?
US: Oh, probably mid tier. 10K SAS would be good enough.
HP: Well, SAS only comes in 15K, and the next option down is 7.2K SATA. And really, the entire storage market is moving to SAS.
Note the lack of Fibre Channel drives. Those it seems are being depreciated. Two years ago the storage tier looked like this:
Back in 2003 when we bought that EVA3000 for the new 6 node NetWare cluster, clustering required shared storage. In 2003, shared storage meant one of two things:
- SCSI and SCSI disks, if using 2 nodes.
- Fibre Channel and FC Disks if using more than 2 nodes.
Now if only we had some LTO drives to back it all up.