Monday, February 08, 2010

OpenSUSE Survey

It's time for another openSUSE survey! If you're an openSUSE user (or even a user of SLES/SLED) it's a good idea to take this thing. They set development priorities based on these surveys, so if you have an area that needs buffing up this is the place to tell them. Or if you want to tell them 'works great!' this is where you do it too.

http://www.surveymonkey.com/s/6MJYV7T

Labels: ,


Friday, February 05, 2010

Dealing with User 2.0

The SANS Diary had a post this morning with the same title as this post. The bulk of the article is about how user attitudes have changed over time, from the green-screen era to today where any given person has 1-2 computing devices on them at all times. The money quote for my purposes is this one:

User 2.0 has different expectations of their work environment. Social and work activities are blurred, different means of communications are used. Email is dated, IM, twitter, facebook, myspace, etc are the tools to use to communicate. There is also an expectation/desire to use own equipment. Own phone, own laptop, own applications. I can hear the cries of "over my dead body" from security person 0.1 through to 1.9 all the way over here in AU. But really, why not? when is the last time you told your plumber to only use the tools you provide? We already allow some of this to happen anyway. We hire consultants, who often bring their own tools and equipment, it generally makes them more productive. Likewise for User 2.0, if using Windows is their desire, then why force them to use a Mac? if they prefer Openoffice to Word, why should't they use it? if it makes them more productive the business will benefit.

Here in the office several of us have upgraded to User 2.0 from previous versions. Happily, our office is somewhat accommodating for this, and this is good. I may be an 80% Windows Administrator these days, but that isn't stopping me running Linux as the primary OS on my desktop. A couple of us have Macs, though they both manage non-Windows operating systems so that's to be expected ;). I have seen more than one iPod touch used to manage servers. Self-owned laptops are present in every meeting we have. See us use our own tools for increased productivity.

The SANS Diary entry closed with this challenge:

So here is you homework for the weekend. How will you deal with User 2.0? How are you going to protect your corporate data without saying "Nay" to things like facebook, IM, own equipment, own applications, own …….? How will you sort data leakage, remote access, licensing issues, malware in an environment where you maybe have no control or access over the endpoint? Do you treat everyone with their own equipment as strangers and place them of the "special" VLAN? How do you deal with the Mac users that insist their machines cannot be infected? Enjoy thinking about User 2.0, if you send in your suggestions I'll collate them and update the diary.


Being a University we've always had a culture that was supportive of the individual, that Academic Freedom thing rearing its head again. So we've had to be accommodating to this kind of user for quite some time. What's more, we put a Default-Deny firewall between us and the internet really late in the game. When I got here in 2003 I was shocked and appalled to learn that the only thing standing between my workstation and the Internet were a few router rules blocking key ports; two months later I was amazed at just how survivable that ended up being. What all this means is that end-user factors have been trumping or modifying security decisions for a very long time, so we have experience with these kinds of "2.0" users.

When it comes to end-user internet access? Anything goes. If we get a DMCA notice, we'll handle that when it arrives. What we don't do is block any sites of any kind. Want to surf hard-core porn on the job? Go ahead, we'll deal with it when we get the complaints.

Inbound is another story entirely, and we've finally got religion about that. Our externally facing firewall only allows access to specific servers on specific ports. While we may have a Class B IP block and therefore every device on our network has a 'routable' address, that does not mean you can get there from the outside.

As for Faculty/Staff computer config, there are some limits there. The simple expedient of budget pressure forces a certain homogeneity in hardware config, but software config is another matter and depends very largely on the department in question. We do not enforce central software there beyond anti-virus. End users can still use Netscape 4.71 if they really, really, really want to.

Our network controls are evolving. We've been using port-level security for some time, which eliminates the ability of students to unplug the ethernet cable connected to a lab machine and plug it into their laptop. That doesn't stop conference rooms where such multi-access is expected. And we only allow one MAC address per end-port, which eliminates the usage of hubs and switches to multiply a port (and also annoy VMWare users). We have a 'Network Access Control' client installed, but all we're doing with it so far is monitor; efforts to do something with it have hit a wall. Our WLAN requires a WWU login for use, and nodes there can't get everywhere on the wired side. Our Telecom group has worked up a LimboVLAN for exiling 'bad' devices, but it is not in use because of a disagreement over what constitutes a 'bad' device.

However, if given the choice I can guarantee certain office managers would simply love to slam the bar down on non-work related internet access. What's preventing them from doing so are professors and Academic Freedom. We could have people doing legitimate research that involves viewing hard core porn, so that has to be allowed. So the 'restrict everything' reflex is still alive and strong around here, it has just been waylaid by historic traditions of free access.

And finally, student workers. They are a second class citizen around here, there is no denying that. However, they are the very definition of 'User 2.0' and they're in our offices providing yet another counter-weight to 'restrict-everything'. Our Helpdesk has a lot of student workers, so we end up with a fair amount of that attitude in IT itself which helps even more.

Universities. We're the future, man.

Labels: ,


Wednesday, February 03, 2010

Free information, followup

As for the previous post, my information sharing has in large part been facilitated by my place of work. I work for a publicly funded institution of higher learning. Because of this, I have two biiiig things working in my favor:
  1. Academic freedom. This has been a tradition for longer than 'information wants to be free' has been a catch-phrase. While I'm on the business side rather than the academic side, some of that liberalism splashes over. Which means I can talk about what I do every day.
  2. I work for the state. In theory everything I do in any given day can be published by way of a Freedom of Information Act request, or as they're called here in Washington State a Public Records Request. Which means that even if I wanted to hide what I was doing, any inquisitive citizen could find it out anyway. So why bother hiding things?
If I were working for a firm that has significant trade secrets I'm pretty sure I couldn't blog about a lot of the break/fix stuff I've blogged about. Opinion, yes. Examples from my work life? Not so much.

I passed my 6 year blogaversary earlier last month, and if it is one thing I've learned is that people appreciate examples. It's one thing to describe how to fix a problem, and quite another (more useful) thing to provide the context in which a problem arose. It's the examples that are hard to provide when you have to protect trade secrets.

So, yes. I'm creating free information, in significant part because I work somewhere that values free information.

Labels:


Free information

Charles Stross had a nice piece this morning about that long time hacker slogan, "Information wants to be free". It's a good read, so I'll wait while you go read it. It focuses on the different definitions of free. One means, "no cost," like those real-estate fliers you see at the grocery store. The other means, "free to move," like Amazon MP3 Store mp3 files. Different, see.

Part of his point is that it is one thing to enable information to be free, and quite another to create free information. Information creation is the ultimate validation of this credo. In his case, he can work with his publishers to release novels in a non-DRMed format; something he has done once and will do again soon.

But he closes with a question:
What have you created and released lately?
That's a very good question. The quick answer to that is this blog. My experiences wrestling with technology have proven useful to others. The search key-words that drive people here have evolved over time, but give a nice snapshot for what issues people are having and are looking for answers about. For a long time that was news about the Novell client for Vista. Right this moment the top trending keywords all include two of the following terms 'cifs', 'Windows 7', 'Netware', and 'OES', strongly suggests people looking for how to connect Vista/Win7 to NetWare/OES. Comments I've received have also proven that what I've posted here has been useful to others.

But what about beyond that? I've written a couple of AppNotes for Novell over the years covering topics that the NetWare-using community didn't have adequate coverage over. Novell has always had a stake in 'community', which fosters this sort of information sharing.

I've also been active on ServerFault, a sort of peer-support community for system administrators. I don't get as good data about what my contributions there are being used for, but I do still get comments on accepted answers months after their original posting. I'm in the top 25 for reputation there, so that's something.

It doesn't look like a lot, but it is free information out there. In both senses of the word.

Labels:


Tuesday, February 02, 2010

Budget plans

Washington State has a $2.6 Billion deficit for this year. In fact, the finance people point out that if something isn't done the WA treasury will run dry some time in September and we'll have to rely on short-term loans. As this is not good, the Legislature is attempting to come up with some way to fill the hole.

As far as WWU is concerned, we know we'll be passed some kind of cut. We don't know the size, nor do we know what other strings may be attached to the money we do get. So we're planning for various sizes of cuts.

One thing that is definitely getting bandied about is the idea of 'sweeping' unused funds at end-of-year in order to reduce the deficits. As anyone who has ever worked in a department subject to a budget knows, the idea of having your money taken away from you for being good with your money runs counter to every bureaucratic instinct. I have yet to meet the IT department that considers themselves fully funded. My old job did that; our Fiscal year ended 12/15, which meant that we bought a lot of stuff in October and November with the funds we'd otherwise have to give back (a.k.a. "Christmas in October"). Since WWU's fiscal year starts 7/1, this means that April and May will become 'use it or lose it' time.

Sweeping funds is a great way to reduce fiscal efficiency.

In the end, what this means is that the money tree is actually producing at the moment. We have a couple of crying needs that may actually get addressed this year. It's enough to completely fix our backup environment, OR do some other things. We still have to dicker over what exactly we'll fix. The backup environment needs to be made better at least somewhat, that much I know. We have a raft of servers that fall off of cheap maintenance in May (i.e. they turn 5). We have a need for storage that costs under $5/GB but is still fast enough for 'online' storage (i.e. not SATA). As always, the needs are many, and the resources few.

At least we HAVE resources at the moment. It's a bad sign when you have to commiserate with your end-users over not being able to do cool stuff, or tell researchers they can't do that particular research since we have no where to store their data. Baaaaaad. We haven't quite gotten there yet, but we can see it from where we are.

Labels: , , ,


Thursday, January 28, 2010

Evolving best-practice

As of this morning, everyone's home-directory is now on the Microsoft cluster. The next Herculean task is to sort out the shared volume. And this, this is the point where past-practice runs smack into both best-practice, and common-practice.

You see, since we've been a NetWare shop since, uh, I don't know when, we have certain habits ingrained into our thinking. I've already commented on some of it, but that thinking will haunt us for some time to come.

The first item I've touched on already, and that's how you set permissions at the top of a share/volume. In the Land of NetWare, practically no one has any rights to the very top level of the volume. This runs contrary to both Microsoft and Posix/Unix ways of doing it, since both environments require a user to have at least read rights to that top level for anything to work at all. NetWare got around this problem by creating traverse rights based on rights granted lower down the directory structure. Therefore, giving a right 4 directories deep gave an inplicit 'read' to the top of the volume. Microsoft and Posix both don't do this weirdo 'implicit' thing.

The second item is the fact that Microsoft Windows allows you to declare a share pretty much anywhere, and NetWare was limited to the 'share' being the volume. This changed a bit when Novell introduced CIFS to NetWare, as they introduced the ability to declare a share anywhere; however, NCP networking still required root-of-volume only. At the same time, Novell also allowed the 'map root' to pretend there is a share anywhere but it isn't conceptually the same. The side-effect of being able to declare a share anywhere is that if you're not careful, Windows networks have share-proliferation to a very great extent.

In our case, past-practice has been to restrict who gets access to top-level directories, greatly limit who can create top-level directories, and generally grow more permissive/specific rights-wise the deeper you get in a directory tree. Top level is zilch, first tier of directories is probably read-only, second tier is read/write. Also, we have one (1) shared volume upon which everyone resides for ease of sharing.

Now, common-practice among Microsoft networks is something I'm not that familiar with. What I do know is that shares proliferate, and many, perhaps most, networks have the shares as the logical equivalent of what we use top-level directories for. Where we may have a structure like this, \\cluster-facshare\facshare\HumRes, Microsoft networks tend to develop structures like \\cluster-facshare\humres instead. Microsoft networks rely a lot on browsing to find resources. It is common for people to browse to \\cluster-facshare\ and look at the list of shares to get what they want. We don't do that.

One thing that really gets in the way of this model is Apple OSX. You see, the Samba version on OSX machines can't browse cluster-shares. If we had 'real' servers instead of virtual servers this sort of browse-to-the-resource trick would work. But since we have a non-trivial amount of Macs all over the place, we have to pay attention to the fact that all a Mac sees when they browse to \\cluster-facshare\ is a whole lot of nothing. We're already running into this, and we only have our user-directories migrated so far. We have to train our Mac users to enter the share as well. For this reason, we really need to stick to the top-level-directory model as much as possible, instead of the more commonly encountered MS-model of shares. Maybe a future Mac-Samba version will fix this. But 10.6 hasn't fixed it, so we're stuck for another year or two. Or maybe until Apple shoves Samba 4 into OSX.

Since we're on a fundamentally new architecture, and can't use common-practice, our sense of best-practice is still evolving. We come up with ideas. We're trying them out. Time will tell just how far up our heads are up our butts, since we can't tell from here just yet. So far we're making extensive use of advanced NTFS permissions (those permissions beyond just read, modify, full-control) in order to do what we need to do. Since this is a deviation from how the Windows industry does things, it is pretty easy for someone who is not completely familiar with how we do things to mess things up out of ignorance. We're doing it this way due to past-practice and all those Macs.

In 10 years I'm pretty sure we'll look a lot more like a classic Windows network than we do now. 10 years is long enough for even end-users to change how they think, and is long enough for industry-practice to erode our sense of specialness more into a compliant shape.

In the mean time, as the phone ringing off the hook today foretold, there is a LOT of learning, decision-making, and mind-changing to go through.

Labels: , ,


Monday, January 25, 2010

Storage tiers

Events have pushed us to give a serious look at cheaper storage solutions. What's got our attention most recently is HP's new LeftHand products. That's some nice looking kit, there. But there was an exchange there that really demonstrated how the storage market has changed in the last two years:

HP: What kind disk are you thinking of?
US: Oh, probably mid tier. 10K SAS would be good enough.
HP: Well, SAS only comes in 15K, and the next option down is 7.2K SATA. And really, the entire storage market is moving to SAS.

Note the lack of Fibre Channel drives. Those it seems are being depreciated. Two years ago the storage tier looked like this:
  1. SATA
  2. SAS/SCSI
  3. FC
Now the top end has been replaced.
  1. SATA
  2. SAS
  3. SSD
We don't have anything that requires SSD-levels of performance. Our VMWare stack could run quite happily on sufficient SAS drives.

Back in 2003 when we bought that EVA3000 for the new 6 node NetWare cluster, clustering required shared storage. In 2003, shared storage meant one of two things:
  1. SCSI and SCSI disks, if using 2 nodes.
  2. Fibre Channel and FC Disks if using more than 2 nodes.
With 6 nodes in the cluster, Fibre Channel was our only choice. So that's what we have. Here we are 6+ years later, and our I/O loads are very much mid-tier. We don't need HPC-level I/O ops. CPU on our EVA controllers rarely goes above 20%. Our I/O is significantly randomized, so SATA is no good. But we need a lot of it, so SSDs become prohibitive. Therefore SAS is what we should be using if we buy new.

Now if only we had some LTO drives to back it all up.

Labels: ,


Thursday, January 21, 2010

Migrating knowledge bases

This morning we moved the main class volume from NetWare to Windows. We knew we were going to have problems with this since some departments hadn't migrated key groups into AD yet, so the rights-migration script we wrote just plain missed bits. Those have been fixed all morning.

However, it is becoming abundantly clear that we're going to have to retrain a large portion of campus Desktop IT in just what it means to be dealing with Windows networking. We'd thought we'd done a lot of it, but it turns out we were wrong. It doesn't help that some departments had delegated 'access control' rights to professors to set up creative permissioning schemes, this morning the very heated calls were coming in from the professors and not the IT people.

There are two things that are tripping people up. One has been tripping people up on the Exchange side since forever, but the second one is new.
  1. In AD, you have to log out and back in again for new group-memberships to take.
  2. NTFS permissions do not grant the pass-through right that NSS permissions do. So if you grant a group rights to \Science\biology\BIOL1234, members of that group will NOT be able to pass through Science and Biology to get to BIOL1234.
We have a few spots here and there where for one reason or another rights were set at the 2nd level directories instead of the top level dirs. Arrangements like that just won't work in NTFS without busting out the advanced permissions.

An area we haven't had problems yet, but I'm pretty certain we will have some are places where rights are granted and then removed. With NSS that could be done two ways: an Inherited Rights Filter, or a direct trustee grant with no permissions. With NTFS the only way to do that is to block rights inheritance, copy the rights you want, and remove the ones you don't. That sounds simple, but here is the case I'm worried about:

\Humres\JobReview\VPIT\ITS\JohnSmith\

At 'HumRes' the group grp.hr is granted 'read' rights, and the HR director is granted Modify directly to their user (bad practice, I know. But it's real-world).
At 'JobReview' the group grp.hr.jobreclass is granted 'Modify'
At 'VPIT' Inheritance is Blocked and rights copied.
At 'JohnSmith' the HR user AngieSmith is granted the DENY right due to a conflict of interest.

Time passes. The old director retires, the new director comes in. IT Person gets informed that the new director can't see everything even though they have Modify to the entire \Humres tree. That IT person will go to us and ask, "WTH?" and we will reply with, "Inheritance is blocked at that level, you will need to explicitly grant Modify for the new director on that directory."

So this is a bit of a sleeper issue.

Meanwhile, we're dealing with a community of users who know in their bones that granting access to 'JohnSmith' means they can browse down from \HumRes to that directory just on that access-grant alone. Convincing them that it doesn't work that way, and working with them to rearrange directory structures to accommodate that lack will take time. Lots of time.

Labels: ,


This page is powered by Blogger. Isn't yours?