Wednesday, November 04, 2009

Novell federates with Google

It seems that Novell is the first company out of the gate to interoperate with Google Wave. Meet Novell Pulse. Like wave, it'll be a cloud-hosted service for enterprise collaboration at first, but will come out in a software package later. Not at all surprisingly, this will be a commercial product Novell will attempt to make bank with. Also, it is not open source. Unlike Google, Novell makes its money from subscription costs not advertising.

That said, with Pulse offering interoperability with Wave, it is entirely possible for extra-organizational users to collaborate with in-organization users on specific items. Sort of an Open-ID enabled version of SharePoint perhaps. This could be good.

Labels:


<permalink> posted by riedesg : 12:29 PM  1 comments

Thursday, October 22, 2009

Windows 7 releases!

Or rather, its retail availability is today. We're on a Microsoft agreement, so we've had it since late August. And boy do I know that. I've been having a trickle of calls and emails ever since the beta released about various ways Win7 isn't working in my environment and whether I have any thoughts about that. Well, I do. As a matter of fact, Technical Services and ATUS both have thoughts on that:

Don't use it yet. We're not ready. Things will break. Don't call us when it does.

But as with any brand new technology there is demand. Couple that with the loose 'corporate controls' inherent in a public Higher Ed institution and we have it coming in anyway. And I get calls when people can't get to stuff.

The main generator of calls is our replacement of the Novell Login Script. I've spoken about how we feel about our login script in the past. Back on July 9, 2004 I had a long article about that. The environment has changed, but it still largely stands. Microsoft doesn't have a built in login script the same way NetWare/OES has had since the 80's, but there are hooks we can leverage. One of my co-workers has built a cunning .VBS file that we're using for our login script, and does the kinds of things we need out of a login script:
  • Run a series of small applications we need to run, which drive the password change notification process among other things.
  • Maps drives based on group membership.
  • Maps home directories.
  • Allows shelling out to other scripts, which allows less privileged people to manage scripts for their own users.
A fair amount of engineering did go into that script, but it works. Mostly. And that's the problem. It works good enough that at least one department on campus decided to put Vista in their one computer lab and rely on this script to get drive mappings. So I got calls shortly after quarter-start to the effect of, "your script don't work, how can this be fixed." To which my reply was (summarized), "You're on Vista and we told y'all not to do that. This isn't working because of XYZ, you'll have to live with it." And they have, for which I am greatful.

Which brings me to XYZ and Win7.

The main incompatibility has to do with the NetWare CIFS stack. Which I describe here. The NetWare CIFS stack doesn't speak NTLMv2, only LM and NTLM. In this instance, it makes it similar to much older Samba versions. This conflicts with Vista and Windows 7, which both default their LAN Manager Authentication Level to "NTLMv2 Responses Only." Which means that out of the box both Vista and Win7 will require changes to talk to our NetWare servers at all. This is fine, so long as they're domained we've set a Group Policy to change that level down to something the NetWare servers speak.

That's not all of it, though. Windows 7 introduced some changes into the SMB/CIFS stack that make talking to NetWare a bit less of a sure thing even with the LAN Man Auth level set right. Perhaps this is SMB2 negotiations getting in the way. I don't know. But for whatever reason, the NetWare CIFS stack and Win7 don't get along as well as the Vista's SMB/CIFS stack did.

The main effect of this is that the user's home-directory will fail to mount a lot more often on Win7 than on Vista. Also, other static drive mappings will fail more often. It is reasons like these that we are not recommending removing the Novell Client and relying on our still in testing Windows Login Script.

That said, I can understand why people are relying on the crufty script rather than the just-works Novell Login Script. Due to how our environment works, The Vista/Win7 Novell Client is dog slow. Annoyingly slow. So annoyingly slow that not getting some drives when you log in is preferable to dealing with it.

This will all change once we move the main file-serving cluster to Windows 2008. At that point, the Windows script should Just Work (tm). At that point, getting rid of the Novell Client will allow a more functional environment. We are not at that point yet.

Labels: , , ,


<permalink> posted by riedesg : 9:13 AM  2 comments

Tuesday, October 06, 2009

BrainShare returns for 2010?

Novell just posted the Call For Participation, essentially soliciting session proposals, for BrainShare 2010. So it sounds like they're at least planning on going for it for 2010. Obviously, what with this little project I'm working on I won't be going. But it is nice to see it up and running.

Posting will be light. I was out sick last week, and I have family arriving later this week and in to next week.

Labels: ,


<permalink> posted by riedesg : 12:12 PM  0 comments

Wednesday, September 30, 2009

I have a degree in this stuff

I have a CompSci degree. This qualified me for two things:
  • A career in academics
  • A career in programming
You'll note that Systems Administration is not on that list. My degree has helped my career by getting me past the "4 year degree in a related field" requirement of jobs like mine. An MIS degree would be more appropriate, but there were very few of those back when I graduated. It has indirectly helped me in troubleshooting, as I have a much better foundation about how the internals work than your average computer mechanic.

Anyway. Every so often I stumble across something that causes me to go Ooo! ooo! over the sheer computer science of it. Yesterday I stumbled across Barrelfish, and this paper. If I weren't sick today I'd have finished it, but even as far as I've gotten into it I can see the implications of what they're trying to do.

The core concept behind the Barrelfish operating system is to assume that each computing core does not share memory and has access to some kind of message passing architecture. This has the side effect of having each computing core running its own kernel, which is why they're calling Barrelfish a 'multikernel operating system'. In essence, they're treating the insides of your computer like the distributed network that it is, and using already existing distributed computing methods to improve it. The type of multi-core we're doing now, SMP, ccNUMA, uses shared memory techniques rather than message passing, and it seems that this doesn't scale as far as message passing does once core counts go higher.

They go into a lot more detail in the paper about why this is. A big one is hetergenaity of CPU architectures out there in the marketplace, and they're not just talking just AMD vs Intel vs CUDA, this is also Core vs Core2 vs Nehalem. This heterogenaity in the marketplace makes it very hard for a traditional Operating System to be optimized for a specific platform.

A multikernel OS would use a discrete kernel for each microarcitecture. These kernels would communicate with each other using OS-standardized message passing protocols. On top of these microkernels would be created the abstraction called an Operating System upon which applications would run. Due to the modularity at the base of it, it would take much less effort to provide an optimized microkernel for a new microarcitecture.

The use of message passing is very interesting to me. Back in college, parallel computing was my main focus. I ended up not pursuing that area of study in large part because I was a strictly C student in math, parallel computing was a largely academic endeavor when I graduated, and you needed to be at least a B student in math to hack it in grad school. It still fired my imagination, and there was squee when the Pentium Pro was released and you could do 2 CPU multiprocessing.

In my Databases class, we were tasked with creating a database-like thingy in code and to write a paper on it. It was up to us what we did with it. Having just finished my Parallel Computing class, I decided to investigate distributed databases. So I exercised the PVM extensions we had on our compilers thanks to that class. I then used the six Unix machines I had access to at the time to create a 6-node distributed database. I used statically defined tables and queries since I didn't have time to build a table parser or query processor and needed to get it working so I could do some tests on how optimization of table positioning impacted performance.

Looking back on it 14 years later (eek) I can see some serious faults about my implementation. But then, I've spent the last... 12 years working with a distributed database in the form of Novell's NDS and later eDirectory. At the time I was doing this project, Novell was actively developing the first version of NDS. They had some problems with their implementation too.

My results were decidedly inconclusive. There was a noise factor in my data that I was not able to isolate and managed to drown out what differences there were between my optimized and non-optimized runs (in hindsight I needed larger tables by an order of magnitude or more). My analysis paper was largely an admission of failure. So when I got an A on the project I was confused enough I went to the professor and asked how this was possible. His response?
"Once I realized you got it working at all, that's when you earned the A. At that point the paper didn't matter."
Dude. PVM is a message passing architecture, like most distributed systems. So yes, distributed systems are my thing. And they're talking about doing this on the motherboard! How cool is that?

Both Linux and Windows are adopting more message-passing architectures in their internal structures, as they scale better on highly parallel systems. In Linux this involved reducing the use of the Big Kernel Lock in anything possible, as invoking the BKL forces the kernel into single-threaded mode and that's not a good thing with, say, 16 cores. Windows 7 involves similar improvements. As more and more cores sneak into everyday computers, this becomes more of a problem.

An operating system working without the assumption of shared memory is a very different critter. Operating state has to be replicated to each core to facilitate correct functioning, you can't rely on a common memory address to handle this. It seems that the form of this state is key to performance, and is very sensitive to microarchitecture changes. What was good on a P4, may suck a lot on a Phenom II. The use of a per-core kernel allows the optimal structure to be used on each core, with changes replicated rather than shared which improves performance. More importantly, it'll still be performant 5 years after release assuming regular per-core kernel updates.

You'd also be able to use the 1.75GB of GDDR3 on your GeForce 295 as part of the operating system if you really wanted to! And some might.

I'd burble further, but I'm sick so not thinking straight. Definitely food for thought!

Labels: , , , , ,


<permalink> posted by riedesg : 4:32 PM  1 comments

Friday, September 25, 2009

More thoughts on the Novell support change

Something struck me in comments on the last post about this that I think needs repeating on a full post.

Novell spent quite a bit of time attempting to build up their 'community' forums for peer-support. Even going so far as to seed the community with supported 'sysops' who helped catalyze others into participating, and creating a vibrant peer support community. This made sense because it built both goodwill and brand loyalty, but also reduced the cost-center known as 'support'. All those volunteers were taking the minor-issue load off of the call-in support! Money saved!

Fast forward several years. Novell bought SuSE and got heavily into Open Source. Gradually, as the OSS products started to take off commercially, the support contracts became the main money maker instead of product licenses. Just as suddenly, this vibrant goodwill-generating peer-support community is taking vital business away from the revenue-stream known as 'support'. Money lost!

Just a simple shift in the perception of where 'support' fits in the overall cost/revenue stream makes this move make complete sense.

Novell will absolutely be keeping the peer support forums going because they do provide a nice goodwill bonus to those too cheap to pay for support. However.... with 'general support' product-patches going behind a pay-wall, the utility of those forums decreases somewhat. Not all questions, or even most of them for that matter, require patches. But anyone who has called in for support knows the first question to be asked is, "are you on the latest code," and that applies to forum posts as well.

Being unable to get at the latest code for your product version means that the support forum volunteers will have to troubleshoot your problem based on code they may already be well past, or not have had recent experience with. This will necessarily degrade their accuracy, and therefore the quality of the peer support offered. This will actively hurt the utility of the peer-support forums. Unfortunately, this is as designed.

For users of Novell's active-development but severe underdog products such as GroupWise, OES2, and Teaming+Conferencing, the added cost of paying for a maintenance/support contract can be used by internal advocates of Exchange, Windows, and SharePoint as evidence that it is time to jump ship. For users of Novell's industry-leading products such as Novell Identity Management, it will do exactly as designed and force these people into maintaining maintenance contracts.

The problem Novell is trying to address are the kinds of companies that only buy product licenses when they need to upgrade, and don't bother with maintenance unless they're very sure that a software upgrade will fall within the maintenance period. I know many past and present Novell shops who pay for their software this way. It has its disadvantages because it requires convincing upper management to fork over big bucks every two to five years, and you have to justify Novell's existence every time. The requirement to have a maintenance contract in order for your highly skilled staff to get at TIDs and patches, something that used to be both free and very effective, is a real-world major added expense.

This is the kind of thing that can catalyze migration events. A certain percentage will pony up and pay for support every year, and grumble about it. Others, who have been lukewarm towards Novell for some time due adherence to the underdog products, may take it as the sign needed to ditch these products and go for the industry leader instead.

This move will hurt their underdog-product market-share more than it will their mid-market and top-market products.

If you've read Novell financial statements in the past few years you will have noticed that they're making a lot more money on 'subscriptions' these days. This is intentional. They, like most of the industry right now, don't want you to buy your software in episodic bursts every couple years. They want you to put a yearly line-item in your budget that reads, "Send money to Novell," that you forget about because it is always there. These are the subscriptions, and they're the wave of the future!

Labels: , ,


<permalink> posted by riedesg : 3:00 PM  0 comments

Wednesday, September 23, 2009

Novell Support: Now even MORE behind a pay-wall!

I first ran into this on Bucky's Blog. Specifically, Novell is changing what non-paying users can get out of Novell's support options. The details are still being hashed out, but they made the mistake of running afoul of one of the major no-no's of support: Pay-for-patches, or at least the suggestion of it. They caught a lot of flack about that with requiring a support contract to use the auto-update channels for their Linux products, but this will go even farther and put even support packs behind the maintenance-contract pay-wall.

So if you're a NetWare customer that hasn't paid maintenance in umpteen years since your server Just Works (TM), you'll now have to buy maintenance if you want to apply the latest Service Pack. Or if your server is throwing abends that can be fixed with a patch that you learned about in the peer support forums, you'll need a contract to be able to access it. This was done intentionally to pull in these free-loaders into paid support, but it does represent a potentially steep cost that can catalyze more migrations off of Novell products. This will hurt the shoe-string IT departments more than the big-bucks one. And since that describes a goodly percentage of 'small businesses' this could be a major problem in the future.

What's causing some confusion is their intent to put some of the KB articles behind the pay-wall as well. As described by Novell's support-community coordinator:
FACT: Only about 8% of the TIDs in the knowledgebase will be closed off
for entitled customers. Those are the TIDS for the products under "General
Support" ( http://support.novell.com/lifecycle ). All other TIDS will
remain open to the general public. As products move from general support
to extended and self support, all TIDS will become public.
So the 20+ year history of NetWare TIDs will still be there as NetWare is nolonger on general support per-se, but TIDs about currently in support closed-source items like Novell Identity Manager and the entire ZenWorks line is another story. One beef I have about this is that even if you do have a maintenance contract, it means that anyone who could possibly search the KB for articles has to have:
  1. A novell.com login
  2. Their novell.com login associated with a maintenance contract
This doesn't always happen. I've had to add a few people to our contract so they can use the Customer Center to get license codes or register SLES machines against our support. But the large majority of our historic NetWare admins aren't on the contract because they haven't needed it. This move will force organizations such as ours to much more actively manage our Customer Center contract/username associations. That can be a lot of bother.

The end effect of all of this is that the value of 'peer support' is markedly reduced for currently-shipping products. Once upon a time Novell was a company that really encouraged peer support since it took load off of their support engineers, customers liked it since it was free, and it encouraged quite a lot of goodwill. Now they seem to have realized that this was a drain on the bottom line and are dismantling the system in favor of everyone paying for support. This destroys goodwill, as they're now learning in the support forums.

Labels:


<permalink> posted by riedesg : 12:55 PM  5 comments

Tuesday, September 01, 2009

NetWare and Snow Leopard

In case you hadn't heard, the early release of Snow Leopard has tripped up Novell a bit.

http://www.novell.com/products/openenterpriseserver/snowleopard.html

What's interesting is that they'll be releasing a fix for NetWare too, not just OES. This suggests that the breakage isn't something like depreciating older authentication protocols, rather changing how such protocols are handled. That way the amount of engineering required is a lot less than trying to get Diffie Helman into NetWare.

Labels: ,


<permalink> posted by riedesg : 1:51 PM  0 comments

Thursday, August 06, 2009

Another nice How-To

On Novell Cool-Solutions:

Setting up Novell LDAP Libraries for C#

Another one of those things I went, "Ooh! USEFUL! Oh wait, we don't care any more. Drat. I bet I can blog that, though." So I am.

Use VisualStudio for developing applications (we did)? Need to talk to eDirectory? Why not use LDAP and Novell's tools for doing so! We've used their elderly ActiveX controls to do great things, and this should to about half of what we do with those. File manipulations will need to be another library, though.

Update: And how to set it up to use SSL. It requires Mono.

Labels: ,


<permalink> posted by riedesg : 12:12 PM  0 comments

Thursday, July 23, 2009

Fixing links and history

I just went through the 1072 past posts to this blog looking for links in posts to earlier posts. I do that a lot, it seems. It took a LONG time. I do wonder how many words I've committed to this blog in the five and a half years I've been doing it. There are some long essays back there! Also, I started back when Blogger didn't have:
  • Post-pages, the per-post link for direct linking to posts
  • Tags, or labels as they call it
  • Subjects, though it may have been there and I didn't elect to use it.
I seem to have covered, "the future of [netware|novell]" a lot (5/31/05, 11/9/05, 4/12/06 and that's just the posts with that as the title). There are a few other recurring themes as well. It's always interesting to look back like that.

Labels: ,


<permalink> posted by riedesg : 2:47 PM  0 comments

Tuesday, July 21, 2009

Digesting Novell financials

It's a perennial question, "why would anyone use Novell any more?" Typically coming from people who only know Novell as "That NetWare company," or perhaps, "the company that we replaced with Exchange." These are the same people who are convinced Novell is a dying company who just doesn't know it yet.

Yeah, well. Wrong. Novell managed to turn the corner and wean themselves off of the NetWare cash-cow. Take the last quarterly statement, which you can read in full glory here. I'm going to excerpt some bits, but it'll get long. First off, their description of their market segments. I'll try to include relevant products where I know them.

We are organized into four business unit segments, which are Open Platform Solutions, Identity and Security Management, Systems and Resource Management, and Workgroup. Below is a brief update on the revenue results for the second quarter and first six months of fiscal 2009 for each of our business unit segments:


•

Within our Open Platform Solutions business unit segment, Linux and open source products remain an important growth business. We are using our Open Platform Solutions business segment as a platform for acquiring new customers to which we can sell our other complementary cross-platform identity and management products and services. Revenue from our Linux Platform Products category within our Open Platform Solutions business unit segment increased 25% in the second quarter of fiscal 2009 compared to the prior year period. This product revenue increase was partially offset by lower services revenue of 11%, such that total revenue from our Open Platform Solutions business unit segment increased 18% in the second quarter of fiscal 2009 compared to the prior year period.

Revenue from our Linux Platform Products category within our Open Platform Solutions business unit segment increased 24% in the first six months of fiscal 2009 compared to the prior year period. This product revenue increase was partially offset by lower services revenue of 17%, such that total revenue from our Open Platform Solutions business unit segment increased 15% in the first six months of fiscal 2009 compared to the prior year period.

[sysadmin1138: Products include: SLES/SLED]

•

Our Identity and Security Management business unit segment offers products that we believe deliver a complete, integrated solution in the areas of security, compliance, and governance issues. Within this segment, revenue from our Identity, Access and Compliance Management products increased 2% in the second quarter of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 45%, such that total revenue from our Identity and Security Management business unit segment decreased 16% in the second quarter of fiscal 2009 compared to the prior year period.

Revenue from our Identity, Access and Compliance Management products decreased 3% in the first six months of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 40%, such that total revenue from our Identity and Security Management business unit segment decreased 18% in the first six months of fiscal 2009 compared to the prior year period.

[sysadmin1138: Products include: IDM, Sentinal, ZenNAC, ZenEndPointSecurity]

•

Our Systems and Resource Management business unit segment strategy is to provide a complete “desktop to data center” offering, with virtualization for both Linux and mixed-source environments. Systems and Resource Management product revenue decreased 2% in the second quarter of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 10%, such that total revenue from our Systems and Resource Management business unit segment decreased 3% in the second quarter of fiscal 2009 compared to the prior year period. In the second quarter of fiscal 2009, total business unit segment revenue was higher by 8%, compared to the prior year period, as a result of our acquisitions of Managed Object Solutions, Inc. (“Managed Objects”) which we acquired on November 13, 2008 and PlateSpin Ltd. (“PlateSpin”) which we acquired on March 26, 2008.

Systems and Resource Management product revenue increased 3% in the first six months of fiscal 2009 compared to the prior year period. The total product revenue increase was partially offset by lower services revenue of 14% in the first six months of fiscal 2009 compared to the prior year period. Total revenue from our Systems and Resource Management business unit segment increased 1% in the first six months of fiscal 2009 compared to the prior year period. In the first six months of fiscal 2009 total business unit segment revenue was higher by 12% compared to the prior year period as a result of our Managed Objects and PlateSpin acquisitions.

[sysadmin1138: Products include: The rest of the ZEN suite, PlateSpin]

•

Our Workgroup business unit segment is an important source of cash flow and provides us with the potential opportunity to sell additional products and services. Our revenue from Workgroup products decreased 14% in the second quarter of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 39%, such that total revenue from our Workgroup business unit segment decreased 17% in the second quarter of fiscal 2009 compared to the prior year period.

Our revenue from Workgroup products decreased 12% in the first six months of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 39%, such that total revenue from our Workgroup business unit segment decreased 15% in the first six months of fiscal 2009 compared to the prior year period.

[sysadmin1138: Products include: Open Enterprise Server, GroupWise, Novell Teaming+Conferencing,

The reduction in 'services' revenue is, I believe, a reflection in a decreased willingness for companies to pay Novell for consulting services. Also, Novell has changed how they advertise their consulting services which seems to also have had an impact. That's the economy for you. The raw numbers:
Three months ended


April 30, 2009

April 30, 2008

(In thousands)
Net revenue
Gross
profit

Operating
income (loss)

Net revenue
Gross
profit

Operating
income (loss)

Open Platform Solutions
$ 44,112
$ 34,756

$ 21,451

$ 37,516
$ 26,702

$ 12,191

Identity and Security Management

38,846

27,559


18,306


46,299

24,226


12,920

Systems and Resource Management

45,354

37,522


26,562


46,769

39,356


30,503

Workgroup

87,283

73,882


65,137


105,082

87,101


77,849

Common unallocated operating costs

—

(3,406 )

(113,832 )

—

(2,186 )

(131,796 )





















Total per statements of operations
$ 215,595
$ 170,313

$ 17,624

$ 235,666
$ 175,199

$ 1,667



























Six months ended


April 30, 2009

April 30, 2008

(In thousands)
Net revenue
Gross
profit

Operating
income (loss)

Net revenue
Gross
profit

Operating
income (loss)

Open Platform Solutions
$ 85,574
$ 68,525

$ 40,921

$ 74,315
$ 52,491

$ 24,059

Identity and Security Management

76,832

52,951


35,362


93,329

52,081


29,316

Systems and Resource Management

90,757

74,789


52,490


90,108

74,847


58,176

Workgroup

177,303

149,093


131,435


208,840

173,440


155,655

Common unallocated operating costs

—

(7,071 )

(228,940 )

—

(4,675 )

(257,058 )





















Total per statements of operations
$ 430,466
$ 338,287

$ 31,268

$ 466,592
$ 348,184

$ 10,148

So, yes. Novell is making money, even in this economy. Not lots, but at least they're in the black. Their biggest growth area is Linux, which is making up for deficits in other areas of the company. Especially the sinking 'Workgroup' area. Once upon a time, "Workgroup," constituted over 90% of Novell revenue.
Revenue from our Workgroup segment decreased in the first six months of fiscal 2009 compared to the prior year period primarily from lower combined OES and NetWare-related revenue of $13.7 million, lower services revenue of $10.5 million and lower Collaboration product revenue of $6.3 million. Invoicing for the combined OES and NetWare-related products decreased 25% in the first six months of fiscal 2009 compared to the prior year period. Product invoicing for the Workgroup segment decreased 21% in the first six months of fiscal 2009 compared to the prior year period.
Which is to say, companies dropping OES/NetWare constituted the large majority of the losses in the Workgroup segment. Yet that loss was almost wholly made up by gains in other areas. So yes, Novell has turned the corner.

Another thing to note in the section about Linux:
The invoicing decrease in the first six months of 2009 reflects the results of the first quarter of fiscal 2009 when we did not sign any large deals, many of which have historically been fulfilled by SUSE Linux Enterprise Server (“SLES”) certificates delivered through Microsoft.
Which is pretty clear evidence that Microsoft is driving a lot of Novell's Operating System sales these days. That's quite a reversal, and a sign that Microsoft is officially more comfortable with this Linux thing.

Labels: , , , , , , , ,


<permalink> posted by riedesg : 4:38 PM  0 comments

Thursday, July 09, 2009

Using GroupWise as a generic mail client

Tired of Thunderbird's quirks? Want something else but don't like either Evolution or Outlook? You have another option.

GroupWise@Home!

As near as I can figure, this is a GroupWise client tweaked for use without a GroupWise server. This'll allow you to do IMAP/POP email and have all those other nifty GroupWise features like richly featured rules. I haven't tried it myself, but I am sorely tempted.

Labels:


<permalink> posted by riedesg : 12:41 PM  0 comments

Monday, May 11, 2009

Rebuilds and nwadmin

Friday afternoon the Kala server, one of our three primary eDirectory replica servers, died. In event I've never seen before, one hard drive of a mirrored pair failed in such a way that bad data got committed. This server had to be rebuilt.

Happily for me, this is a procedure I can do without having to look things up in the Novell KB. This is part of the reason the letters "CNE" follow my name. The procedure is pretty straight-forward and I've done it before.
  1. Remove dead server's objects from the tree
  2. Designate a new server as the Master for any replica this server was the master of (all of them, as it happened)
  3. Install server fresh
The details change somewhat over time, but that's the same workflow it has been since the NetWare 4 days. In my case I did hit the KB to see if there was a way to do step 2 in iMonitor. I couldn't find one, so I did it through DSREPAIR which works just fine.

As for the install... this server is an HP BL20P G3, which means I used the procedure I documented a while back (Novell, local copy). A few minor steps changed (the INSERT Linux I used then now correctly handles SmartArray cards), but otherwise that's what I did. Still works.

For a wonder, our SSL administrator still had the custom SSL certificate we created for this server three years ago. That saved me the step of creating a CSR and setting up all the Subject Alternate Names we needed.

And today I fired up NWADMIN for the first time in not nearly long enough to associate the SLP scope to this server, since it was one of our two DA's. I could probably have done the same thing in iManager with "Other" attributes, but... why risk not getting all the right attributes associated when I have a tool that has all the built-in rules. This is the one thing that I still have NWAdmin around for. SLP-on-NetWare management.

Labels: , , ,


<permalink> posted by riedesg : 1:29 PM  0 comments

Wednesday, May 06, 2009

Windows 7 and NetWare CIFS

Now that RC1 is out we're trying things. Aaaaaand it doesn't work, even when set to the least restrictive Lan Man authentication level. Also? Windows 7 has a lot more NTLM tweakables in the policy settings that we don't understand. But one thing is clear, Windows 7 will not talk to NetWare CIFS out of the box. The Win7 workstation will need some kind of tweaks.

I may need to break out Wireshark and see what the heck is going on at the packet level.

Life on the bleeding edge, I tell you.

Update: Know what? It was a name-resolution issue. It seems that once you went to the resource with its FQDN rather than the short name, the short names started working. Kind of odd, but that's what did it. A bit of packet sniffing may illuminate why the short-name method didn't work at first (it should) which just might illuminate either a bug in Win7, or a simple feature of Windows name-resolution protocols.

The only change that needed to be made was drop the LAN Manager Authentication Level to not offer NTLMv2 responses unless negotiated for it.

Labels: , , ,


<permalink> posted by riedesg : 9:07 AM  2 comments

Thursday, April 30, 2009

Windows 7 RC is out

And they're saying that Win7 will ship well before the Jan-2010 Vista timeframe mentioned before. Well, we kind of expected that. We've also been doing a lot with our network to make it more Win7 (and Vista) friendly, since we know we'll get a LOT of Win7 once it shows up for real.

The biggest concern is that Microsoft still hasn't fixed the issue that makes the Novell Client for Vista so darned slow. This is a major deal-breaker for us, so we've been informed from on high to Do Something so our Vista/Win7 clients can have fast file-serving and printing.

That "Something" has been to turn on the CIFS stack on our NetWare servers, with domain integrated login. The Vista and Win7 clients will have to turn their LanMan Authentication Level from the default (and secure) setting of, "Send NTLMv2 Response Only" to at most, "Send NTLM Response Only." The NetWare CIFS stack can't handle NTLMv2, nor will it ever. Those people who have been suffering through the NCV get downright bouncy when they see how fast it is.

Printing... we'll see. A LOT of the printing in fac/staff land is direct-IP which has no Novell dependencies. There are a few departments out there that have enough print volume that a print-server is a good idea, so I'm hoping there is an iPrint client for Win7 out pretty fast.

All in all, we're expecting uptake of Win7 to be a lot faster than Vista ever was. In this sense Win7 is a lot like Win98SE. All the press saying that Win7 is a lot better than Vista will help drive the push away from WinXP.

Labels: , , ,


<permalink> posted by riedesg : 9:11 AM  0 comments

Wednesday, April 22, 2009

Novell wants your BrainShare input

Just posted on the Cool Solutions community page:

Novell BrainShare 2010 Advisory Board

Since BrainShare took 2009 off, they're planning on bringing it back in 2010. And they're looking for end user input into what it should look like. Should it stay in Salt Lake City? Should events be dropped? Should events be added? This looks to be an online colaboration rather than physical presence, so proximity to Provo, UT shouldn't be a problem. Though, proximity to the US Mountain Timezone may be a good idea.

If you get selected for the board, a perk is a pass for BrainShare 2010.

Labels: ,


<permalink> posted by riedesg : 12:09 PM  0 comments

Tuesday, April 21, 2009

Zen Asset Inventory

A while back we installed Zen Asset Inventory (but not Asset Management) since it came with our Novell bundle, and inventory is a nice thing to have. At the beginning of this quarter it started to crash while inventorying certain workstations. After sending the logs to Novell, it turned out to be crashing on a lot of workstations.

Novell said that the reason for the crashes was excessive duplicate workstations. ZAM is supposed to handle this, but it seems 2 years of quarterly lab reimaging seems to have finally overwhelmed the de-dup process. The fix is fairly straight forward, but very labor intensive:
  1. Clean out the Zenworks database
  2. Force a WorkstationOID change on all workstations
The second took quite a while. Those steps are:
  1. Stop the Collection Client service
  2. Delete a specific registry key
  3. Start the Collection Client service
These three steps can be done by way of Powershell (or the 'pstools' suite of command-line utilities if you want to rock it old school). One at a time. As we have on the order of 3,700 workstations, this took a few days and I'm sure I missed some. I did get all of the lab machines, though. That's important.

Cleaning out the database proved to be more complicated than I thought. At first I thought I just had to delete all the workstations from the Manager tool. But that would be wrong. Actually looking at the database tables showed a LOT of data in a supposedly clean database.

The very first thing I tried was to remove all the workstations from the database by way of the manager, and restart inventory. The theory here is that this would eliminate all the duplicate entries, so we'd just start the clock ticking again until the imaging caught us out. Since I had modified our imaging procedures, this shouldn't happen again any way. Tada!

Only the inventory process started crashing. Crap.

The second thing I tried was to strobe through the Lab workstations with the WorkstationOID-reset script I worked up in PowerShell (this is not something I could have done without an Active Directory domain, by the way). These are the stations with the most images, and getting them reset should clear the problem. Couple that with a clearing of the database by way of the Manager, and we should be good!

Only the inventory process started crashing. It took a bit longer, but it still crashed pretty quickly.

Try number three... run the powershell script across the ENTIRE DOMAIN. This took close to four days. Empty the database via Manager again, restart.

It crashed. It took until the second day to crash, but it still crashed.

As I had reset the WorkstationOID on all domained machines (or at least a very large percentage of them), the remaining dups were probably in the non-domained labs I have no control over. So why the heck was I still getting duplication problems with a supposedly clean database? So I went into SQL Studio to look at the database tables themselves. The NC_Workstation table itself had over 15,000 workstations in it. Whaaa?

However, this would explain the duplication problems I'd been having! If it had been doing the de-dup processing on historical data that included a freighter full of duplicates already, it was going to crash. Riiiiight. So. How do I clean out the tables? Due to foreign key references and full tables elsewhere, I had to build a script that would purge leaf tables, then core tables. The leaf tables (things like NC_BIOS) could be Truncated, handy when a table contains over a million rows. Core tables (NC_Component) have to be deleted line-by-line, which for the 2.7 million row NC_Component table took close to 24 hours to fully delete and reindex.

With a squeaky clean database, and the large majority of WorkstationOID values reset enterprise wide, I have restarted the inventory process. The Zenworks database is growing at a great pace as the Component tables repopulate. This morning we have 3,750 workstations and growing. We inventoried close to 3,300 stations yesterday and didn't get a single inventory crash. This MAY have fixed it!

I'm keeping these SQL scripts for later use if I need 'em.

They key learning here? Removing the workstations from the Manager doesn't actually purge the workstation from the database itself.

Labels: , ,


<permalink> posted by riedesg : 9:52 AM  1 comments

Wednesday, April 15, 2009

Windows 7 forces major change

I've said before that you'll have to pry the login-script out of our cold dead hands. The simple Novell login-script is the single most pervasive workstation management tool we have, since EVERYONE needs the Novell Client to talk to their file servers. Its one reason we have computer labs when others are paring down or getting rid of theirs. People can live without the Zen agents if they work at it, but they can't live without the Novell Client. Therefore, we do a lot of our workstation management through the login-script.

The Vista client has been vexing in this regard since it is so painfully slow in our clustered environment. The reason it is slow is the same reason the first WinXP clients were slow, the Microsoft and Novell name-resolution processes conmpete in bad ways. As each drive letter we map is its own virtual-server, every time you attempt to display a Save/Open box or open Windows Explorer it has to resolve-timeout-resolve each and every drive letter. This means that opening a Save/Open box on a Vista machine running the Novell client can take upwards of 5 minutes to display thanks to the timeouts. Novell knows about this issue, and has reported it to Microsoft. This is something Microsoft has to fix, and they haven't yet.

This is vexing enough that certain highly influential managers want to make sure that the same thing doesn't happen again for Windows 7. As anyone who follows any piece of the tech media knows, Windows 7 has been deemed, "Vista done right," and we expect a lot faster uptake of Win7 than WinVista. So we need to make sure our network can accommodate that on release-day. Make it so, said the highly placed manager. Yessir, we said.

So last night I turned CIFS on for all the file services on the cluster. It was that or migrate our entire file-serving function to Windows. The choice, as you can expect, was an easy one.

This morning our Mac users have been decidedly gleeful, as CIFS has long password support where AFP didn't. The one sysadmin here in techservices running Vista as his primary desktop has uninstalled the Novell Client and is also cheerful. Happily for us, the directive from said highly placed manager was accompanied by a strong suggestion to all departments that domaining PCs into the AD domain would be a Really Good Idea. This allows us to use the AD login-script, as well as group-policies, for those Windows machines that lack a Novell Client.

Ultimately, I expect the Novell Client to slowly fade away as a mandatory install. So that clientless-future I said we couldn't take part in? Microsoft managed to push us there.

Labels: , , , ,


<permalink> posted by riedesg : 10:49 AM  0 comments

Friday, April 03, 2009

Open-sourcing eDirectory?

The topic of open-sourcing eDirectory comes up every so often. The answer is always the same, it can't be done. Novell NDS and the eDirectory that followed it uses technology licensed from RSA, and RSA will not allow their code to be open-sourced. And that's it.

However... it isn't the RSA technology that allows eDirectory to scale as far as it does. To the best of my knowledge, that's pure Novell IP, based on close to 20 years of distributed directory experience. The RSA stuff is used in the password process, specifically the NDS Password, as well as authenticating eDirectory servers to the tree and each other. The RSA code is a key glue to holding the directory together.

If Novell really wanted to, they could produce another directory that scales as far as eDirectory does. This directory would be fundamentally incompatible with eDir because it would have to be made without any RSA code, which eDirectory requires. This hypothetical open-source directory could scale as far as eDir does, but would have to use a security process that is also open-source.

This would take a lot of engineering on the part of Novell. The RSA stuff has been central to both NDS and eDir for all of those close to 20 years, and the dependency tree is probably very large. The RSA code even is involved in the NCP protocol that eDir uses to talk with other eDir servers, so a new network protocol would probably have to be created from scratch. At the pace Novell is developing software these days, this project would probably take 2-3 years.

Since it would take a lot of developer time, I don't see Novell creating an open-source eDir-clone any time soon. Too much effort for what'll essentially be a good-will move with little revenue generating potential. That's capitalism for you.

Labels: , ,


<permalink> posted by riedesg : 10:37 AM  3 comments

Thursday, March 12, 2009

GroupWise survey

Novell just posted on Cool Solutions a GroupWise survey they're running. It sounds like they're looking for futures for the product. As GW8 is out, I'm guessing Novell is looking to refine their GroupWise road map. So, if you're a GW user/admin, go forth and take it!

I didn't do the survey because, erm, I'm no longer a GroupWise admin. But that's no reason not to share!

Labels:


<permalink> posted by riedesg : 12:41 PM  1 comments

Tuesday, March 10, 2009

But what about GroupWise

Today I picked up my dead tree version of NetworkWorld, and saw an item on the cover:

Looking to exchange Exchange?
Joel Snyder tested six alternatives to Microsoft's Exchange 2007. OUr findings: the Exchange alternatives are adequate for midsized networks, but Exchange offers the most comprehensive set of features and management hooks for networks of all sizes. Page 22
(online version)

GroupWise was NOT in this test. This surprised me greatly, as the Big Three mailers have always been Exchange, Notes, and GroupWise. Notes was also left out of this test. The online version already has a few comments regarding GroupWise, and Joel Snyder replied with this:
By Joel Snyder on Tue, 03/10/2009 - 10:09am.

Sorry, Groupwise fans, but Novell just didn't show up on our radar in the mid-size email business.

When you're looking at this space, Microsoft and Lotus together own 96% of the on-site mail service in businesses (the numbers that IDC, Ferris, and Radicati offer all vary a lot, but no one seems to give the non-MS/non-Lotus camp more than 10% total for everyone). Slicing up the remaining piece is a pretty difficult task, with lots of little players. While Groupwise used to be a major mover-and-shaker, there is no obvious "#3" in this business anymore.

It's clear that we've got a pile of Groupwise fans (why am I remembering the Windows vs. Netware war of about 10 years ago???) here, so maybe we should take a quick look at Groupwise and see how it stacks up.

Considering Novell has spent quite a lot of effort trying to convince people that they're the number three behind the MS/IBM duopoly, this is somewhat concerning. I have no idea what the real market-share numbers are for the mid-size enterprise groupware market.

Labels: ,


<permalink> posted by riedesg : 1:21 PM  7 comments

Thursday, February 26, 2009

SLES11 will be out soon.

For Novell has posted a preview release of SLES11. It says "RC4", which I suspect means we're within a month or so of release. This release is just in time for BrainShare 2009, were it actually happening. SLES11 would have been the major message of BS09.

Labels: , ,


<permalink> posted by riedesg : 11:38 AM  0 comments

Tuesday, February 17, 2009

tsatest and incrementals

Today I learned how to tell TSATEST to do an incremental backup. I also learned that the /path parameter requires the DOS namespace name. Example:

tsatest /V=SHARE: /path=FACILI~1 /U=.username.for.backup /c=2

That'll do an incremental (files with the Archive bit set) backup of that specific directory, on that specific volume.

Labels: , , ,


<permalink> posted by riedesg : 10:43 AM  0 comments

Wednesday, February 11, 2009

High availability

64-bit OES provides some options to highly available file serving. Now that we've split the non-file services out of the main 6-node cluster, all that cluster is doing is NCP and some trivial other things. What kinds of things could we do with this should we get a pile of money to do whatever we want?

Disclaimer: Due to the budget crisis, it is very possible we will not be able to replace the cluster nodes when they turn 5 years old. It may be easier to justify eating the greatly increased support expenses. Won't know until we try and replace them. This is a pure fantasy exercise as a result.

The stats of the 6-node cluster are impressive:
  • 12 P4 cores, with an average of 3GHz per core (36GHz).
  • A total of 24GB of RAM
  • About 7TB of active data
The interesting thing is that you can get a similar server these days:
  • HP ProLiant DL580 (4 CPU sockets)
  • 4x Quad Core Xeon E7330 Processors (2.40GHz per core, 38.4GHz total)
  • 24 GB of RAM
  • The usual trimmings
  • Total cost: No more than $16,547 for us
With OES2 running in 64-bit mode, this monolithic server could handle what six 32-bit nodes are handling right now. The above is just a server that matches the stats of the existing cluster. If I were to really replace the 6 node cluster with a single device I would make a few changes to the above. Such as moving to 32GB of RAM at minimum, and using a 2-socket server instead of a 4-socket server; 8 cores should be plenty for a pure file-server this big.

A single server does have a few things to recommend it. By doing away with the virtual servers, all of the NCP volumes would be hosted on the same server. Right now each virtual-server/volume pair causes a new connection to each cluster node. Right now if I fail all the volumes to the same cluster node, that cluster node will legitimately have on the order of 15,000 concurrent connections. If I were to move all the volumes to a single server itself, the concurrent connection count would drop to only ~2500.

Doing that would also make one of the chief annoyances of the Vista Client for Novell much less annoying. Due to name cache expiration, if you don't look at Windows Explorer or that file dialog in the Vista client once every 10 minutes, it'll take a freaking-long time to open that window when you do. This is because the Vista client has to enumerate/resolve the addresses of each mapped drive. Because of our cluster, each user gets no less than 6 drive mappings to 6 different virtual servers. Since it takes Vista 30-60 seconds per NCP mapping to figure out the address (it has to try Windows resolution methods before going to Novell resolution methods, and unlike WinXP there is no way to reverse that order), this means a 3-5 minute pause before Windows Explorer opens.

By putting all of our volumes on the same server, it'd only pause 30-60 seconds. Still not great, but far better.

However, putting everything on a single server is not what you call "highly available". OES2 is a lot more stable now, but it still isn't to the legendary stability of NetWare 3. Heck, NetWare 6.5 isn't at that legendary stability either. Rebooting for patches takes everything down for minutes at a time. Not viable.

With a server this beefy it is quite doable to do a cluster-in-a-box by way of Xen. Lay a base of SLES10-Sp2 on it, run the Xen kernel, and create four VMs for NCS cluster nodes. Give each 64-bit VM 7.75GB of RAM for file-caching, and bam! Cluster-in-a-box, and highly available.

However, this is a pure fantasy solution, so chances are real good that if we had the money we would use VMWare ESX instead XEN for the VM. The advantage to that is that we don't have to keep the VM/Host kernel versions in lock-step, which reduces downtime. There would be some performance degradation, and clock skew would be a problem, but at least uptime would be good; no need to perform a CLUSTER DOWN when updating kernels.

Best case, we'd have two physical boxes so we can patch the VM host without having to take every VM down.

But I still find it quite interesting that I could theoretically buy a single server with the same horsepower as the six servers driving our cluster right now.

Labels: , , , , , , ,


<permalink> posted by riedesg : 9:25 AM  1 comments

Monday, February 09, 2009

Novell marketing

Looks like they're continuing to follow their grassroots marketing strategy. Novell recently created a Novell channel on YouTube. They always made many videos for BrainShare, and I see some vids from BrainShares past up there already. So if you were wandering the internet looking for the Novell PC/Mac spoofs, now you have an official channel to find them.

Labels: ,


<permalink> posted by riedesg : 1:18 PM  1 comments

Wednesday, February 04, 2009

The mystery of lsimpe.cdm

Last night I turned on multi-path support for the main NetWare file cluster. This has been a long time coming. When we upgraded the EVA3000 to an EVA6100 it gained the ability to do active/active IO on the controllers, something that the new EVA4400 can also do.

What's more, the two Windows backup-to-disk servers we've attached to the EVA4400 (and the MSA1500 for that matter) have the HP MPIO drivers installed, which are extensions of the Microsoft MPIO stack. Looking at the bandwidth chart on the fiber-channel fabric I see that these Windows servers are also doing load balancing over both of the paths. This is nifty! Also, when I last updated the XCS code on the EVA4400 both of those servers didn't even notice the controller reboots. EVEN NICER!

I want to do the same thing with NetWare. On the surface, turning on MPIO support is dead easy:

Startup.ncf file:
SET MULTI-PATH SUPPORT = ON
LOAD QL2X00.HAM SLOT=10001 /LUNS /ALLPATHS /PORTNAMES

Tada. Reboot, present both paths in your zoning, and issue the "list failover devices" command on the console, and you'll get a list. In theory should one go away, it'll seamlessly move over to the other.

But what it won't do is load-balance. Unfortunately, the documentation on NetWare's multi-path support is rather scanty, focusing more on configuring path failover priority. The fact that the QL2X00.HAM driver itself can do it all on its own without letting NetWare know (the "allpaths" and "portnames" options tell it to not do that and let NetWare do the work) is a strong hint that MPIO is a fairly light weight protocol.

On the support forums you'll get several references to the LSIMPE.CDM file. With interesting phrases like, "that's the multipath driver", and, "Yeah, it isn't well documented." The data on the file itself is scanty, but suggestive:
LSIMPE.CDM
Loaded from [C:\NWSERVER\DRIVERS\] on Feb 4, 2009   3:32:13 am
(Address Space = OS)
LSI Multipath Enhancer
Version 1.02.02 September 5, 2006
Copyright 1989-2006 Novell, Inc.
But the exact details of what it does remain unclear. One thing I do know, it doesn't do the load-balancing trick.

Labels: , , ,


<permalink> posted by riedesg : 1:48 PM  0 comments

Thursday, January 22, 2009

NetStorage and IE7

Looks like there is a bug in NetStorage (NW65SP7, not sure if SP8 fixes it or not) and IE7. When you're browsing along, select a file for download, then go to File -> Download, you get a login screen. No matter what you enter, it won't let you download the file. Access forbidden!

You can get around it by double-clicking on the file you want to download.

However, this also breaks upload and there is no workaround for that.

Works just fine in non IE7 browsers. I understand this issue is known by Novell.

On the server side, I can see a few signs of this in the log-files. There is a line like this for a failed download attempt:

140.160.246.45 - - [22/Jan/2009:11:15:40 -0800] "GET /oneNet/netstorage/Home@WWU/ac228.tgz HTTP/1.1" 401 1370 "-" "Java/1.4.2_13"

That IP is the server's IP address, not the clients. The user agent is Java. Clearly (to me anyway) Tomcat is proxying the download request and thus creating the new user-agent string. The rest of this session is with a normal IE7/WinXP user-agent.

Now a successful download (firefox):

140.160.246.45 - username [22/Jan/2009:11:16:39 -0800] "GET /oneNet/NetStorage/Home@WWU/cert.txt HTTP/1.1" 200 3329 "-" "Java/1.4.2_13"

The observant may notice some case differences there. I thought of the same thing, and did some poking around in IE to get this:

140.160.246.45 - - [22/Jan/2009:11:18:14 -0800] "GET /oneNet/NetStorage/Home@WWU/cert.txt HTTP/1.1" 401 3329 "-" "Java/1.4.2_13"

Same case as the Firefox access, but still failed. I don't know why this is doing this, but clearly something inside Tomcat isn't happy with how IE7 is handling the POST request that requests the download.

Labels: ,


<permalink> posted by riedesg : 11:22 AM  0 comments

Thursday, December 18, 2008

BrainShare, what next

As promised, some additional information.

The BrainShare Forum announcement of the cancellation. I've met Mr. Groneman. He works with the BrainShare crew, as does a lot of Novell, as he has to get the Support and SysOp areas set up. He's also the guy who is (I believe) in charge of the whole forums.novell.com thing. He states clearly that this is not the end of BrainShare for good, this year's step was taken due to the apparently marked decrease in registrations they've received to date.

Brainstorming some replacement ideas for the short term. One item mentioned a lot in the above thread was to have an ATT Live event during the same week as BrainShare is now, presumably to allow the people with nonrefundable air tickets to get SOMETHING out of the travel.

GWAVACon is offering discounts to BrainShare attendees, now that their travel budgets suddenly have more room. They claim they have more than just GWAVA and GroupWise there.

There are some hard economic factors that figure in here, I'm sure. Until this year, I understand that BrainShare attendance had been growing. From personal experience, the groups that had seen the most growth were the IDM and Linux portions of the conference. More than once I've heard BrainShare attendees say it feels more like a Linux conference than anything else, any more. Still, they were seeing regular growth. To see a sharp decline? That's clearly overall economics at work.

I know that the Sponsors for BrainShare expect to reach a certain number of attendees, and if Novell can't deliver the head-count, it could go poorly. Also, the $1695 reg-fee is heavily subsidized by both Novell and sponsors. While direct per-person costs such as food-service are rather low, the sunk costs of the Salt Palace and other such items are a fixed cost, so having fewer people around means the apportioned fixed-cost-per-head is much higher when attendance is down a lot. A lot of those contracts were probably signed months ago and have no back-out clause, or those that do have punative penalties associated, so Novell isn't saving a lot of money by doing this.

Also, Novell is not the only group cancelling a big trade show. MacWorld will be Jobs-free this next time around, and there are others. I guess travel-bans are one of the first things that employers are putting in place in light of reduced income, and that's really taking a bite out of the convention-business.

I wonder how the big Las Vegas hotels will make out?

Labels: ,


<permalink> posted by riedesg : 1:38 PM  0 comments

Novell has canceled BrainShare 2009

Duuuuuuude.

I didn't find out about it until this morning, as I was snowed in yesterday and wasn't paying as much attention as I should. But still. I'm one of those people who cited travel-budget shortages for not coming. This is major news!

This begs the question, "is BrainShare dead, or just this year?"

It could be either, but from what I'm hearing in the Novell Forums it sounds like there are still plans to put on a BrainShare 2010, should conditions improve. I can't go to BrainShare 2010, those aforementioned economic concerns ($5.6Bn budget shortfall in the state budget for fiscal 09-11), so it's kind of moot for me. But still.

There is a lot of stuff going on over in the forums. As they're web-based these days, and the interface doesn't suck, I might be able to link to interesting bits. In another post.

Still. duuuuuuuuuuuude.

Labels: ,


<permalink> posted by riedesg : 7:39 AM  1 comments

Tuesday, December 09, 2008

The price of storage

I've had cause to do the math lately, which I'll spare you :). But as of the best numbers I have, the cost of 1GB of space on the EVA6100 is about $16.22. Probably more, since this 6100 was created out of the carcass of an EVA3000, and I don't know what percentage of parts from the old 3000 are still in the 6100 and thus can't apportion the costs right.

For the EVA4400, which we have filled with FATA drives, the cost is $3.03.

Suddenly, the case for Dynamic Storage Technology (formerly known as Shadow Volumes) in OES can be made in economic terms. Yowza.

The above numbers do not include backup rotation costs. Those costs can vary from $3/GB to $15/GB depending on what you're doing with the data in the backup rotation.

Why is the cost of the EVA6100 so much greater than the EVA4400?
  1. The EVA6100 uses 10K RPM 300GB FibreChannel disks, where the the EVA4400 uses 7.2K RPM 1TB (or is it 450GB?) FATA drives. The cost-per-gig on FC is vastly higher than it is on fibre-ATA.
  2. Most of the drives in the EVA6100 were purchased back when 300GB FC drives cost over $2000 each.
  3. The EVA6100 controller and cabinets just plain cost more than the EVA4400, because it can expand farther.
To put it into a bit of perspective, lets take the example of a 1TB volume of, "unorganized file data", the seemingly official term for "file-server". If you place that 1TB of data on the EVA6100, that data consumes $16609.28 worth of storage. So what if 70% of that data hasn't been modified in a year (not unreasonable), and is then put on the EVA4400 instead? So you'd have 307GB on the 6100 and 717GB on the 4400. Your storage cost now drops to $5909.75. That's real money.

Labels: , , , ,


<permalink> posted by riedesg : 1:30 PM  1 comments

Wednesday, December 03, 2008

Infiltrating the market

Over on The Open Road there is a very interesting blog post. It talks about how Microsoft and Red Hat approach the market, and touches on Novell.
Microsoft offers a full ecosystem of software to would-be buyers, but its greatest success may actually result from its strategy to present customers with an "and" decision when initially purchasing Microsoft technology, rather than a difficult "or" decision.
And I really see this. The argument has been made internally that what you get from a Microsoft Enterprise CAL is worlds above what we can get from a Novell academic seat license, which follows into cost-effective discussions (and not good ones). It is soooooo easy to go all Microsoft, whereas a pure Linux solution requires a lot of stitching, and translation-glue.

The article goes on to point out that Red Hat's targeting people looking to do forklift upgrades from Unix to Linux. And then points out that Microsoft wins more of that traffic than Red Hat does, by a good margin. Largely because the Microsoft family of products is very complete.

As it points out, Novell figured this out a few years ago when they launched their collaboration with Microsoft. The fruits of which arrived today with OES2 SP1, and the Novell CIFS stack and Domain Services for Windows. This allows OES2 to do something you can't do with Samba (yet), pretend to be a full up AD Domain Controller.

And yeah, Novell's current marketing slogan is, "Making IT work as One," which is a clear embracing of the "and" concept described. If they could make DSfW work on plain SLES, it may make it an even more attractive product for people.

Labels: , ,


<permalink> posted by riedesg : 3:38 PM  2 comments

OES2 SP1 ships!

Full announcement.

It's out!

Labels: , , , , , ,


<permalink> posted by riedesg : 8:42 AM  0 comments

Tuesday, December 02, 2008

The NetWare 7 that never was

My last post generated some comments lamenting where NetWare has gone. I hear ya.

I have friends and have spoken with people at BrainShare who were closer to things than I was regarding how the next version of NetWare evolved. And to be truthful, it sounded a lot like how Microsoft moved from XP to Vista. If you'll recall, "the version of Windows after XP," was something of a moving target for many years. I recall media reports of Microsoft scrapping the whole project and starting afresh at least once.

My very first BrainShare was 2001, and that was the release party for NetWare 6. It was in 2003 when Novell bought Ximian, and bought SuSE, so it is clear when Novell probably decided to bet the house on this whole Linux thing. Yet at BS01 there was talk about NW7, or if there would be a NW6.1 version out. The rumors I remember from back then had NW7 being a progression towards a more application-friendly environment. I also remember hearing the L word around once or twice.

What we actually got was NetWare 6.5, which solidified NetWare 6 and made the core services better and more mature. What it wasn't was any more application friendly than NetWare 6 was (or even NetWare 5.1 for that matter). NetWare 6.5 released in August of 2003, the same month as the Ximian purchase. This is what tells me that Novell had decided on a path for NetWare 7, and it was green, not red. Open Enterprise Server arrived in 2005, which gives OES a solid year and a half dev-time between when SuSE was bought and when we started seeing public betas of OES. The NetWare version of OES was NetWare 6.5 SP3.

What happened to NetWare 7? It got lost on the roadmap. When NW6 came out, Novell probably had 6.5 on the roadmap as the next rev, with NW7 next down. The rumors we were hearing were very provisional, as the spot on the map held by NW7 was at least 3 years away. Sometime between BrainShare 2001 and when Novell started buying its way into the Linux world NW7 was dropped and the decision was made to port to a completely different Kernel. That decision was probably made in the summer of 2003, as the NetWare 6.5 development was entering final beta, and the task of allocating developer resources for the next full rev needed to be made.

Which brings us to today. OES2 SP1 is going to drop any day now, probably in time for Novell's quarterly earnings report. SP1 finally brings the Linux-kernel 'NetWare Services' to feature-comparable with the NetWare kernel. There are still a few things missing, like an eDirectory integrated SLP server, but now all the major points are covered. If you count it up, this has taken Novell a bit over 5 years to get to this point.

In my opinion, that's about right for an organization the size of Novell. Porting over the proprietary NetWare services to completely new kernel requires a LOT of developer attention, and Novell is a lot smaller than Microsoft. Also of note, it took Microsoft 5 years to give us Vista after XP, including the presumed nuke-and-rewrite they did. Novell got a boost in that they had already ported eDirectory to Linux, so that helped out the NCP side. But that didn't help the NSS folks, who had to figure out a way to do a NetWare-style rich metadata file-system on a kernel and driver model that expects POSIX-spartan file-systems. The problems Novell had with this were amply displayed in the performance problems reported with OES1-FCS. Samba doesn't scale to the same levels as CIFS-on-NetWare did, so that meant Novell had to create their own CIFS stack from scratch. The AFP stack on Linux is a joke, and the resurgence of Apple since 2003 meant they had to do something about that as well; by making a proprietary AFP stack. All of this represents nuke-and-rebuild-from-spec, which takes time.

Novell probably should have started the migration in 2000 instead of 2003. They already knew that Exchange 5.5 upgrades were driving a LOT of customers into Active Directory, which was triggering migrations away from NetWare. But, there are business concerns here. Novell managed to survive the fall of NetWare by diversifying their product portfolio enough that GroupWise, Zen, and Identity Management could support the company. It took until this year to return to the black, but they did it. Had they shot the NetWare cash cow two years earlier, it is entirely possible that Novell couldn't have survived the lean years.

Labels: , , , ,


<permalink> posted by riedesg : 11:16 AM  1 comments

Saturday, November 29, 2008

10,000 hours

I read an excerpt of a book a week or so ago. Always dangerous, as it lacks context. But the general principal of the book was the observation that to get really really good at something requires about 10,000 hours of practice. There are no 'naturals', just people who are naturally more pig-headed than others who can get to 10K hours.

10K hours is 2-4 hours a day for 10 years.

The studies were about things like child prodigies, or top tier athletes who get Olympic gold at age 22, and retire by 30. That sort of thing. It seems that almost all of these people started their thing by age 6, and by age 8 there was already a break between the kids who'd ultimately reach the peak of their field and those who'd merely be very good. The ones destined for peak were giving 2-3 hours a day at age 8, where the other group had cut back.

I believe this also applies to technical expertise. As anyone who has done any job searching in my field knows, there are real breaks for experience levels; 1-3 years, 4-6 years, 10+. Those of us in the 10+ area (and by now I am there with NetWare, and by the end of December I can claim that with Windows) are pretty much technical experts. We've put in the time over the years to get good.

However, we work in a field where, "Change or Die," is an accurate mantra. The IT industry of 2008 is markedly different than it was in 1998. Windows NT installs right now are laughed at. Very, very little of the operating systems and software in active use in 1998 is still able to be on a support contract. It is hard to be a 10K-hour expert in something in our field, you have to put in 8 hours a day for 5 years.

My first real exposure to NetWare was in a class I took for my CNA back in the Autumn of 1996. That was on NetWare 4.0, so at least my first experience was with NDS. In fact, my first job with NetWare was with 3.x, so I had to learn bindary on-the-job.

I consider myself to be an expert in NetWare. I've been actively administering it for 11 years now, so if I'm not across the 10K line I'm really close to it. This is only possible because the 'change or die' mantra has not applied to NetWare over the years. Lets take a look at the biggest disruptions of how things work in NetWare (kernel). This isn't incremental changes, this is fundamental re-learnings of how things work. Sort of like what all the Windows engineers had to go through when Active Directory came onto the scene.
  1. The move to TCP/IP. This by far is the biggest disruption since 1996. NetWare 5.0(?) introduced the ability to do NCP over TCP/IP natively, and not tunneled IPX-over-IP. This required replacing IPX SAP, something the routers just did, with SLP, a service that needed configuration and setup.
  2. The NSS file-system. This was a much lesser move than the TCP/IP one, as it worked on a general level (trustees, quotas, etc) the same as TFS did. Tweaking it for performance, however, was a dark art for many years and much learning was derived out of this.
  3. Protected memory. A concept familiar to anyone who has used Windows or Linux, and all NetWare admins are by now administering one or both of these OS's. While some modules can't use it for whatever reason (iPrint, NetStorage) others (GroupWise) could.
  4. Native File Access Pack. NetWare could do AFP since the NW3 days, the same for NFS. SMB was another story. It was with NetWare 5.1 that NFAP came on to the scene, and NetWare 6.0 where it came built in and performed much better. The ability to use protocols other than NCP for your Windows clients was embraced by many shops.
There were more changes, but in my mind these are the biggest four. You will note the complete lack of OES in this list. That's because this is a list of the changes to NetWare, and OES-Linux is not NetWare. OES-Linux represents the sort of "change everything you expect" that the rest of the industry does, that the Novell ecosystem hasn't had.

Over the last 12 years NetWare has remained markedly static. This has allowed enough time for people who don't do this every waking moment to achieve a high level of expertise with NetWare. While this is good for NetWare, it unfortunately shows how NetWare has lagged behind the rest of the industry.

It is my opinion that OES-Linux represents a decade of pent up change that needed to happen in NetWare but didn't. This is why old time NetWare admins are having such trouble moving to Linux, they're being asked to support an Operating System that they don't have anywhere near the same level of expertise in and that is uncomfortable. I know I'm moving from an OS that I know exceedingly well to one where there are still, "here be monsters," marked out on my mental map. I'm also having to give up, "10+ year experience with NetWare," in favor of, "2-4 years of experience with Linux," and that doesn't feel good professionally.

But... that is the nature of our field. Just when we get really good at something, it's time to throw it out and learn something new. That something may be an incremental change from what we know (Windows 2003 vs Windows 2000) or a complete break (NT Domains vs AD Tree). But, learn we must. Us NetWare wonks have just been sheltered from it for some time.

Labels: , , ,


<permalink> posted by riedesg : 3:05 PM  4 comments

Monday, November 17, 2008

Signs and portents

Last Thursday I was over on download.novell.com looking for an eDirectory patch. I was staging up a new NetWare box and needed to see what the latest edir levels were. I knew 8.8.3 came out in August, and we're not there yet, so I needed 8.8.2 ftf2. However, I noticed that one of the searchable versions was 8.8.4. There was nothing in the list, but it was an option. It's not there right now, but it was then.

Thus emboldened I checked around a few more places. NetWare 6.5 SP8 was in the list, and still is right now. As is Open Enterprise Server 2 SP1. Both have the public betas posted, though.

But 8.8.4 was there. I saw it. Must have been a test or something. All this tells me that OES2 SP1 (a.k.a. NW65SP8) is just around the corner. Since we were told back at BrainShare that Sp1 would be in the Q4 time-frame, it's about due.

Labels: , , ,


<permalink> posted by riedesg : 10:15 PM  1 comments

Monday, November 10, 2008

NetStorage, WebDav, and Vista

I figured out how to get it working! You need KB907306. This updates the Web Folders in Vista to support how Novell does WebDav through NetStorage.

In our case you'll also need to add the CA that serves the SSL certificate that's on top of NetStorage (a.k.a. MyFiles). But, it works.

Labels: , , , ,


<permalink> posted by riedesg : 4:22 PM  0 comments

No BrainShare for me

Last week I asked my new boss if getting me to BrainShare was in the cards. I also threw out the alternative of the ATT Live sessions, which are fewer days and relatively cheaper than BrainShare. I also floated the possibility of me covering the plane expenses. As it happens Provo is about as expensive a city as Spokane, and Salt Lake City during BrainShare is equivalent to Bellevue. This is important since the travel ban is for 'out of state travel'. If I cover the plane tickets, that makes the cost the equivalent to having it in Spokane/Bellevue! Thus, I can go!

Unfortunately, not really. Quoth the new boss:
I highly doubt this'll work. Even Vice-Presidents are canceling travel plans.
If the grand high muckitymucks are honoring the travel ban, then the chances of me getting permission is next to zero. And is fully zero if I don't get buy off from my boss. Drat. Considering that the budget outlook for 2010 is even more grim than it is for 2009, the next probable BrainShare I can get to is 2011. Double drat.

Labels: ,


<permalink> posted by riedesg : 10:42 AM  2 comments

Monday, October 20, 2008

Dorm printing

On my post about finally running vista patrickbuller asked:
So you have printers that students in the dorms can print to? Wow. Do you audit all those and charge the numbers of pages against the student?
The answer to that is that we make big use of AND Technology's PCounter product. When paired with their PrintStations, it makes a very nice way to put a lid on unrestricted 'free' printing in the dorms. The PrintStations also make sure that only jobs people want to pick up get printed, which saves a serious amount of paper.

PCounter is core to our student printing. We'll only move our NDPS/iPrint infrastructure over to OES2-linux when Pcounter is supported on that platform, not before. We'll keep a 2 node NetWare cluster around just for printing if we have to. Since accounting support is one of the features that's supposed to be in OES2-SP1, it is my hope that PCounter will support OES2-Linux within a year after SP1's release. But I haven't heard any specifics.

Labels: , , , , ,


<permalink> posted by riedesg : 9:11 AM  2 comments

Wednesday, October 15, 2008

OES2 SP1 (public beta) has been posted

The public beta of OES2 SP1 has been posted.

I believe the NDA has lifted, but I'm not 100% on that. Will check. But, some of the new stuff in SP1:
  • An AFP stack that doesn't suck. Or more specifically, an AFP stack that scales beyond 100 users and is eDirectory integrated.
  • A new CIFS stack written by Novell, so it can scale well past the Samba limit.
  • A migration toolkit in one UI, rather than a cluster of scripts.
  • A new version of iFolder
  • EDirectory integrated DNS/DHCP. But no eDir integrated SLP yet, open-source politics you know.
  • IIRC a beta of eDir 8.8 SP4.
  • The ability to put iPrint-for-Linux on NSS volumes (handy for Clustering).
  • And lots more I can't remember off the top of my head.
Go forth, and have fun. There is a beta-feedback box on the beta page I linked to above in case you find a bug and want to tell Novell about it.

One thing I think it is safe to say, is that even though it says "Beta4" on it, it's really a release-candidate. Only major bugs are getting quashed right now. UI freeze was a month or more ago, and strange, annoying behaviors may get "fixed in doc" rather than getting true fixes which will have to wait for SP2. Still report them anyway, since it'll go on the list to fix in the next SP.

Labels: , , , , ,


<permalink> posted by riedesg : 9:33 AM  0 comments

Tuesday, October 07, 2008

Erm, about the budget

From an email sent to all points from the U President this afternoon:

In the OFM spreadsheets received today, we were stunned to find that targets had been set for higher education. Western, today, is now expected, from the sorts of measures outlined in the August 4 memorandum, to "save" $1,827,000 in the current fiscal year. (This major reduction applies across all budgets, including instructional budgets.)

Add that to the earlier number, and our total budget reduction is NOT the $176,000 representing 1% of non-instructional budgets. It is $2,003,682.

Aaaaaand...

Further, we have been advised to expect these reductions to be permanent; that is, to also be a part of our 2009-11 budget.

Pardon me whilst I mutter things.

This means that it is nearly certain that we will NOT be getting any new hardware for the Novell cluster next summer. We'll have to do it on hardware we already own right now. This means I won't be able to partake of that lovely 64-bit goodness. Drat drat drat.

We're already under-funded for where we need to be, this won't help. Even with the storage arrays we just bought, in terms of total disk-space we've managed to fully commit all of it. There is no excess capacity. What's more, there is no easy way to ADD new capacity since any significant amounts will require purchasing new storage shelves.

In the intermediate term, this means that WWU will now descend into bureaucratic charge-back warfare. As service-providing departments like ours try to find ways to finance the needed growth, we'll start being hard-ass about charging for exceptional services. And they'll do it to us too. So if the College of Arts and Sciences comes to us and asks us for space to host 2TB of, say, NASA data, we'll have to bill them for it. And that cost will be a 'total cost' which will by necessity include the backup costs. In return, if we need 16 ethernet jacks added to the AC datacenter, Telecom may start billing us.

And I get a new boss Thursday. Happily, since there is overlap between outgoing and incoming they've been briefing a lot. This is to prepare the new guy for the challenges he'll face in his first few weeks flying solo. There may even be the odd phone-call for advice, we'll see.

Gonna get real interesting around here.

Labels: , , ,


<permalink> posted by riedesg : 9:24 PM  0 comments

Wednesday, September 24, 2008

Fickle fortune

I lost a RAID card in one of my Beta servers. Crap. These beasties are all old beasties since that's the only hardware that could be released for the beta. And with crap servers, comes a crap failure rate. This is the second RAID card I've lost, and I've lost one hard-drive too. It isn't common to lose more RAID cards than hard-drives. Arrg.

This puts a kink into things. This was going to be an edirectory host, so I could host my replicas on one set of servers and abuse the crap out of the non-replica application servers. I may have to dual host. Icky icky.

Labels: , ,


<permalink> posted by riedesg : 2:38 PM  0 comments

Wednesday, September 10, 2008

That darned budget

This is where I whine about not having enough money.

It has been a common complaint amongst my co-workers that WWU wants enterprise level service for a SOHO budget. Especially for the Win/Novell environments. Our Solaris stuff is tied in closely to our ERP product, SCT Banner, and that gets big budget every 5 years to replace. We really need the same kind of thing for the Win/Novell side of the house, such as this disk-array replacement project we're doing right now.

The new EVAs are being paid for by Student Tech Fee, and not out of a general budget request. This is not how these devices should be funded, since the scope of this array is much wider than just student-related features. Unfortunately, STF is the only way we could get them funded, and we desperately need the new arrays. Without the new arrays, student service would be significantly impacted over the next fiscal year.

The problem is that the EVA3000 contains between 40-45% directly student-related storage. The other 55-60% is Fac/Staff storage. And yet, the EVA3000 was paid for by STF funds in 2003. Huh.

The summer of 2007 saw a Banner Upgrade Project, when the servers that support SCT Banner were upgraded. This was a quarter million dollar project and it happens every 5 years. They also got a disk-array upgrade to a pair of StorageTek (SUN, remember) arrays, DR replicated between our building and the DR site in Bond Hall. I believe they're using Solaris-level replication rather than Array-level replication.

The disk-array upgrade we're doing now got through the President's office just before the boom went down on big expensive purchases. It languished in the Purchasing department due to summer-vacation related under-staffing. I hate to think how late it would have gone had it been subjected to the added paperwork we now have to go through for any purchase over $1000. Under no circumstances could we have done it before Fall quarter. Which would have been bad, since we were too short to deal with the expected growth of storage for Fall quarter.

Now that we're going deep into the land of VMWare ESX, centralized storage-arrays are line of business. Without the STF funded arrays, we'd be stuck with "Departmental" and "Entry-level" arrays such as the much maligned MSA1500, or building our own iSCSI SAN from component parts (a DL385, with 2x 4-channel SmartArray controller cards, 8x MSA70 drive enclosures, running NetWare or Linux as an iSCSI target, with bonded GigE ports for throughput). Which would blow chunks. As it is, we're still stuck using SATA drives for certain 'online' uses, such as a pair of volumes on our NetWare cluster that are low usage but big consumers of space. Such systems are not designed for the workloads we'd have to subject them to, and are very poor performers when doing things like LUN expansions.

The EVA is exactly what we need to do what we're already doing for high-availability computing, yet is always treated as an exceptional budget request when it comes time to do anything big with it. Since these things are hella expensive, the budgetary powers-that-be balk at approving them and like to defer them for a year or two. We asked for a replacement EVA in time for last year's academic year, but the general-budget request got denied. For this year we went, IIRC, both with general-fund and STF proposals. The general fund got denied, but STF approved it. This needs to change.

By October, every person between and Governor Gregoir will be new. My boss is retiring in October. My grandboss was replaced last year, my great grand boss also has been replaced in the last year, and the University President stepped down on September 1st. Perhaps the new people will have a broader perspective on things and might permit the budget priorities to be realigned to the point that our disk-arrays are classified as the critical line-of-business investments they are.

Labels: , , , , , , , , , , , ,


<permalink> posted by riedesg : 10:50 AM  1 comments

Monday, August 11, 2008

Novell Client for Vista, the ecosystem

I just reported a bug in the beta that surprised me. I can't talk details about it, but it strikes me as the kind of bug that should have been at least reported shortly after the client released. Perhaps it was just so overall buggy that it got lost in the forest, but still. The Vista client has been out for some time now.

Having said the following rant several times over the past few days, I figure it's time to post it ;).

The problem we're running in to is that the number of users of the Vista Client is a small, small sub-set of the overall users of the Novell Client, which are by now a minority of overall users of Novell NCP file-servers. Novell spent years hyping 'clientless' approaches to file-serving, through the CIFS stack on NetWare. A lot of places bought in to that. Because of this, the percentage of NCP-client Vista users among the overall Novell File-Server market is a rather small one.

And small means you don't get a lot of testing done by people-who-are-not-us, and seemingly obvious bugs showing up in the beta Sp1 builds. I don't have any Vista workstations, so I've done exactly zero testing of the Vista Client; this particular bug was reported and troubleshot by someone who is not me (I just filed it). Even though we have beta builds of the Vista client as part of this beta, I'm not testing it. All things considered, I probably should.

Since we're wedded hard to the Novell Client, it's probably time for us to start devoting resources to the ecosystem in order to keep it alive.

Labels: , ,


<permalink> posted by riedesg : 3:27 PM  4 comments

Friday, July 25, 2008

Handling eDirectory core-files on linux

If you've been getting core files generated by ndsd on your Linux servers, and want to call Novell Support about it, there are a few things you can do to maximize what Novell will get out of the files themselves. You may not get much, but these will help the people with the debug symbols figure out what's going on.

Packaging the Core


First and foremost, you already have the tool to package core files for delivery to Novell already on your system. TID3078409 describes the details of how to use 'novell-getcore.sh'. It is included on 8.7.3.x installations as well as 8.8.x installations.

Running it looks like this:
edirsrv1:~ # novell-getcore -b /var/opt/novell/eDirectory/data/dib/core.31448 /opt/novell/eDirectory/sbin/ndsd
Novell GetCore Utility 1.1.34 [Linux]
Copyright (C) 2007 Novell, Inc. All rights reserved.


[*] User specified binary that generated core: /opt/novell/eDirectory/sbin/ndsd
[*] Processing '/var/opt/novell/eDirectory/data/dib/core.31448' with GDB...
[*] PreProcessing GDB output...
[*] Parsing GDB output...
[*] Core file /var/opt/novell/eDirectory/data/dib/core.31448 is a valid Linux core
[*] Core generated by: /opt/novell/eDirectory/sbin/ndsd
[*] Obtaining names of shared libraries listed in core...
[*] Counting number of shared libraries listed in core...
[*] Total number of shared libraries listed in core: 72
[*] Corefile bundle: core_20080725_092227_linux_ndsd_edirsrv1
[*] Generating GDBINIT commands to open core remotely...
[*] Generating ./opencore.sh...
[*] Gathering package info...
[*] Creating core_20080725_092227_linux_ndsd_edirsrv1.tar...
[*] GZipping ./core_20080725_092227_linux_ndsd_edirsrv1.tar...
[*] Done. Corefile bundle is ./core_20080725_092227_linux_ndsd_edirsrv1.tar.gz

Once you have the packaged core, you can upload it to ftp.novell.com/incoming as part of your service-request.

Including More Data


If you're lucky enough to be able to cause the core file to drop on demand, or it just plain happens often enough that repetition isn't a problem, there is one more thing you can do to include better data in the core you ship to Novell. TID3113982 describes a setting you can add to the ndsd launch script (/etc/init.d/ndsd) that'll include more data. The TID describes what is being done pretty well. In essence, you're using an alternate malloc call that fails with better information than the normal one. You don't want to run with this set for very long, especially in busy environments, as it impacts performance. But if you have a repeatable core, the information it can provide is better than a 'naked' core. Setting MALLOC_CHECK_=2 is my recommendation.

Be sure to unset this once you're done troubleshooting. As I said, it can impact performance of your eDirectory server.

Labels: , , , , ,


<permalink> posted by riedesg : 9:34 AM  0 comments

Wednesday, July 16, 2008

Patching SLES

Last night I attempted to patch one of our OES2 servers. This particular server is an elderly beast, a P3 1GHz machine. So I wasn't expecting anything like fastness out of it. Especially with rug.

But still, it was painful!
normandy: ~#: rug lu
Waking up ZMD...
[8 minutes later]
[list of one update, libzypp]
normandy: ~#: rug update
Resolving Dependencies....
[8 minutes later]
Install this update? (y/N)
y
[12 minutes later]
Restarting ZMD...
[8 minutes later]
normandy: ~#: rug lu
[list of updates. No need to wait 8 minutes this time.]
normandy: ~#: rug update
Resolving Dependencies...
[8 minutes later]
Dependency resolution failed for bind-util and bind-libs. libdns-whatzihoozit required by bind-util is provided by bind-libs. Please fix you hoser.
[insert swearing here]
normandy: ~#: rug in bind-util bind-libs
Resolving Dependencies....
[8 minutes later]
Install these updates? (y/N)
y
[12 minutes later]
normandy: ~#: exit
As this had taken far longer than even I was expecting, I stopped. I'll finish up tonight. As this is an OES2 server, this means SLES10-SP1. I can attest that SLES10-SP2 on identical hardware is MUCH faster. I can't wait until OES2-SP1 comes out and this dinosaur can get faster patching.

Labels: , , ,


<permalink> posted by riedesg : 4:19 PM  0 comments

Monday, June 30, 2008

Novell Client for Linux, packaged for OpenSUSE

It has been mentioned many places, and I've done some of the mentioning, that since openSUSE is the foundation for SLED, it makes sense for Novell to distribute an NCL for openSUSE. It turns out they're working on just that. And here is the Novell beta page. I'm soooo going to try this out, since I'm running openSUSE 10.3 on my work desktop and won't be moving to openSUSE 11 until I can run the client on it (oh, wait, I can).

It should also be mentioned that Ubuntu is a very frequently requested target for another NCL, but I have reason to believe that'll never happen. First of all, any Novell Client involves closed source 3rd party licensed code, which makes it hard to port to Linux in the first place (a relic of being based on code from the days when open-source was just an ethical standpoint rather than a tangible market force). Second, Novell has proven to be rather light in developer resources in certain areas, and linux integration with non-SUSE linux distros is very minimal.

Labels: , , , , ,


<permalink> posted by riedesg : 12:29 PM  0 comments

Tuesday, June 24, 2008

Backing up NSS, note for the future

According to this documentation, the storing of NSS/NetWare metadata in xattrs is turned off by default. You turn it on for OES2 servers through the "nss /ListXattrNWMetadata" command. This allows linux level utilities (i.e. cp, tar) to be able to access and copy the NSS metadata. This also allows backup software that isn't SMS enabled for OES2 to be able to backup the NSS information.

This is handy, as HP DataProtector doesn't support NSS backup on Linux. I need to remember this.

Labels: , , , , ,


<permalink> posted by riedesg : 9:10 AM  3 comments

Monday, June 16, 2008

A good article on trustees

Over on the Novell Cool Solutions site, Marcel Cox just posted an article about how Trustees are handled on the Novell Filesystems (TFS and NFS). If you wanted to know the fundamentals of how ACLs are done on NSS volumes and how it relates to eDirectory, this is a good start.

Labels: , , , , , ,


<permalink> posted by riedesg : 12:55 PM  1 comments

Thursday, May 29, 2008

OES2 and SLES10-SP2

Per Novell:

Updating OES2

OES2 systems should NOT be updated to SLES10 SP2 at this time!
Very true. And most especially true if you're running virtualized NetWare! The paravirtualization components in NW65SP7 are designed around the version of Xen that's in SLES10-SP1, and SP2 contains a much newer version of Xen (trying to play catch-up to VMWare means a fast dev cycle, after all). So, expect problems if you do it.

Also, the OES2 install does contain some kernel packages, such as those relating to NSS.

OES2 systems need to wait until either Novell gives the all clear for SP2 deployments on OES2-fcs, or OES2-SP1 ships. OES2-SP1 is built around SLES10-Sp2.

Labels: , , , , ,


<permalink> posted by riedesg : 9:45 AM  0 comments

Friday, May 23, 2008

Problem with SLES10-SP2

Just this morning Novell posted a new TID:

Updates catalogs missing after updating libzypp

I've heard on the grape-vine that this particular libzypp update was put into the SLES10-SP1 channel in order to prepare for SP2's release. Those fine folk out there that have turned on Auto Updating on their SLE[S|D] boxes have very probably already been bit by it. I hope Novell gets this one fixed, and posts recovery steps, soon.

Labels: , , , ,


<permalink> posted by riedesg : 10:07 AM  0 comments

Wednesday, May 21, 2008

SLES10 SP2 shipped

According to Novell, SLES10 SP2 has shipped.

This means that the ongoing OES2 SP1 beta I'm a part of will be done on released code for the SLES side of it. So any bugs we find there may end up as patches on the SP2 channel.

One nice thing in the new code?

"rug refresh --clean"

This will do what I posted about a few days ago. It'll nuke the zmd database and rebuild it fresh! Niiiice! Unfortunately, a truly better version of rug won't come until "Code 11".

Labels: , , ,


<permalink> posted by riedesg : 2:22 PM  0 comments

Wednesday, May 14, 2008

NetWare and Xen

Here is something I didn't really know about in virtualized NetWare:

Guidelines for using NSS in a virtual environment

Towards the bottom of this document, you get this:

Configuring Write Barrier Behavior for NetWare in a Guest Environment

Write barriers are needed for controlling I/O behavior when writing to SATA and ATA/IDE devices and disk images via the Xen I/O drivers from a guest NetWare server. This is not an issue when NetWare is handling the I/O directly on a physical server.

The XenBlk Barriers parameter for the SET command controls the behavior of XenBlk Disk I/O when NetWare is running in a virtual environment. The setting appears in the Disk category when you issue the SET command in the NetWare server console.

Valid settings for the XenBlk Barriers parameter are integer values from 0 (turn off write barriers) to 255, with a default value of 16. A non-zero value specifies the depth of the driver queue, and also controls how often a write barrier is inserted into the I/O stream. A value of 0 turns off XenBlk Barriers.

A value of 0 (no barriers) is the best setting to use when the virtual disks assigned to the guest server’s virtual machine are based on physical SCSI, Fibre Channel, or iSCSI disks (or partitions on those physical disk types) on the host server. In this configuration, disk I/O is handled so that data is not exposed to corruption in the event of power failure or host crash, so the XenBlk Barriers are not needed. If the write barriers are set to zero, disk I/O performance is noticeably improved.

Other disk types such as SATA and ATA/IDE can leave disk I/O exposed to corruption in the event of power failure or a host crash, and should use a non-zero setting for the XenBlk Barriers parameter. Non-zero settings should also be used for XenBlk Barriers when writing to Xen LVM-backed disk images and Xen file-backed disk images, regardless of the physical disk type used to store the disk images.

Nice stuff there! The "xenblk barriers" can also have an impact on the performance of your virtualized NetWare server. If your I/O stream runs the server out of cache, performance can really suffer if barriers are non-zero. If it fits in cache, the server can reorder the I/O stream to the disks to the point that you don't notice the performance hit.

So, keep in mind where your disk files are! If you're using one huge XFS partition and hosting all the disks for your VM-NW systems on that, then you'll need barriers. If you're presenting a SAN LUN directly to the VM, then you'll need to "SET XENBLK BARRIERS = 0", as they're set to 16 by default. This'll give you better performance.

Labels: , , , , , ,


<permalink> posted by riedesg : 10:31 AM  0 comments

Tuesday, May 06, 2008

Being annoyed by rug?

Rug/zmd in SLES10-SP1 is still a headache maker. Novell knows this, but I strongly suspect that we'll have to wait until SLES11 before we get anything improved. OpenSUSE now has zypper which works pretty good, and I think you can do it in SLES if you want, but I haven't tried.

One of the chief annoyances of rug is that the zmd.db file kept in /var/lib/zmd/zmd.db gets corrupted far too easily. And when that happens, rug can take HOURS to return anything. If it returns anything at all.

The fix for it is easy, stop zmd, delete the zmd.db file, restart zmd. Since I'm doing this fairly often, I've whipped up a bash script to do it for me.

nukezmd
#!/bin/sh
#
# For killing ZMD when it is clearly hung. An all too often occurance.
#

declare PIDZMD

# First get the PID of ZMD

printf "Getting PID... "
let PIDZMD=`rczmd showpid`
printf "$PIDZMD\n"
# Then unconditionally kill it

printf "Killing zmd hard... \n"
kill -9 $PIDZMD

# Remove the old, inconsistent database

printf "Nuking old database... \n"
rm /var/lib/zmd/zmd.db

# Restart ZMD, which will build a new, consistent database

printf "Restarting ZMD\n"
rczmd start
Simple, to the point. Works.

Labels: , , ,


<permalink> posted by riedesg : 2:57 PM  0 comments

Monday, May 05, 2008

Linux @ Home

My laptop at home dual-boots between openSUSE and WinXP. There are a few reasons why I don't boot the Linux side very often, some of them work related. And, what the heck, here are the two reasons.

1: Wireless driver problems
I have an intel 3945 WLAN card. It works just fine in linux, well supported. What throws it for a loop, however, are sleep and hibernate states. It can go one, two, four, maybe five cycles through sleep before it will require a reboot in order to find the home wireless again. If it doesn't lock the laptop up hard. Since my usage patterns are heavily dependent upon Sleep mode, this is a major, major disincentive to keep the Linux side booted.

I understand the 2.6.25 kernel is a lot better about this particular driver. Thus, I wait with eager anticipation the release of openSUSE 11.0. This driver is currently the ipw3945 driver, and will eventually turn into iwl3945 driver once it comes down the pipe. What little I've read about it suggests that the iwl driver is more stable through power states.

2: NetWare remote console
I use rconip for remote console to NetWare. Back when Novell first created the IP-based rconsole, they also released rconj along side ConsoleOne to provide it. As this was written in Java, it was mind bogglingly slow. This little .exe file was vastly faster, and I've come to use it extensively. Unless I get Wine working, this tool will have to stay on my Windows XP partition. It works great, and I haven't found a good linux-based replacement yet.

Time has moved on. Hardware has gotten faster, and the 'java penalty' has reduced markedly. RconJ is actually usable, but I still don't use it. Plus, it would require me to install ConsoleOne onto my laptop. It's 32-bit, so that's actually possible, but I really don't want to do that.

The Remote Console through the Novell Remote Monitor (that service out on :8009) has a nice remote-console utility, but it also requires Java. I'm still biased against java, and java-on-linux still seems fairly unstable to me. I don't trust it yet. It also doesn't scale well. When I'm service-packing, it is a LOT nicer looking to have 6 rconip windows up than 6 browser-based NRM java-consoles open. Plus, rconip will allow me access to the server console if DS is locked, something that NRM can't do and is invaluable in an emergency.

Once the wireless driver problems are fixed, I'll boot the linux side much more often. Remote-X over SSH actually makes some of my remote management a touch easier than it is in WinXP. And if I really really need to use Windows, my work XP VM is accessible over RDesktop. There are a few other non-work reasons why I don't boot Linux very often, but I'll not go into those here.

So, oddly, NetWare is partly responsible for keeping me in Windows at home. But only partly.

Labels: , , , ,


<permalink> posted by riedesg : 4:32 PM  1 comments

Thursday, April 17, 2008

NetWare and Novell, changing a company

A couple days ago Richard Bliss had a long blog entry about, "Novell's Cash Cow - How NetWare almost killed the company". It had some very interesting points. Some we knew:
We are all familiar with NetWare, the dominate Network Operating system of the 1980s and 1990s. We are all familiar with Microsoft's tactics of penetrating the NOS market with Windows NT by focusing on using Windows as an application platform.
Apparently Richard worked for Novell around 2001. I find that interesting since my first BrainShare was 2001, and that was when they announced the release of NetWare 6.0. While there he saw what seemed to be an outright denial that NetWare had been passed up by Windows and something new needed to be done.

In 2001 I knew that Windows had for all intents and purposes won. The only place you ever really saw NetWare servers were as file-servers, or running GroupWise or the small handful of apps that used NetWare as an application server. The stalwart loyalists among us saw this as annoying, but not a major problem.

It was also good for Novell's bottom line. NetWare still accounted for a large percentage of their revenues. Even though the writing was on the wall, they were still making real money on it so didn't see a need to change. This is why NetWare 6.0 introduced the AMP stack to NetWare, as a way to better make NetWare an application server and to slow the loss of customers. At BrainShare 2001 there was open speculation about "NetWare 7.0" and what it would look like.

And there still was until 2005 when Novell announced what the next version of NetWare would be. This being after the SUSE and Ximian purchases, it would be based on Linux. This move had been rumored, and alternately derided and lauded, for some time. There was a great wailing and gnashing of teeth on the part of the stalwart NetWare loyalists. It also started an exodus of customers, as Novell's financial reports at the time point out.

Fortunately for the company, they started actively promoting (for certain values of 'active' that are higher than they were previously, but still in the theme of Novell Stealth Marketing) and developing their other products, like GroupWise, Novell Identity Management, ZenWorks, and most especially their Linux business. It took them until last quarter to turn in a quarter in the black, and NetWare revenues are under 20% of total now. So, they've turned the corner and are no longer dependent on the NetWare cash cow. They have a couple of them in the field now, which is a MUCH healthier place to be.

It's a funny thing, but one of the reasons why NetWare is such a kick-butt file-server compared to everything else is why it's a challenging environment to develop in. Had Novell seen the light earlier and bought SUSE (or rolled their own Linux distro) in... 1999 instead, right after the NW5.1 release, they still would have run into the fundamental architectural problems in 32-bit linux that make it an inferior file-serving platform for large environments. By 2008 their server could have been a LOT more mature, and perfectly poised to take advantage of 64-bit Linux.

Novell in the 1990's is not an example of a 'nimble' company. It is trying to get there now through diversification. Not many companies (especially tech companies) have survived the loss of their prime money earner; Apple has done it through OSX, which required a fanatically loyal fan base to survive the dark years. This is the prime reason people kept predicting the imminent demise or buyout of Novell. Now that they're earning profits again, and have diversified away from just the OS sector, they're not going to be going out of business any time soon.

Now if only they had better SMB packages and programs. I hear repeatedly from peers who support SMBs that Novell's packages and programs in that space are lacking or exploitative. Significant revenue, and more importantly mindshare, are in the SMB market. Plus, today's SMB is tomorrow's large or global enterprise.

Labels: , ,


<permalink> posted by riedesg : 10:53 AM  0 comments

Tuesday, April 01, 2008

Slow blogging

I found out at BrainShare that WWU has been accepted as a Novell Authorized Beta site for OES2 SP1. And that's what I've been doing for the better part of the past week. Due to the NDA required, I can't talk about it. So, not much bloggable stuff to bring forward.

We requested entry into the program in part because of what I learned at BrainShare 2007. Specifically, Novell doesn't test for our scales of users. Therefore, it is in our best interest to make sure that organizations like us are in the beta. We have the hardware to make a go of it right now (all those new ESX boxes are liberating some still-useful 3-5 year old servers), and I have the time. Unfortunately, the only 64-bit testing we'll be doing will be in VMWare, so the newest of the new code will have to be really tested by other people.

That's why I've been quiet.

Labels: , ,


<permalink> posted by riedesg : 7:19 PM  3 comments

Tuesday, March 25, 2008

IPv6 vs IPX

In a session last week came the following comment from a presenter (paraphrased):
How may of you in the room have been at this long enough to do IPX? Ok, great. Now how many of you have done anything with IPv6? Doesn't that look JUST like IPX?
And he's right, to a point. IPX addresses are of the form network-number:node-number, such as:

00008021:0002a540d0e1

Where 'node number' is the MAC address of the network card in question. It's up to the routers to figure out where network-numbers live, and advertised services issue full-network broadcasts to advertise said service, which is the primary reason that IPX just doesn't scale if WAN links are in the mix. But that's by the by.

IPv6 addresses work similarly:

2001:0db8:85a3:08d3:1319:8a2e:0370:7334

The last 48 bits are the MAC address and the bits ahead of it constitute the network number. Except... the IPv6 designers knew about the failings of IPX and worked around them. The last 48 bits don't have to be the MAC address, though as I understand it that address has to exist for each physical interface. Unlike IPX, IPv6 has the ability to have 'secondary' addresses. The lack of this ability was the main reason that Novell Cluster Services only worked on IP networks, which caused its own wave of grief when clustering was introduced in the NetWare 5.1 era. Secondary IPv6 numbers don't have to follow the MAC format, which in my opinion is a good thing!

Yes, when I first read about IPv6 addressing I had that same, "wow, this is just like IPX," moment the BrainShare presenter had. Only, more scalable, and more flexible.

Labels: , , , ,


<permalink> posted by riedesg : 1:57 PM  0 comments

Thursday, March 20, 2008

BrainShare Thursday

Not a good day. My first course, "Advanced BASH," could more accurately be described as, "BASH scripting tips & tricks". I then proceeded to skip the other three sessions I had signed up for.
  • Novell Open Enterprise Server 2 Interoperability with Windows and AD. All about Domain Services for Windows and Samba. Neither of which we'll ever use. No idea why I wanted to be in this session.
  • Rapid Deployment of ZENworks Configuration Management. Other people around here have suggested that if we haven't moved yet, wait until at least SP3 before moving. If then. So, demotivated. Plus I was rather tired.
  • Configuring Samba on OES2. CIFS will do what we need, I don't need Samba. Don't need this one. Skipped.
DL236: Advanced BASH Course
BASH tips and tricks. I got a lot out of it, but the developers around me were quietly derisive.

ZEN Overview and Features
Not so much with the futures, but it did explain Novell's overall ZEN strategy. It isn't a coincidence that most of Novell's recent purchases have been for ZEN products.

TUT303: OES2 Clusters, from beginning to extremes
This was great. They had a full demo rig, and they showed quite a bit in it. Including using Novell Cluster Services to migrate Xen VM's around. They STRONGLY recommended using AutoYast to set up your cluster nodes to ensure they are simply identical except for the bits you explicitly want different (hostname, IP). And also something else I've heard before, you want one LUN for each NSS Pool. Really. Plus, the presenters were rather funny. A nice cap for the day.

And tonight, Meet the Experts!

Labels: , , , , , , ,


<permalink> posted by riedesg : 4:05 PM  2 comments

BrainShare Wednesday

The Wednesday keynote was, indeed, a bunch of demos. It was also mostly pointless as far as the technology I'm concerned with. Lots of GroupWise (don't care), lots and lots of PlateSpin (can't afford it), lots of Zen (not the bits I'd use).

That said, the new GroupWise WebAccess is gorgeous. I wish Exchange had their non-ActiveX pages look that good.

TUT175: RBAC: Avoiding the horror, getting past the hype
Mostly about IDM as it turned out. Only minimally interesting from an abstract viewpoint about roles in general.

TUT 277: Advanced eDirectory Configuration, new features, and tuning for performance
I learned a few things I didn't know, such as the fact that each object as an "AncestorList" attribute listing who their parent objects are. This apparently greatly speeds up searching. SP3, coming out this Summer, will have faster LDAP binds for a couple of reasons. Right now Novell is recommending 2 million objects as a reasonable maximum size for a partition for performance reasons.

And also they reiterated something I've heard before...
You know how back in the NetWare 4 days, we said to design your tree by geography at the first level, and then get to departments? Um, sorry about that. It was great back then, but for LDAP or IDM it really, really slows things down.
Yep. I took my first class for my CNA when 'Green River' was just coming out, or was just out. So I remember that.

TUT221: iPrint on Linux, what Novell Support wants you to know
A nice session from a mainline support guy about the ways people don't do iPrint on linux correctly. We're not going there until pcounter can run in linux, so this is still somewhat abstract. But, nice to know.
  • The reason that some print jobs render differently than direct-print jobs, is because of how Windows is designed. Direct-print jobs render with the 'local print provider', and iPrint jobs render with the 'network print provider'. This is a Microsoft thing, not an iPrint thing. You can duplicate it by setting up a microsoft IPP printer (assuming you're not mandating SSL like we are) and printing to the same printer with the same driver.
  • The Manager on Linux doesn't use a Broker, it uses a 'driver store'.
  • The Manager on NetWare doesn't always bind to the same broker. I didn't know that.
  • It is recommended to have only one Broker, or one driver store per tree.
  • Novell recommends using DNS rather than IP for your printer-agents, check your manager load scripts.

Labels: , , , , , , , ,


<permalink> posted by riedesg : 3:54 PM  0 comments

Tuesday, March 18, 2008

BrainShare Tuesday

Today started off with a bit of panic, as I hadn't set my alarm. Me being a west-coaster, 7:20 (when I woke up) is an entirely reasonable time to get up as far as my body is concerned. Only, I needed to get dressed and breakfasted before my first session at 8:30. Aie! I had to eat quick, but I got there. Didn't get a chance to check work email, though.

ATT326: Advanced Linux Troubleshooting
An ATT, therefore hard to summarize. But I learned about a few new commands I didn't know about before. Like strace. And vimdiff.

TUT130: Challenges in Storage I/O in Virtualization
Another nice one, but an emergency at work (printing down in a dorm, during finals week) distracted me heavily during the first half of it. Which resulted in the following note in my notes:
NPIV looks really nifty. Look into it.
NPIV being how you can use fibre-channel zoning to zone off VM's, rather than HBA's. Highly useful. I also learned about a neat new thing called Virtual Fabrics. Virtual Fabrics work kind of like VLANS for fabrics. You can segregate your fabrics into fabrics that share hardware but nothing else. Handy if your, say, Solaris admins don't want you mucking about with their zoning, while saving money through consolidated hardware.

TUT216: OES2 SP1 Architectural Overview
There is a LOT of new stuff in SP1.
  • It will include eDir 8.8.4 (8.8.3 will ship this summer sometime)
  • NCP and eDir will be fully 64-bit
  • OES2 SP1 will be based on SLES SP2, which will be releasing about the same time
  • AFP Support
    • AFP 3.1
    • Uses Diffie-Helman 1 for password exchange, meaning the 8-character password problem is solved.
    • Fully SMP-safe
    • Has cross-protocol locking with NCP. CIFS doesn't have cross-protocol locking yet, but IIRC, Samba does
    • Does not need LUM enabled users
  • CIFS Support
    • NTLMv1, but v2 is a possibility if enough people ask, so file those enhancement requests!!
    • CIFS is separate from Samba, therefore can not be used in conjunction with Domain Services for Windows
    • As with AFP, fully SMP safe
  • EDir 8.8.4
    • LDAP auditing enhanced
    • "newer auth protocols", but they didn't say what.
I should also mention that they're still deploying Novell Integrated Samba, which is what you'll have to use to get Domain Services For Windows. Samba still doesn't scale as far as I'd like ('only' 700-800 concurrent users), so that may be an issue for higher ed types who want high concurrency CIFS and also DSFW on the same box.

TUT211: Enhanced Protocol Support in OES2 SP1
This is the session where they went into detail about the AFP and CIFS support. They said that netatalk, the existing AFP stack on Linux, gets really slow once you go over the 20 concurrent users. Whoa! I can soooo understand why Novell felt the need to make a new one.
  • The 8 character password limit has been fixed! They now support DH1 for passing passwords.
  • The 'afptcp' daemon can use one password protocol at a time, so you can only use DH1, or one of the other three I can't remember.
  • Support for OSX 10.1 and 10.2 is scanty, and 10.5 is limited but users may not notice anyway.
  • Passwords will be case sensitive.
  • Kerberos will be in a future release
  • Performance is faster than NetWare, partly due to the ability to multi-thread
  • Can register services by way of SLP
  • Only supports NSS for the time being, the other Linux file-systems will be a future feature.
  • Can support 500 concurrent users, and 1000+ in the future. This fits our current AFP loads.
  • We can configure more about how it works than we could on NetWare, such as how many worker threads to spawn.
  • Has meaninful debug logs!
  • Has a new command, 'afpstat' that works like 'netstat' for giving a snapshot of afp connections.
And then some CIFS stuff. We can't use it for political reasons so I didn't pay attention. Sorry.

Tonight was the night formerly known as 'Sponsor Night,' but has a new name now that everyone who gets a booth is no longer a 'sponsor'. Some are sponsors, some are exhibiters. I can't keep track. Anyway, today was their party. "World of Novellcraft!" Homage to vid-gaming.

Lots of Wii, lots of Rock Band, some Halo, lots of women dressed in Renaissance Festival gear getting their pictures taken by the 90%+ male audience. I've blogged before about my ambivalence about Sponsor Night. I lasted until about 7, when I came back to the hotel.

Tomorrow I have an actual LUNCH BREAK in my schedule! Ooo! And Soul Asylum Soul Coughing Collective Soul plays the concert! I've been listening to two of their CD's for the past two months so I think I may even know a few songs by now.

Labels: , , , , , ,


<permalink> posted by riedesg : 7:21 PM  0 comments

Monday, March 17, 2008

Today at Brainshare

Monday. Opening day. I had trouble getting to sleep last night due to a poor choice of bed-time reading (don't read action, don't read action, don't read action). And had to get up at 6am body time in order to get breakfast before the morning keynote. There be zombies.

Breakfast was uninspired. As per usual, the hashbrowns had cooled to a gellid mass before I found everything and got a seat.

The Monday keynotes are always the CxO talks about strategy and where we're going. Today a mess of press releases from Novell give a good idea what the talks were about. Hovsepian was first, of course, and was actually funny. He gave some interesting tid-bits of knowledge.
  • Novell's group of partners is growing, adding a couple hundred new ones since last year. This shows the Novell 'ecosystem' is strong.
  • 8700 new customers last year
  • Novell press mentions are now only 5% negative.
Jeff Jaffe came on to give the big wow-wow speech about Novell's "Fossa" project, which I'm too lazy to link to right now. The big concern is agility. He also identified several "megatrends" in the industry:
  • High Capacity Computing
  • Policy Engines
  • Orchestration
  • Convergence
  • Mobility
I'm not sure what 'Convergence' is, but the others I can take a stab at. Note the lack of 'virtualization' in this list. That's soooo 2007. The big problem is now managing the virtualization, thus Orchestration. And Policy Engines.

Another thing he mentioned several times in association with Fossa and agility, is mergers and acquisitions. This is not something us Higher Ed types ever have to deal with, but it is an area in .COM land that requires a certain amount of IT agility to accommodate successfully. He mentioned this several times, which suggests that this strategy is aimed squarely at for-profit industry.

Also, SAP has apparently selected SLES as their primary platform for the SMB products.

Pat Hume from SAP also spoke. But as we're on Banner, and it'll take a sub-megaton nuclear strike to get us off of it, I didn't pay attention and used the time to send some emails.

Oh, and Honeywell? They're here because they have hardware that works with IDM. That way the same ID you use for your desktop login can be tied to the RFID card in your pocket that gets you into the datacenter. Spiffy.

ATT375 Advanced Tips & Tricks for Troubleshooting eDir 8.8
A nice session. Hard to summarize. That said, they needed more time as the Laptops with VMWare weren't fast enough for us to get through many of the exercises. They also showed us some nifty iMonitor tricks. And where the high-yield shoot-your-foot-off weapons are kept.

BUS202 Migrating a NetWare Cluster to OES2
Not a good session. The presenter had a short slide deck, and didn't really present anything new to me other than areas where other people have made major mistakes. And to PLAN on having one of the linux migrations go all lost-data on you. He recommended SAN snapshots. It shortly digressed into "Migrating a NetWare Cluster to Linux HA", which is a different session all together. So I left.

TUT215 Integrating Macintosh with Novell
A very good session. The CIO of Novell Canada was presenting it, and he is a skilled speaker. Apparently Novell has written a new AFP stack from scratch for OES2 Sp1, since NETATALK is comparatively dog slow. And, it seems, the AFP stack is currently out performing the NCP stack on OES2 SP1. Whoa! Also, the Banzai GroupWise client for Mac is apparently gorgeous. He also spent quite a long time (18 minutes) on the Kanaka client from Condrey Consulting. The guy who wrote that client was in the back of the room and answered some questions.

Labels: , , , , , ,


<permalink> posted by riedesg : 7:33 PM  0 comments

Thursday, March 13, 2008

Brainshare Sponsors

In order to keep costs to us walking sales leads down, Novell solicits sponsors for BrainShare to help subsidize the whole event. There is nothing wrong with that, it means a lot of potential freebies for the people who are good at saying No politely ;).

So I'm offering this list of companies who have booths at BrainShare, what Novell product they're primarily interested in, and how it relates to me. The PDF I'm sucking this off of is this one of the Sponsor Hall.

  • SAP. The 'Cornerstone Sponsor'. I think everyone who reads my blog knows what they do. At a guess, their primary interest is in Identity Manager. SCT Banner is the ERP for the .EDU space, so we don't use 'em.
  • IBM. From last year, it's clear this is their Hardware division. So their primary interest is in SLES. We're on a different hardware platform, but... it's hardware. I'll still drop by to look at the pretty.
  • GWAVA. They make message filtering software for GroupWise. If you need anti-spam/virus for your GW installation, you're probably running GWAVA. We don't use GroupWise, so they have nothing I need.
  • GroupLink HelpDesk. A Helpdesk product that appears to be cross-platform. Their product is probably Linux, but it wouldn't surprise me to learn that they still have a lot of NetWare hiding back there. We use Magic Helpdesk for that function.
  • Microsoft. You know who they are. Officially their product is SLES but... who knows what they'll bring. We use a LOT of them around here, what with being an Exchange deployment and owning 96% of the desktops.
  • Messaging Architects. They are a more general email security and archiving provider. Their product is GroupWise, but they also sell some appliances that I could theoretically use in front of our Exchange servers. We've settled on a product from a much bigger vendor for that function, but still.
  • Novacoast IT. A consulting firm specializing in Novell. Their products are a wide gamut of Novell stuff, SLES, ZEN, IDM, and GroupWise. We're a poor .EDU, and can't afford consultants.
  • Honeywell. Honeywell is kind of like GE and IBM, they do a little of everything. I don't know what their Novell tie-in is.
  • Syncsort. They were one of the first backup products to fully support OES1. They are arguably the backup software that supports Novell stuff the best. Their products are SLES, OES, and NetWare. We looked at them when we were looking for a new backup vendor, but they didn't quite measure up for various reasons. I just might drop by.
  • Omni. Another consulting firm that specializes in Novell products, but they also have some discrete products. Their web-site says they do SLES, OES, NetWare, GroupWise, and NetMail (now a Messaging Architects product). We're a poor .EDU, and can't afford consultants.
  • HP. They do hardware. Their booth isn't as big as it was last year, so there will be less pretty to look at. Their product is SLES/OES. They're our hardware vendor, so I'll be talking real good with these folks.
  • Condrey Corporation. Another consulting company specializing in Novell products. They do IDM, Novell Storage Manager, NetWare, and probably OES/SLES. Poor .edu, can't afford 'em. yadda yadda. Also, we built our own IDM stuff so don't need no steeenkin other stuff.
And a bunch more vendors in smaller booths. Some big names (Blackberry), some not so big (idEngines).

There are exceedingly few (two, really) vendors there that can expect to see any of WWU's money any time soon. Nor is that at all likely to change. Our user head-count (21,000+) and FTE count (13,000+) combine to mean that anything that charges per-user is going to be out of our price-range pretty quickly, or will be subjected to a bidding process. We build our own solutions to problems a lot of the time because of this.

Which means that I'm a very poor sales lead.

It also means I feel a bit guilty trading my contact info for Shiny! during Vendor Night since those vendors are sooo going to strike out when they call me in April.

Labels: ,


<permalink> posted by riedesg : 9:36 AM  2 comments

Tuesday, March 11, 2008

New novell.com web site

Novell just updated their web-site.

As in, updated in the last 12 hours or so, so expect some broken links for a while.

Another thing I noticed is a very slight rendering difference between Linux and Windows.

Top left of Novell.com, from Linux
The page as rendered in SeaMoneky from Linux

Top left of Novell.com, from Opera
The page as rendered in Opera from Linux

Top left of Novell.com, from WinXP
The page as rendered in SeaMonkey from WinXP


It's a very simple lay-out thing, but it does indent the page that much. I kinda like it.

What I don't like is that the front page is very flash-heavy. I've had issues with flash on x86-64 machines, so I'm a bit burned by it. That said, I do realize that flash is about as prevalent as the ability to render .PNG files so it's a valid web technology.

Labels:


<permalink> posted by riedesg : 10:03 AM  0 comments

Thursday, February 28, 2008

Novell posted 1st Q financials

Normally I don't cover this, but there is a significant thing in there.
Income from continuing operations in the first fiscal quarter 2008 was $15 million, or $0.04 per share. This compares to a loss from continuing operations of $12 million, or $0.04 loss per share, for the first fiscal quarter 2007.
What this means is that Novell just posted a quarter in the black. If I'm remembering right, this is the first one in a couple of years. My pure guess is that this represents a couple of factors:
  • The loss of NetWare customers has slowed down significantly. The large majority of who is going to jump ship already has, and the remaining ship-jumpers represent a small part of the overall Novell picture. They will still cause the NetWare unit to lose money, but the loss is balanced in other areas now.
  • The SLES business has increased significantly, making up for the loss in NetWare customers. Novell has made many press releases about how well SLES/SLED is doing in the market, and point to the Microsoft deal as a key part of that.
  • Identity Manager, the central software that is in the Gartner Magic Quadrant, continues to do rather well.
Which is good, since it means that Novell will be around for quite some time. It just won't be "that NetWare company" any more. I just hope the OES services continue to see development. We won't know how large the OES segment is until Novell files their SEC paperwork.

Labels:


<permalink> posted by riedesg : 2:52 PM  1 comments

Monday, February 25, 2008

First OES2

This weekend I upgraded the one replica server running OES1-Linux to OES2-Linux. It already was at eDir 8.8.2 so the only real changes were to the base OS. It went rather well. The upgrade documentation provided by Novell was just fine. Really, a simple upgrade.

It being done on a Pentium III 1.2GHz machine meant it took a while. But very little in the way of complication. The one hitch was that it changed the certificate the NLDAP server loads to the default, which I didn't catch until a certain service we wrote failed. But that was a very easy fix.

Labels: , , ,


<permalink> posted by riedesg : 10:02 AM  0 comments

Thursday, February 14, 2008

OES2-SP1 soon to be in closed beta

Novell just announced that OES2 SP1 is going into closed beta.

"What is in this release of Open Enterprise Server

Novell Open Enterprise Server 2 Support Pack 1 refreshes the SUSE Linux Enterprise Server 10 distribution with SLES10 SP2, fixes defects found since the release of OES2 and also adds in the following functionality:

  • Novell engineered CIFS and AFP protocols
  • New version of iFolder (3.7)
  • Updated iPrint with an accounting API
  • 64-bit version of eDirectory
  • Enhanced migration tools and migration GUI
  • Improved performance of the XEN hypervisor
  • Domain Services for Windows
  • NetWare 6.5 Support Pack 8

Note that although Domain Services for Windows is part of OES2 SP1, a separate beta program will be run in order to collate DSfW feedback."

Novell engineered CIFS? I soooo want to know what that is. Is is a completely new CIFS stack, or is it Samba with Novell extensions whacked on? I want to know! The other important bit of information:

The beta test program is currently scheduled to begin mid March and run through October.
Which means there won't be product for my 2008 upgrade window. Fie. Well, at least we'll have ample time to prototype and test for the 2009 upgrade window.

Update 9/2008: Novell has posted on their beta site that a public beta is 'coming soon'.

Update 10/2008: The public beta for OES2 SP1 has been posted.

Labels: , , ,


<permalink> posted by riedesg : 11:21 AM  0 comments

Tuesday, February 05, 2008

Exchange vs Groupwise

A post on CoolSolutions today quoted another blog about why GroupWise makes sense over Exchange. This is some of the same stuff I've seen over the years. A faaaaavorite theme is to point to mass mailer worms taking out Exchange, leaving everyone else fat and running.

On 1/7/07 I wrote about just this sort of thing. A quote:
The days of viruses and other crud scaring people off of Exchange are long gone. Now the fight has to be taken up on, unfortunately, features and mind-share. In the absence of a scare like Melissa provided, migrations from Exchange to something else will be driven by migration events. Microsoft may be providing just that threshold in the future, as they've said that they will be integrating Exchange in with SharePoint to create the End All Be All of groupware applications. Companies that aren't comfortable with that, or haven't deployed SharePoint for whatever reason may see that as an excuse to jump the Microsoft ship for something else. Unfortunately, it'll be executives looking for an excuse rather than executives seeing much better features in, say, GroupWise.
Which, 13 months later, is still mostly true. Mass mailer worms are no longer the scourge they used to be, and are well handled by commercial AV packages. Mass mailer worms even look different these days, preferring to infest and send mail independent of the mail client directly to the internet, thus neatly bypassing the poor meltable Exchange servers. The fear of mass mailers is FUD leftovers from years ago, not a current threat or reason to get off of the dominant platform.

The other thing I mentioned 13 months ago was 'migration events'. We're coming up on one, in the form of Exchange 2007. As the other blog mentioned, the hardware requirements for Exchange 2007 are a bit higher than for 2003. Speaking as an administrator with a sizable Exchange deployment, the requirement of 64-bit OS is something of a non issue since I'd be using one anyway. For a small office with only 200 users, though, forking out for Windows Server 2003 64 would be expensive.

Another point mentioned is that GroupWise can run on anything, and Exchange (especially Exch2007) won't. Again, as a mail admin for a largish Exchange system that doesn't matter to me since I'll be using newer servers to keep up with the load anyway. Again, for small offices who upgrade their servers whenever the old one completely bakes off, this is a bigger concern.

The other migration point is the Public Folders that Microsoft dropped in Exchange 2007. Or rather, made a lot harder to manage. Their users roasted their account managers hotly enough that Exchange 2007 SP1 reintroduces Public Folder management. We make some use of Public Folders, but I can see an office that makes extensive use of them looking at Exchange 2007 as not a simple plonk-in upgrade that Exchange 2003 was from Exch 2000. GroupWise doesn't have a similar concept to Public Folders (Resources might be, but only sort of), so this doesn't help GW much, but is the sort of event that makes an organization really think about what they're moving to.

As for productivity, we haven't had problems. Our Exchange has about 4300 accounts in it right now. This is supported by three administrators and a lot of automation. That said, during summer vacation season when I'm the only one of us three here I can go whole days without touching anything Exchange. It just works. This is a claim I frequently hear from GroupWise shops, so... Microsoft can do it too eh?

Another thing on CoolSolutions lately has been a few pieces on marketing GroupWise. In short, it makes more sense for Novell to pitch GroupWise as the #2 player than it is to pitch it as fundamentally better than Exchange. This has some good points. There are some markets that GroupWise is a better fit than Exchange, and the small, infrequently upgraded office is one of them. As are organizations looking really closely at Linux. GroupWise can very well be the #1 mail product in the Linux space, so long as Novell can convince people that paying for email services in Linux is a good idea.

I close out my previous post 13 months ago with a paragraph that still stands:
So, Exchange will be with us a long time. What'll start making the throne wobble is if non-Windows desktops start showing up in great numbers in the workplace. THEN we could see some non-MS groupware application threaten Exchange the way that Mac (and Linux) are threatening the desktop.

Labels: , ,


<permalink> posted by riedesg : 2:36 PM  1 comments

Monday, February 04, 2008

Today's 18 year olds...

Over the time I've been here there has occasionally been a list posted in the break-room. This list is the, "Incoming freshman today...." list of things they know, experience, or haven't experienced. It contains things like:
  • Were born in 1990
  • ...have never known life without cable or satellite TV.
  • ...probably have never seen a rotary dial phone.
  • ...have had internet access for most of their school life.
And other such things. Ostensibly this is to help foster an understanding of where incoming freshman are coming from, but generally they just cause faculty and staff to just feel a bit old. In tech circles this sparks conversations about the first computers we used.

Which got me thinking about a few things. One of the items that is frequently put forth about Kids These Days (tm) is that they don't KNOW anything, they just know how to FIND things. There is some debate about this, but it is a common sentiment. I believe that kids these days (KTD) have figured out keyword based searching, and the search engines have gotten good enough at mind-reading that arcane search incantations aren't needed nearly as often as they were in the past.

Before Google, there was AltaVista. This was an era of the internet where boolean search incantations were needed to really narrow down to what you wanted. I didn't switch to Google for a long time because Google didn't have the NEAR search term, which I used on AltaVista as a way to narrow results to be more relevant. I didn't know at the time that Google effectively threw that term in on every search.

Those of us who lived through that era of the internet built up searching skills. I remember some searches I did back then that were pretty complex. I can't remember the exact terms used, but they looked like this:

bootes AND (antaries OR proxima) AND (fulcrum NEAR pinnacle)

I had a logic class in college, so these sorts of parenthetical statements made sense to me. Still do, I just don't end up needing to uncork the boolean logic to find what I need anymore as the search engines have gotten good enough that I don't NEED to do it. I know google allows much of the above, but I haven't had to do it so I don't know the syntax for it.

So I posit that yes, KTD don't know anything, but neither are their search skills robust.

Which brings me to Novell. I got to thinking what a NetWare administrator in 1990 had to know to do their job, and how I could fit into such a hypothetical time.

Right now if I don't know the answer to a problem I have a few methods to figure it out.
  1. Hit the online Novell Knowledge Base over at novell.com/support
  2. Hit the peer-support forums over at forums.novell.com (or nntp://forums.novell.com/ if you prefer old-school)
  3. Pay for a support incident
  4. Ask around the office
In 1990 the options were similar, but a key player was missing:
  1. Hit the peer-support forums over on CompuServe, which required a modem and a CompuServe account.
  2. See if the problem is mentioned in the book-shelf of manuals, which was a big investment to own.
  3. Pay for a support incident.
  4. Ask around the office.
When I first started this Novell Administrator gig in 1997 most of the admins I knew had CompuServe accounts, even though the support forums had officially moved to NNTP. There was still plenty of traffic on the CS servers, though those died out fairly quickly. The office I started in had a subscription to a monthly publication from Novell of their support knowledge base, which I made extensive use of. Somewhere in there Novell made the archives web-searchable and I stopped using the CD's.

As I see it, a NetWare admin of 1990 was on average more knowledgeable about their product than the NetWare admin of 2008. Such administrators avoided the cost of paying for support incidents by having the manuals in hard-copy form, and plonking down real money for CompuServe accounts. If I have a weird problem I'll hit up the Novell KB to see if there is a TID on it, then check the support forums to see if it is mentioned there, before I'll expend an incident on the thing. In time I've managed to teach myself how NetWare works in some very basic ways, simply by troubleshooting oddball problems. This is why I typically end up talking to backline support when I call in, unless the problem is a known issue in the private KB. My skills are probably on par with what was normal 'back in the day'.

I think this holds true for a lot of the tech field. Back then there was a lot of stuff you just had to KNOW. Or failing that, have spent the money to get the backup resources in place (manuals, support contracts). These days a base understanding of how things work is the key to phrasing the right search queries in the online knowledge bases, and less rote memorization (training) can be effective in solving a greater list of problems.

Prosthetic memory! Prosthetic training! The tools of geeks everywhere.

Labels: , , ,


<permalink> posted by riedesg : 10:40 AM  1 comments

Wednesday, January 30, 2008

I don't think it works that way

We've been having some abend issues on the cluster lately, something with the network services rather than file serving. It seems to be triggered more by iPrint/NDPS than by MyFiles, but both are associated with it. The abend itself is in WS2_32.NLM, so it's in the network stack. I have a call open with Novell.

I finally, finally managed to get a meaningful packet-capture after it fails, and I found some traffic that... doesn't look right. Take a look:

-> NCP Connection Destroy
<- R OK 
<- FIN, PSH, ACK 
<- RST, ACK 
-> ACK (to R OK)
<- RST

Note the three packets in the middle. The responding server is tearing down the connection twice for some reason. Compare this with a 'normal' tear-down:

-> NCP Connection Destroy
<- R OK
-> FIN, PSH, ACK
<- ACK
-> RST, ACK

The first example I gave is the last traffic on the wire before the server abends, so is of course highly suspicious. The pattern that leaps right out is that the responding server is issuing the FIN,PSH,ACK and RST,ACK pair, rather than the sending server, and doing so before the sending server can say "I got it" to the connection close packet.

Now I need to catch it in the act again to prove this theory.

Labels: ,


<permalink> posted by riedesg : 3:49 PM  0 comments

Friday, January 25, 2008

BrainShare social networking

I am going to BrainShare this year!

It has been interesting to watch the social networking thingy related to BrainShare over the years.

Two years ago, and for many years before that, the primary social group for BrainShare was novell.community.brainshare. This was an NNTP (you remember Usenet?) group hosted on the same servers that host the Novell Support Forums. BrainShare 2006 saw an increase in a certain kind of anti-Novell traffic that was already fairly common in the lead up to BrainShare 2005. The denizens of the group tend to be old time Novell hands, and as you can imagine they were pretty upset about Novell's plans for NetWare. A few very vocal people managed to raise enough of a stink that there wasn't a lot going on in the group for 2006. Unsurprisingly, novell.community.brainshare was removed from the NNTP servers around May 2006 (though the google-groups version of it is still around, see the link).

Last year Novell came up with BrainShare Connect as the social networking thingy. It had forums, blogs, and various other things to try and get attendees hooked up with each other and interacting. It got a reasonable amount of traffic, but many folks who had been regulars of the NNTP group were not there. I checked in every few days to see if anything new was up. For 2006 and 2005 I had checked the NNTP group daily, since there really was that much going on.

This year BrainShare Connect is back, but... they didn't do it right. The same outsourced firm is handling it, but even though it has Web 2.0 stamped all over it the interface is markedly worse than last year. There are no blogs. There are no polls. The interest finders are... weak and obfuscated. The forums are implemented on PhpBB, but done wrong. As an example of the wrong, take a look at this screen shot of me Replying to a thread:

Reply pop-over obscuring everything

What am I replying to? I can't tell. That window can't be moved or resized. I better hope my memory is good. I don't know if this is a new PhpBB feature, a new version came out a while ago, or some customized mod from WingateWeb. Whatever it is, it isn't a good thing. The ability to see what you're replying to greatly eases the flow of conversation.

And the logout screen is particularly interesting, too.

The logout window with weird buttons

What ever happened to "Cancel/OK"? Hasn't that been a de facto standard since, like, the Mac Classic came out 24 years ago? Proceed? I think that's the first time I've ever seen that particular word in that particular spot in an application developed by professionals.

The NNTP group had plenty going for it, but it was spoiled by a few vociferous critics. In the last few months Novell has released a brand new HTTP interface for the support forums that is worlds better than what was there before. Novell could bring this function back in-house if they really wanted to, and I'd support that decision. That said, I do understand why they need/want WingateWeb to handle that function. I just wish they did it better.

Labels: , ,


<permalink> posted by riedesg : 3:40 PM  0 comments

A needed patch.

Novell has released a patch for the "ConsoleOne sorting problem."

The sorting problem happens when you have eDir 8.8 installed. Suddenly C1 starts sorting things by creation date rather than as you've ever seen it before. This is... confusing. ConsoleOne 1.3h helped some of it for us, but not all. And now, we have a patch!

Let ConsoleOne Sort Correctly!

Labels: , , , ,


<permalink> posted by riedesg : 2:26 PM  0 comments

Wednesday, January 16, 2008

NetWare library patches

Novell recently split the libc and clib patches for NetWare. For a long time patches like "nwlib6a" included both. Now, they're split.

This just caused me a problem. It turns out that if you have libcsp6b (the LibC patch) applied and not nwlib6k (the CLib patch), there is an abend possibility. It happened yesterday. It turns out that in that case, a badly formed network broadcast can cause an abend. This caused three of my six cluster nodes to fall on their butts at the same time. That was fun. Strange (but good) thing is, I had already applied both patches to these three servers but hadn't gotten around to rebooting them yet. So, by killing themselves they actually fixed the problem.

The abend, key details:

EIP in SERVER.NLM at code start +0015FD27h

Heh heh heh. Oops.

And now a bit of history. Long time NetWare admins can ignore this part.

Q: Why are there two C libraries?

CLIB is the library NetWare started with. It began life in the dark and misty past, probably in the late 1980's. It is the deepest, darkest bowels of NetWare from the era when Novell was it when it came to office networking. Being so old, its APIs are very mature. Applications developed against CLIB generally speaking just plain work.

CLIB is also depreciated since it is highly proprietary, and doesn't play well with others. "Just plain works" in this instance means an assumption of 8.3 names, with kludging to support long file names if at all possible. CLIB applications have a tendency to have IPX dependencies for no good reason.

LIBC was created, IIRC, around the release of NetWare 5.0 when it became possible for NetWare to operate in a "pure IP" environment. LIBC was designed with the concept of POSIX semantics in mind, which CLIB was not. LIBC was created from scratch with long file name support. By now, as of NetWare 6.5 SP7, most of the NetWare kernel is written against LIBC rather than CLIB.

As an example of LIBC vs CLIB, take the 'MyWeb' service this blog is served by. When I did this the first time, it was on NetWare 6.0, using Apache 1.3. Apache 1.3 was linked against CLIB and was very stable. The service notes for the Apache Modules I needed to run to make it work made it clear that supporting long file-names on remote servers was something that only recently started working.

When the migration to NetWare 6.5 came around, it meant I had to migrate MyWeb to Apache 2.0. Apache 2.0 is linked against LIBC and used a different apache module to make things work. I had troubles. The LibC functions were not nearly as mature as their CLIB counterparts, and it showed. 3.5 years later things are now a lot more stable then back then.

Labels: , , ,


<permalink> posted by riedesg : 8:58 AM  0 comments

Monday, January 07, 2008

I/O starvation on NetWare, another update

I've spoken before about my latency problems on the MSA1500cs. Since my last update I've spoken with Novell at length. Their own back-line HP people were thinking firmware issues to, and recommended I open another case with HP support. And if HP again tries to lay the blame on NetWare, to point their techs at the NetWare backline tech. Who will then have a talk about why exactly it is that NetWare isn't the problem in this case.

This time when I opened the case I mentioned that we see performance problems on the backup-to-disk server, which is Windows. Which is true, when the problem occurs B2D speeds drop through the floor; last Friday a 525GB backup that normally completes in 6 hours took about 50 hours. Since I'm seeing problems on more than one operating system, clearly this is a problem with the storage device.

The first line tech agreed, and escalated. The 2nd line tech said (paraphrased):
I'm seeing a lot of parity RAID LUNs out there. This sort of RAID uses CPU on the MSA1000 controllers, so the results you're seeing are normal for this storage system.
Which, if true, puts the onus of putting up with a badly behaved I/O system onto NetWare again. The tech went on to recommend RAID1 for the LUNs that need high performance when doing array operations that disable the internal cache. Which, as far as I can figure, would work. We're not bottlenecking on I/O to the physical disks, the bottleneck is CPU on the MSA1000 controller that's active. Going RAID1 on the LUNs would keep speeds very fast even when doing array operations.

That may be where we have to go with this. Unfortunately, I don't think we have 16TB of disk-drives available to fully mirror the cluster. That'll be a significant expense. So, I think we have some rethinking to do regarding what we use this device for.

Labels: , , , ,


<permalink> posted by riedesg : 9:54 AM  2 comments

Wednesday, January 02, 2008

Where NetWare Fits

NetWare 6.5 still holds top honors in one server niche. Even though it is a 32-bit operating system. That niche is the "large file-server" segment. I define "large" as, "lots of data, way-lots of concurrent users". Yeah, that's highly scientific. But "way-lots" means "over 1000 concurrent" to my thinking.

We regularly run between 1200-6000 concurrent connections on our cluster nodes. This is a density that just doesn't happen all that often in the market. If you have 6000 users close enough together to all talk to the same file-server at LAN speeds using a protocol designed for file-serving (such as NCP, SMB/CIFS, or AFP), you're a big organization. 6000 is a large corporate campus, a large governmental entity of some kind, or a larger .EDU like us. Nationally, the number of 'large' file-servers like that is peanuts compared to the number of 'workgroup' (i.e. under 300 concurrent users) servers out there.

It is therefore no surprise to me that Novell is not devoting a lot of engineering to supporting the top end of this market. While it may pay well, there just isn't enough revenue coming from these customers to try and handle the hardest-to-test use-case: very high concurrency. I find it disappointing because I AM one of those customers (a larger .EDU), but I understand the business drivers supporting the decision.

For the moment, NetWare 6.5 (32bit) is the top-dog performance wise for our environment. That isn't going to stay true for much longer. It would not surprise me to find out that a Windows Enterprise Server (x86_64) with 16GB of RAM can out-perform a NetWare 6.5 (32bit) server with 4GB of RAM, simply due to the added room for a file-cache. What I don't know is how CPU-bound file-serving I/O is on a Windows Enterprise Server, that's the one area that could keep NetWare 6.5 (32bit) on top. I already know that OES2-Linux out-performs NetWare for NCP traffic, so long as you stay within CPU bounds.

For high-concurrency applications, as far as I know NetWare still wins.

Labels: , , , ,


<permalink> posted by riedesg : 9:39 AM  0 comments

Friday, December 28, 2007

NetWare and Hyperthreading, again

I talked about this one last year during some benchmarking. I just had a classic case crop up today.

It has long been consensus in the support forums that, (paraphrased) "If you have hyperthreading turned on and get an I/O thread stuck on a logical process, woe be unto you."

I have a server that I've been backing up for a fellow admin in another department. This particular server has 525GB of storage to back up, so it's going to take some time. It has been vexing figuring this one out. Until today, when I finally twigged to the fact that this server has HT turned on. I turn HT off as almost the first thing I do when setting up a server, so I don't think about it when troubleshooting.

Between 1000 and 1215 today, the backup got 882MB of data. Yeah, very crappy.

At 1215 I turned off the logical processors. This is a handy feature NetWare has, and I used it in the article I linked above.

At 1222 when I checked back the backup was up to 4.0GB.

At 1417 it is now up to 71GB backed up.

The only thing that changed was me turning off the logical processors. That's it. At that rate, this server should be backed up in around 15 hours, which is a far cry from the 30+ hours it was doing before.

Turn Hyperthreading off on your NetWare servers. Just do it.

Labels: , , ,


<permalink> posted by riedesg : 2:17 PM  0 comments

Thursday, December 27, 2007

eDirectory certificate server changes

The new eDir 8.8 has introduced some changes into my environment, and from the looks of it some of them were there before I did the upgrade. Specifically to the CA in the tree. In googling around, I found this excerpt from the CA documentation:

With Certificate Server 3.2 and later, in order to completely backup the Certificate Authority, it is necessary to back up the CRL database and the Issued Certificates database. On NetWare, these files are located in the sys:system\certserv directory.

For other platforms, both of these databases are located in the same directory as the eDirectory dib files. The defaults for these locations are as follows:

  • Windows: c:\novell\nds\dibfiles

  • Linux/AIX/Solaris: /var/opt/novell/edirectory/data/dib

These defaults can be changed at the time that eDirectory is installed.

The files to back up for the CRL database are crl.db, crl.lck, crl.01 and the crl.rfl directory. The files to back up for the Issue Certificates database are cert.db, cert.lck, cert.01, and the cert.rfl directory.

I didn't know about that directory. I also didn't know that the CA is publishing a certificate-revocation-list to sys:apache2\htdocs\crl\. Time to twiddle the backup jobs.

Labels: , , ,


<permalink> posted by riedesg : 12:39 PM  0 comments

Thursday, December 20, 2007

eDir 8.8, Priority Sync

One of the things that grabbed by attention with 8.8 is 'priority sync'. The documentation has an overview of it:
6.0 Priority Sync

Priority Sync is a new feature in Novell® eDirectory 8.8™ that is complimentary to the current synchronization process in eDirectory. Through Priority Sync, you can synchronize the modified critical data, such as passwords, immediately.

You can sync your critical data through Priority Sync when you cannot wait for normal synchronization. The Priority Sync process is faster than the normal synchronization process. Priority Sync is supported only between two or more eDirectory 8.8 or later servers hosting the same partition.

6.1 Need for Priority Sync

Normal synchronization can take some time, during which the modified data would not be available on other servers. For example, suppose that in your setup you have different applications talking to the directory. You change your password on Server1. With normal synchronization, it is some time before this change is synchronized with Server2. Therefore, a user would still be able to authenticate to the directory through an application talking to Server2, using the old password.

In large deployments, when the critical data of an object is modified, changes need to be synchronized immediately. The Priority Sync process resolves this issue.
Which sounds spiffy. Instant sync of passwords? I'm all for that. Then I remembered, 'wasn't that happening already? That's right, that's the "SYNC_IMMEDIATE" flag in schema.' And that's what's described in this older CoolSolutions article.

Looking at iMonitor I see this:
Screencap of iMonitor screen
As 90-95% of our user objects are in either the root container or the students container, those are the statistics I'm interested in. The "maximum ring delta" number is very, very rarely over 30 seconds for these two partitions. With it being intersession right now, we're seeing some higher numbers than usual right now but it is still kept in close sync. As we have 24 hour computer labs, and a simple login causes several user-object attributes to update, we have a continual flow of directory changes. In our case, using Priority Sync would buy us a few seconds at most. We're not under any sort of regulatory mandate to do things 'instantly' so that isn't an issue, and our password-change process is well known to our end users for taking "up to 5 minutes".

Still, I like the idea even if it isn't a good fit for us.

Labels: , , ,


<permalink> posted by riedesg : 9:40 AM  0 comments

Wednesday, December 19, 2007

eDir 8.8 is in

And as far as upgrades go, it was pretty much a non-event.

Whenever you do upgrades like this you always wonder if those balls you're juggling are tennis-balls or grenades. It took about a half hour per server and didn't have any significant hitches. The one problem that did surface is that the OES1-linux server's LDAP server had its certificate change from the one it was using to SSL CertificateDNS. This was not good, as that certificate doesn't have the subject-name we need and this caused some S/LDAP binds to fail due to SSL validation problems. That was an easy fix. The LDAP servers on the NetWare boxes didn't change.

This was a tennis-ball upgrade. So far.

We haven't turned on case-sensitive LDAP binds yet, but soon. Soon.

One unexpected side-effect of getting all three eDir servers upgraded to 8.8 like this, is that the Change Cache is now cleaned of those permanent residents we've had for ages. Woo!

Labels: , , ,


<permalink> posted by riedesg : 1:06 PM  0 comments

Monday, December 17, 2007

Not dead.

Wow, last post was the 30th? Jeez. I was on vacation all last week, which accounts for some of it. And it's looking like I'll be out sick for at least a pair of days with a crud I got while wandering about. Not sharing that with work, nosir.

On my list of things to do during the winter inter-session is to get eDir 8.8 deployed in the production tree. I just need to have ALL the servers in the tree (all, not just replica holders due to backlink updates) up and talking when I do the first one, and that could take some scheduling. This is the first step to OES2, which will be deployed on the eDir servers first.

As soon as I get some new hardware, since they're getting old.

Labels: , , , ,


<permalink> posted by riedesg : 6:29 PM  0 comments

Friday, November 30, 2007

OES2 SP1 timing

Novell just posted the third draft of their OES2 Best Practices guide. Which you can locate here. In that guide is this text:
Domain Services for Windows, which is scheduled to ship with OES 2 SP1 (currently scheduled for late 2008), will also offer some clear advantages.
"Late 2008" means they WILL NOT have SP1 out by August of 2008. This means that the upgrade of our 6 node cluster to OES will have to wait until 2009. Grrarrr!

Another 21 months of a 32-bit operating system on the single biggest storage consumer on campus. We'll have at least one hardware refresh before then for some of the nodes, and... boy I hope they have NetWare drivers for that. The very limited testing I did with NetWare-in-Xen was not encouraging from a performance stand-point. If it looks like I'll have to deploy that way for the next servers we get in the cluster, I'll have to do more real testing to characterize the performance hit (if any). The idea of a 64-bit memory space for file-caching makes me drool. Not getting it for 21 months is painful.

That said, if Novell releases the eDirectory enabled AFP server for OES2-Linux outside of the service-pack I could still make the 2008 window. That's our only dependency for SP1.

Update (09/08/08): Looks like 'late October' is the date for SP1's release. Should be in public beta before then.

Update (12/03/08): It's out!

Labels: , , , , ,


<permalink> posted by riedesg : 9:04 AM  1 comments

Wednesday, November 28, 2007

I/O starvation on NetWare, HP update

Last week I talked about a problem we're having with the HP MSA1500cs and our NetWare cluster. The problem is still there, of course. I've opened cases with both HP and Novell to handle this one. HP because I really thing that such command latencies are a defect, and Novell since they're having starvation issues with clusters.

This morning I got a voice-mail from HP, an update for our case. Greatly summarized:
The MSA team has determined that your device is working perfectly, and can find no defects. They've referred the case to the NetWare software team.
Or...
Working as designed. Fix your software. Talk to Novell.
Which I'm doing. Now to see if I can light a fire on the back-channels, or if we've just made HP admit that these sorts of command latencies are part of the design and need to be engineered around in software. Highly frustrating.

Especially since I don't think I've made back-line on the Novell case yet. They're involved, but I haven't been referred to a new support engineer yet.

Labels: , , , , , , ,


<permalink> posted by riedesg : 2:45 PM  0 comments

Wednesday, November 21, 2007

I/O starvation on NetWare

The MSA1500cs we've had for a while has shown a bad habit. It is visible when you connect a serial cable to the management port on the MSA1000 controller, and doing a "show perf" after starting performance tracking. The line in question is "Avg Command Latency:", which is a measure of how long it takes to execute an I/O operation. Under normal circumstances this metric stays between 5-30ms. When things go bad, I've seen it as far as 270ms.

This is a problem with our cluster nodes. Our cluster nodes can seen LUNs on both the MSA1500cs and the EVA3000. The EVA is where the cluster has been housed since it started, and the MSA has taken up two low-I/O-volume volumes to make space on the EVA.

IF the MSA is in the high Avg Command Latency state, and
IF a cluster node is doing a large Write to the MSA (such as a DVD ISO image, or B2D operation),
THEN "Concurrent Disk Requests" in Monitor go north of 1000

This is a dangerous state. If this particular cluster node is housing some higher trafficked volumes, such as FacShare:, the laggy I/O is competing with regular (fast) I/O to the EVA. If this sort of mostly-Read I/O is concurrent with the above heavy Write situation it can cause the cluster node to not write to the Cluster Partition on time and trigger a poison-pill from the Split Brain Detector. In short, the storage heart-beat to the EVA (where the Cluster Partition lives) gets starved out in the face of all the writes to the laggy MSA.

Users definitely noticed when the cluster node was in such a heavy usage state. Writes and Reads took a loooong time on the LUNs hosted on the fast EVA. Our help-desk recorded several "unable to map drive" calls when the nodes were in that state, simply because a drive-mapping involves I/O and the server was too busy to do it in the scant seconds it normally does.

This is sub-optimal. This also doesn't seem to happen on Windows, but I'm not sure of that.

This is something that a very new feature in the Linux kernel could help out, that that's to introduce the concept of 'priority I/O' to the storage stack. I/O with a high priority, such as cluster heart-beats, gets serviced faster than I/O of regular priority. That could prevent SBD abends. Unfortunately, as the NetWare kernel is no longer under development and just under Maintenance, this is not likely to be ported to NetWare.

I/O starvation. This shouldn't happen, but neither should 270ms I/O command latencies.

Labels: , , , , , , ,


<permalink> posted by riedesg : 12:40 PM  0 comments

Tuesday, October 23, 2007

Large eDirectory installs

There was a nice post about a real install of a large eDir tree in the support forums recently:

Check it out.

Also? Novell has a real HTTP interface for the forums now.

Labels: ,


<permalink> posted by riedesg : 11:21 AM  0 comments

Tuesday, October 02, 2007

NCL 2.0 beta and Xen, pt 2

I now have server side and client side captures. The server side shows what's really going on. It is clear that those jumbos I talked about earlier are being disassembled in the Xen network stack. The client traces look similar to this

-> NCP to server
<- Ack
-> NCP to server
-> NCP to server
<- Ack
<- Ack
-> NCP to server
<- Ack

With variations on the order. The NCP to server packets are all jumbos. From the server side it looks a lot different. The same sequence from the server side:

-> NCP to server
-> NCP to server
-> NCP to server
<- Ack
-> NCP to server
-> NCP to server
-> NCP to server
<- Ack
-> NCP to server
-> NCP to server
-> NCP to server
<- Ack
-> NCP to server
-> NCP to server
-> NCP to server
<- Ack

What's more, the server sees a marked delay in packets between the <- Ack and the first -> NCP to server. On the client side the delays are between the -> NCP to server and the responding <- Ack. I interpret this to show a packet-disassembly delay in the Xen stack.

What I can't figure out is how are the jumbos getting on the network stack at all? The configured network interfaces (except for loopback) in the Dom0 all have MTU values of 1500. I suspect NCL throughput for higher record sizes will improve markedly if it didn't force the Xen layer to disassemble the jumbos. Overriding the MTU on an interface is something that can only be done in kernel-space (I think) which would point to the novfs kernel module.

Labels: , ,


<permalink> posted by riedesg : 3:30 PM  0 comments

Monday, October 01, 2007

NCL 2.0 beta and Xen

I found a good candidate for why the Novell Client for Linux 2.0-beta is so crappy in a Xen kernel. Take a look at this:

Frame 6 (4434 bytes on wire, 4434 bytes captured)

What the heck? What it should look like is this:

Frame 6 (1514 bytes on wire, 1514 bytes captured)

So I go to our network guys and ask, "Have we turned on jumbo frames anywhere?" No, we haven't. Anywhere. Which I pretty much knew, since we're not doing any iSCSI. So where the heck is that jumbo coming from? The only thing I can think of is that the sniffing layer I'm getting this at is above the layer that'd grab what actually hits the wire, and something between the sniffer and the wire is converting these jumbos to the normal 1514B ethernet MTU, and that's where my lag is coming from.

This is a case where I'd like to span a port and get a sniff of what actually hits the wire so I can compare.

Labels: , ,


<permalink> posted by riedesg : 10:47 AM  1 comments

Friday, September 28, 2007

Novell client for Linux, part 2

I'm doing a test with the vanilla openSUSE kernel and the results are VASTLY BETTER. A snippet:
    KB  reclen   write rewrite    read    reread
51200     256    9178    9189     9130     8898
Compare that with the earlier numbers for that record size and you can see the problem.
    KB  reclen   write rewrite    read    reread
51200     256     595     587     8551     8491
See? It has to be something in the Xen Dom0 environment.

This would be a lot easier if Wireshark would stop freezing hard when I try to save a capture.

Labels: , ,


<permalink> posted by riedesg : 2:51 PM  0 comments

Novell Client for Linux

I'm running the beta of it, but keep in mind I'm also running it on openSUSE, which is not what it has been specifically designed for. That said, I do have some performance numbers. Yesterday I ran a very simple IOZONE test on a 50MB file. This isn't a great test since it fits entirely inside the server's cache. But I'm not worried about that, I just want to see how fast this code can peddle.

Performance was... not good. Unfortunately, right now I can't tell if that is a side effect of running it on openSUSE 10.2, inside a Dom0 Xen domain. I want to re-run it running the standard kernel and see if the performance also doth suck.
                                   random  random
   KB  reclen   write rewrite    read    reread
51200       4    3259    3218     3875     4026
51200       8    5144    4925     5404     5215
51200      16     193     196     6980     6788
51200      32     593     599     8371     8309
51200      64     609     594     8213     8437
51200     128     589     590     8399     8432
51200     256     595     587     8551     8491
51200     512     598     595     8528     8439
51200    1024     596     588     8580     8295
51200    2048     594     596     8595     8542
51200    4096     597     609     8258     8548
51200    8192     595     599     8683     8683
51200   16384     608     599     8673     8585
As you can see, performance at record sizes over 8 is not great for writes. Reads, on the other hand, are quite zippy. Still not as zippy as the Windows client, which I've seen get up to 11000 on those tests. But still, zippy. I don't know where the problem is. I did some sniffing to try and figure it out, but nothing really stuck out as a cause. I'm seeing some 160ms delays in ACKs, but they're coming from the server. I can't tell what condition of the client-side write is causing the delayed ACK. Need more testing.

Labels: , ,


<permalink> posted by riedesg : 8:41 AM  0 comments

Wednesday, September 26, 2007

OES2 release date

Just got out of the WebCast they had. First, the important stuff:

OES2 will be released on October 5th.
OES2-SP1 is targeted for mid-April, 2008.
AFP integration will be in SP1.

I sooooooooo hope they don't push SP1 past July. If that happens, my main migration of our cluster will have to be pushed to 2009. Ick. We're already running out of effective file-cache in 32-bit memory space. I need 64-bit to really give good performance. Hope hope hope.

A few other minor points:
  • Around the release of SP1, Prosoft and Condrey Consulting (Kanaka) will release an NCP client for Mac.
  • The clearing of throats next to a mic is a sign of someone who doesn't do a lot of work in front of mics.
  • OES2 is fully 64-bit optimized (on Linux)
  • They claim EVEN BETTER NSS performance on OES2. I hope to try that out, soon as I can figure out how to get SLES10/OES2-beta5 to talk to my SAN luns. It hates me.

Labels: , ,


<permalink> posted by riedesg : 12:00 PM  0 comments

Tuesday, September 25, 2007

OES2 Web-chat tomorrow

This isn't exactly widely spread, but here it is:

Open Enterprise Server 2 Live Webcast

Tomorrow, September 26th at 11AM PDT.

They'll be talking about all the spiffy thats in OES2, and some new info about code releases. I think this is the 'event' they mentioned a while back.

Labels: , ,


<permalink> posted by riedesg : 10:58 AM  0 comments

The perils of a manual process

Yesterday I found the root cause of a rather perplexing problem. We had a user, happily for me one of the other sysadmins at WWU, who couldn't get their eDir password changed. No matter how many times he ran the identity management process, his AD PW would change, but eDir would not even though the success on the event was good.

A word of note:

We do not use Novell Identity Management. We've built our own. When Novell first came out with DirXML 1.0, we already had the foundation of what we have right now. So, when I talk about IDM, I'm actually referring to our own self-built system not Novell's IDM.

To troubleshoot, I ran many tests. The longest one was to turn on dstrace logging on the root replica server, and push changes to the object. I'd push a change, watch the logs, then parse through the log for the user's object.
  • Changing it via LDAP made a sync.
  • Changing it via the IDM did not make a sync.
  • Changing it via iManager made a sync.
  • Changing it via ConsoleOne on the IDM server made a sync
This would point to some flaw in the IDM process. This is unlikely, as the password change logic has been largely unchanged for close to 7 years. The underlying libraries have also been unchanged for close to 3 years. Very unlikely to be that. What it could be, though, is some odd-ball untrapped error.

To figure out what that is, I needed to sniff packets. PKTSCAN to the rescue. On the IDM server I turned off connections to all but the server holding the Master replicas of everything. Then on the master replica server I loaded PKTSCAN. I turned on sniffing, make the change, wait 5 seconds just to be safe, turn off the sniff, save the sniff, and load the sniff in Wireshark. The two cases I tested:
  • Change the concurrent connections attribute through IDM
  • Change the concurrent connections attribute through ConsoleOne on the IDM server
This is what showed my problem. When I did it through IDM, it was attempting to change the Concurrent Connections attribute of T=WWU. Ahem. When I did it through ConsoleOne, it was attempting to change the Concurrent Connections attribute of CN=[username].OU=Users.O=WWU. AHAH!

Looking at the details of T=WWU, I saw that it had an aux class associated with it. It was posixAccount. Thus, was I illuminated.

This particular sysadmin requested to have his account 'turned on for linux'. Which is code for having the posixAccount aux-class associated and the uid, gid, cn, and shell attributes added. This is still a manual process for us since requests are few and far between, though that is changing. It would seem that when I did it, I right-clicked on the wrong object. Whoopsie poo! Easily fixed, though.

I removed the aux-class from the tree root object, and suddenly... IDM changes started applying to the right object! Hooray! I think the IDM code was keying off of commonName rather than CN for some reason, which is why the aux-class got in the way.

Labels: , , ,


<permalink> posted by riedesg : 7:34 AM  0 comments

Monday, September 24, 2007

Neat eDir trick

One thing that I learned at BrainShare years ago is that eDir 8.7 permits LDAP clients to register against events. Probably the most widely applicable devnet thing is the LDAP Classes for Java. From my understanding, this sort of technology is used in both Novell Identity Manager and NSure Audit.

So, what the heck is it? From the documentation:
The event system extension allows the client to specify the events for which it wants to receive notification. This information is sent in the extension request. If the extension request specifies valid events, the LDAP server keeps the connection open and uses the intermediate extended response to notify the client when events occur. Any data associated with an event is also sent in the response. If an error occurs when processing the extended request or during the subsequent processing of events, the server sends an extended response to the client containing error information and then terminates the processing of the request.
It's an extension to LDAP that Novell created to permit event monitoring. It monitors events in eDirectory, from object changes, to internal eDirectory statuses like obituary processing. For example, you can set up a connection and tell the LDAP server to tell you of all changes to the "member" attribute, and track all group modifications. Or track the "last login time" attribute, and create a robust login audit log.

Stuff like this is downright handy in identity management situations. If a change is made to "phoneNumber" in the Identity tree, that change can be trapped by the monitor, and propagated to the production eDir tree, Active Directory, and NIS+. What's now a batch process can be event based.

I'm not a java programmer so I'm limited in what *I* can do with it. However, I have coworkers who DO speak java, and can probably do wonderful things with it.

Labels: , ,


<permalink> posted by riedesg : 11:16 AM  0 comments

Mod_edir issues again

As I mentioned last week, I'm seeing a LOT of connections related to mod_edir. Late Friday I updated to the 1.0.13 build, which updates from 1.0.12 that comes with SP6. That doesn't seem to have fixed the problem. On the plus side, it would seem that the mod_edir developers know of this problem. On the down side, I don't see a fix.

Right now I'm suspecting libc, as that's been my problem in the past. Perhaps the connection tear-down code in mod_edir isn't "taking" somehow.

Unfortunately, I'm not sure if I can call in an incident against mod_edir, or if I'll have to work with the devs (somehow) and call in against libc. If I reboot the web-servers every couple of days that causes the connections to close, but that is not a fun solution.

Labels: ,


<permalink> posted by riedesg : 9:20 AM  1 comments

Thursday, September 20, 2007

That... is a lot of connections!

Oh, NetWare fans, take a look at this:
MONITOR Snapshot of an impressive number. Concurrent Connections = 13075Yep. Check the Concurrent Connections number. That is a very big number. During term we run between 1500 and 4000 concurrent connections. Yet... that is way above that. What's more, going into the Novell Remote Manager, I find this pair of very interesting numbers:

Connection Slots Allocated: 44000
Connection Slots Being Used: 43982

Looking at the connections shows me what the problem is. All those 'extra' connections are for the user account that allows MyWeb (what you're reading this through ultimately) to work. Somehow... and this is a guess... mod_edir seems to be creating a new connection for each request coming in, rather than reusing the old ones. Or perhaps it isn't cleaning up after itself. Probably since I put SP6 in.

This would explain why this particular server has an unreasonably high memory allocation to CONNMGR. Must Poke More.

Labels: ,


<permalink> posted by riedesg : 1:39 PM  1 comments

Tuesday, September 18, 2007

OES2: clustering

I made a cluster inside Xen! Two NetWare VM's inside a Xen container. I had to use a SAN LUN as the shared device since I couldn't make it work doing it just to a single file. Not sure what's up with that. But, it's a cluster, the volume moves between the two just fine.

Another thing about speeds, now that I have some data to play with. I copied a bunch of user directory data over to the shared LUN. It's a piddly 10GB LUN so it filled quick. That's OK, it should give me some ideas of transfer times. Doing a TSATEST backup from one cluster-node to the other (i.e. inside the Xen bridge) gave me speeds on the order of 1000MB/Min. Doing a TSATEST backup from a server in our production tree to the cluster node (i.e. over the LAN) gave me speeds of about 350MB/Min. Not so good.

For comparison, doing a TSATEST backup from the same host only drawing data from one of the USER volumes on the EVA (highly fragmented, but must faster, storage) gives a rate of 550 MB/Min.

I also discovered the VAST DIFFERENCES between our production eDirectory tree, which has been in existence since 1995 if the creation timestamp on the tree object is to be believed, and the brand new eDir 8.8 tree the OES2 cluster is living in. We have a heckova lot more attributes and classes in the prod tree than in this new one. Whoa. It made for some interesting challenges when importing users into it.

Labels: , , , ,


<permalink> posted by riedesg : 1:21 PM  1 comments

OES2-beta progress

As mentioned before, I have the OES2 beta. Right now I have two NetWare servers parked in Xen VM's on SLES10SP1. This is how it is supposed to work!

I haven't gotten very far in my testing, but a few things are showing. I managed to do a TSATEST-based throughput run of a backup of SYS. That's about a gig of data. Throughputs for just one stream to one of the servers was around 500 MB/min, which is passible and within the realm of real performance for slower hardware. The downside of that is that the CPU reported by "xm top" was around 45%, where the CPU reported in MONITOR was closer to 25%. That's way higher than I expected, but could be related to all the disk I/O ops. This I/O was to a file in the file-system, not a physical device like a LUN on the SAN (that comes later).

Now I'm trying to get Novell Cluster Services installed. I want to get a weensy 2-node cluster set up to prove that it can be done. I suspect it can, but actually seeing it will be very nice.

Labels: , , ,


<permalink> posted by riedesg : 8:53 AM  0 comments

Thursday, September 13, 2007

OES2: virtualization

I have the beta up and running. I have a pair of OES2-NW servers running in Xen on SLES10SP1. And it loads just spiffy. Haven't done any performance testing on it, kind of hard to really interpret results at this point anyway.

What I HAVE been spending time on is seeing if it is possible to get a cluster set up. Clusters, of course, rely on shared storage. And if it works the way I need it to work, I need multiple Xen machines talking to the same LUNs. It may be doable, but I'm having a hard time figuring it out. The documentation on Xen isn't what you'd call complete. Novell has some in the SLES10SP1 documentation, but the stuff in the OES2 documentation is... decidedly overview-oriented. This is the most annoying thing, as I can't just put my nose to a manual and find it.

So, looking for Xen manual. It has to be around somewhere. Google-foo failed me today.

Labels: , , ,


<permalink> posted by riedesg : 4:15 PM  0 comments

Tuesday, September 11, 2007

What's been keeping me up late

Unfortunately, I haven't had time to crack open the OES2 beta. I'd really like to, but no time for that.

Instead what's been getting my attention are dsrepair.log entries like this:

ERROR: Inconsistent: Transitive Vector on partitionID: 00008204, DN: OU=[not that one].O=wwu.T=WWU
=>> Purging: 6 invalid entries from partition: OU=[not that one].O=wwu.T=WWU
=>> on server: CN=HERA
(1)Time stamp: 6-27-2006 3:12:46 pm; rep # = 0004; event = 0001
(2)Time stamp: 7-08-2004 9:00:57 am; rep # = 0005; event = 0001
(3)Time stamp: 5-09-2055 8:31:14 pm; rep # = 13F7; event = 0000
(4)Time stamp: 10-09-2002 1:37:16 pm; rep # = 0006; event = 0000
(5)Time stamp: 2-24-2070 7:43:44 am; rep # = 0070; event = 0034
(6)Time stamp: 8-25-1999 4:03:01 pm; rep # = 0007; event = 0000

=>> Updated: Transitive Vector on partition: OU=[not that one].O=wwu.T=WWU,
=>> with: 3 entries for server: CN=HERA
(1)Time stamp: 8-16-2007 12:02:19 pm; rep # = 0001; event = 0004
(2)Time stamp: 9-10-2007 9:56:11 pm; rep # = 0002; event = 0028
(3)Time stamp: 9-10-2007 10:00:03 pm; rep # = 0003; event = 0003

Those weren't alone. In fact, all of our partitions had some invalid transitive vectors. I get to go on another stomp fest tonight. Maybe this time they'll clear.

Labels: ,


<permalink> posted by riedesg : 3:23 PM  0 comments

Monday, September 10, 2007

OES2 public beta is out

Jason Williams said so.

This looks to be Beta5. They released both the Linux and NetWare parts of it. The NW65SP7 overlay iso is 1.1GB in size. I sooooooooooooooooooooo gotta get DVD drives into my servers.

Rumor has it release is now mid-October. So who knows what's going on with the 'launch' on the 26th.

Labels: , ,


<permalink> posted by riedesg : 3:06 PM  1 comments

Friday, September 07, 2007

The mystery of the OES2 release date

Various sources have pointed at evidence that Novell will be launching OES2 on the 26th. As has been pointed out, "Launch" and "Release" are different things. And yet, and the same time rumor has it to "watch for events this Monday".

I don't know what to make of that.

It COULD be that the open beta will be out Monday. I have doubts about that, as that leaves very little time for reports to come back from the field for incorporation into OES2-release, presumably on or about the 26th.

It COULD be that it'll be released Monday, and the major PR push for launch will be two and a half weeks later. I have my doubts about that, Novell will be scooped by the likes of me as we put the new product to the test, but it could happen.

It COULD be that Monday is a red herring and Novell will announce a ship date on the 26th, and the opening of the beta. I put more stock into this possibility. The likes of me will swoop up the beta code, run it through its paces and send feedback about what we manage to break, for a presumed ship of OES in November or so.

Or it could be none of these. I guess we'll find out Monday or something.

Labels: , ,


<permalink> posted by riedesg : 12:52 PM  1 comments

Friday, August 31, 2007

SSL puzzler resolved!

The anonymous commenter on the previous post nailed it. Disabling TLS in firefox & seamonkey removed the delay. Now that they've mentioned it, I knew this! This is one of the reasons why we're discouraging students from using Vista! I just didn't know that the newer Mozilla browsers were also affected by NetWare's inability to do TLSv1.

Labels: ,


<permalink> posted by riedesg : 9:49 AM  3 comments

Here's an interesting thing

Novell is putting together a Best Practices guide for migrating to OES2 from NetWare. Obviously this is OES2-Linux, as there is not much that needs migrating when going from OES-NW to OES2-NW. They're soliciting community input for the guides, and will be offering Cool Solutions reward points for contributions.

This is interesting. I know that the Novell Support Forum Sysops tend to build up their own micro guides based on problems people report in the forums, and this is a way to better formalize that. Some of the sysops have taken to using the Cool Solutions Wiki as a place to park boiler-plate answers and forward questioners to those pages. This is an interesting concept.

More interesting as OES2 isn't out yet, even in an open-beta form. Where are we going to get our experience from, eh? This implies that shortly we'll have at least an open beta to try out. I hope so.

I can't contribute much to this document because my main migration is contingent on AFP being eDir integrated, and they've said that'll not happen until probably SP1. If I do anything it'll be the eDir servers, and those are relatively easy migrations. DFS is the only sticking point for that.

Labels: , ,


<permalink> posted by riedesg : 9:16 AM  0 comments

This page is powered by Blogger. Isn't yours?