Tuesday, July 21, 2009
Digesting Novell financials
It's a perennial question, "why would anyone use Novell any more?" Typically coming from people who only know Novell as "That NetWare company," or perhaps, "the company that we replaced with Exchange." These are the same people who are convinced Novell is a dying company who just doesn't know it yet.
Yeah, well. Wrong. Novell managed to turn the corner and wean themselves off of the NetWare cash-cow. Take the last quarterly statement, which you can read in full glory here. I'm going to excerpt some bits, but it'll get long. First off, their description of their market segments. I'll try to include relevant products where I know them.
So, yes. Novell is making money, even in this economy. Not lots, but at least they're in the black. Their biggest growth area is Linux, which is making up for deficits in other areas of the company. Especially the sinking 'Workgroup' area. Once upon a time, "Workgroup," constituted over 90% of Novell revenue.
Another thing to note in the section about Linux:
Yeah, well. Wrong. Novell managed to turn the corner and wean themselves off of the NetWare cash-cow. Take the last quarterly statement, which you can read in full glory here. I'm going to excerpt some bits, but it'll get long. First off, their description of their market segments. I'll try to include relevant products where I know them.
The reduction in 'services' revenue is, I believe, a reflection in a decreased willingness for companies to pay Novell for consulting services. Also, Novell has changed how they advertise their consulting services which seems to also have had an impact. That's the economy for you. The raw numbers:We are organized into four business unit segments, which are Open Platform Solutions, Identity and Security Management, Systems and Resource Management, and Workgroup. Below is a brief update on the revenue results for the second quarter and first six months of fiscal 2009 for each of our business unit segments:
•
Within our Open Platform Solutions business unit segment, Linux and open source products remain an important growth business. We are using our Open Platform Solutions business segment as a platform for acquiring new customers to which we can sell our other complementary cross-platform identity and management products and services. Revenue from our Linux Platform Products category within our Open Platform Solutions business unit segment increased 25% in the second quarter of fiscal 2009 compared to the prior year period. This product revenue increase was partially offset by lower services revenue of 11%, such that total revenue from our Open Platform Solutions business unit segment increased 18% in the second quarter of fiscal 2009 compared to the prior year period.
Revenue from our Linux Platform Products category within our Open Platform Solutions business unit segment increased 24% in the first six months of fiscal 2009 compared to the prior year period. This product revenue increase was partially offset by lower services revenue of 17%, such that total revenue from our Open Platform Solutions business unit segment increased 15% in the first six months of fiscal 2009 compared to the prior year period.
[sysadmin1138: Products include: SLES/SLED]
•
Our Identity and Security Management business unit segment offers products that we believe deliver a complete, integrated solution in the areas of security, compliance, and governance issues. Within this segment, revenue from our Identity, Access and Compliance Management products increased 2% in the second quarter of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 45%, such that total revenue from our Identity and Security Management business unit segment decreased 16% in the second quarter of fiscal 2009 compared to the prior year period.
Revenue from our Identity, Access and Compliance Management products decreased 3% in the first six months of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 40%, such that total revenue from our Identity and Security Management business unit segment decreased 18% in the first six months of fiscal 2009 compared to the prior year period.
[sysadmin1138: Products include: IDM, Sentinal, ZenNAC, ZenEndPointSecurity]
•
Our Systems and Resource Management business unit segment strategy is to provide a complete “desktop to data center” offering, with virtualization for both Linux and mixed-source environments. Systems and Resource Management product revenue decreased 2% in the second quarter of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 10%, such that total revenue from our Systems and Resource Management business unit segment decreased 3% in the second quarter of fiscal 2009 compared to the prior year period. In the second quarter of fiscal 2009, total business unit segment revenue was higher by 8%, compared to the prior year period, as a result of our acquisitions of Managed Object Solutions, Inc. (“Managed Objects”) which we acquired on November 13, 2008 and PlateSpin Ltd. (“PlateSpin”) which we acquired on March 26, 2008.
Systems and Resource Management product revenue increased 3% in the first six months of fiscal 2009 compared to the prior year period. The total product revenue increase was partially offset by lower services revenue of 14% in the first six months of fiscal 2009 compared to the prior year period. Total revenue from our Systems and Resource Management business unit segment increased 1% in the first six months of fiscal 2009 compared to the prior year period. In the first six months of fiscal 2009 total business unit segment revenue was higher by 12% compared to the prior year period as a result of our Managed Objects and PlateSpin acquisitions.
[sysadmin1138: Products include: The rest of the ZEN suite, PlateSpin]
•
Our Workgroup business unit segment is an important source of cash flow and provides us with the potential opportunity to sell additional products and services. Our revenue from Workgroup products decreased 14% in the second quarter of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 39%, such that total revenue from our Workgroup business unit segment decreased 17% in the second quarter of fiscal 2009 compared to the prior year period.
Our revenue from Workgroup products decreased 12% in the first six months of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 39%, such that total revenue from our Workgroup business unit segment decreased 15% in the first six months of fiscal 2009 compared to the prior year period.
[sysadmin1138: Products include: Open Enterprise Server, GroupWise, Novell Teaming+Conferencing,
| | | Three months ended | | |||||||||||||||||||
| | | April 30, 2009 | | | April 30, 2008 | | ||||||||||||||||
| (In thousands) | | Net revenue | | Gross profit | | | Operating income (loss) | | | Net revenue | | Gross profit | | | Operating income (loss) | | ||||||
| Open Platform Solutions | | $ | 44,112 | | $ | 34,756 | | | $ | 21,451 | | | $ | 37,516 | | $ | 26,702 | | | $ | 12,191 | |
| Identity and Security Management | | | 38,846 | | | 27,559 | | | | 18,306 | | | | 46,299 | | | 24,226 | | | | 12,920 | |
| Systems and Resource Management | | | 45,354 | | | 37,522 | | | | 26,562 | | | | 46,769 | | | 39,356 | | | | 30,503 | |
| Workgroup | | | 87,283 | | | 73,882 | | | | 65,137 | | | | 105,082 | | | 87,101 | | | | 77,849 | |
| Common unallocated operating costs | | | — | | | (3,406 | ) | | | (113,832 | ) | | | — | | | (2,186 | ) | | | (131,796 | ) |
| | | | ||||||||||||||||||||
| Total per statements of operations | | $ | 215,595 | | $ | 170,313 | | | $ | 17,624 | | | $ | 235,666 | | $ | 175,199 | | | $ | 1,667 | |
| | | | ||||||||||||||||||||
| | | Six months ended | | |||||||||||||||||||
| | | April 30, 2009 | | | April 30, 2008 | | ||||||||||||||||
| (In thousands) | | Net revenue | | Gross profit | | | Operating income (loss) | | | Net revenue | | Gross profit | | | Operating income (loss) | | ||||||
| Open Platform Solutions | | $ | 85,574 | | $ | 68,525 | | | $ | 40,921 | | | $ | 74,315 | | $ | 52,491 | | | $ | 24,059 | |
| Identity and Security Management | | | 76,832 | | | 52,951 | | | | 35,362 | | | | 93,329 | | | 52,081 | | | | 29,316 | |
| Systems and Resource Management | | | 90,757 | | | 74,789 | | | | 52,490 | | | | 90,108 | | | 74,847 | | | | 58,176 | |
| Workgroup | | | 177,303 | | | 149,093 | | | | 131,435 | | | | 208,840 | | | 173,440 | | | | 155,655 | |
| Common unallocated operating costs | | | — | | | (7,071 | ) | | | (228,940 | ) | | | — | | | (4,675 | ) | | | (257,058 | ) |
| | | | ||||||||||||||||||||
| Total per statements of operations | | $ | 430,466 | | $ | 338,287 | | | $ | 31,268 | | | $ | 466,592 | | $ | 348,184 | | | $ | 10,148 | |
So, yes. Novell is making money, even in this economy. Not lots, but at least they're in the black. Their biggest growth area is Linux, which is making up for deficits in other areas of the company. Especially the sinking 'Workgroup' area. Once upon a time, "Workgroup," constituted over 90% of Novell revenue.
Revenue from our Workgroup segment decreased in the first six months of fiscal 2009 compared to the prior year period primarily from lower combined OES and NetWare-related revenue of $13.7 million, lower services revenue of $10.5 million and lower Collaboration product revenue of $6.3 million. Invoicing for the combined OES and NetWare-related products decreased 25% in the first six months of fiscal 2009 compared to the prior year period. Product invoicing for the Workgroup segment decreased 21% in the first six months of fiscal 2009 compared to the prior year period.Which is to say, companies dropping OES/NetWare constituted the large majority of the losses in the Workgroup segment. Yet that loss was almost wholly made up by gains in other areas. So yes, Novell has turned the corner.
Another thing to note in the section about Linux:
The invoicing decrease in the first six months of 2009 reflects the results of the first quarter of fiscal 2009 when we did not sign any large deals, many of which have historically been fulfilled by SUSE Linux Enterprise Server (“SLES”) certificates delivered through Microsoft.Which is pretty clear evidence that Microsoft is driving a lot of Novell's Operating System sales these days. That's quite a reversal, and a sign that Microsoft is officially more comfortable with this Linux thing.
Labels: linux, microsoft, netware, novell, OES, sled, sles, sysadmin, zenworks
Tuesday, February 17, 2009
tsatest and incrementals
Today I learned how to tell TSATEST to do an incremental backup. I also learned that the /path parameter requires the DOS namespace name. Example:
tsatest /V=SHARE: /path=FACILI~1 /U=.username.for.backup /c=2
That'll do an incremental (files with the Archive bit set) backup of that specific directory, on that specific volume.
tsatest /V=SHARE: /path=FACILI~1 /U=.username.for.backup /c=2
That'll do an incremental (files with the Archive bit set) backup of that specific directory, on that specific volume.
Wednesday, February 11, 2009
High availability
64-bit OES provides some options to highly available file serving. Now that we've split the non-file services out of the main 6-node cluster, all that cluster is doing is NCP and some trivial other things. What kinds of things could we do with this should we get a pile of money to do whatever we want?
Disclaimer: Due to the budget crisis, it is very possible we will not be able to replace the cluster nodes when they turn 5 years old. It may be easier to justify eating the greatly increased support expenses. Won't know until we try and replace them. This is a pure fantasy exercise as a result.
The stats of the 6-node cluster are impressive:
A single server does have a few things to recommend it. By doing away with the virtual servers, all of the NCP volumes would be hosted on the same server. Right now each virtual-server/volume pair causes a new connection to each cluster node. Right now if I fail all the volumes to the same cluster node, that cluster node will legitimately have on the order of 15,000 concurrent connections. If I were to move all the volumes to a single server itself, the concurrent connection count would drop to only ~2500.
Doing that would also make one of the chief annoyances of the Vista Client for Novell much less annoying. Due to name cache expiration, if you don't look at Windows Explorer or that file dialog in the Vista client once every 10 minutes, it'll take a freaking-long time to open that window when you do. This is because the Vista client has to enumerate/resolve the addresses of each mapped drive. Because of our cluster, each user gets no less than 6 drive mappings to 6 different virtual servers. Since it takes Vista 30-60 seconds per NCP mapping to figure out the address (it has to try Windows resolution methods before going to Novell resolution methods, and unlike WinXP there is no way to reverse that order), this means a 3-5 minute pause before Windows Explorer opens.
By putting all of our volumes on the same server, it'd only pause 30-60 seconds. Still not great, but far better.
However, putting everything on a single server is not what you call "highly available". OES2 is a lot more stable now, but it still isn't to the legendary stability of NetWare 3. Heck, NetWare 6.5 isn't at that legendary stability either. Rebooting for patches takes everything down for minutes at a time. Not viable.
With a server this beefy it is quite doable to do a cluster-in-a-box by way of Xen. Lay a base of SLES10-Sp2 on it, run the Xen kernel, and create four VMs for NCS cluster nodes. Give each 64-bit VM 7.75GB of RAM for file-caching, and bam! Cluster-in-a-box, and highly available.
However, this is a pure fantasy solution, so chances are real good that if we had the money we would use VMWare ESX instead XEN for the VM. The advantage to that is that we don't have to keep the VM/Host kernel versions in lock-step, which reduces downtime. There would be some performance degradation, and clock skew would be a problem, but at least uptime would be good; no need to perform a CLUSTER DOWN when updating kernels.
Best case, we'd have two physical boxes so we can patch the VM host without having to take every VM down.
But I still find it quite interesting that I could theoretically buy a single server with the same horsepower as the six servers driving our cluster right now.
Disclaimer: Due to the budget crisis, it is very possible we will not be able to replace the cluster nodes when they turn 5 years old. It may be easier to justify eating the greatly increased support expenses. Won't know until we try and replace them. This is a pure fantasy exercise as a result.
The stats of the 6-node cluster are impressive:
- 12 P4 cores, with an average of 3GHz per core (36GHz).
- A total of 24GB of RAM
- About 7TB of active data
- HP ProLiant DL580 (4 CPU sockets)
- 4x Quad Core Xeon E7330 Processors (2.40GHz per core, 38.4GHz total)
- 24 GB of RAM
- The usual trimmings
- Total cost: No more than $16,547 for us
A single server does have a few things to recommend it. By doing away with the virtual servers, all of the NCP volumes would be hosted on the same server. Right now each virtual-server/volume pair causes a new connection to each cluster node. Right now if I fail all the volumes to the same cluster node, that cluster node will legitimately have on the order of 15,000 concurrent connections. If I were to move all the volumes to a single server itself, the concurrent connection count would drop to only ~2500.
Doing that would also make one of the chief annoyances of the Vista Client for Novell much less annoying. Due to name cache expiration, if you don't look at Windows Explorer or that file dialog in the Vista client once every 10 minutes, it'll take a freaking-long time to open that window when you do. This is because the Vista client has to enumerate/resolve the addresses of each mapped drive. Because of our cluster, each user gets no less than 6 drive mappings to 6 different virtual servers. Since it takes Vista 30-60 seconds per NCP mapping to figure out the address (it has to try Windows resolution methods before going to Novell resolution methods, and unlike WinXP there is no way to reverse that order), this means a 3-5 minute pause before Windows Explorer opens.
By putting all of our volumes on the same server, it'd only pause 30-60 seconds. Still not great, but far better.
However, putting everything on a single server is not what you call "highly available". OES2 is a lot more stable now, but it still isn't to the legendary stability of NetWare 3. Heck, NetWare 6.5 isn't at that legendary stability either. Rebooting for patches takes everything down for minutes at a time. Not viable.
With a server this beefy it is quite doable to do a cluster-in-a-box by way of Xen. Lay a base of SLES10-Sp2 on it, run the Xen kernel, and create four VMs for NCS cluster nodes. Give each 64-bit VM 7.75GB of RAM for file-caching, and bam! Cluster-in-a-box, and highly available.
However, this is a pure fantasy solution, so chances are real good that if we had the money we would use VMWare ESX instead XEN for the VM. The advantage to that is that we don't have to keep the VM/Host kernel versions in lock-step, which reduces downtime. There would be some performance degradation, and clock skew would be a problem, but at least uptime would be good; no need to perform a CLUSTER DOWN when updating kernels.
Best case, we'd have two physical boxes so we can patch the VM host without having to take every VM down.
But I still find it quite interesting that I could theoretically buy a single server with the same horsepower as the six servers driving our cluster right now.
Labels: hp, linux, netware, novell, NSS, OES, sysadmin, virtualization
Tuesday, December 09, 2008
The price of storage
I've had cause to do the math lately, which I'll spare you :). But as of the best numbers I have, the cost of 1GB of space on the EVA6100 is about $16.22. Probably more, since this 6100 was created out of the carcass of an EVA3000, and I don't know what percentage of parts from the old 3000 are still in the 6100 and thus can't apportion the costs right.
For the EVA4400, which we have filled with FATA drives, the cost is $3.03.
Suddenly, the case for Dynamic Storage Technology (formerly known as Shadow Volumes) in OES can be made in economic terms. Yowza.
The above numbers do not include backup rotation costs. Those costs can vary from $3/GB to $15/GB depending on what you're doing with the data in the backup rotation.
Why is the cost of the EVA6100 so much greater than the EVA4400?
For the EVA4400, which we have filled with FATA drives, the cost is $3.03.
Suddenly, the case for Dynamic Storage Technology (formerly known as Shadow Volumes) in OES can be made in economic terms. Yowza.
The above numbers do not include backup rotation costs. Those costs can vary from $3/GB to $15/GB depending on what you're doing with the data in the backup rotation.
Why is the cost of the EVA6100 so much greater than the EVA4400?
- The EVA6100 uses 10K RPM 300GB FibreChannel disks, where the the EVA4400 uses 7.2K RPM 1TB (or is it 450GB?) FATA drives. The cost-per-gig on FC is vastly higher than it is on fibre-ATA.
- Most of the drives in the EVA6100 were purchased back when 300GB FC drives cost over $2000 each.
- The EVA6100 controller and cabinets just plain cost more than the EVA4400, because it can expand farther.
Labels: hp, novell, OES, shadowvolumes, storage
Wednesday, December 03, 2008
Infiltrating the market
Over on The Open Road there is a very interesting blog post. It talks about how Microsoft and Red Hat approach the market, and touches on Novell.
The article goes on to point out that Red Hat's targeting people looking to do forklift upgrades from Unix to Linux. And then points out that Microsoft wins more of that traffic than Red Hat does, by a good margin. Largely because the Microsoft family of products is very complete.
As it points out, Novell figured this out a few years ago when they launched their collaboration with Microsoft. The fruits of which arrived today with OES2 SP1, and the Novell CIFS stack and Domain Services for Windows. This allows OES2 to do something you can't do with Samba (yet), pretend to be a full up AD Domain Controller.
And yeah, Novell's current marketing slogan is, "Making IT work as One," which is a clear embracing of the "and" concept described. If they could make DSfW work on plain SLES, it may make it an even more attractive product for people.
Microsoft offers a full ecosystem of software to would-be buyers, but its greatest success may actually result from its strategy to present customers with an "and" decision when initially purchasing Microsoft technology, rather than a difficult "or" decision.And I really see this. The argument has been made internally that what you get from a Microsoft Enterprise CAL is worlds above what we can get from a Novell academic seat license, which follows into cost-effective discussions (and not good ones). It is soooooo easy to go all Microsoft, whereas a pure Linux solution requires a lot of stitching, and translation-glue.
The article goes on to point out that Red Hat's targeting people looking to do forklift upgrades from Unix to Linux. And then points out that Microsoft wins more of that traffic than Red Hat does, by a good margin. Largely because the Microsoft family of products is very complete.
As it points out, Novell figured this out a few years ago when they launched their collaboration with Microsoft. The fruits of which arrived today with OES2 SP1, and the Novell CIFS stack and Domain Services for Windows. This allows OES2 to do something you can't do with Samba (yet), pretend to be a full up AD Domain Controller.
And yeah, Novell's current marketing slogan is, "Making IT work as One," which is a clear embracing of the "and" concept described. If they could make DSfW work on plain SLES, it may make it an even more attractive product for people.
OES2 SP1 ships!
Full announcement.
It's out!
It's out!
Tuesday, December 02, 2008
The NetWare 7 that never was
My last post generated some comments lamenting where NetWare has gone. I hear ya.
I have friends and have spoken with people at BrainShare who were closer to things than I was regarding how the next version of NetWare evolved. And to be truthful, it sounded a lot like how Microsoft moved from XP to Vista. If you'll recall, "the version of Windows after XP," was something of a moving target for many years. I recall media reports of Microsoft scrapping the whole project and starting afresh at least once.
My very first BrainShare was 2001, and that was the release party for NetWare 6. It was in 2003 when Novell bought Ximian, and bought SuSE, so it is clear when Novell probably decided to bet the house on this whole Linux thing. Yet at BS01 there was talk about NW7, or if there would be a NW6.1 version out. The rumors I remember from back then had NW7 being a progression towards a more application-friendly environment. I also remember hearing the L word around once or twice.
What we actually got was NetWare 6.5, which solidified NetWare 6 and made the core services better and more mature. What it wasn't was any more application friendly than NetWare 6 was (or even NetWare 5.1 for that matter). NetWare 6.5 released in August of 2003, the same month as the Ximian purchase. This is what tells me that Novell had decided on a path for NetWare 7, and it was green, not red. Open Enterprise Server arrived in 2005, which gives OES a solid year and a half dev-time between when SuSE was bought and when we started seeing public betas of OES. The NetWare version of OES was NetWare 6.5 SP3.
What happened to NetWare 7? It got lost on the roadmap. When NW6 came out, Novell probably had 6.5 on the roadmap as the next rev, with NW7 next down. The rumors we were hearing were very provisional, as the spot on the map held by NW7 was at least 3 years away. Sometime between BrainShare 2001 and when Novell started buying its way into the Linux world NW7 was dropped and the decision was made to port to a completely different Kernel. That decision was probably made in the summer of 2003, as the NetWare 6.5 development was entering final beta, and the task of allocating developer resources for the next full rev needed to be made.
Which brings us to today. OES2 SP1 is going to drop any day now, probably in time for Novell's quarterly earnings report. SP1 finally brings the Linux-kernel 'NetWare Services' to feature-comparable with the NetWare kernel. There are still a few things missing, like an eDirectory integrated SLP server, but now all the major points are covered. If you count it up, this has taken Novell a bit over 5 years to get to this point.
In my opinion, that's about right for an organization the size of Novell. Porting over the proprietary NetWare services to completely new kernel requires a LOT of developer attention, and Novell is a lot smaller than Microsoft. Also of note, it took Microsoft 5 years to give us Vista after XP, including the presumed nuke-and-rewrite they did. Novell got a boost in that they had already ported eDirectory to Linux, so that helped out the NCP side. But that didn't help the NSS folks, who had to figure out a way to do a NetWare-style rich metadata file-system on a kernel and driver model that expects POSIX-spartan file-systems. The problems Novell had with this were amply displayed in the performance problems reported with OES1-FCS. Samba doesn't scale to the same levels as CIFS-on-NetWare did, so that meant Novell had to create their own CIFS stack from scratch. The AFP stack on Linux is a joke, and the resurgence of Apple since 2003 meant they had to do something about that as well; by making a proprietary AFP stack. All of this represents nuke-and-rebuild-from-spec, which takes time.
Novell probably should have started the migration in 2000 instead of 2003. They already knew that Exchange 5.5 upgrades were driving a LOT of customers into Active Directory, which was triggering migrations away from NetWare. But, there are business concerns here. Novell managed to survive the fall of NetWare by diversifying their product portfolio enough that GroupWise, Zen, and Identity Management could support the company. It took until this year to return to the black, but they did it. Had they shot the NetWare cash cow two years earlier, it is entirely possible that Novell couldn't have survived the lean years.
I have friends and have spoken with people at BrainShare who were closer to things than I was regarding how the next version of NetWare evolved. And to be truthful, it sounded a lot like how Microsoft moved from XP to Vista. If you'll recall, "the version of Windows after XP," was something of a moving target for many years. I recall media reports of Microsoft scrapping the whole project and starting afresh at least once.
My very first BrainShare was 2001, and that was the release party for NetWare 6. It was in 2003 when Novell bought Ximian, and bought SuSE, so it is clear when Novell probably decided to bet the house on this whole Linux thing. Yet at BS01 there was talk about NW7, or if there would be a NW6.1 version out. The rumors I remember from back then had NW7 being a progression towards a more application-friendly environment. I also remember hearing the L word around once or twice.
What we actually got was NetWare 6.5, which solidified NetWare 6 and made the core services better and more mature. What it wasn't was any more application friendly than NetWare 6 was (or even NetWare 5.1 for that matter). NetWare 6.5 released in August of 2003, the same month as the Ximian purchase. This is what tells me that Novell had decided on a path for NetWare 7, and it was green, not red. Open Enterprise Server arrived in 2005, which gives OES a solid year and a half dev-time between when SuSE was bought and when we started seeing public betas of OES. The NetWare version of OES was NetWare 6.5 SP3.
What happened to NetWare 7? It got lost on the roadmap. When NW6 came out, Novell probably had 6.5 on the roadmap as the next rev, with NW7 next down. The rumors we were hearing were very provisional, as the spot on the map held by NW7 was at least 3 years away. Sometime between BrainShare 2001 and when Novell started buying its way into the Linux world NW7 was dropped and the decision was made to port to a completely different Kernel. That decision was probably made in the summer of 2003, as the NetWare 6.5 development was entering final beta, and the task of allocating developer resources for the next full rev needed to be made.
Which brings us to today. OES2 SP1 is going to drop any day now, probably in time for Novell's quarterly earnings report. SP1 finally brings the Linux-kernel 'NetWare Services' to feature-comparable with the NetWare kernel. There are still a few things missing, like an eDirectory integrated SLP server, but now all the major points are covered. If you count it up, this has taken Novell a bit over 5 years to get to this point.
In my opinion, that's about right for an organization the size of Novell. Porting over the proprietary NetWare services to completely new kernel requires a LOT of developer attention, and Novell is a lot smaller than Microsoft. Also of note, it took Microsoft 5 years to give us Vista after XP, including the presumed nuke-and-rewrite they did. Novell got a boost in that they had already ported eDirectory to Linux, so that helped out the NCP side. But that didn't help the NSS folks, who had to figure out a way to do a NetWare-style rich metadata file-system on a kernel and driver model that expects POSIX-spartan file-systems. The problems Novell had with this were amply displayed in the performance problems reported with OES1-FCS. Samba doesn't scale to the same levels as CIFS-on-NetWare did, so that meant Novell had to create their own CIFS stack from scratch. The AFP stack on Linux is a joke, and the resurgence of Apple since 2003 meant they had to do something about that as well; by making a proprietary AFP stack. All of this represents nuke-and-rebuild-from-spec, which takes time.
Novell probably should have started the migration in 2000 instead of 2003. They already knew that Exchange 5.5 upgrades were driving a LOT of customers into Active Directory, which was triggering migrations away from NetWare. But, there are business concerns here. Novell managed to survive the fall of NetWare by diversifying their product portfolio enough that GroupWise, Zen, and Identity Management could support the company. It took until this year to return to the black, but they did it. Had they shot the NetWare cash cow two years earlier, it is entirely possible that Novell couldn't have survived the lean years.
Labels: brainshare, linux, netware, novell, OES
Monday, November 17, 2008
Signs and portents
Last Thursday I was over on download.novell.com looking for an eDirectory patch. I was staging up a new NetWare box and needed to see what the latest edir levels were. I knew 8.8.3 came out in August, and we're not there yet, so I needed 8.8.2 ftf2. However, I noticed that one of the searchable versions was 8.8.4. There was nothing in the list, but it was an option. It's not there right now, but it was then.
Thus emboldened I checked around a few more places. NetWare 6.5 SP8 was in the list, and still is right now. As is Open Enterprise Server 2 SP1. Both have the public betas posted, though.
But 8.8.4 was there. I saw it. Must have been a test or something. All this tells me that OES2 SP1 (a.k.a. NW65SP8) is just around the corner. Since we were told back at BrainShare that Sp1 would be in the Q4 time-frame, it's about due.
Thus emboldened I checked around a few more places. NetWare 6.5 SP8 was in the list, and still is right now. As is Open Enterprise Server 2 SP1. Both have the public betas posted, though.
But 8.8.4 was there. I saw it. Must have been a test or something. All this tells me that OES2 SP1 (a.k.a. NW65SP8) is just around the corner. Since we were told back at BrainShare that Sp1 would be in the Q4 time-frame, it's about due.
Monday, November 10, 2008
NetStorage, WebDav, and Vista
I figured out how to get it working! You need KB907306. This updates the Web Folders in Vista to support how Novell does WebDav through NetStorage.
In our case you'll also need to add the CA that serves the SSL certificate that's on top of NetStorage (a.k.a. MyFiles). But, it works.
In our case you'll also need to add the CA that serves the SSL certificate that's on top of NetStorage (a.k.a. MyFiles). But, it works.
Monday, October 20, 2008
Dorm printing
On my post about finally running vista patrickbuller asked:
PCounter is core to our student printing. We'll only move our NDPS/iPrint infrastructure over to OES2-linux when Pcounter is supported on that platform, not before. We'll keep a 2 node NetWare cluster around just for printing if we have to. Since accounting support is one of the features that's supposed to be in OES2-SP1, it is my hope that PCounter will support OES2-Linux within a year after SP1's release. But I haven't heard any specifics.
So you have printers that students in the dorms can print to? Wow. Do you audit all those and charge the numbers of pages against the student?The answer to that is that we make big use of AND Technology's PCounter product. When paired with their PrintStations, it makes a very nice way to put a lid on unrestricted 'free' printing in the dorms. The PrintStations also make sure that only jobs people want to pick up get printed, which saves a serious amount of paper.
PCounter is core to our student printing. We'll only move our NDPS/iPrint infrastructure over to OES2-linux when Pcounter is supported on that platform, not before. We'll keep a 2 node NetWare cluster around just for printing if we have to. Since accounting support is one of the features that's supposed to be in OES2-SP1, it is my hope that PCounter will support OES2-Linux within a year after SP1's release. But I haven't heard any specifics.
Wednesday, October 15, 2008
OES2 SP1 (public beta) has been posted
The public beta of OES2 SP1 has been posted.
I believe the NDA has lifted, but I'm not 100% on that. Will check. But, some of the new stuff in SP1:
One thing I think it is safe to say, is that even though it says "Beta4" on it, it's really a release-candidate. Only major bugs are getting quashed right now. UI freeze was a month or more ago, and strange, annoying behaviors may get "fixed in doc" rather than getting true fixes which will have to wait for SP2. Still report them anyway, since it'll go on the list to fix in the next SP.
I believe the NDA has lifted, but I'm not 100% on that. Will check. But, some of the new stuff in SP1:
- An AFP stack that doesn't suck. Or more specifically, an AFP stack that scales beyond 100 users and is eDirectory integrated.
- A new CIFS stack written by Novell, so it can scale well past the Samba limit.
- A migration toolkit in one UI, rather than a cluster of scripts.
- A new version of iFolder
- EDirectory integrated DNS/DHCP. But no eDir integrated SLP yet, open-source politics you know.
- IIRC a beta of eDir 8.8 SP4.
- The ability to put iPrint-for-Linux on NSS volumes (handy for Clustering).
- And lots more I can't remember off the top of my head.
One thing I think it is safe to say, is that even though it says "Beta4" on it, it's really a release-candidate. Only major bugs are getting quashed right now. UI freeze was a month or more ago, and strange, annoying behaviors may get "fixed in doc" rather than getting true fixes which will have to wait for SP2. Still report them anyway, since it'll go on the list to fix in the next SP.
Tuesday, October 07, 2008
Erm, about the budget
From an email sent to all points from the U President this afternoon:
This means that it is nearly certain that we will NOT be getting any new hardware for the Novell cluster next summer. We'll have to do it on hardware we already own right now. This means I won't be able to partake of that lovely 64-bit goodness. Drat drat drat.
We're already under-funded for where we need to be, this won't help. Even with the storage arrays we just bought, in terms of total disk-space we've managed to fully commit all of it. There is no excess capacity. What's more, there is no easy way to ADD new capacity since any significant amounts will require purchasing new storage shelves.
In the intermediate term, this means that WWU will now descend into bureaucratic charge-back warfare. As service-providing departments like ours try to find ways to finance the needed growth, we'll start being hard-ass about charging for exceptional services. And they'll do it to us too. So if the College of Arts and Sciences comes to us and asks us for space to host 2TB of, say, NASA data, we'll have to bill them for it. And that cost will be a 'total cost' which will by necessity include the backup costs. In return, if we need 16 ethernet jacks added to the AC datacenter, Telecom may start billing us.
And I get a new boss Thursday. Happily, since there is overlap between outgoing and incoming they've been briefing a lot. This is to prepare the new guy for the challenges he'll face in his first few weeks flying solo. There may even be the odd phone-call for advice, we'll see.
Gonna get real interesting around here.
In the OFM spreadsheets received today, we were stunned to find that targets had been set for higher education. Western, today, is now expected, from the sorts of measures outlined in the August 4 memorandum, to "save" $1,827,000 in the current fiscal year. (This major reduction applies across all budgets, including instructional budgets.)
Add that to the earlier number, and our total budget reduction is NOT the $176,000 representing 1% of non-instructional budgets. It is $2,003,682.
Aaaaaand...
Pardon me whilst I mutter things.Further, we have been advised to expect these reductions to be permanent; that is, to also be a part of our 2009-11 budget.
This means that it is nearly certain that we will NOT be getting any new hardware for the Novell cluster next summer. We'll have to do it on hardware we already own right now. This means I won't be able to partake of that lovely 64-bit goodness. Drat drat drat.
We're already under-funded for where we need to be, this won't help. Even with the storage arrays we just bought, in terms of total disk-space we've managed to fully commit all of it. There is no excess capacity. What's more, there is no easy way to ADD new capacity since any significant amounts will require purchasing new storage shelves.
In the intermediate term, this means that WWU will now descend into bureaucratic charge-back warfare. As service-providing departments like ours try to find ways to finance the needed growth, we'll start being hard-ass about charging for exceptional services. And they'll do it to us too. So if the College of Arts and Sciences comes to us and asks us for space to host 2TB of, say, NASA data, we'll have to bill them for it. And that cost will be a 'total cost' which will by necessity include the backup costs. In return, if we need 16 ethernet jacks added to the AC datacenter, Telecom may start billing us.
And I get a new boss Thursday. Happily, since there is overlap between outgoing and incoming they've been briefing a lot. This is to prepare the new guy for the challenges he'll face in his first few weeks flying solo. There may even be the odd phone-call for advice, we'll see.
Gonna get real interesting around here.
Wednesday, September 24, 2008
Fickle fortune
I lost a RAID card in one of my Beta servers. Crap. These beasties are all old beasties since that's the only hardware that could be released for the beta. And with crap servers, comes a crap failure rate. This is the second RAID card I've lost, and I've lost one hard-drive too. It isn't common to lose more RAID cards than hard-drives. Arrg.
This puts a kink into things. This was going to be an edirectory host, so I could host my replicas on one set of servers and abuse the crap out of the non-replica application servers. I may have to dual host. Icky icky.
This puts a kink into things. This was going to be an edirectory host, so I could host my replicas on one set of servers and abuse the crap out of the non-replica application servers. I may have to dual host. Icky icky.
Wednesday, September 10, 2008
That darned budget
This is where I whine about not having enough money.
It has been a common complaint amongst my co-workers that WWU wants enterprise level service for a SOHO budget. Especially for the Win/Novell environments. Our Solaris stuff is tied in closely to our ERP product, SCT Banner, and that gets big budget every 5 years to replace. We really need the same kind of thing for the Win/Novell side of the house, such as this disk-array replacement project we're doing right now.
The new EVAs are being paid for by Student Tech Fee, and not out of a general budget request. This is not how these devices should be funded, since the scope of this array is much wider than just student-related features. Unfortunately, STF is the only way we could get them funded, and we desperately need the new arrays. Without the new arrays, student service would be significantly impacted over the next fiscal year.
The problem is that the EVA3000 contains between 40-45% directly student-related storage. The other 55-60% is Fac/Staff storage. And yet, the EVA3000 was paid for by STF funds in 2003. Huh.
The summer of 2007 saw a Banner Upgrade Project, when the servers that support SCT Banner were upgraded. This was a quarter million dollar project and it happens every 5 years. They also got a disk-array upgrade to a pair of StorageTek (SUN, remember) arrays, DR replicated between our building and the DR site in Bond Hall. I believe they're using Solaris-level replication rather than Array-level replication.
The disk-array upgrade we're doing now got through the President's office just before the boom went down on big expensive purchases. It languished in the Purchasing department due to summer-vacation related under-staffing. I hate to think how late it would have gone had it been subjected to the added paperwork we now have to go through for any purchase over $1000. Under no circumstances could we have done it before Fall quarter. Which would have been bad, since we were too short to deal with the expected growth of storage for Fall quarter.
Now that we're going deep into the land of VMWare ESX, centralized storage-arrays are line of business. Without the STF funded arrays, we'd be stuck with "Departmental" and "Entry-level" arrays such as the much maligned MSA1500, or building our own iSCSI SAN from component parts (a DL385, with 2x 4-channel SmartArray controller cards, 8x MSA70 drive enclosures, running NetWare or Linux as an iSCSI target, with bonded GigE ports for throughput). Which would blow chunks. As it is, we're still stuck using SATA drives for certain 'online' uses, such as a pair of volumes on our NetWare cluster that are low usage but big consumers of space. Such systems are not designed for the workloads we'd have to subject them to, and are very poor performers when doing things like LUN expansions.
The EVA is exactly what we need to do what we're already doing for high-availability computing, yet is always treated as an exceptional budget request when it comes time to do anything big with it. Since these things are hella expensive, the budgetary powers-that-be balk at approving them and like to defer them for a year or two. We asked for a replacement EVA in time for last year's academic year, but the general-budget request got denied. For this year we went, IIRC, both with general-fund and STF proposals. The general fund got denied, but STF approved it. This needs to change.
By October, every person between and Governor Gregoir will be new. My boss is retiring in October. My grandboss was replaced last year, my great grand boss also has been replaced in the last year, and the University President stepped down on September 1st. Perhaps the new people will have a broader perspective on things and might permit the budget priorities to be realigned to the point that our disk-arrays are classified as the critical line-of-business investments they are.
It has been a common complaint amongst my co-workers that WWU wants enterprise level service for a SOHO budget. Especially for the Win/Novell environments. Our Solaris stuff is tied in closely to our ERP product, SCT Banner, and that gets big budget every 5 years to replace. We really need the same kind of thing for the Win/Novell side of the house, such as this disk-array replacement project we're doing right now.
The new EVAs are being paid for by Student Tech Fee, and not out of a general budget request. This is not how these devices should be funded, since the scope of this array is much wider than just student-related features. Unfortunately, STF is the only way we could get them funded, and we desperately need the new arrays. Without the new arrays, student service would be significantly impacted over the next fiscal year.
The problem is that the EVA3000 contains between 40-45% directly student-related storage. The other 55-60% is Fac/Staff storage. And yet, the EVA3000 was paid for by STF funds in 2003. Huh.
The summer of 2007 saw a Banner Upgrade Project, when the servers that support SCT Banner were upgraded. This was a quarter million dollar project and it happens every 5 years. They also got a disk-array upgrade to a pair of StorageTek (SUN, remember) arrays, DR replicated between our building and the DR site in Bond Hall. I believe they're using Solaris-level replication rather than Array-level replication.
The disk-array upgrade we're doing now got through the President's office just before the boom went down on big expensive purchases. It languished in the Purchasing department due to summer-vacation related under-staffing. I hate to think how late it would have gone had it been subjected to the added paperwork we now have to go through for any purchase over $1000. Under no circumstances could we have done it before Fall quarter. Which would have been bad, since we were too short to deal with the expected growth of storage for Fall quarter.
Now that we're going deep into the land of VMWare ESX, centralized storage-arrays are line of business. Without the STF funded arrays, we'd be stuck with "Departmental" and "Entry-level" arrays such as the much maligned MSA1500, or building our own iSCSI SAN from component parts (a DL385, with 2x 4-channel SmartArray controller cards, 8x MSA70 drive enclosures, running NetWare or Linux as an iSCSI target, with bonded GigE ports for throughput). Which would blow chunks. As it is, we're still stuck using SATA drives for certain 'online' uses, such as a pair of volumes on our NetWare cluster that are low usage but big consumers of space. Such systems are not designed for the workloads we'd have to subject them to, and are very poor performers when doing things like LUN expansions.
The EVA is exactly what we need to do what we're already doing for high-availability computing, yet is always treated as an exceptional budget request when it comes time to do anything big with it. Since these things are hella expensive, the budgetary powers-that-be balk at approving them and like to defer them for a year or two. We asked for a replacement EVA in time for last year's academic year, but the general-budget request got denied. For this year we went, IIRC, both with general-fund and STF proposals. The general fund got denied, but STF approved it. This needs to change.
By October, every person between and Governor Gregoir will be new. My boss is retiring in October. My grandboss was replaced last year, my great grand boss also has been replaced in the last year, and the University President stepped down on September 1st. Perhaps the new people will have a broader perspective on things and might permit the budget priorities to be realigned to the point that our disk-arrays are classified as the critical line-of-business investments they are.
Labels: budget, clustering, exchange, hp, linux, msa, netware, novell, OES, opinion, storage, sysadmin, virtualization
Thursday, August 14, 2008
Virtualization and Fileservers
There are some workloads that fit well within VM of any kind, and others that are very tricky. Fileservers are one area that are not good candidates for VM. In some cases they qualify as highly transactional. In others, the memory required to do fileserving well makes them very expensive. When you can fit 40 web-servers on a VM host, but only 4 fileservers, it makes the calculus obvious.
This is on my mind since we're running into memory problems on our NetWare cluster. We've just plain outgrown the 32-bit memory space for file-cache. NW can use memory above the 4GB line, it does have PAE support, but memory access above there is markedly slower than it is below the line. Last I heard the conventional wisdom is that 12GB is about the point where it starts earning you performance gains again. eek!
So, I'm looking forward to 64-bit memory spaces and OES2. 6GB should do us for a few years. That said, 6GB of actually-used RAM in a virtual-host means that I could fit... two of them on a VM server with 16GB of RAM.
16GB of RAM in, say, an ESX cluster is enough to host 10 other servers. Especially with memory deduplication. In the case of my hypothetical 6GB file-servers, 5.5GB of that RAM will be consumed by file-cache that will be unique to that server and thus very little gains from memory de-dup.
In the end, how well a fileserver fits in a VM environment is based on how large of a 'working set' your users have. If the working set it large enough, it can mean that you'll get small gains for virtualization. However, I realize fileserving on the scale we do it is somewhat rare, so for departmental fileservers VM can be a good-sized win. As always, know your environment.
In light of the budgetary woes we'll be having, I don't know what we'll do. Last I heard the State is projected to have a 2.7 billion deficit for the 2009-2011 (fiscal year starts July 1) budget cycle. So it may very well be possible that the only way I'll get access to 64-bit memory spaces is in an ESX context. That may mean a 6 node cluster on 3 physical hosts. And that's assuming I can get new hardware at all. If it gets bad enough I'll have to limp along until 2010 and play partitioning games to load-balance my data-loads across all 6 nodes. By 2011 all of our older hardware falls off of cheap-maintenance and we'll have to replace it, so worst-case that's when I can do my migration to 64-bit. Arg.
This is on my mind since we're running into memory problems on our NetWare cluster. We've just plain outgrown the 32-bit memory space for file-cache. NW can use memory above the 4GB line, it does have PAE support, but memory access above there is markedly slower than it is below the line. Last I heard the conventional wisdom is that 12GB is about the point where it starts earning you performance gains again. eek!
So, I'm looking forward to 64-bit memory spaces and OES2. 6GB should do us for a few years. That said, 6GB of actually-used RAM in a virtual-host means that I could fit... two of them on a VM server with 16GB of RAM.
16GB of RAM in, say, an ESX cluster is enough to host 10 other servers. Especially with memory deduplication. In the case of my hypothetical 6GB file-servers, 5.5GB of that RAM will be consumed by file-cache that will be unique to that server and thus very little gains from memory de-dup.
In the end, how well a fileserver fits in a VM environment is based on how large of a 'working set' your users have. If the working set it large enough, it can mean that you'll get small gains for virtualization. However, I realize fileserving on the scale we do it is somewhat rare, so for departmental fileservers VM can be a good-sized win. As always, know your environment.
In light of the budgetary woes we'll be having, I don't know what we'll do. Last I heard the State is projected to have a 2.7 billion deficit for the 2009-2011 (fiscal year starts July 1) budget cycle. So it may very well be possible that the only way I'll get access to 64-bit memory spaces is in an ESX context. That may mean a 6 node cluster on 3 physical hosts. And that's assuming I can get new hardware at all. If it gets bad enough I'll have to limp along until 2010 and play partitioning games to load-balance my data-loads across all 6 nodes. By 2011 all of our older hardware falls off of cheap-maintenance and we'll have to replace it, so worst-case that's when I can do my migration to 64-bit. Arg.
Labels: linux, netware, OES, sysadmin, virtualization
Monday, August 11, 2008
Novell Client for Vista, the ecosystem
I just reported a bug in the beta that surprised me. I can't talk details about it, but it strikes me as the kind of bug that should have been at least reported shortly after the client released. Perhaps it was just so overall buggy that it got lost in the forest, but still. The Vista client has been out for some time now.
Having said the following rant several times over the past few days, I figure it's time to post it ;).
The problem we're running in to is that the number of users of the Vista Client is a small, small sub-set of the overall users of the Novell Client, which are by now a minority of overall users of Novell NCP file-servers. Novell spent years hyping 'clientless' approaches to file-serving, through the CIFS stack on NetWare. A lot of places bought in to that. Because of this, the percentage of NCP-client Vista users among the overall Novell File-Server market is a rather small one.
And small means you don't get a lot of testing done by people-who-are-not-us, and seemingly obvious bugs showing up in the beta Sp1 builds. I don't have any Vista workstations, so I've done exactly zero testing of the Vista Client; this particular bug was reported and troubleshot by someone who is not me (I just filed it). Even though we have beta builds of the Vista client as part of this beta, I'm not testing it. All things considered, I probably should.
Since we're wedded hard to the Novell Client, it's probably time for us to start devoting resources to the ecosystem in order to keep it alive.
Having said the following rant several times over the past few days, I figure it's time to post it ;).
The problem we're running in to is that the number of users of the Vista Client is a small, small sub-set of the overall users of the Novell Client, which are by now a minority of overall users of Novell NCP file-servers. Novell spent years hyping 'clientless' approaches to file-serving, through the CIFS stack on NetWare. A lot of places bought in to that. Because of this, the percentage of NCP-client Vista users among the overall Novell File-Server market is a rather small one.
And small means you don't get a lot of testing done by people-who-are-not-us, and seemingly obvious bugs showing up in the beta Sp1 builds. I don't have any Vista workstations, so I've done exactly zero testing of the Vista Client; this particular bug was reported and troubleshot by someone who is not me (I just filed it). Even though we have beta builds of the Vista client as part of this beta, I'm not testing it. All things considered, I probably should.
Since we're wedded hard to the Novell Client, it's probably time for us to start devoting resources to the ecosystem in order to keep it alive.
Friday, July 25, 2008
Handling eDirectory core-files on linux
If you've been getting core files generated by ndsd on your Linux servers, and want to call Novell Support about it, there are a few things you can do to maximize what Novell will get out of the files themselves. You may not get much, but these will help the people with the debug symbols figure out what's going on.
First and foremost, you already have the tool to package core files for delivery to Novell already on your system. TID3078409 describes the details of how to use 'novell-getcore.sh'. It is included on 8.7.3.x installations as well as 8.8.x installations.
Running it looks like this:
Once you have the packaged core, you can upload it to ftp.novell.com/incoming as part of your service-request.
If you're lucky enough to be able to cause the core file to drop on demand, or it just plain happens often enough that repetition isn't a problem, there is one more thing you can do to include better data in the core you ship to Novell. TID3113982 describes a setting you can add to the ndsd launch script (/etc/init.d/ndsd) that'll include more data. The TID describes what is being done pretty well. In essence, you're using an alternate malloc call that fails with better information than the normal one. You don't want to run with this set for very long, especially in busy environments, as it impacts performance. But if you have a repeatable core, the information it can provide is better than a 'naked' core. Setting MALLOC_CHECK_=2 is my recommendation.
Be sure to unset this once you're done troubleshooting. As I said, it can impact performance of your eDirectory server.
Packaging the Core
First and foremost, you already have the tool to package core files for delivery to Novell already on your system. TID3078409 describes the details of how to use 'novell-getcore.sh'. It is included on 8.7.3.x installations as well as 8.8.x installations.
Running it looks like this:
edirsrv1:~ # novell-getcore -b /var/opt/novell/eDirectory/data/dib/core.31448 /opt/novell/eDirectory/sbin/ndsd
Novell GetCore Utility 1.1.34 [Linux]
Copyright (C) 2007 Novell, Inc. All rights reserved.
[*] User specified binary that generated core: /opt/novell/eDirectory/sbin/ndsd
[*] Processing '/var/opt/novell/eDirectory/data/dib/core.31448' with GDB...
[*] PreProcessing GDB output...
[*] Parsing GDB output...
[*] Core file /var/opt/novell/eDirectory/data/dib/core.31448 is a valid Linux core
[*] Core generated by: /opt/novell/eDirectory/sbin/ndsd
[*] Obtaining names of shared libraries listed in core...
[*] Counting number of shared libraries listed in core...
[*] Total number of shared libraries listed in core: 72
[*] Corefile bundle: core_20080725_092227_linux_ndsd_edirsrv1
[*] Generating GDBINIT commands to open core remotely...
[*] Generating ./opencore.sh...
[*] Gathering package info...
[*] Creating core_20080725_092227_linux_ndsd_edirsrv1.tar...
[*] GZipping ./core_20080725_092227_linux_ndsd_edirsrv1.tar...
[*] Done. Corefile bundle is ./core_20080725_092227_linux_ndsd_edirsrv1.tar.gzOnce you have the packaged core, you can upload it to ftp.novell.com/incoming as part of your service-request.
Including More Data
If you're lucky enough to be able to cause the core file to drop on demand, or it just plain happens often enough that repetition isn't a problem, there is one more thing you can do to include better data in the core you ship to Novell. TID3113982 describes a setting you can add to the ndsd launch script (/etc/init.d/ndsd) that'll include more data. The TID describes what is being done pretty well. In essence, you're using an alternate malloc call that fails with better information than the normal one. You don't want to run with this set for very long, especially in busy environments, as it impacts performance. But if you have a repeatable core, the information it can provide is better than a 'naked' core. Setting MALLOC_CHECK_=2 is my recommendation.
Be sure to unset this once you're done troubleshooting. As I said, it can impact performance of your eDirectory server.
Wednesday, July 16, 2008
Patching SLES
Last night I attempted to patch one of our OES2 servers. This particular server is an elderly beast, a P3 1GHz machine. So I wasn't expecting anything like fastness out of it. Especially with rug.
But still, it was painful!
As this had taken far longer than even I was expecting, I stopped. I'll finish up tonight. As this is an OES2 server, this means SLES10-SP1. I can attest that SLES10-SP2 on identical hardware is MUCH faster. I can't wait until OES2-SP1 comes out and this dinosaur can get faster patching.
But still, it was painful!
normandy: ~#: rug lu
Waking up ZMD...
[8 minutes later]
[list of one update, libzypp]
normandy: ~#: rug update
Resolving Dependencies....
[8 minutes later]
Install this update? (y/N)
y
[12 minutes later]
Restarting ZMD...
[8 minutes later]
normandy: ~#: rug lu
[list of updates. No need to wait 8 minutes this time.]
normandy: ~#: rug update
Resolving Dependencies...
[8 minutes later]
Dependency resolution failed for bind-util and bind-libs. libdns-whatzihoozit required by bind-util is provided by bind-libs. Please fix you hoser.
[insert swearing here]
normandy: ~#: rug in bind-util bind-libs
Resolving Dependencies....
[8 minutes later]
Install these updates? (y/N)
y
[12 minutes later]
normandy: ~#: exitAs this had taken far longer than even I was expecting, I stopped. I'll finish up tonight. As this is an OES2 server, this means SLES10-SP1. I can attest that SLES10-SP2 on identical hardware is MUCH faster. I can't wait until OES2-SP1 comes out and this dinosaur can get faster patching.
Monday, June 30, 2008
Novell Client for Linux, packaged for OpenSUSE
It has been mentioned many places, and I've done some of the mentioning, that since openSUSE is the foundation for SLED, it makes sense for Novell to distribute an NCL for openSUSE. It turns out they're working on just that. And here is the Novell beta page. I'm soooo going to try this out, since I'm running openSUSE 10.3 on my work desktop and won't be moving to openSUSE 11 until I can run the client on it (oh, wait, I can).
It should also be mentioned that Ubuntu is a very frequently requested target for another NCL, but I have reason to believe that'll never happen. First of all, any Novell Client involves closed source 3rd party licensed code, which makes it hard to port to Linux in the first place (a relic of being based on code from the days when open-source was just an ethical standpoint rather than a tangible market force). Second, Novell has proven to be rather light in developer resources in certain areas, and linux integration with non-SUSE linux distros is very minimal.
It should also be mentioned that Ubuntu is a very frequently requested target for another NCL, but I have reason to believe that'll never happen. First of all, any Novell Client involves closed source 3rd party licensed code, which makes it hard to port to Linux in the first place (a relic of being based on code from the days when open-source was just an ethical standpoint rather than a tangible market force). Second, Novell has proven to be rather light in developer resources in certain areas, and linux integration with non-SUSE linux distros is very minimal.
Tuesday, June 24, 2008
Backing up NSS, note for the future
According to this documentation, the storing of NSS/NetWare metadata in xattrs is turned off by default. You turn it on for OES2 servers through the "nss /ListXattrNWMetadata" command. This allows linux level utilities (i.e. cp, tar) to be able to access and copy the NSS metadata. This also allows backup software that isn't SMS enabled for OES2 to be able to backup the NSS information.
This is handy, as HP DataProtector doesn't support NSS backup on Linux. I need to remember this.
This is handy, as HP DataProtector doesn't support NSS backup on Linux. I need to remember this.
Thursday, May 29, 2008
OES2 and SLES10-SP2
Per Novell:
Also, the OES2 install does contain some kernel packages, such as those relating to NSS.
OES2 systems need to wait until either Novell gives the all clear for SP2 deployments on OES2-fcs, or OES2-SP1 ships. OES2-SP1 is built around SLES10-Sp2.
Very true. And most especially true if you're running virtualized NetWare! The paravirtualization components in NW65SP7 are designed around the version of Xen that's in SLES10-SP1, and SP2 contains a much newer version of Xen (trying to play catch-up to VMWare means a fast dev cycle, after all). So, expect problems if you do it.Updating OES2
OES2 systems should NOT be updated to SLES10 SP2 at this time!
Also, the OES2 install does contain some kernel packages, such as those relating to NSS.
OES2 systems need to wait until either Novell gives the all clear for SP2 deployments on OES2-fcs, or OES2-SP1 ships. OES2-SP1 is built around SLES10-Sp2.
Wednesday, May 21, 2008
SLES10 SP2 shipped
According to Novell, SLES10 SP2 has shipped.
This means that the ongoing OES2 SP1 beta I'm a part of will be done on released code for the SLES side of it. So any bugs we find there may end up as patches on the SP2 channel.
One nice thing in the new code?
"rug refresh --clean"
This will do what I posted about a few days ago. It'll nuke the zmd database and rebuild it fresh! Niiiice! Unfortunately, a truly better version of rug won't come until "Code 11".
This means that the ongoing OES2 SP1 beta I'm a part of will be done on released code for the SLES side of it. So any bugs we find there may end up as patches on the SP2 channel.
One nice thing in the new code?
"rug refresh --clean"
This will do what I posted about a few days ago. It'll nuke the zmd database and rebuild it fresh! Niiiice! Unfortunately, a truly better version of rug won't come until "Code 11".
Wednesday, May 14, 2008
NetWare and Xen
Here is something I didn't really know about in virtualized NetWare:
Guidelines for using NSS in a virtual environment
Towards the bottom of this document, you get this:
So, keep in mind where your disk files are! If you're using one huge XFS partition and hosting all the disks for your VM-NW systems on that, then you'll need barriers. If you're presenting a SAN LUN directly to the VM, then you'll need to "SET XENBLK BARRIERS = 0", as they're set to 16 by default. This'll give you better performance.
Guidelines for using NSS in a virtual environment
Towards the bottom of this document, you get this:
Nice stuff there! The "xenblk barriers" can also have an impact on the performance of your virtualized NetWare server. If your I/O stream runs the server out of cache, performance can really suffer if barriers are non-zero. If it fits in cache, the server can reorder the I/O stream to the disks to the point that you don't notice the performance hit.Configuring Write Barrier Behavior for NetWare in a Guest Environment
Write barriers are needed for controlling I/O behavior when writing to SATA and ATA/IDE devices and disk images via the Xen I/O drivers from a guest NetWare server. This is not an issue when NetWare is handling the I/O directly on a physical server.
The XenBlk Barriers parameter for the SET command controls the behavior of XenBlk Disk I/O when NetWare is running in a virtual environment. The setting appears in the Disk category when you issue the SET command in the NetWare server console.
Valid settings for the XenBlk Barriers parameter are integer values from 0 (turn off write barriers) to 255, with a default value of 16. A non-zero value specifies the depth of the driver queue, and also controls how often a write barrier is inserted into the I/O stream. A value of 0 turns off XenBlk Barriers.
A value of 0 (no barriers) is the best setting to use when the virtual disks assigned to the guest server’s virtual machine are based on physical SCSI, Fibre Channel, or iSCSI disks (or partitions on those physical disk types) on the host server. In this configuration, disk I/O is handled so that data is not exposed to corruption in the event of power failure or host crash, so the XenBlk Barriers are not needed. If the write barriers are set to zero, disk I/O performance is noticeably improved.
Other disk types such as SATA and ATA/IDE can leave disk I/O exposed to corruption in the event of power failure or a host crash, and should use a non-zero setting for the XenBlk Barriers parameter. Non-zero settings should also be used for XenBlk Barriers when writing to Xen LVM-backed disk images and Xen file-backed disk images, regardless of the physical disk type used to store the disk images.
So, keep in mind where your disk files are! If you're using one huge XFS partition and hosting all the disks for your VM-NW systems on that, then you'll need barriers. If you're presenting a SAN LUN directly to the VM, then you'll need to "SET XENBLK BARRIERS = 0", as they're set to 16 by default. This'll give you better performance.
Labels: benchmarking, netware, novell, NSS, OES, storage, virtualization
