Thursday, October 22, 2009
Windows 7 releases!
Or rather, its retail availability is today. We're on a Microsoft agreement, so we've had it since late August. And boy do I know that. I've been having a trickle of calls and emails ever since the beta released about various ways Win7 isn't working in my environment and whether I have any thoughts about that. Well, I do. As a matter of fact, Technical Services and ATUS both have thoughts on that:
Don't use it yet. We're not ready. Things will break. Don't call us when it does.
But as with any brand new technology there is demand. Couple that with the loose 'corporate controls' inherent in a public Higher Ed institution and we have it coming in anyway. And I get calls when people can't get to stuff.
The main generator of calls is our replacement of the Novell Login Script. I've spoken about how we feel about our login script in the past. Back on July 9, 2004 I had a long article about that. The environment has changed, but it still largely stands. Microsoft doesn't have a built in login script the same way NetWare/OES has had since the 80's, but there are hooks we can leverage. One of my co-workers has built a cunning .VBS file that we're using for our login script, and does the kinds of things we need out of a login script:
Which brings me to XYZ and Win7.
The main incompatibility has to do with the NetWare CIFS stack. Which I describe here. The NetWare CIFS stack doesn't speak NTLMv2, only LM and NTLM. In this instance, it makes it similar to much older Samba versions. This conflicts with Vista and Windows 7, which both default their LAN Manager Authentication Level to "NTLMv2 Responses Only." Which means that out of the box both Vista and Win7 will require changes to talk to our NetWare servers at all. This is fine, so long as they're domained we've set a Group Policy to change that level down to something the NetWare servers speak.
That's not all of it, though. Windows 7 introduced some changes into the SMB/CIFS stack that make talking to NetWare a bit less of a sure thing even with the LAN Man Auth level set right. Perhaps this is SMB2 negotiations getting in the way. I don't know. But for whatever reason, the NetWare CIFS stack and Win7 don't get along as well as the Vista's SMB/CIFS stack did.
The main effect of this is that the user's home-directory will fail to mount a lot more often on Win7 than on Vista. Also, other static drive mappings will fail more often. It is reasons like these that we are not recommending removing the Novell Client and relying on our still in testing Windows Login Script.
That said, I can understand why people are relying on the crufty script rather than the just-works Novell Login Script. Due to how our environment works, The Vista/Win7 Novell Client is dog slow. Annoyingly slow. So annoyingly slow that not getting some drives when you log in is preferable to dealing with it.
This will all change once we move the main file-serving cluster to Windows 2008. At that point, the Windows script should Just Work (tm). At that point, getting rid of the Novell Client will allow a more functional environment. We are not at that point yet.
Don't use it yet. We're not ready. Things will break. Don't call us when it does.
But as with any brand new technology there is demand. Couple that with the loose 'corporate controls' inherent in a public Higher Ed institution and we have it coming in anyway. And I get calls when people can't get to stuff.
The main generator of calls is our replacement of the Novell Login Script. I've spoken about how we feel about our login script in the past. Back on July 9, 2004 I had a long article about that. The environment has changed, but it still largely stands. Microsoft doesn't have a built in login script the same way NetWare/OES has had since the 80's, but there are hooks we can leverage. One of my co-workers has built a cunning .VBS file that we're using for our login script, and does the kinds of things we need out of a login script:
- Run a series of small applications we need to run, which drive the password change notification process among other things.
- Maps drives based on group membership.
- Maps home directories.
- Allows shelling out to other scripts, which allows less privileged people to manage scripts for their own users.
Which brings me to XYZ and Win7.
The main incompatibility has to do with the NetWare CIFS stack. Which I describe here. The NetWare CIFS stack doesn't speak NTLMv2, only LM and NTLM. In this instance, it makes it similar to much older Samba versions. This conflicts with Vista and Windows 7, which both default their LAN Manager Authentication Level to "NTLMv2 Responses Only." Which means that out of the box both Vista and Win7 will require changes to talk to our NetWare servers at all. This is fine, so long as they're domained we've set a Group Policy to change that level down to something the NetWare servers speak.
That's not all of it, though. Windows 7 introduced some changes into the SMB/CIFS stack that make talking to NetWare a bit less of a sure thing even with the LAN Man Auth level set right. Perhaps this is SMB2 negotiations getting in the way. I don't know. But for whatever reason, the NetWare CIFS stack and Win7 don't get along as well as the Vista's SMB/CIFS stack did.
The main effect of this is that the user's home-directory will fail to mount a lot more often on Win7 than on Vista. Also, other static drive mappings will fail more often. It is reasons like these that we are not recommending removing the Novell Client and relying on our still in testing Windows Login Script.
That said, I can understand why people are relying on the crufty script rather than the just-works Novell Login Script. Due to how our environment works, The Vista/Win7 Novell Client is dog slow. Annoyingly slow. So annoyingly slow that not getting some drives when you log in is preferable to dealing with it.
This will all change once we move the main file-serving cluster to Windows 2008. At that point, the Windows script should Just Work (tm). At that point, getting rid of the Novell Client will allow a more functional environment. We are not at that point yet.
Tuesday, September 01, 2009
NetWare and Snow Leopard
In case you hadn't heard, the early release of Snow Leopard has tripped up Novell a bit.
http://www.novell.com/products/openenterpriseserver/snowleopard.html
What's interesting is that they'll be releasing a fix for NetWare too, not just OES. This suggests that the breakage isn't something like depreciating older authentication protocols, rather changing how such protocols are handled. That way the amount of engineering required is a lot less than trying to get Diffie Helman into NetWare.
http://www.novell.com/products/openenterpriseserver/snowleopard.html
What's interesting is that they'll be releasing a fix for NetWare too, not just OES. This suggests that the breakage isn't something like depreciating older authentication protocols, rather changing how such protocols are handled. That way the amount of engineering required is a lot less than trying to get Diffie Helman into NetWare.
Thursday, August 06, 2009
Permission differences
In part, this blog post could have been written in 1997. We haven't exactly beaten down the door migrating away from NetWare.
Anyway, there are two areas that are vexing me regarding the different permissioning models between how Novell does it, and how Microsoft does it. The first has been around since the NT days, and relates to the differences (vast differences) between NTFS and the Trustee model. The second has to do with Active Directory permissions.
First, NTFS. As most companies contemplating a move from NetWare to Microsoft undoubtedly find out, Microsoft does permissions differently. First and foremost, NTFS doesn't have the concept of the 'visibility list', which is what allows NetWare to do this:
Grant a permission w-a-y down a directory tree.
Members of that rights grant will be able to browse from volume-root to that directory. They will see each directory entry along the path, and nothing else. Even if they have no rights to the intervening directories.
NTFS doesn't do that. In order to fake it you need two things:
Example: if I grant the group "StateAuditors" the write access to this directory:
\\accountServ\SharedFiles\Accounting\StandardsOffice\DocTeam\Procedures\
If I just grant the right directly on "Procedures", the StateAuditors won't be able to get to that directory by way of that share. I could just create a new share on that spot, and it'd work. Otherwise, I'll have to grant the above mentioned rights to each of DocTeam, StandardsOffice, and Accounting.
It can be done, and it can even be scripted, but it represents a significant change in thinking required when it comes to handling permissions. As most permissions are handled by our Desktop group, this will require retraining on their part.
Second, AD permissions. AD, unlike eDirectory, does not allow the permissions short-cut of assigning a right to an OU. In eDirectory, this allowed anything in that OU access to the whatever. In AD, you can't grant the permission in the first place without a lot of trouble, and it won't work like you expect even if you do manage to assign it.
This is going to be a problem with printers. In the past, when creating new print objects for Faculty/Staff printers, I'd grant the users.wwu OU rights to use the printer. As students aren't in the access list, they can't print to it unless they're in a special printer-access group. All staff can print, but only special students can. As it should be. No biggie.
AD doesn't allow that. In order to allow "all staff but no students" to print to a printer, I'd have to come up with a group of some kind that contains all staff. That's going to be too unwieldy for words, so we have to go to the 'printer access group for everyone' model. Since I'm the one that sets up printer permissions, this is something *I* have to keep in mind.
Anyway, there are two areas that are vexing me regarding the different permissioning models between how Novell does it, and how Microsoft does it. The first has been around since the NT days, and relates to the differences (vast differences) between NTFS and the Trustee model. The second has to do with Active Directory permissions.
First, NTFS. As most companies contemplating a move from NetWare to Microsoft undoubtedly find out, Microsoft does permissions differently. First and foremost, NTFS doesn't have the concept of the 'visibility list', which is what allows NetWare to do this:
Grant a permission w-a-y down a directory tree.
Members of that rights grant will be able to browse from volume-root to that directory. They will see each directory entry along the path, and nothing else. Even if they have no rights to the intervening directories.
NTFS doesn't do that. In order to fake it you need two things:
- Access Based Enumeration turned on on the share (default in Server 2008, and add-on option in Server 2003)
- A specific rights grant on each directory between the share and the directory with the rights grant. The "Read" simple right granted to "this directory only".
Example: if I grant the group "StateAuditors" the write access to this directory:
\\accountServ\SharedFiles\Accounting\StandardsOffice\DocTeam\Procedures\
If I just grant the right directly on "Procedures", the StateAuditors won't be able to get to that directory by way of that share. I could just create a new share on that spot, and it'd work. Otherwise, I'll have to grant the above mentioned rights to each of DocTeam, StandardsOffice, and Accounting.
It can be done, and it can even be scripted, but it represents a significant change in thinking required when it comes to handling permissions. As most permissions are handled by our Desktop group, this will require retraining on their part.
Second, AD permissions. AD, unlike eDirectory, does not allow the permissions short-cut of assigning a right to an OU. In eDirectory, this allowed anything in that OU access to the whatever. In AD, you can't grant the permission in the first place without a lot of trouble, and it won't work like you expect even if you do manage to assign it.
This is going to be a problem with printers. In the past, when creating new print objects for Faculty/Staff printers, I'd grant the users.wwu OU rights to use the printer. As students aren't in the access list, they can't print to it unless they're in a special printer-access group. All staff can print, but only special students can. As it should be. No biggie.
AD doesn't allow that. In order to allow "all staff but no students" to print to a printer, I'd have to come up with a group of some kind that contains all staff. That's going to be too unwieldy for words, so we have to go to the 'printer access group for everyone' model. Since I'm the one that sets up printer permissions, this is something *I* have to keep in mind.
Tuesday, July 21, 2009
Digesting Novell financials
It's a perennial question, "why would anyone use Novell any more?" Typically coming from people who only know Novell as "That NetWare company," or perhaps, "the company that we replaced with Exchange." These are the same people who are convinced Novell is a dying company who just doesn't know it yet.
Yeah, well. Wrong. Novell managed to turn the corner and wean themselves off of the NetWare cash-cow. Take the last quarterly statement, which you can read in full glory here. I'm going to excerpt some bits, but it'll get long. First off, their description of their market segments. I'll try to include relevant products where I know them.
So, yes. Novell is making money, even in this economy. Not lots, but at least they're in the black. Their biggest growth area is Linux, which is making up for deficits in other areas of the company. Especially the sinking 'Workgroup' area. Once upon a time, "Workgroup," constituted over 90% of Novell revenue.
Another thing to note in the section about Linux:
Yeah, well. Wrong. Novell managed to turn the corner and wean themselves off of the NetWare cash-cow. Take the last quarterly statement, which you can read in full glory here. I'm going to excerpt some bits, but it'll get long. First off, their description of their market segments. I'll try to include relevant products where I know them.
The reduction in 'services' revenue is, I believe, a reflection in a decreased willingness for companies to pay Novell for consulting services. Also, Novell has changed how they advertise their consulting services which seems to also have had an impact. That's the economy for you. The raw numbers:We are organized into four business unit segments, which are Open Platform Solutions, Identity and Security Management, Systems and Resource Management, and Workgroup. Below is a brief update on the revenue results for the second quarter and first six months of fiscal 2009 for each of our business unit segments:
•
Within our Open Platform Solutions business unit segment, Linux and open source products remain an important growth business. We are using our Open Platform Solutions business segment as a platform for acquiring new customers to which we can sell our other complementary cross-platform identity and management products and services. Revenue from our Linux Platform Products category within our Open Platform Solutions business unit segment increased 25% in the second quarter of fiscal 2009 compared to the prior year period. This product revenue increase was partially offset by lower services revenue of 11%, such that total revenue from our Open Platform Solutions business unit segment increased 18% in the second quarter of fiscal 2009 compared to the prior year period.
Revenue from our Linux Platform Products category within our Open Platform Solutions business unit segment increased 24% in the first six months of fiscal 2009 compared to the prior year period. This product revenue increase was partially offset by lower services revenue of 17%, such that total revenue from our Open Platform Solutions business unit segment increased 15% in the first six months of fiscal 2009 compared to the prior year period.
[sysadmin1138: Products include: SLES/SLED]
•
Our Identity and Security Management business unit segment offers products that we believe deliver a complete, integrated solution in the areas of security, compliance, and governance issues. Within this segment, revenue from our Identity, Access and Compliance Management products increased 2% in the second quarter of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 45%, such that total revenue from our Identity and Security Management business unit segment decreased 16% in the second quarter of fiscal 2009 compared to the prior year period.
Revenue from our Identity, Access and Compliance Management products decreased 3% in the first six months of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 40%, such that total revenue from our Identity and Security Management business unit segment decreased 18% in the first six months of fiscal 2009 compared to the prior year period.
[sysadmin1138: Products include: IDM, Sentinal, ZenNAC, ZenEndPointSecurity]
•
Our Systems and Resource Management business unit segment strategy is to provide a complete “desktop to data center” offering, with virtualization for both Linux and mixed-source environments. Systems and Resource Management product revenue decreased 2% in the second quarter of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 10%, such that total revenue from our Systems and Resource Management business unit segment decreased 3% in the second quarter of fiscal 2009 compared to the prior year period. In the second quarter of fiscal 2009, total business unit segment revenue was higher by 8%, compared to the prior year period, as a result of our acquisitions of Managed Object Solutions, Inc. (“Managed Objects”) which we acquired on November 13, 2008 and PlateSpin Ltd. (“PlateSpin”) which we acquired on March 26, 2008.
Systems and Resource Management product revenue increased 3% in the first six months of fiscal 2009 compared to the prior year period. The total product revenue increase was partially offset by lower services revenue of 14% in the first six months of fiscal 2009 compared to the prior year period. Total revenue from our Systems and Resource Management business unit segment increased 1% in the first six months of fiscal 2009 compared to the prior year period. In the first six months of fiscal 2009 total business unit segment revenue was higher by 12% compared to the prior year period as a result of our Managed Objects and PlateSpin acquisitions.
[sysadmin1138: Products include: The rest of the ZEN suite, PlateSpin]
•
Our Workgroup business unit segment is an important source of cash flow and provides us with the potential opportunity to sell additional products and services. Our revenue from Workgroup products decreased 14% in the second quarter of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 39%, such that total revenue from our Workgroup business unit segment decreased 17% in the second quarter of fiscal 2009 compared to the prior year period.
Our revenue from Workgroup products decreased 12% in the first six months of fiscal 2009 compared to the prior year period. In addition, services revenue was lower by 39%, such that total revenue from our Workgroup business unit segment decreased 15% in the first six months of fiscal 2009 compared to the prior year period.
[sysadmin1138: Products include: Open Enterprise Server, GroupWise, Novell Teaming+Conferencing,
| | | Three months ended | | |||||||||||||||||||
| | | April 30, 2009 | | | April 30, 2008 | | ||||||||||||||||
| (In thousands) | | Net revenue | | Gross profit | | | Operating income (loss) | | | Net revenue | | Gross profit | | | Operating income (loss) | | ||||||
| Open Platform Solutions | | $ | 44,112 | | $ | 34,756 | | | $ | 21,451 | | | $ | 37,516 | | $ | 26,702 | | | $ | 12,191 | |
| Identity and Security Management | | | 38,846 | | | 27,559 | | | | 18,306 | | | | 46,299 | | | 24,226 | | | | 12,920 | |
| Systems and Resource Management | | | 45,354 | | | 37,522 | | | | 26,562 | | | | 46,769 | | | 39,356 | | | | 30,503 | |
| Workgroup | | | 87,283 | | | 73,882 | | | | 65,137 | | | | 105,082 | | | 87,101 | | | | 77,849 | |
| Common unallocated operating costs | | | — | | | (3,406 | ) | | | (113,832 | ) | | | — | | | (2,186 | ) | | | (131,796 | ) |
| | | | ||||||||||||||||||||
| Total per statements of operations | | $ | 215,595 | | $ | 170,313 | | | $ | 17,624 | | | $ | 235,666 | | $ | 175,199 | | | $ | 1,667 | |
| | | | ||||||||||||||||||||
| | | Six months ended | | |||||||||||||||||||
| | | April 30, 2009 | | | April 30, 2008 | | ||||||||||||||||
| (In thousands) | | Net revenue | | Gross profit | | | Operating income (loss) | | | Net revenue | | Gross profit | | | Operating income (loss) | | ||||||
| Open Platform Solutions | | $ | 85,574 | | $ | 68,525 | | | $ | 40,921 | | | $ | 74,315 | | $ | 52,491 | | | $ | 24,059 | |
| Identity and Security Management | | | 76,832 | | | 52,951 | | | | 35,362 | | | | 93,329 | | | 52,081 | | | | 29,316 | |
| Systems and Resource Management | | | 90,757 | | | 74,789 | | | | 52,490 | | | | 90,108 | | | 74,847 | | | | 58,176 | |
| Workgroup | | | 177,303 | | | 149,093 | | | | 131,435 | | | | 208,840 | | | 173,440 | | | | 155,655 | |
| Common unallocated operating costs | | | — | | | (7,071 | ) | | | (228,940 | ) | | | — | | | (4,675 | ) | | | (257,058 | ) |
| | | | ||||||||||||||||||||
| Total per statements of operations | | $ | 430,466 | | $ | 338,287 | | | $ | 31,268 | | | $ | 466,592 | | $ | 348,184 | | | $ | 10,148 | |
So, yes. Novell is making money, even in this economy. Not lots, but at least they're in the black. Their biggest growth area is Linux, which is making up for deficits in other areas of the company. Especially the sinking 'Workgroup' area. Once upon a time, "Workgroup," constituted over 90% of Novell revenue.
Revenue from our Workgroup segment decreased in the first six months of fiscal 2009 compared to the prior year period primarily from lower combined OES and NetWare-related revenue of $13.7 million, lower services revenue of $10.5 million and lower Collaboration product revenue of $6.3 million. Invoicing for the combined OES and NetWare-related products decreased 25% in the first six months of fiscal 2009 compared to the prior year period. Product invoicing for the Workgroup segment decreased 21% in the first six months of fiscal 2009 compared to the prior year period.Which is to say, companies dropping OES/NetWare constituted the large majority of the losses in the Workgroup segment. Yet that loss was almost wholly made up by gains in other areas. So yes, Novell has turned the corner.
Another thing to note in the section about Linux:
The invoicing decrease in the first six months of 2009 reflects the results of the first quarter of fiscal 2009 when we did not sign any large deals, many of which have historically been fulfilled by SUSE Linux Enterprise Server (“SLES”) certificates delivered through Microsoft.Which is pretty clear evidence that Microsoft is driving a lot of Novell's Operating System sales these days. That's quite a reversal, and a sign that Microsoft is officially more comfortable with this Linux thing.
Labels: linux, microsoft, netware, novell, OES, sled, sles, sysadmin, zenworks
Monday, May 11, 2009
Rebuilds and nwadmin
Friday afternoon the Kala server, one of our three primary eDirectory replica servers, died. In event I've never seen before, one hard drive of a mirrored pair failed in such a way that bad data got committed. This server had to be rebuilt.
Happily for me, this is a procedure I can do without having to look things up in the Novell KB. This is part of the reason the letters "CNE" follow my name. The procedure is pretty straight-forward and I've done it before.
As for the install... this server is an HP BL20P G3, which means I used the procedure I documented a while back (Novell, local copy). A few minor steps changed (the INSERT Linux I used then now correctly handles SmartArray cards), but otherwise that's what I did. Still works.
For a wonder, our SSL administrator still had the custom SSL certificate we created for this server three years ago. That saved me the step of creating a CSR and setting up all the Subject Alternate Names we needed.
And today I fired up NWADMIN for the first time in not nearly long enough to associate the SLP scope to this server, since it was one of our two DA's. I could probably have done the same thing in iManager with "Other" attributes, but... why risk not getting all the right attributes associated when I have a tool that has all the built-in rules. This is the one thing that I still have NWAdmin around for. SLP-on-NetWare management.
Happily for me, this is a procedure I can do without having to look things up in the Novell KB. This is part of the reason the letters "CNE" follow my name. The procedure is pretty straight-forward and I've done it before.
- Remove dead server's objects from the tree
- Designate a new server as the Master for any replica this server was the master of (all of them, as it happened)
- Install server fresh
As for the install... this server is an HP BL20P G3, which means I used the procedure I documented a while back (Novell, local copy). A few minor steps changed (the INSERT Linux I used then now correctly handles SmartArray cards), but otherwise that's what I did. Still works.
For a wonder, our SSL administrator still had the custom SSL certificate we created for this server three years ago. That saved me the step of creating a CSR and setting up all the Subject Alternate Names we needed.
And today I fired up NWADMIN for the first time in not nearly long enough to associate the SLP scope to this server, since it was one of our two DA's. I could probably have done the same thing in iManager with "Other" attributes, but... why risk not getting all the right attributes associated when I have a tool that has all the built-in rules. This is the one thing that I still have NWAdmin around for. SLP-on-NetWare management.
Wednesday, May 06, 2009
Windows 7 and NetWare CIFS
I may need to break out Wireshark and see what the heck is going on at the packet level.
Life on the bleeding edge, I tell you.
Update: Know what? It was a name-resolution issue. It seems that once you went to the resource with its FQDN rather than the short name, the short names started working. Kind of odd, but that's what did it. A bit of packet sniffing may illuminate why the short-name method didn't work at first (it should) which just might illuminate either a bug in Win7, or a simple feature of Windows name-resolution protocols.
The only change that needed to be made was drop the LAN Manager Authentication Level to not offer NTLMv2 responses unless negotiated for it.
Monday, May 04, 2009
Cooperative multitasking and security, a browser perspective
An article over on Ars Technica goes into some detail about a dispute between two Firefox extensions that has gotten nasty. It seems that the extension environment inside Firefox (and that other Mozilla browser, SeaMonkey) is not sandboxed to any significant degree. The developer of NoScript was able to write in a complete disabling of the AdBlock Pro extension.
In some ways this reminds me strongly of how NetWare works. NetWare uses cooperative multitasking, rather than the preemptive multitasking used is pretty much every other modern server-class OS. This is part of the reason NetWare can squeeze out the performance it does under high load. The Firefox extensions run as children of the main Firefox process, can freely interact with each other, and when they crash hard can take the whole environment down with it.
Another way it reminds me of NetWare is the seeming lack of memory protection. In NetWare, unless you specifically declare that a module is to run in a protected memory space, it runs in the Kernel's memory space. This means that one program can access the memory of another program, so long as they're in the same memory space. This stands in contrast to other operating systems which provide a protected memory space for each process. It sounds to me like Firefox has a single memory space equivalent, and all process have access to it. This allows the extension-war described above to be possible without resorting to outright hacking around security features.
Moving away from a cooperative multitasking model made Windows at lot more stable (Win3.11 to Win95) as did the introduction of true memory protection (Win98 to Win2K). Memory protection is a major security barrier, and is something that Firefox seems to lack. If a Firefox install is unlucky enough to have an evil extension added to it, all the data that passes through that browser could be copied to persons nefarious, much like the Browser Helper Objects in IE.
Does it seem odd to you that I'm talking about Operating System protection features being built into browsers? These days browsers are in large part operating systems on their own, albeit ones missing the key feature of having exclusive control of the hardware.
These problems to appear to be on the verge of changing. Both Chrome and IE8 launch each browser tab in its own process, which adds a barrier between processes spawned in those tabs. That way when a flash game starts consuming all available CPU, you can kill the tab and keep the rest of your browser session running. Unfortunately, process separation still won't stop NoScript from killing AdBlocker. For that, more work needs to be done.
In some ways this reminds me strongly of how NetWare works. NetWare uses cooperative multitasking, rather than the preemptive multitasking used is pretty much every other modern server-class OS. This is part of the reason NetWare can squeeze out the performance it does under high load. The Firefox extensions run as children of the main Firefox process, can freely interact with each other, and when they crash hard can take the whole environment down with it.
Another way it reminds me of NetWare is the seeming lack of memory protection. In NetWare, unless you specifically declare that a module is to run in a protected memory space, it runs in the Kernel's memory space. This means that one program can access the memory of another program, so long as they're in the same memory space. This stands in contrast to other operating systems which provide a protected memory space for each process. It sounds to me like Firefox has a single memory space equivalent, and all process have access to it. This allows the extension-war described above to be possible without resorting to outright hacking around security features.
Moving away from a cooperative multitasking model made Windows at lot more stable (Win3.11 to Win95) as did the introduction of true memory protection (Win98 to Win2K). Memory protection is a major security barrier, and is something that Firefox seems to lack. If a Firefox install is unlucky enough to have an evil extension added to it, all the data that passes through that browser could be copied to persons nefarious, much like the Browser Helper Objects in IE.
Does it seem odd to you that I'm talking about Operating System protection features being built into browsers? These days browsers are in large part operating systems on their own, albeit ones missing the key feature of having exclusive control of the hardware.
These problems to appear to be on the verge of changing. Both Chrome and IE8 launch each browser tab in its own process, which adds a barrier between processes spawned in those tabs. That way when a flash game starts consuming all available CPU, you can kill the tab and keep the rest of your browser session running. Unfortunately, process separation still won't stop NoScript from killing AdBlocker. For that, more work needs to be done.
Thursday, April 30, 2009
Windows 7 RC is out
And they're saying that Win7 will ship well before the Jan-2010 Vista timeframe mentioned before. Well, we kind of expected that. We've also been doing a lot with our network to make it more Win7 (and Vista) friendly, since we know we'll get a LOT of Win7 once it shows up for real.
The biggest concern is that Microsoft still hasn't fixed the issue that makes the Novell Client for Vista so darned slow. This is a major deal-breaker for us, so we've been informed from on high to Do Something so our Vista/Win7 clients can have fast file-serving and printing.
That "Something" has been to turn on the CIFS stack on our NetWare servers, with domain integrated login. The Vista and Win7 clients will have to turn their LanMan Authentication Level from the default (and secure) setting of, "Send NTLMv2 Response Only" to at most, "Send NTLM Response Only." The NetWare CIFS stack can't handle NTLMv2, nor will it ever. Those people who have been suffering through the NCV get downright bouncy when they see how fast it is.
Printing... we'll see. A LOT of the printing in fac/staff land is direct-IP which has no Novell dependencies. There are a few departments out there that have enough print volume that a print-server is a good idea, so I'm hoping there is an iPrint client for Win7 out pretty fast.
All in all, we're expecting uptake of Win7 to be a lot faster than Vista ever was. In this sense Win7 is a lot like Win98SE. All the press saying that Win7 is a lot better than Vista will help drive the push away from WinXP.
The biggest concern is that Microsoft still hasn't fixed the issue that makes the Novell Client for Vista so darned slow. This is a major deal-breaker for us, so we've been informed from on high to Do Something so our Vista/Win7 clients can have fast file-serving and printing.
That "Something" has been to turn on the CIFS stack on our NetWare servers, with domain integrated login. The Vista and Win7 clients will have to turn their LanMan Authentication Level from the default (and secure) setting of, "Send NTLMv2 Response Only" to at most, "Send NTLM Response Only." The NetWare CIFS stack can't handle NTLMv2, nor will it ever. Those people who have been suffering through the NCV get downright bouncy when they see how fast it is.
Printing... we'll see. A LOT of the printing in fac/staff land is direct-IP which has no Novell dependencies. There are a few departments out there that have enough print volume that a print-server is a good idea, so I'm hoping there is an iPrint client for Win7 out pretty fast.
All in all, we're expecting uptake of Win7 to be a lot faster than Vista ever was. In this sense Win7 is a lot like Win98SE. All the press saying that Win7 is a lot better than Vista will help drive the push away from WinXP.
Wednesday, April 22, 2009
A new version of BIND
I saw on the SANS log today that the ISC is starting work on BIND10. A list of the new stuff can be found here. A couple of those items are very interesting to me. Specifically the Modularity and Clustering items.
Modularity:
Clustering:
The big thing about using a database of some kind as the back-end for DNS is that you no longer have to create Secondary servers and muck about with Zone Transfers. For domains that change on a second by second basis, such as an AD DNS domain with dynamic updates enabled and thousands of computers during morning power-on, it is entirely possible for a BIND secondary-server to be missing many, many DNS updates. Microsoft has known about this issue, which is why they have their own directory-integrated DNS service.
This also shows just how creaky the NetWare DNS service really is. That's based on BIND8 code, which is now over 10 years old. Very creaky.
I'm looking forward to BIND10. It is a needed update that addresses DNS as it is done today, and would better enable BIND to handle large Active Directory domains.
Modularity:
...the selection of a variety of back-ends for data storage, be it the current in-memory database, a traditional SQL-based server, an embedded database engine or back-ends for specific applications such as a high performance, pre-compiled answer database.Which makes me think of eDirectory backed DNS. Novell has had this for ages with NetWare, and from what I recall it was based on BIND. But... BIND8. BIND10 would formalize this in the linux base, which would further allow Novell to publish a more 'pure' eDir-integrated BIND.
Clustering:
run on multiple but related systems simultaneously, using a pluggable, open-source architecture to enable backbone communications between individual members of the cluster. These coordination services would enable a server farm to maintain consistency and coherence.This is exactly what AD-integrated DNS and the DNS on NetWare has been doing for over 8 years now. Glad to see BIND catch up.
The big thing about using a database of some kind as the back-end for DNS is that you no longer have to create Secondary servers and muck about with Zone Transfers. For domains that change on a second by second basis, such as an AD DNS domain with dynamic updates enabled and thousands of computers during morning power-on, it is entirely possible for a BIND secondary-server to be missing many, many DNS updates. Microsoft has known about this issue, which is why they have their own directory-integrated DNS service.
This also shows just how creaky the NetWare DNS service really is. That's based on BIND8 code, which is now over 10 years old. Very creaky.
I'm looking forward to BIND10. It is a needed update that addresses DNS as it is done today, and would better enable BIND to handle large Active Directory domains.
Labels: clustering, linux, microsoft, netware, sysadmin
Wednesday, April 15, 2009
Windows 7 forces major change
I've said before that you'll have to pry the login-script out of our cold dead hands. The simple Novell login-script is the single most pervasive workstation management tool we have, since EVERYONE needs the Novell Client to talk to their file servers. Its one reason we have computer labs when others are paring down or getting rid of theirs. People can live without the Zen agents if they work at it, but they can't live without the Novell Client. Therefore, we do a lot of our workstation management through the login-script.
The Vista client has been vexing in this regard since it is so painfully slow in our clustered environment. The reason it is slow is the same reason the first WinXP clients were slow, the Microsoft and Novell name-resolution processes conmpete in bad ways. As each drive letter we map is its own virtual-server, every time you attempt to display a Save/Open box or open Windows Explorer it has to resolve-timeout-resolve each and every drive letter. This means that opening a Save/Open box on a Vista machine running the Novell client can take upwards of 5 minutes to display thanks to the timeouts. Novell knows about this issue, and has reported it to Microsoft. This is something Microsoft has to fix, and they haven't yet.
This is vexing enough that certain highly influential managers want to make sure that the same thing doesn't happen again for Windows 7. As anyone who follows any piece of the tech media knows, Windows 7 has been deemed, "Vista done right," and we expect a lot faster uptake of Win7 than WinVista. So we need to make sure our network can accommodate that on release-day. Make it so, said the highly placed manager. Yessir, we said.
So last night I turned CIFS on for all the file services on the cluster. It was that or migrate our entire file-serving function to Windows. The choice, as you can expect, was an easy one.
This morning our Mac users have been decidedly gleeful, as CIFS has long password support where AFP didn't. The one sysadmin here in techservices running Vista as his primary desktop has uninstalled the Novell Client and is also cheerful. Happily for us, the directive from said highly placed manager was accompanied by a strong suggestion to all departments that domaining PCs into the AD domain would be a Really Good Idea. This allows us to use the AD login-script, as well as group-policies, for those Windows machines that lack a Novell Client.
Ultimately, I expect the Novell Client to slowly fade away as a mandatory install. So that clientless-future I said we couldn't take part in? Microsoft managed to push us there.
The Vista client has been vexing in this regard since it is so painfully slow in our clustered environment. The reason it is slow is the same reason the first WinXP clients were slow, the Microsoft and Novell name-resolution processes conmpete in bad ways. As each drive letter we map is its own virtual-server, every time you attempt to display a Save/Open box or open Windows Explorer it has to resolve-timeout-resolve each and every drive letter. This means that opening a Save/Open box on a Vista machine running the Novell client can take upwards of 5 minutes to display thanks to the timeouts. Novell knows about this issue, and has reported it to Microsoft. This is something Microsoft has to fix, and they haven't yet.
This is vexing enough that certain highly influential managers want to make sure that the same thing doesn't happen again for Windows 7. As anyone who follows any piece of the tech media knows, Windows 7 has been deemed, "Vista done right," and we expect a lot faster uptake of Win7 than WinVista. So we need to make sure our network can accommodate that on release-day. Make it so, said the highly placed manager. Yessir, we said.
So last night I turned CIFS on for all the file services on the cluster. It was that or migrate our entire file-serving function to Windows. The choice, as you can expect, was an easy one.
This morning our Mac users have been decidedly gleeful, as CIFS has long password support where AFP didn't. The one sysadmin here in techservices running Vista as his primary desktop has uninstalled the Novell Client and is also cheerful. Happily for us, the directive from said highly placed manager was accompanied by a strong suggestion to all departments that domaining PCs into the AD domain would be a Really Good Idea. This allows us to use the AD login-script, as well as group-policies, for those Windows machines that lack a Novell Client.
Ultimately, I expect the Novell Client to slowly fade away as a mandatory install. So that clientless-future I said we couldn't take part in? Microsoft managed to push us there.
Labels: clustering, microsoft, netware, novell, sysadmin
Tuesday, February 17, 2009
tsatest and incrementals
Today I learned how to tell TSATEST to do an incremental backup. I also learned that the /path parameter requires the DOS namespace name. Example:
tsatest /V=SHARE: /path=FACILI~1 /U=.username.for.backup /c=2
That'll do an incremental (files with the Archive bit set) backup of that specific directory, on that specific volume.
tsatest /V=SHARE: /path=FACILI~1 /U=.username.for.backup /c=2
That'll do an incremental (files with the Archive bit set) backup of that specific directory, on that specific volume.
Wednesday, February 11, 2009
Performance tuning Data Protector for NetWare
HP Data Protector has a client for NetWare (and OES2, but I'm not backing up any of those yet). This is proving to take a bit of TSA tuning to work out right. I haven't figured out where the problem exactly is, but I've worked around it.
The following settings are what I've got running right now, and seems to work. I may tweak later:
tsafs /readthreadsperjob=1
tsafs /readaheadthrottle=1
This seems to get around a contention issue I'm seeing with more aggressive settings, where the TSAFS memory will go the max allowed by the /cachememorythreshold setting and sit there, not passing data to the DP client. This makes backups go really long. The above setting somehow prevent this from happening.
If these prove stable, I may up the readaheadthrottle setting to see if it halts on that. This is an EVA6100 after all, so I should be able to go up to at least 18 if not 32 for that setting.
The following settings are what I've got running right now, and seems to work. I may tweak later:
tsafs /readthreadsperjob=1
tsafs /readaheadthrottle=1
This seems to get around a contention issue I'm seeing with more aggressive settings, where the TSAFS memory will go the max allowed by the /cachememorythreshold setting and sit there, not passing data to the DP client. This makes backups go really long. The above setting somehow prevent this from happening.
If these prove stable, I may up the readaheadthrottle setting to see if it halts on that. This is an EVA6100 after all, so I should be able to go up to at least 18 if not 32 for that setting.
High availability
64-bit OES provides some options to highly available file serving. Now that we've split the non-file services out of the main 6-node cluster, all that cluster is doing is NCP and some trivial other things. What kinds of things could we do with this should we get a pile of money to do whatever we want?
Disclaimer: Due to the budget crisis, it is very possible we will not be able to replace the cluster nodes when they turn 5 years old. It may be easier to justify eating the greatly increased support expenses. Won't know until we try and replace them. This is a pure fantasy exercise as a result.
The stats of the 6-node cluster are impressive:
A single server does have a few things to recommend it. By doing away with the virtual servers, all of the NCP volumes would be hosted on the same server. Right now each virtual-server/volume pair causes a new connection to each cluster node. Right now if I fail all the volumes to the same cluster node, that cluster node will legitimately have on the order of 15,000 concurrent connections. If I were to move all the volumes to a single server itself, the concurrent connection count would drop to only ~2500.
Doing that would also make one of the chief annoyances of the Vista Client for Novell much less annoying. Due to name cache expiration, if you don't look at Windows Explorer or that file dialog in the Vista client once every 10 minutes, it'll take a freaking-long time to open that window when you do. This is because the Vista client has to enumerate/resolve the addresses of each mapped drive. Because of our cluster, each user gets no less than 6 drive mappings to 6 different virtual servers. Since it takes Vista 30-60 seconds per NCP mapping to figure out the address (it has to try Windows resolution methods before going to Novell resolution methods, and unlike WinXP there is no way to reverse that order), this means a 3-5 minute pause before Windows Explorer opens.
By putting all of our volumes on the same server, it'd only pause 30-60 seconds. Still not great, but far better.
However, putting everything on a single server is not what you call "highly available". OES2 is a lot more stable now, but it still isn't to the legendary stability of NetWare 3. Heck, NetWare 6.5 isn't at that legendary stability either. Rebooting for patches takes everything down for minutes at a time. Not viable.
With a server this beefy it is quite doable to do a cluster-in-a-box by way of Xen. Lay a base of SLES10-Sp2 on it, run the Xen kernel, and create four VMs for NCS cluster nodes. Give each 64-bit VM 7.75GB of RAM for file-caching, and bam! Cluster-in-a-box, and highly available.
However, this is a pure fantasy solution, so chances are real good that if we had the money we would use VMWare ESX instead XEN for the VM. The advantage to that is that we don't have to keep the VM/Host kernel versions in lock-step, which reduces downtime. There would be some performance degradation, and clock skew would be a problem, but at least uptime would be good; no need to perform a CLUSTER DOWN when updating kernels.
Best case, we'd have two physical boxes so we can patch the VM host without having to take every VM down.
But I still find it quite interesting that I could theoretically buy a single server with the same horsepower as the six servers driving our cluster right now.
Disclaimer: Due to the budget crisis, it is very possible we will not be able to replace the cluster nodes when they turn 5 years old. It may be easier to justify eating the greatly increased support expenses. Won't know until we try and replace them. This is a pure fantasy exercise as a result.
The stats of the 6-node cluster are impressive:
- 12 P4 cores, with an average of 3GHz per core (36GHz).
- A total of 24GB of RAM
- About 7TB of active data
- HP ProLiant DL580 (4 CPU sockets)
- 4x Quad Core Xeon E7330 Processors (2.40GHz per core, 38.4GHz total)
- 24 GB of RAM
- The usual trimmings
- Total cost: No more than $16,547 for us
A single server does have a few things to recommend it. By doing away with the virtual servers, all of the NCP volumes would be hosted on the same server. Right now each virtual-server/volume pair causes a new connection to each cluster node. Right now if I fail all the volumes to the same cluster node, that cluster node will legitimately have on the order of 15,000 concurrent connections. If I were to move all the volumes to a single server itself, the concurrent connection count would drop to only ~2500.
Doing that would also make one of the chief annoyances of the Vista Client for Novell much less annoying. Due to name cache expiration, if you don't look at Windows Explorer or that file dialog in the Vista client once every 10 minutes, it'll take a freaking-long time to open that window when you do. This is because the Vista client has to enumerate/resolve the addresses of each mapped drive. Because of our cluster, each user gets no less than 6 drive mappings to 6 different virtual servers. Since it takes Vista 30-60 seconds per NCP mapping to figure out the address (it has to try Windows resolution methods before going to Novell resolution methods, and unlike WinXP there is no way to reverse that order), this means a 3-5 minute pause before Windows Explorer opens.
By putting all of our volumes on the same server, it'd only pause 30-60 seconds. Still not great, but far better.
However, putting everything on a single server is not what you call "highly available". OES2 is a lot more stable now, but it still isn't to the legendary stability of NetWare 3. Heck, NetWare 6.5 isn't at that legendary stability either. Rebooting for patches takes everything down for minutes at a time. Not viable.
With a server this beefy it is quite doable to do a cluster-in-a-box by way of Xen. Lay a base of SLES10-Sp2 on it, run the Xen kernel, and create four VMs for NCS cluster nodes. Give each 64-bit VM 7.75GB of RAM for file-caching, and bam! Cluster-in-a-box, and highly available.
However, this is a pure fantasy solution, so chances are real good that if we had the money we would use VMWare ESX instead XEN for the VM. The advantage to that is that we don't have to keep the VM/Host kernel versions in lock-step, which reduces downtime. There would be some performance degradation, and clock skew would be a problem, but at least uptime would be good; no need to perform a CLUSTER DOWN when updating kernels.
Best case, we'd have two physical boxes so we can patch the VM host without having to take every VM down.
But I still find it quite interesting that I could theoretically buy a single server with the same horsepower as the six servers driving our cluster right now.
Labels: hp, linux, netware, novell, NSS, OES, sysadmin, virtualization
Wednesday, February 04, 2009
The mystery of lsimpe.cdm
Last night I turned on multi-path support for the main NetWare file cluster. This has been a long time coming. When we upgraded the EVA3000 to an EVA6100 it gained the ability to do active/active IO on the controllers, something that the new EVA4400 can also do.
What's more, the two Windows backup-to-disk servers we've attached to the EVA4400 (and the MSA1500 for that matter) have the HP MPIO drivers installed, which are extensions of the Microsoft MPIO stack. Looking at the bandwidth chart on the fiber-channel fabric I see that these Windows servers are also doing load balancing over both of the paths. This is nifty! Also, when I last updated the XCS code on the EVA4400 both of those servers didn't even notice the controller reboots. EVEN NICER!
I want to do the same thing with NetWare. On the surface, turning on MPIO support is dead easy:
Startup.ncf file:
Tada. Reboot, present both paths in your zoning, and issue the "list failover devices" command on the console, and you'll get a list. In theory should one go away, it'll seamlessly move over to the other.
But what it won't do is load-balance. Unfortunately, the documentation on NetWare's multi-path support is rather scanty, focusing more on configuring path failover priority. The fact that the QL2X00.HAM driver itself can do it all on its own without letting NetWare know (the "allpaths" and "portnames" options tell it to not do that and let NetWare do the work) is a strong hint that MPIO is a fairly light weight protocol.
On the support forums you'll get several references to the LSIMPE.CDM file. With interesting phrases like, "that's the multipath driver", and, "Yeah, it isn't well documented." The data on the file itself is scanty, but suggestive:
What's more, the two Windows backup-to-disk servers we've attached to the EVA4400 (and the MSA1500 for that matter) have the HP MPIO drivers installed, which are extensions of the Microsoft MPIO stack. Looking at the bandwidth chart on the fiber-channel fabric I see that these Windows servers are also doing load balancing over both of the paths. This is nifty! Also, when I last updated the XCS code on the EVA4400 both of those servers didn't even notice the controller reboots. EVEN NICER!
I want to do the same thing with NetWare. On the surface, turning on MPIO support is dead easy:
Startup.ncf file:
SET MULTI-PATH SUPPORT = ON
LOAD QL2X00.HAM SLOT=10001 /LUNS /ALLPATHS /PORTNAMESTada. Reboot, present both paths in your zoning, and issue the "list failover devices" command on the console, and you'll get a list. In theory should one go away, it'll seamlessly move over to the other.
But what it won't do is load-balance. Unfortunately, the documentation on NetWare's multi-path support is rather scanty, focusing more on configuring path failover priority. The fact that the QL2X00.HAM driver itself can do it all on its own without letting NetWare know (the "allpaths" and "portnames" options tell it to not do that and let NetWare do the work) is a strong hint that MPIO is a fairly light weight protocol.
On the support forums you'll get several references to the LSIMPE.CDM file. With interesting phrases like, "that's the multipath driver", and, "Yeah, it isn't well documented." The data on the file itself is scanty, but suggestive:
LSIMPE.CDMBut the exact details of what it does remain unclear. One thing I do know, it doesn't do the load-balancing trick.
Loaded from [C:\NWSERVER\DRIVERS\] on Feb 4, 2009 3:32:13 am
(Address Space = OS)
LSI Multipath Enhancer
Version 1.02.02 September 5, 2006
Copyright 1989-2006 Novell, Inc.
Labels: clustering, netware, novell, storage
Wednesday, January 14, 2009
NTP on NetWare
A while back I did some work setting up an ntp peer-group on a pair of SLES servers (SLES9 and SLES10). That worked pretty good, and I managed to get autokey security working, which I thought was nifty. Then my thoughts turned to the OES environment.
If/when we get off of NetWare and move to the Linux kernel, NTP becomes the only way to do timesync. So I figured I'd see how amenable NetWare's xntpd was to secure configuration. Turns out it can do it, but there are some caveats.
First of all, it seems that the NTP for NetWare is based on NTPv3, not NTPv4, which means it doesn't support autokey and only supports symmetric keys. This also means that some other items on the ntp.conf file on the sles servers couldn't be carried over.
As it happens, the following sys:/etc/ntp.conf file works pretty well:
Populating the ntp.keys file couldn't be done from NetWare directly, I had to do that on a SLES server and copy it over. But once I did that, the ntppeer1 server and the NetWare server correctly authenticated to each other.
Interestingly, when I pointed an NTPv4 linux machine at the NetWare NTP setup I got complaints on the NetWare server about the incoming timehost not having the correct key and not being able to sync time. This is interesting because this linux machine was NOT one of the specified time-hosts. When I put in the 'restrict' line above with the 'nopeer' flag on it, those messages stopped.
The above configuration was successful in enabling a peer relationship between the two timehosts. This is loosely analogous to a PRIMARY group in traditional NetWare TIMESYNC setup. Should one or both of these hosts lose connection to the non-WWU time-servers (which are in essence equivalent to REFERENCE servers in Timesync, but unlike Timesync you can have more than one), they can negotiate time between themselves. This is important, as it prevents them from going out of sync, which would have dire consequences if allowed to happen more than a few minutes.
If/when we get off of NetWare and move to the Linux kernel, NTP becomes the only way to do timesync. So I figured I'd see how amenable NetWare's xntpd was to secure configuration. Turns out it can do it, but there are some caveats.
First of all, it seems that the NTP for NetWare is based on NTPv3, not NTPv4, which means it doesn't support autokey and only supports symmetric keys. This also means that some other items on the ntp.conf file on the sles servers couldn't be carried over.
As it happens, the following sys:/etc/ntp.conf file works pretty well:
server ntpserver1
server ntpserver2 minpoll 6 maxpoll 13
peer ntppeer1 key 1
enable auth monitor
keys sys:\etc\ntp.keys
trustedkey 1
requestkey 1
restrict default ignore
restrict 140.160.0.0 mask 255.255.0.0 nomodify nopeer
restrict 127.0.0.1
restrict [ip of ntpserver1]
restrict [ip of ntpserver2]
restrict [ip of ntppeer1]
Populating the ntp.keys file couldn't be done from NetWare directly, I had to do that on a SLES server and copy it over. But once I did that, the ntppeer1 server and the NetWare server correctly authenticated to each other.
Interestingly, when I pointed an NTPv4 linux machine at the NetWare NTP setup I got complaints on the NetWare server about the incoming timehost not having the correct key and not being able to sync time. This is interesting because this linux machine was NOT one of the specified time-hosts. When I put in the 'restrict' line above with the 'nopeer' flag on it, those messages stopped.
The above configuration was successful in enabling a peer relationship between the two timehosts. This is loosely analogous to a PRIMARY group in traditional NetWare TIMESYNC setup. Should one or both of these hosts lose connection to the non-WWU time-servers (which are in essence equivalent to REFERENCE servers in Timesync, but unlike Timesync you can have more than one), they can negotiate time between themselves. This is important, as it prevents them from going out of sync, which would have dire consequences if allowed to happen more than a few minutes.
Wednesday, December 03, 2008
OES2 SP1 ships!
Full announcement.
It's out!
It's out!
Tuesday, December 02, 2008
The NetWare 7 that never was
My last post generated some comments lamenting where NetWare has gone. I hear ya.
I have friends and have spoken with people at BrainShare who were closer to things than I was regarding how the next version of NetWare evolved. And to be truthful, it sounded a lot like how Microsoft moved from XP to Vista. If you'll recall, "the version of Windows after XP," was something of a moving target for many years. I recall media reports of Microsoft scrapping the whole project and starting afresh at least once.
My very first BrainShare was 2001, and that was the release party for NetWare 6. It was in 2003 when Novell bought Ximian, and bought SuSE, so it is clear when Novell probably decided to bet the house on this whole Linux thing. Yet at BS01 there was talk about NW7, or if there would be a NW6.1 version out. The rumors I remember from back then had NW7 being a progression towards a more application-friendly environment. I also remember hearing the L word around once or twice.
What we actually got was NetWare 6.5, which solidified NetWare 6 and made the core services better and more mature. What it wasn't was any more application friendly than NetWare 6 was (or even NetWare 5.1 for that matter). NetWare 6.5 released in August of 2003, the same month as the Ximian purchase. This is what tells me that Novell had decided on a path for NetWare 7, and it was green, not red. Open Enterprise Server arrived in 2005, which gives OES a solid year and a half dev-time between when SuSE was bought and when we started seeing public betas of OES. The NetWare version of OES was NetWare 6.5 SP3.
What happened to NetWare 7? It got lost on the roadmap. When NW6 came out, Novell probably had 6.5 on the roadmap as the next rev, with NW7 next down. The rumors we were hearing were very provisional, as the spot on the map held by NW7 was at least 3 years away. Sometime between BrainShare 2001 and when Novell started buying its way into the Linux world NW7 was dropped and the decision was made to port to a completely different Kernel. That decision was probably made in the summer of 2003, as the NetWare 6.5 development was entering final beta, and the task of allocating developer resources for the next full rev needed to be made.
Which brings us to today. OES2 SP1 is going to drop any day now, probably in time for Novell's quarterly earnings report. SP1 finally brings the Linux-kernel 'NetWare Services' to feature-comparable with the NetWare kernel. There are still a few things missing, like an eDirectory integrated SLP server, but now all the major points are covered. If you count it up, this has taken Novell a bit over 5 years to get to this point.
In my opinion, that's about right for an organization the size of Novell. Porting over the proprietary NetWare services to completely new kernel requires a LOT of developer attention, and Novell is a lot smaller than Microsoft. Also of note, it took Microsoft 5 years to give us Vista after XP, including the presumed nuke-and-rewrite they did. Novell got a boost in that they had already ported eDirectory to Linux, so that helped out the NCP side. But that didn't help the NSS folks, who had to figure out a way to do a NetWare-style rich metadata file-system on a kernel and driver model that expects POSIX-spartan file-systems. The problems Novell had with this were amply displayed in the performance problems reported with OES1-FCS. Samba doesn't scale to the same levels as CIFS-on-NetWare did, so that meant Novell had to create their own CIFS stack from scratch. The AFP stack on Linux is a joke, and the resurgence of Apple since 2003 meant they had to do something about that as well; by making a proprietary AFP stack. All of this represents nuke-and-rebuild-from-spec, which takes time.
Novell probably should have started the migration in 2000 instead of 2003. They already knew that Exchange 5.5 upgrades were driving a LOT of customers into Active Directory, which was triggering migrations away from NetWare. But, there are business concerns here. Novell managed to survive the fall of NetWare by diversifying their product portfolio enough that GroupWise, Zen, and Identity Management could support the company. It took until this year to return to the black, but they did it. Had they shot the NetWare cash cow two years earlier, it is entirely possible that Novell couldn't have survived the lean years.
I have friends and have spoken with people at BrainShare who were closer to things than I was regarding how the next version of NetWare evolved. And to be truthful, it sounded a lot like how Microsoft moved from XP to Vista. If you'll recall, "the version of Windows after XP," was something of a moving target for many years. I recall media reports of Microsoft scrapping the whole project and starting afresh at least once.
My very first BrainShare was 2001, and that was the release party for NetWare 6. It was in 2003 when Novell bought Ximian, and bought SuSE, so it is clear when Novell probably decided to bet the house on this whole Linux thing. Yet at BS01 there was talk about NW7, or if there would be a NW6.1 version out. The rumors I remember from back then had NW7 being a progression towards a more application-friendly environment. I also remember hearing the L word around once or twice.
What we actually got was NetWare 6.5, which solidified NetWare 6 and made the core services better and more mature. What it wasn't was any more application friendly than NetWare 6 was (or even NetWare 5.1 for that matter). NetWare 6.5 released in August of 2003, the same month as the Ximian purchase. This is what tells me that Novell had decided on a path for NetWare 7, and it was green, not red. Open Enterprise Server arrived in 2005, which gives OES a solid year and a half dev-time between when SuSE was bought and when we started seeing public betas of OES. The NetWare version of OES was NetWare 6.5 SP3.
What happened to NetWare 7? It got lost on the roadmap. When NW6 came out, Novell probably had 6.5 on the roadmap as the next rev, with NW7 next down. The rumors we were hearing were very provisional, as the spot on the map held by NW7 was at least 3 years away. Sometime between BrainShare 2001 and when Novell started buying its way into the Linux world NW7 was dropped and the decision was made to port to a completely different Kernel. That decision was probably made in the summer of 2003, as the NetWare 6.5 development was entering final beta, and the task of allocating developer resources for the next full rev needed to be made.
Which brings us to today. OES2 SP1 is going to drop any day now, probably in time for Novell's quarterly earnings report. SP1 finally brings the Linux-kernel 'NetWare Services' to feature-comparable with the NetWare kernel. There are still a few things missing, like an eDirectory integrated SLP server, but now all the major points are covered. If you count it up, this has taken Novell a bit over 5 years to get to this point.
In my opinion, that's about right for an organization the size of Novell. Porting over the proprietary NetWare services to completely new kernel requires a LOT of developer attention, and Novell is a lot smaller than Microsoft. Also of note, it took Microsoft 5 years to give us Vista after XP, including the presumed nuke-and-rewrite they did. Novell got a boost in that they had already ported eDirectory to Linux, so that helped out the NCP side. But that didn't help the NSS folks, who had to figure out a way to do a NetWare-style rich metadata file-system on a kernel and driver model that expects POSIX-spartan file-systems. The problems Novell had with this were amply displayed in the performance problems reported with OES1-FCS. Samba doesn't scale to the same levels as CIFS-on-NetWare did, so that meant Novell had to create their own CIFS stack from scratch. The AFP stack on Linux is a joke, and the resurgence of Apple since 2003 meant they had to do something about that as well; by making a proprietary AFP stack. All of this represents nuke-and-rebuild-from-spec, which takes time.
Novell probably should have started the migration in 2000 instead of 2003. They already knew that Exchange 5.5 upgrades were driving a LOT of customers into Active Directory, which was triggering migrations away from NetWare. But, there are business concerns here. Novell managed to survive the fall of NetWare by diversifying their product portfolio enough that GroupWise, Zen, and Identity Management could support the company. It took until this year to return to the black, but they did it. Had they shot the NetWare cash cow two years earlier, it is entirely possible that Novell couldn't have survived the lean years.
Labels: brainshare, linux, netware, novell, OES
Saturday, November 29, 2008
10,000 hours
I read an excerpt of a book a week or so ago. Always dangerous, as it lacks context. But the general principal of the book was the observation that to get really really good at something requires about 10,000 hours of practice. There are no 'naturals', just people who are naturally more pig-headed than others who can get to 10K hours.
10K hours is 2-4 hours a day for 10 years.
The studies were about things like child prodigies, or top tier athletes who get Olympic gold at age 22, and retire by 30. That sort of thing. It seems that almost all of these people started their thing by age 6, and by age 8 there was already a break between the kids who'd ultimately reach the peak of their field and those who'd merely be very good. The ones destined for peak were giving 2-3 hours a day at age 8, where the other group had cut back.
I believe this also applies to technical expertise. As anyone who has done any job searching in my field knows, there are real breaks for experience levels; 1-3 years, 4-6 years, 10+. Those of us in the 10+ area (and by now I am there with NetWare, and by the end of December I can claim that with Windows) are pretty much technical experts. We've put in the time over the years to get good.
However, we work in a field where, "Change or Die," is an accurate mantra. The IT industry of 2008 is markedly different than it was in 1998. Windows NT installs right now are laughed at. Very, very little of the operating systems and software in active use in 1998 is still able to be on a support contract. It is hard to be a 10K-hour expert in something in our field, you have to put in 8 hours a day for 5 years.
My first real exposure to NetWare was in a class I took for my CNA back in the Autumn of 1996. That was on NetWare 4.0, so at least my first experience was with NDS. In fact, my first job with NetWare was with 3.x, so I had to learn bindary on-the-job.
I consider myself to be an expert in NetWare. I've been actively administering it for 11 years now, so if I'm not across the 10K line I'm really close to it. This is only possible because the 'change or die' mantra has not applied to NetWare over the years. Lets take a look at the biggest disruptions of how things work in NetWare (kernel). This isn't incremental changes, this is fundamental re-learnings of how things work. Sort of like what all the Windows engineers had to go through when Active Directory came onto the scene.
Over the last 12 years NetWare has remained markedly static. This has allowed enough time for people who don't do this every waking moment to achieve a high level of expertise with NetWare. While this is good for NetWare, it unfortunately shows how NetWare has lagged behind the rest of the industry.
It is my opinion that OES-Linux represents a decade of pent up change that needed to happen in NetWare but didn't. This is why old time NetWare admins are having such trouble moving to Linux, they're being asked to support an Operating System that they don't have anywhere near the same level of expertise in and that is uncomfortable. I know I'm moving from an OS that I know exceedingly well to one where there are still, "here be monsters," marked out on my mental map. I'm also having to give up, "10+ year experience with NetWare," in favor of, "2-4 years of experience with Linux," and that doesn't feel good professionally.
But... that is the nature of our field. Just when we get really good at something, it's time to throw it out and learn something new. That something may be an incremental change from what we know (Windows 2003 vs Windows 2000) or a complete break (NT Domains vs AD Tree). But, learn we must. Us NetWare wonks have just been sheltered from it for some time.
10K hours is 2-4 hours a day for 10 years.
The studies were about things like child prodigies, or top tier athletes who get Olympic gold at age 22, and retire by 30. That sort of thing. It seems that almost all of these people started their thing by age 6, and by age 8 there was already a break between the kids who'd ultimately reach the peak of their field and those who'd merely be very good. The ones destined for peak were giving 2-3 hours a day at age 8, where the other group had cut back.
I believe this also applies to technical expertise. As anyone who has done any job searching in my field knows, there are real breaks for experience levels; 1-3 years, 4-6 years, 10+. Those of us in the 10+ area (and by now I am there with NetWare, and by the end of December I can claim that with Windows) are pretty much technical experts. We've put in the time over the years to get good.
However, we work in a field where, "Change or Die," is an accurate mantra. The IT industry of 2008 is markedly different than it was in 1998. Windows NT installs right now are laughed at. Very, very little of the operating systems and software in active use in 1998 is still able to be on a support contract. It is hard to be a 10K-hour expert in something in our field, you have to put in 8 hours a day for 5 years.
My first real exposure to NetWare was in a class I took for my CNA back in the Autumn of 1996. That was on NetWare 4.0, so at least my first experience was with NDS. In fact, my first job with NetWare was with 3.x, so I had to learn bindary on-the-job.
I consider myself to be an expert in NetWare. I've been actively administering it for 11 years now, so if I'm not across the 10K line I'm really close to it. This is only possible because the 'change or die' mantra has not applied to NetWare over the years. Lets take a look at the biggest disruptions of how things work in NetWare (kernel). This isn't incremental changes, this is fundamental re-learnings of how things work. Sort of like what all the Windows engineers had to go through when Active Directory came onto the scene.
- The move to TCP/IP. This by far is the biggest disruption since 1996. NetWare 5.0(?) introduced the ability to do NCP over TCP/IP natively, and not tunneled IPX-over-IP. This required replacing IPX SAP, something the routers just did, with SLP, a service that needed configuration and setup.
- The NSS file-system. This was a much lesser move than the TCP/IP one, as it worked on a general level (trustees, quotas, etc) the same as TFS did. Tweaking it for performance, however, was a dark art for many years and much learning was derived out of this.
- Protected memory. A concept familiar to anyone who has used Windows or Linux, and all NetWare admins are by now administering one or both of these OS's. While some modules can't use it for whatever reason (iPrint, NetStorage) others (GroupWise) could.
- Native File Access Pack. NetWare could do AFP since the NW3 days, the same for NFS. SMB was another story. It was with NetWare 5.1 that NFAP came on to the scene, and NetWare 6.0 where it came built in and performed much better. The ability to use protocols other than NCP for your Windows clients was embraced by many shops.
Over the last 12 years NetWare has remained markedly static. This has allowed enough time for people who don't do this every waking moment to achieve a high level of expertise with NetWare. While this is good for NetWare, it unfortunately shows how NetWare has lagged behind the rest of the industry.
It is my opinion that OES-Linux represents a decade of pent up change that needed to happen in NetWare but didn't. This is why old time NetWare admins are having such trouble moving to Linux, they're being asked to support an Operating System that they don't have anywhere near the same level of expertise in and that is uncomfortable. I know I'm moving from an OS that I know exceedingly well to one where there are still, "here be monsters," marked out on my mental map. I'm also having to give up, "10+ year experience with NetWare," in favor of, "2-4 years of experience with Linux," and that doesn't feel good professionally.
But... that is the nature of our field. Just when we get really good at something, it's time to throw it out and learn something new. That something may be an incremental change from what we know (Windows 2003 vs Windows 2000) or a complete break (NT Domains vs AD Tree). But, learn we must. Us NetWare wonks have just been sheltered from it for some time.
Monday, November 17, 2008
Signs and portents
Last Thursday I was over on download.novell.com looking for an eDirectory patch. I was staging up a new NetWare box and needed to see what the latest edir levels were. I knew 8.8.3 came out in August, and we're not there yet, so I needed 8.8.2 ftf2. However, I noticed that one of the searchable versions was 8.8.4. There was nothing in the list, but it was an option. It's not there right now, but it was then.
Thus emboldened I checked around a few more places. NetWare 6.5 SP8 was in the list, and still is right now. As is Open Enterprise Server 2 SP1. Both have the public betas posted, though.
But 8.8.4 was there. I saw it. Must have been a test or something. All this tells me that OES2 SP1 (a.k.a. NW65SP8) is just around the corner. Since we were told back at BrainShare that Sp1 would be in the Q4 time-frame, it's about due.
Thus emboldened I checked around a few more places. NetWare 6.5 SP8 was in the list, and still is right now. As is Open Enterprise Server 2 SP1. Both have the public betas posted, though.
But 8.8.4 was there. I saw it. Must have been a test or something. All this tells me that OES2 SP1 (a.k.a. NW65SP8) is just around the corner. Since we were told back at BrainShare that Sp1 would be in the Q4 time-frame, it's about due.
Monday, November 10, 2008
NetStorage, WebDav, and Vista
I figured out how to get it working! You need KB907306. This updates the Web Folders in Vista to support how Novell does WebDav through NetStorage.
In our case you'll also need to add the CA that serves the SSL certificate that's on top of NetStorage (a.k.a. MyFiles). But, it works.
In our case you'll also need to add the CA that serves the SSL certificate that's on top of NetStorage (a.k.a. MyFiles). But, it works.
Wednesday, October 15, 2008
OES2 SP1 (public beta) has been posted
The public beta of OES2 SP1 has been posted.
I believe the NDA has lifted, but I'm not 100% on that. Will check. But, some of the new stuff in SP1:
One thing I think it is safe to say, is that even though it says "Beta4" on it, it's really a release-candidate. Only major bugs are getting quashed right now. UI freeze was a month or more ago, and strange, annoying behaviors may get "fixed in doc" rather than getting true fixes which will have to wait for SP2. Still report them anyway, since it'll go on the list to fix in the next SP.
I believe the NDA has lifted, but I'm not 100% on that. Will check. But, some of the new stuff in SP1:
- An AFP stack that doesn't suck. Or more specifically, an AFP stack that scales beyond 100 users and is eDirectory integrated.
- A new CIFS stack written by Novell, so it can scale well past the Samba limit.
- A migration toolkit in one UI, rather than a cluster of scripts.
- A new version of iFolder
- EDirectory integrated DNS/DHCP. But no eDir integrated SLP yet, open-source politics you know.
- IIRC a beta of eDir 8.8 SP4.
- The ability to put iPrint-for-Linux on NSS volumes (handy for Clustering).
- And lots more I can't remember off the top of my head.
One thing I think it is safe to say, is that even though it says "Beta4" on it, it's really a release-candidate. Only major bugs are getting quashed right now. UI freeze was a month or more ago, and strange, annoying behaviors may get "fixed in doc" rather than getting true fixes which will have to wait for SP2. Still report them anyway, since it'll go on the list to fix in the next SP.
Tuesday, September 23, 2008
That darned 32-bit limit
Today I learned that the disk-space counters NetWare provides in SNMP use signed integers for its disk-space monitoring. These are stats published at a table at OID .1.3.6.1.4.1.23.2.28.2.14.1.3. Having just expanded our FacShare volume past 2TB, it went negative-space according to the monitors. A simple integer overflow since apparently Novell is using a signed integer for a number that can never be legitimately negative.
I've pointed this out on an enhancement request. This being NetWare, they may not chose to fix it if it is more than a two-line fix. We'll see.
This also means that volumes over 4TB can not be effectively monitored with SNMP. Since NSS can have up to 8TB volumes on NetWare, this could potentially be a problem. We're not there yet.
I've pointed this out on an enhancement request. This being NetWare, they may not chose to fix it if it is more than a two-line fix. We'll see.
This also means that volumes over 4TB can not be effectively monitored with SNMP. Since NSS can have up to 8TB volumes on NetWare, this could potentially be a problem. We're not there yet.
Wednesday, September 10, 2008
That darned budget
This is where I whine about not having enough money.
It has been a common complaint amongst my co-workers that WWU wants enterprise level service for a SOHO budget. Especially for the Win/Novell environments. Our Solaris stuff is tied in closely to our ERP product, SCT Banner, and that gets big budget every 5 years to replace. We really need the same kind of thing for the Win/Novell side of the house, such as this disk-array replacement project we're doing right now.
The new EVAs are being paid for by Student Tech Fee, and not out of a general budget request. This is not how these devices should be funded, since the scope of this array is much wider than just student-related features. Unfortunately, STF is the only way we could get them funded, and we desperately need the new arrays. Without the new arrays, student service would be significantly impacted over the next fiscal year.
The problem is that the EVA3000 contains between 40-45% directly student-related storage. The other 55-60% is Fac/Staff storage. And yet, the EVA3000 was paid for by STF funds in 2003. Huh.
The summer of 2007 saw a Banner Upgrade Project, when the servers that support SCT Banner were upgraded. This was a quarter million dollar project and it happens every 5 years. They also got a disk-array upgrade to a pair of StorageTek (SUN, remember) arrays, DR replicated between our building and the DR site in Bond Hall. I believe they're using Solaris-level replication rather than Array-level replication.
The disk-array upgrade we're doing now got through the President's office just before the boom went down on big expensive purchases. It languished in the Purchasing department due to summer-vacation related under-staffing. I hate to think how late it would have gone had it been subjected to the added paperwork we now have to go through for any purchase over $1000. Under no circumstances could we have done it before Fall quarter. Which would have been bad, since we were too short to deal with the expected growth of storage for Fall quarter.
Now that we're going deep into the land of VMWare ESX, centralized storage-arrays are line of business. Without the STF funded arrays, we'd be stuck with "Departmental" and "Entry-level" arrays such as the much maligned MSA1500, or building our own iSCSI SAN from component parts (a DL385, with 2x 4-channel SmartArray controller cards, 8x MSA70 drive enclosures, running NetWare or Linux as an iSCSI target, with bonded GigE ports for throughput). Which would blow chunks. As it is, we're still stuck using SATA drives for certain 'online' uses, such as a pair of volumes on our NetWare cluster that are low usage but big consumers of space. Such systems are not designed for the workloads we'd have to subject them to, and are very poor performers when doing things like LUN expansions.
The EVA is exactly what we need to do what we're already doing for high-availability computing, yet is always treated as an exceptional budget request when it comes time to do anything big with it. Since these things are hella expensive, the budgetary powers-that-be balk at approving them and like to defer them for a year or two. We asked for a replacement EVA in time for last year's academic year, but the general-budget request got denied. For this year we went, IIRC, both with general-fund and STF proposals. The general fund got denied, but STF approved it. This needs to change.
By October, every person between and Governor Gregoir will be new. My boss is retiring in October. My grandboss was replaced last year, my great grand boss also has been replaced in the last year, and the University President stepped down on September 1st. Perhaps the new people will have a broader perspective on things and might permit the budget priorities to be realigned to the point that our disk-arrays are classified as the critical line-of-business investments they are.
It has been a common complaint amongst my co-workers that WWU wants enterprise level service for a SOHO budget. Especially for the Win/Novell environments. Our Solaris stuff is tied in closely to our ERP product, SCT Banner, and that gets big budget every 5 years to replace. We really need the same kind of thing for the Win/Novell side of the house, such as this disk-array replacement project we're doing right now.
The new EVAs are being paid for by Student Tech Fee, and not out of a general budget request. This is not how these devices should be funded, since the scope of this array is much wider than just student-related features. Unfortunately, STF is the only way we could get them funded, and we desperately need the new arrays. Without the new arrays, student service would be significantly impacted over the next fiscal year.
The problem is that the EVA3000 contains between 40-45% directly student-related storage. The other 55-60% is Fac/Staff storage. And yet, the EVA3000 was paid for by STF funds in 2003. Huh.
The summer of 2007 saw a Banner Upgrade Project, when the servers that support SCT Banner were upgraded. This was a quarter million dollar project and it happens every 5 years. They also got a disk-array upgrade to a pair of StorageTek (SUN, remember) arrays, DR replicated between our building and the DR site in Bond Hall. I believe they're using Solaris-level replication rather than Array-level replication.
The disk-array upgrade we're doing now got through the President's office just before the boom went down on big expensive purchases. It languished in the Purchasing department due to summer-vacation related under-staffing. I hate to think how late it would have gone had it been subjected to the added paperwork we now have to go through for any purchase over $1000. Under no circumstances could we have done it before Fall quarter. Which would have been bad, since we were too short to deal with the expected growth of storage for Fall quarter.
Now that we're going deep into the land of VMWare ESX, centralized storage-arrays are line of business. Without the STF funded arrays, we'd be stuck with "Departmental" and "Entry-level" arrays such as the much maligned MSA1500, or building our own iSCSI SAN from component parts (a DL385, with 2x 4-channel SmartArray controller cards, 8x MSA70 drive enclosures, running NetWare or Linux as an iSCSI target, with bonded GigE ports for throughput). Which would blow chunks. As it is, we're still stuck using SATA drives for certain 'online' uses, such as a pair of volumes on our NetWare cluster that are low usage but big consumers of space. Such systems are not designed for the workloads we'd have to subject them to, and are very poor performers when doing things like LUN expansions.
The EVA is exactly what we need to do what we're already doing for high-availability computing, yet is always treated as an exceptional budget request when it comes time to do anything big with it. Since these things are hella expensive, the budgetary powers-that-be balk at approving them and like to defer them for a year or two. We asked for a replacement EVA in time for last year's academic year, but the general-budget request got denied. For this year we went, IIRC, both with general-fund and STF proposals. The general fund got denied, but STF approved it. This needs to change.
By October, every person between and Governor Gregoir will be new. My boss is retiring in October. My grandboss was replaced last year, my great grand boss also has been replaced in the last year, and the University President stepped down on September 1st. Perhaps the new people will have a broader perspective on things and might permit the budget priorities to be realigned to the point that our disk-arrays are classified as the critical line-of-business investments they are.
Labels: budget, clustering, exchange, hp, linux, msa, netware, novell, OES, opinion, storage, sysadmin, virtualization
Monday, September 08, 2008
Web-servers
Looking at usage stats, the amount of data transferred by Myweb for Students has gone down somewhat from its heyday in 2006. I blame Web 2.0. Myweb is a static HTML service. We don't allow any server-side processing of any kind other than server-side includes. This is not how web-development is done anymore. This very blog is database backed, but Blogger publishes static HTML pages to represent that database, which is why I'm able to host this blog on Myweb for FacStaff.
If we were to provide a full-out hosting service for our students (and staff), I'm sure there would be a heck of a lot more uptake. A few years ago there was a push in certain Higher Ed circles to provide a, "portfolio service", which would host a student's work for a certain time after graduation so they could point employers at it as a reference. We never did that for a variety of reasons (cost being a big one), but the sentiment is still there.
If we were to provide not only full-out hosting, but actual domain-hosting for students, it could fill this need quite well. Online brand is important, and if a student can build a body of work on "$studentname.[com|org|net|biz]" it can be quite useful in hunting down employment. Several of the ResTek technicians I know have their own domains hosting their own blogs, so the demand is there.
I've never worked for a company that did web-hosting as a business item, so I've only heard horror stories of how bad it can get. First of all, we'll need a full LAMP stack server-farm to run the thing. That's money. Second, we'll need the organizational experience with the technology to prevent badly configured Wordpress or PhpBB installs from DoSing other cohosted sites from resource-exhaustion by hackers. This is a worker-hours thing.
Then we'd have to figure out the graduated problem. Once a student graduates, do we keep hosting for them? Do we charge them? Do we force them off the system after a specific time? Questions that need answers, and these are the kinds of questions that contributed to the killing of the portfolio-server idea.
Personally, I think this is something we could provide. However, someone needs to kick the money tree hard enough to shake loose the funds to make it happen. Perhaps Student Tech Fee could do it. Perhaps it could be a 'discounted' added-cost service we provide. Who knows. But we could probably do it.
If we were to provide a full-out hosting service for our students (and staff), I'm sure there would be a heck of a lot more uptake. A few years ago there was a push in certain Higher Ed circles to provide a, "portfolio service", which would host a student's work for a certain time after graduation so they could point employers at it as a reference. We never did that for a variety of reasons (cost being a big one), but the sentiment is still there.
If we were to provide not only full-out hosting, but actual domain-hosting for students, it could fill this need quite well. Online brand is important, and if a student can build a body of work on "$studentname.[com|org|net|biz]" it can be quite useful in hunting down employment. Several of the ResTek technicians I know have their own domains hosting their own blogs, so the demand is there.
I've never worked for a company that did web-hosting as a business item, so I've only heard horror stories of how bad it can get. First of all, we'll need a full LAMP stack server-farm to run the thing. That's money. Second, we'll need the organizational experience with the technology to prevent badly configured Wordpress or PhpBB installs from DoSing other cohosted sites from resource-exhaustion by hackers. This is a worker-hours thing.
Then we'd have to figure out the graduated problem. Once a student graduates, do we keep hosting for them? Do we charge them? Do we force them off the system after a specific time? Questions that need answers, and these are the kinds of questions that contributed to the killing of the portfolio-server idea.
Personally, I think this is something we could provide. However, someone needs to kick the money tree hard enough to shake loose the funds to make it happen. Perhaps Student Tech Fee could do it. Perhaps it could be a 'discounted' added-cost service we provide. Who knows. But we could probably do it.
Thursday, August 14, 2008
Virtualization and Fileservers
There are some workloads that fit well within VM of any kind, and others that are very tricky. Fileservers are one area that are not good candidates for VM. In some cases they qualify as highly transactional. In others, the memory required to do fileserving well makes them very expensive. When you can fit 40 web-servers on a VM host, but only 4 fileservers, it makes the calculus obvious.
This is on my mind since we're running into memory problems on our NetWare cluster. We've just plain outgrown the 32-bit memory space for file-cache. NW can use memory above the 4GB line, it does have PAE support, but memory access above there is markedly slower than it is below the line. Last I heard the conventional wisdom is that 12GB is about the point where it starts earning you performance gains again. eek!
So, I'm looking forward to 64-bit memory spaces and OES2. 6GB should do us for a few years. That said, 6GB of actually-used RAM in a virtual-host means that I could fit... two of them on a VM server with 16GB of RAM.
16GB of RAM in, say, an ESX cluster is enough to host 10 other servers. Especially with memory deduplication. In the case of my hypothetical 6GB file-servers, 5.5GB of that RAM will be consumed by file-cache that will be unique to that server and thus very little gains from memory de-dup.
In the end, how well a fileserver fits in a VM environment is based on how large of a 'working set' your users have. If the working set it large enough, it can mean that you'll get small gains for virtualization. However, I realize fileserving on the scale we do it is somewhat rare, so for departmental fileservers VM can be a good-sized win. As always, know your environment.
In light of the budgetary woes we'll be having, I don't know what we'll do. Last I heard the State is projected to have a 2.7 billion deficit for the 2009-2011 (fiscal year starts July 1) budget cycle. So it may very well be possible that the only way I'll get access to 64-bit memory spaces is in an ESX context. That may mean a 6 node cluster on 3 physical hosts. And that's assuming I can get new hardware at all. If it gets bad enough I'll have to limp along until 2010 and play partitioning games to load-balance my data-loads across all 6 nodes. By 2011 all of our older hardware falls off of cheap-maintenance and we'll have to replace it, so worst-case that's when I can do my migration to 64-bit. Arg.
This is on my mind since we're running into memory problems on our NetWare cluster. We've just plain outgrown the 32-bit memory space for file-cache. NW can use memory above the 4GB line, it does have PAE support, but memory access above there is markedly slower than it is below the line. Last I heard the conventional wisdom is that 12GB is about the point where it starts earning you performance gains again. eek!
So, I'm looking forward to 64-bit memory spaces and OES2. 6GB should do us for a few years. That said, 6GB of actually-used RAM in a virtual-host means that I could fit... two of them on a VM server with 16GB of RAM.
16GB of RAM in, say, an ESX cluster is enough to host 10 other servers. Especially with memory deduplication. In the case of my hypothetical 6GB file-servers, 5.5GB of that RAM will be consumed by file-cache that will be unique to that server and thus very little gains from memory de-dup.
In the end, how well a fileserver fits in a VM environment is based on how large of a 'working set' your users have. If the working set it large enough, it can mean that you'll get small gains for virtualization. However, I realize fileserving on the scale we do it is somewhat rare, so for departmental fileservers VM can be a good-sized win. As always, know your environment.
In light of the budgetary woes we'll be having, I don't know what we'll do. Last I heard the State is projected to have a 2.7 billion deficit for the 2009-2011 (fiscal year starts July 1) budget cycle. So it may very well be possible that the only way I'll get access to 64-bit memory spaces is in an ESX context. That may mean a 6 node cluster on 3 physical hosts. And that's assuming I can get new hardware at all. If it gets bad enough I'll have to limp along until 2010 and play partitioning games to load-balance my data-loads across all 6 nodes. By 2011 all of our older hardware falls off of cheap-maintenance and we'll have to replace it, so worst-case that's when I can do my migration to 64-bit. Arg.
Labels: linux, netware, OES, sysadmin, virtualization
Monday, August 11, 2008
Novell Client for Vista, the ecosystem
I just reported a bug in the beta that surprised me. I can't talk details about it, but it strikes me as the kind of bug that should have been at least reported shortly after the client released. Perhaps it was just so overall buggy that it got lost in the forest, but still. The Vista client has been out for some time now.
Having said the following rant several times over the past few days, I figure it's time to post it ;).
The problem we're running in to is that the number of users of the Vista Client is a small, small sub-set of the overall users of the Novell Client, which are by now a minority of overall users of Novell NCP file-servers. Novell spent years hyping 'clientless' approaches to file-serving, through the CIFS stack on NetWare. A lot of places bought in to that. Because of this, the percentage of NCP-client Vista users among the overall Novell File-Server market is a rather small one.
And small means you don't get a lot of testing done by people-who-are-not-us, and seemingly obvious bugs showing up in the beta Sp1 builds. I don't have any Vista workstations, so I've done exactly zero testing of the Vista Client; this particular bug was reported and troubleshot by someone who is not me (I just filed it). Even though we have beta builds of the Vista client as part of this beta, I'm not testing it. All things considered, I probably should.
Since we're wedded hard to the Novell Client, it's probably time for us to start devoting resources to the ecosystem in order to keep it alive.
Having said the following rant several times over the past few days, I figure it's time to post it ;).
The problem we're running in to is that the number of users of the Vista Client is a small, small sub-set of the overall users of the Novell Client, which are by now a minority of overall users of Novell NCP file-servers. Novell spent years hyping 'clientless' approaches to file-serving, through the CIFS stack on NetWare. A lot of places bought in to that. Because of this, the percentage of NCP-client Vista users among the overall Novell File-Server market is a rather small one.
And small means you don't get a lot of testing done by people-who-are-not-us, and seemingly obvious bugs showing up in the beta Sp1 builds. I don't have any Vista workstations, so I've done exactly zero testing of the Vista Client; this particular bug was reported and troubleshot by someone who is not me (I just filed it). Even though we have beta builds of the Vista client as part of this beta, I'm not testing it. All things considered, I probably should.
Since we're wedded hard to the Novell Client, it's probably time for us to start devoting resources to the ecosystem in order to keep it alive.
Monday, June 30, 2008
Novell Client for Linux, packaged for OpenSUSE
It has been mentioned many places, and I've done some of the mentioning, that since openSUSE is the foundation for SLED, it makes sense for Novell to distribute an NCL for openSUSE. It turns out they're working on just that. And here is the Novell beta page. I'm soooo going to try this out, since I'm running openSUSE 10.3 on my work desktop and won't be moving to openSUSE 11 until I can run the client on it (oh, wait, I can).
It should also be mentioned that Ubuntu is a very frequently requested target for another NCL, but I have reason to believe that'll never happen. First of all, any Novell Client involves closed source 3rd party licensed code, which makes it hard to port to Linux in the first place (a relic of being based on code from the days when open-source was just an ethical standpoint rather than a tangible market force). Second, Novell has proven to be rather light in developer resources in certain areas, and linux integration with non-SUSE linux distros is very minimal.
It should also be mentioned that Ubuntu is a very frequently requested target for another NCL, but I have reason to believe that'll never happen. First of all, any Novell Client involves closed source 3rd party licensed code, which makes it hard to port to Linux in the first place (a relic of being based on code from the days when open-source was just an ethical standpoint rather than a tangible market force). Second, Novell has proven to be rather light in developer resources in certain areas, and linux integration with non-SUSE linux distros is very minimal.
Tuesday, June 24, 2008
Backing up NSS, note for the future
According to this documentation, the storing of NSS/NetWare metadata in xattrs is turned off by default. You turn it on for OES2 servers through the "nss /ListXattrNWMetadata" command. This allows linux level utilities (i.e. cp, tar) to be able to access and copy the NSS metadata. This also allows backup software that isn't SMS enabled for OES2 to be able to backup the NSS information.
This is handy, as HP DataProtector doesn't support NSS backup on Linux. I need to remember this.
This is handy, as HP DataProtector doesn't support NSS backup on Linux. I need to remember this.
Monday, June 16, 2008
A good article on trustees
Over on the Novell Cool Solutions site, Marcel Cox just posted an article about how Trustees are handled on the Novell Filesystems (TFS and NFS). If you wanted to know the fundamentals of how ACLs are done on NSS volumes and how it relates to eDirectory, this is a good start.
Labels: coolsolutions, edir, netware, novell, NSS, storage, sysadmin
Thursday, May 29, 2008
OES2 and SLES10-SP2
Per Novell:
Also, the OES2 install does contain some kernel packages, such as those relating to NSS.
OES2 systems need to wait until either Novell gives the all clear for SP2 deployments on OES2-fcs, or OES2-SP1 ships. OES2-SP1 is built around SLES10-Sp2.
Very true. And most especially true if you're running virtualized NetWare! The paravirtualization components in NW65SP7 are designed around the version of Xen that's in SLES10-SP1, and SP2 contains a much newer version of Xen (trying to play catch-up to VMWare means a fast dev cycle, after all). So, expect problems if you do it.Updating OES2
OES2 systems should NOT be updated to SLES10 SP2 at this time!
Also, the OES2 install does contain some kernel packages, such as those relating to NSS.
OES2 systems need to wait until either Novell gives the all clear for SP2 deployments on OES2-fcs, or OES2-SP1 ships. OES2-SP1 is built around SLES10-Sp2.
Wednesday, May 14, 2008
NetWare and Xen
Here is something I didn't really know about in virtualized NetWare:
Guidelines for using NSS in a virtual environment
Towards the bottom of this document, you get this:
So, keep in mind where your disk files are! If you're using one huge XFS partition and hosting all the disks for your VM-NW systems on that, then you'll need barriers. If you're presenting a SAN LUN directly to the VM, then you'll need to "SET XENBLK BARRIERS = 0", as they're set to 16 by default. This'll give you better performance.
Guidelines for using NSS in a virtual environment
Towards the bottom of this document, you get this:
Nice stuff there! The "xenblk barriers" can also have an impact on the performance of your virtualized NetWare server. If your I/O stream runs the server out of cache, performance can really suffer if barriers are non-zero. If it fits in cache, the server can reorder the I/O stream to the disks to the point that you don't notice the performance hit.Configuring Write Barrier Behavior for NetWare in a Guest Environment
Write barriers are needed for controlling I/O behavior when writing to SATA and ATA/IDE devices and disk images via the Xen I/O drivers from a guest NetWare server. This is not an issue when NetWare is handling the I/O directly on a physical server.
The XenBlk Barriers parameter for the SET command controls the behavior of XenBlk Disk I/O when NetWare is running in a virtual environment. The setting appears in the Disk category when you issue the SET command in the NetWare server console.
Valid settings for the XenBlk Barriers parameter are integer values from 0 (turn off write barriers) to 255, with a default value of 16. A non-zero value specifies the depth of the driver queue, and also controls how often a write barrier is inserted into the I/O stream. A value of 0 turns off XenBlk Barriers.
A value of 0 (no barriers) is the best setting to use when the virtual disks assigned to the guest server’s virtual machine are based on physical SCSI, Fibre Channel, or iSCSI disks (or partitions on those physical disk types) on the host server. In this configuration, disk I/O is handled so that data is not exposed to corruption in the event of power failure or host crash, so the XenBlk Barriers are not needed. If the write barriers are set to zero, disk I/O performance is noticeably improved.
Other disk types such as SATA and ATA/IDE can leave disk I/O exposed to corruption in the event of power failure or a host crash, and should use a non-zero setting for the XenBlk Barriers parameter. Non-zero settings should also be used for XenBlk Barriers when writing to Xen LVM-backed disk images and Xen file-backed disk images, regardless of the physical disk type used to store the disk images.
So, keep in mind where your disk files are! If you're using one huge XFS partition and hosting all the disks for your VM-NW systems on that, then you'll need barriers. If you're presenting a SAN LUN directly to the VM, then you'll need to "SET XENBLK BARRIERS = 0", as they're set to 16 by default. This'll give you better performance.
Labels: benchmarking, netware, novell, NSS, OES, storage, virtualization
Monday, May 12, 2008
DataProtector 6 has a problem, continued
I posted last week about DataProtector and its Enhanced Incremental Backup. Remember that "enhincrdb" directory I spoke of? Take a look at this:
See? This is an in-progress count of one of these directories. 1.1 million files, 152MB of space consumed. That comes to an average file-size of 133 bytes. This is significantly under the 4kb block-size for this particular NTFS volume. On another server with a longer serving enhincrdb hive, the average file-size is 831 bytes. So it probably increases as the server gets older.
On the up side, these millions of weensy files won't actually consume more space for quite some time as they expand into the blocks the files are already assigned to. This means that fragmentation on this volume isn't going to be a problem for a while.
On the down side, it's going to park (in this case) 152MB of data on 4.56GB of disk space. It'll get better over time, but in the next 12 months or so it's still going to be horrendous.
This tells me two things:
Since it is highly likely that I'll be using DataProtector for OES2 systems, this is something I need to keep in mind.
See? This is an in-progress count of one of these directories. 1.1 million files, 152MB of space consumed. That comes to an average file-size of 133 bytes. This is significantly under the 4kb block-size for this particular NTFS volume. On another server with a longer serving enhincrdb hive, the average file-size is 831 bytes. So it probably increases as the server gets older.
On the up side, these millions of weensy files won't actually consume more space for quite some time as they expand into the blocks the files are already assigned to. This means that fragmentation on this volume isn't going to be a problem for a while.
On the down side, it's going to park (in this case) 152MB of data on 4.56GB of disk space. It'll get better over time, but in the next 12 months or so it's still going to be horrendous.
This tells me two things:
- When deciding where to host the enhincrdb hive on a Windows server, format that particular volume with a 1k block size.
- If HP supported NetWare as an Enhanced Incremental Backup client, the 4kb block size of NSS would cause this hive to grow beyond all reasonable proportions.
Since it is highly likely that I'll be using DataProtector for OES2 systems, this is something I need to keep in mind.
Wednesday, May 07, 2008
DataProtecter 6 has a problem
We're moving our BackupExec environment to HP DataProtector. Don't ask why, it made sense at the time.
Once of the niiiice things about DP is what's called, "Enhanced Incremental Backup". This is a de-duplication strategy, that only backs up files that have changed, and only stores the changed blocks. From these incremental backups you can construct synthetic full backups, which are just pointer databases to the blocks for that specified point-in-time. In theory, you only need to do one full backup, keep that backup forever, do enhanced incrementals, then periodically construct synthetic full backups.
We've been using it for our BlackBoard content store. That's around... 250GB of file store. Rather than keep 5 full 275GB backup files for the duration of the backup rotation, I keep 2 and construct synthetic fulls for the other 3. In theory I could just go with 1, but I'm paranoid :). This greatly reduces the amount of disk-space the backups consume.
Unfortunately, there is a problem with how DP does this. The problem rests on the client side of it. In the "$InstallDir$\OmniBack\enhincrdb" directory it constructs a file hive. An extensive file hive. In this hive it keeps track of file state data for all the files backed up on that server. This hive is constructed as follows:
The last real full backup I took of the content store backed up just under 1.7 million objects (objects = directory entries in NetWare, or inodes in unix-land). Yet the enhincrdb hive had 2.7 million objects. Why the difference? I'm not sure, but I suspect it was keeping state data for 1 million objects that no longer were present in the backup. I have trouble believing that we managed to churn over 60% of the objects in the store in the time I have backups, so I further suspect that it isn't cleaning out state data from files that no longer have a presence in the backup system.
DataProtector doesn't support Enhanced Incrementals for NetWare servers, only Windows and possibly Linux. Due to how this is designed, were it to support NetWare it would create absolutely massive directory structures on my SYS: volumes. The FACSHARE volume has about 1.3TB of data in it, in about 3.3 million directory entries. The average FacStaff User volume (we have 3) has about 1.3 million, and the average Student User volume has about 2.4 million. Due to how our data works, our Student user volumes have a high churn rate due to students coming and going. If FACSHARE were to share a cluster node with one Student user volume and one FacStaff user volume, they have a combined directory-entry count of 7.0 million directory entries. This would generate, at first, a \enhincrdb directory with 7.0 million files. Given our regular churn rate, within a year it could easily be over 9.0 million.
When you move a volume to another cluster node, it will create a hive for that volume in the \enhincrdb directory tree. We're seeing this on the BlackBoard Content cluster. So given some volumes moving around, and it is quite conceivable that each cluster node will have each cluster volume represented in its own \enhincrdb directory. Which will mean over 15 million directory-entries parked there on each SYS volume, steadily increasing as time goes on taking who knows how much space.
And as anyone who has EVER had to do a consistency check of a volume that size knows (be it vrepair, chkdsk, fsck,or nss /poolrebuild), it takes a whopper of a long time when you get a lot of objects on a file-system. The old Traditional File System on NetWare could only support 16 million directory entries, and DP would push me right up to that limit. Thank heavens NSS can support w-a-y more then that. You better hope that the file-system that the \enhincrdb hive is on never has any problems.
But, Enhanced Incrementals only apply to Windows so I don't have to worry about that. However.... if they really do support Linux (and I think they do), then when I migrate the cluster to OES2 next year this could become a very real problem for me.
DataProtector's "Enhanced Incremental Backup" feature is not designed for the size of file-store we deal with. For backing up the C: drive of application servers or the inetpub directory of IIS servers, it would be just fine. But for file-servers? Good gravy, no! Unfortunately, those are the servers in most need of de-dup technology.
Once of the niiiice things about DP is what's called, "Enhanced Incremental Backup". This is a de-duplication strategy, that only backs up files that have changed, and only stores the changed blocks. From these incremental backups you can construct synthetic full backups, which are just pointer databases to the blocks for that specified point-in-time. In theory, you only need to do one full backup, keep that backup forever, do enhanced incrementals, then periodically construct synthetic full backups.
We've been using it for our BlackBoard content store. That's around... 250GB of file store. Rather than keep 5 full 275GB backup files for the duration of the backup rotation, I keep 2 and construct synthetic fulls for the other 3. In theory I could just go with 1, but I'm paranoid :). This greatly reduces the amount of disk-space the backups consume.
Unfortunately, there is a problem with how DP does this. The problem rests on the client side of it. In the "$InstallDir$\OmniBack\enhincrdb" directory it constructs a file hive. An extensive file hive. In this hive it keeps track of file state data for all the files backed up on that server. This hive is constructed as follows:
- The first level is the mount point. Example: enhincrdb\F\
- The 2nd level are directories named 00-FF which contain the file state data itself
The last real full backup I took of the content store backed up just under 1.7 million objects (objects = directory entries in NetWare, or inodes in unix-land). Yet the enhincrdb hive had 2.7 million objects. Why the difference? I'm not sure, but I suspect it was keeping state data for 1 million objects that no longer were present in the backup. I have trouble believing that we managed to churn over 60% of the objects in the store in the time I have backups, so I further suspect that it isn't cleaning out state data from files that no longer have a presence in the backup system.
DataProtector doesn't support Enhanced Incrementals for NetWare servers, only Windows and possibly Linux. Due to how this is designed, were it to support NetWare it would create absolutely massive directory structures on my SYS: volumes. The FACSHARE volume has about 1.3TB of data in it, in about 3.3 million directory entries. The average FacStaff User volume (we have 3) has about 1.3 million, and the average Student User volume has about 2.4 million. Due to how our data works, our Student user volumes have a high churn rate due to students coming and going. If FACSHARE were to share a cluster node with one Student user volume and one FacStaff user volume, they have a combined directory-entry count of 7.0 million directory entries. This would generate, at first, a \enhincrdb directory with 7.0 million files. Given our regular churn rate, within a year it could easily be over 9.0 million.
When you move a volume to another cluster node, it will create a hive for that volume in the \enhincrdb directory tree. We're seeing this on the BlackBoard Content cluster. So given some volumes moving around, and it is quite conceivable that each cluster node will have each cluster volume represented in its own \enhincrdb directory. Which will mean over 15 million directory-entries parked there on each SYS volume, steadily increasing as time goes on taking who knows how much space.
And as anyone who has EVER had to do a consistency check of a volume that size knows (be it vrepair, chkdsk, fsck,or nss /poolrebuild), it takes a whopper of a long time when you get a lot of objects on a file-system. The old Traditional File System on NetWare could only support 16 million directory entries, and DP would push me right up to that limit. Thank heavens NSS can support w-a-y more then that. You better hope that the file-system that the \enhincrdb hive is on never has any problems.
But, Enhanced Incrementals only apply to Windows so I don't have to worry about that. However.... if they really do support Linux (and I think they do), then when I migrate the cluster to OES2 next year this could become a very real problem for me.
DataProtector's "Enhanced Incremental Backup" feature is not designed for the size of file-store we deal with. For backing up the C: drive of application servers or the inetpub directory of IIS servers, it would be just fine. But for file-servers? Good gravy, no! Unfortunately, those are the servers in most need of de-dup technology.
Monday, May 05, 2008
Linux @ Home
My laptop at home dual-boots between openSUSE and WinXP. There are a few reasons why I don't boot the Linux side very often, some of them work related. And, what the heck, here are the two reasons.
1: Wireless driver problems
I have an intel 3945 WLAN card. It works just fine in linux, well supported. What throws it for a loop, however, are sleep and hibernate states. It can go one, two, four, maybe five cycles through sleep before it will require a reboot in order to find the home wireless again. If it doesn't lock the laptop up hard. Since my usage patterns are heavily dependent upon Sleep mode, this is a major, major disincentive to keep the Linux side booted.
I understand the 2.6.25 kernel is a lot better about this particular driver. Thus, I wait with eager anticipation the release of openSUSE 11.0. This driver is currently the ipw3945 driver, and will eventually turn into iwl3945 driver once it comes down the pipe. What little I've read about it suggests that the iwl driver is more stable through power states.
2: NetWare remote console
I use rconip for remote console to NetWare. Back when Novell first created the IP-based rconsole, they also released rconj along side ConsoleOne to provide it. As this was written in Java, it was mind bogglingly slow. This little .exe file was vastly faster, and I've come to use it extensively. Unless I get Wine working, this tool will have to stay on my Windows XP partition. It works great, and I haven't found a good linux-based replacement yet.
Time has moved on. Hardware has gotten faster, and the 'java penalty' has reduced markedly. RconJ is actually usable, but I still don't use it. Plus, it would require me to install ConsoleOne onto my laptop. It's 32-bit, so that's actually possible, but I really don't want to do that.
The Remote Console through the Novell Remote Monitor (that service out on :8009) has a nice remote-console utility, but it also requires Java. I'm still biased against java, and java-on-linux still seems fairly unstable to me. I don't trust it yet. It also doesn't scale well. When I'm service-packing, it is a LOT nicer looking to have 6 rconip windows up than 6 browser-based NRM java-consoles open. Plus, rconip will allow me access to the server console if DS is locked, something that NRM can't do and is invaluable in an emergency.
Once the wireless driver problems are fixed, I'll boot the linux side much more often. Remote-X over SSH actually makes some of my remote management a touch easier than it is in WinXP. And if I really really need to use Windows, my work XP VM is accessible over RDesktop. There are a few other non-work reasons why I don't boot Linux very often, but I'll not go into those here.
So, oddly, NetWare is partly responsible for keeping me in Windows at home. But only partly.
1: Wireless driver problems
I have an intel 3945 WLAN card. It works just fine in linux, well supported. What throws it for a loop, however, are sleep and hibernate states. It can go one, two, four, maybe five cycles through sleep before it will require a reboot in order to find the home wireless again. If it doesn't lock the laptop up hard. Since my usage patterns are heavily dependent upon Sleep mode, this is a major, major disincentive to keep the Linux side booted.
I understand the 2.6.25 kernel is a lot better about this particular driver. Thus, I wait with eager anticipation the release of openSUSE 11.0. This driver is currently the ipw3945 driver, and will eventually turn into iwl3945 driver once it comes down the pipe. What little I've read about it suggests that the iwl driver is more stable through power states.
2: NetWare remote console
I use rconip for remote console to NetWare. Back when Novell first created the IP-based rconsole, they also released rconj along side ConsoleOne to provide it. As this was written in Java, it was mind bogglingly slow. This little .exe file was vastly faster, and I've come to use it extensively. Unless I get Wine working, this tool will have to stay on my Windows XP partition. It works great, and I haven't found a good linux-based replacement yet.
Time has moved on. Hardware has gotten faster, and the 'java penalty' has reduced markedly. RconJ is actually usable, but I still don't use it. Plus, it would require me to install ConsoleOne onto my laptop. It's 32-bit, so that's actually possible, but I really don't want to do that.
The Remote Console through the Novell Remote Monitor (that service out on :8009) has a nice remote-console utility, but it also requires Java. I'm still biased against java, and java-on-linux still seems fairly unstable to me. I don't trust it yet. It also doesn't scale well. When I'm service-packing, it is a LOT nicer looking to have 6 rconip windows up than 6 browser-based NRM java-consoles open. Plus, rconip will allow me access to the server console if DS is locked, something that NRM can't do and is invaluable in an emergency.
Once the wireless driver problems are fixed, I'll boot the linux side much more often. Remote-X over SSH actually makes some of my remote management a touch easier than it is in WinXP. And if I really really need to use Windows, my work XP VM is accessible over RDesktop. There are a few other non-work reasons why I don't boot Linux very often, but I'll not go into those here.
So, oddly, NetWare is partly responsible for keeping me in Windows at home. But only partly.
Labels: linux, netware, novell, opinion, virtualization
Thursday, April 17, 2008
NetWare and Novell, changing a company
A couple days ago Richard Bliss had a long blog entry about, "Novell's Cash Cow - How NetWare almost killed the company". It had some very interesting points. Some we knew:
In 2001 I knew that Windows had for all intents and purposes won. The only place you ever really saw NetWare servers were as file-servers, or running GroupWise or the small handful of apps that used NetWare as an application server. The stalwart loyalists among us saw this as annoying, but not a major problem.
It was also good for Novell's bottom line. NetWare still accounted for a large percentage of their revenues. Even though the writing was on the wall, they were still making real money on it so didn't see a need to change. This is why NetWare 6.0 introduced the AMP stack to NetWare, as a way to better make NetWare an application server and to slow the loss of customers. At BrainShare 2001 there was open speculation about "NetWare 7.0" and what it would look like.
And there still was until 2005 when Novell announced what the next version of NetWare would be. This being after the SUSE and Ximian purchases, it would be based on Linux. This move had been rumored, and alternately derided and lauded, for some time. There was a great wailing and gnashing of teeth on the part of the stalwart NetWare loyalists. It also started an exodus of customers, as Novell's financial reports at the time point out.
Fortunately for the company, they started actively promoting (for certain values of 'active' that are higher than they were previously, but still in the theme of Novell Stealth Marketing) and developing their other products, like GroupWise, Novell Identity Management, ZenWorks, and most especially their Linux business. It took them until last quarter to turn in a quarter in the black, and NetWare revenues are under 20% of total now. So, they've turned the corner and are no longer dependent on the NetWare cash cow. They have a couple of them in the field now, which is a MUCH healthier place to be.
It's a funny thing, but one of the reasons why NetWare is such a kick-butt file-server compared to everything else is why it's a challenging environment to develop in. Had Novell seen the light earlier and bought SUSE (or rolled their own Linux distro) in... 1999 instead, right after the NW5.1 release, they still would have run into the fundamental architectural problems in 32-bit linux that make it an inferior file-serving platform for large environments. By 2008 their server could have been a LOT more mature, and perfectly poised to take advantage of 64-bit Linux.
Novell in the 1990's is not an example of a 'nimble' company. It is trying to get there now through diversification. Not many companies (especially tech companies) have survived the loss of their prime money earner; Apple has done it through OSX, which required a fanatically loyal fan base to survive the dark years. This is the prime reason people kept predicting the imminent demise or buyout of Novell. Now that they're earning profits again, and have diversified away from just the OS sector, they're not going to be going out of business any time soon.
Now if only they had better SMB packages and programs. I hear repeatedly from peers who support SMBs that Novell's packages and programs in that space are lacking or exploitative. Significant revenue, and more importantly mindshare, are in the SMB market. Plus, today's SMB is tomorrow's large or global enterprise.
We are all familiar with NetWare, the dominate Network Operating system of the 1980s and 1990s. We are all familiar with Microsoft's tactics of penetrating the NOS market with Windows NT by focusing on using Windows as an application platform.Apparently Richard worked for Novell around 2001. I find that interesting since my first BrainShare was 2001, and that was when they announced the release of NetWare 6.0. While there he saw what seemed to be an outright denial that NetWare had been passed up by Windows and something new needed to be done.
In 2001 I knew that Windows had for all intents and purposes won. The only place you ever really saw NetWare servers were as file-servers, or running GroupWise or the small handful of apps that used NetWare as an application server. The stalwart loyalists among us saw this as annoying, but not a major problem.
It was also good for Novell's bottom line. NetWare still accounted for a large percentage of their revenues. Even though the writing was on the wall, they were still making real money on it so didn't see a need to change. This is why NetWare 6.0 introduced the AMP stack to NetWare, as a way to better make NetWare an application server and to slow the loss of customers. At BrainShare 2001 there was open speculation about "NetWare 7.0" and what it would look like.
And there still was until 2005 when Novell announced what the next version of NetWare would be. This being after the SUSE and Ximian purchases, it would be based on Linux. This move had been rumored, and alternately derided and lauded, for some time. There was a great wailing and gnashing of teeth on the part of the stalwart NetWare loyalists. It also started an exodus of customers, as Novell's financial reports at the time point out.
Fortunately for the company, they started actively promoting (for certain values of 'active' that are higher than they were previously, but still in the theme of Novell Stealth Marketing) and developing their other products, like GroupWise, Novell Identity Management, ZenWorks, and most especially their Linux business. It took them until last quarter to turn in a quarter in the black, and NetWare revenues are under 20% of total now. So, they've turned the corner and are no longer dependent on the NetWare cash cow. They have a couple of them in the field now, which is a MUCH healthier place to be.
It's a funny thing, but one of the reasons why NetWare is such a kick-butt file-server compared to everything else is why it's a challenging environment to develop in. Had Novell seen the light earlier and bought SUSE (or rolled their own Linux distro) in... 1999 instead, right after the NW5.1 release, they still would have run into the fundamental architectural problems in 32-bit linux that make it an inferior file-serving platform for large environments. By 2008 their server could have been a LOT more mature, and perfectly poised to take advantage of 64-bit Linux.
Novell in the 1990's is not an example of a 'nimble' company. It is trying to get there now through diversification. Not many companies (especially tech companies) have survived the loss of their prime money earner; Apple has done it through OSX, which required a fanatically loyal fan base to survive the dark years. This is the prime reason people kept predicting the imminent demise or buyout of Novell. Now that they're earning profits again, and have diversified away from just the OS sector, they're not going to be going out of business any time soon.
Now if only they had better SMB packages and programs. I hear repeatedly from peers who support SMBs that Novell's packages and programs in that space are lacking or exploitative. Significant revenue, and more importantly mindshare, are in the SMB market. Plus, today's SMB is tomorrow's large or global enterprise.
Tuesday, March 25, 2008
IPv6 vs IPX
In a session last week came the following comment from a presenter (paraphrased):
00008021:0002a540d0e1
Where 'node number' is the MAC address of the network card in question. It's up to the routers to figure out where network-numbers live, and advertised services issue full-network broadcasts to advertise said service, which is the primary reason that IPX just doesn't scale if WAN links are in the mix. But that's by the by.
IPv6 addresses work similarly:
2001:0db8:85a3:08d3:1319:8a2e:0370:7334
The last 48 bits are the MAC address and the bits ahead of it constitute the network number. Except... the IPv6 designers knew about the failings of IPX and worked around them. The last 48 bits don't have to be the MAC address, though as I understand it that address has to exist for each physical interface. Unlike IPX, IPv6 has the ability to have 'secondary' addresses. The lack of this ability was the main reason that Novell Cluster Services only worked on IP networks, which caused its own wave of grief when clustering was introduced in the NetWare 5.1 era. Secondary IPv6 numbers don't have to follow the MAC format, which in my opinion is a good thing!
Yes, when I first read about IPv6 addressing I had that same, "wow, this is just like IPX," moment the BrainShare presenter had. Only, more scalable, and more flexible.
How may of you in the room have been at this long enough to do IPX? Ok, great. Now how many of you have done anything with IPv6? Doesn't that look JUST like IPX?And he's right, to a point. IPX addresses are of the form network-number:node-number, such as:
00008021:0002a540d0e1
Where 'node number' is the MAC address of the network card in question. It's up to the routers to figure out where network-numbers live, and advertised services issue full-network broadcasts to advertise said service, which is the primary reason that IPX just doesn't scale if WAN links are in the mix. But that's by the by.
IPv6 addresses work similarly:
2001:0db8:85a3:08d3:1319:8a2e:0370:7334
The last 48 bits are the MAC address and the bits ahead of it constitute the network number. Except... the IPv6 designers knew about the failings of IPX and worked around them. The last 48 bits don't have to be the MAC address, though as I understand it that address has to exist for each physical interface. Unlike IPX, IPv6 has the ability to have 'secondary' addresses. The lack of this ability was the main reason that Novell Cluster Services only worked on IP networks, which caused its own wave of grief when clustering was introduced in the NetWare 5.1 era. Secondary IPv6 numbers don't have to follow the MAC format, which in my opinion is a good thing!
Yes, when I first read about IPv6 addressing I had that same, "wow, this is just like IPX," moment the BrainShare presenter had. Only, more scalable, and more flexible.
Labels: brainshare, clustering, netware, novell, sysadmin
Thursday, March 20, 2008
BrainShare Wednesday
The Wednesday keynote was, indeed, a bunch of demos. It was also mostly pointless as far as the technology I'm concerned with. Lots of GroupWise (don't care), lots and lots of PlateSpin (can't afford it), lots of Zen (not the bits I'd use).
That said, the new GroupWise WebAccess is gorgeous. I wish Exchange had their non-ActiveX pages look that good.
TUT175: RBAC: Avoiding the horror, getting past the hype
Mostly about IDM as it turned out. Only minimally interesting from an abstract viewpoint about roles in general.
TUT 277: Advanced eDirectory Configuration, new features, and tuning for performance
I learned a few things I didn't know, such as the fact that each object as an "AncestorList" attribute listing who their parent objects are. This apparently greatly speeds up searching. SP3, coming out this Summer, will have faster LDAP binds for a couple of reasons. Right now Novell is recommending 2 million objects as a reasonable maximum size for a partition for performance reasons.
And also they reiterated something I've heard before...
TUT221: iPrint on Linux, what Novell Support wants you to know
A nice session from a mainline support guy about the ways people don't do iPrint on linux correctly. We're not going there until pcounter can run in linux, so this is still somewhat abstract. But, nice to know.
That said, the new GroupWise WebAccess is gorgeous. I wish Exchange had their non-ActiveX pages look that good.
TUT175: RBAC: Avoiding the horror, getting past the hype
Mostly about IDM as it turned out. Only minimally interesting from an abstract viewpoint about roles in general.
TUT 277: Advanced eDirectory Configuration, new features, and tuning for performance
I learned a few things I didn't know, such as the fact that each object as an "AncestorList" attribute listing who their parent objects are. This apparently greatly speeds up searching. SP3, coming out this Summer, will have faster LDAP binds for a couple of reasons. Right now Novell is recommending 2 million objects as a reasonable maximum size for a partition for performance reasons.
And also they reiterated something I've heard before...
You know how back in the NetWare 4 days, we said to design your tree by geography at the first level, and then get to departments? Um, sorry about that. It was great back then, but for LDAP or IDM it really, really slows things down.Yep. I took my first class for my CNA when 'Green River' was just coming out, or was just out. So I remember that.
TUT221: iPrint on Linux, what Novell Support wants you to know
A nice session from a mainline support guy about the ways people don't do iPrint on linux correctly. We're not going there until pcounter can run in linux, so this is still somewhat abstract. But, nice to know.
- The reason that some print jobs render differently than direct-print jobs, is because of how Windows is designed. Direct-print jobs render with the 'local print provider', and iPrint jobs render with the 'network print provider'. This is a Microsoft thing, not an iPrint thing. You can duplicate it by setting up a microsoft IPP printer (assuming you're not mandating SSL like we are) and printing to the same printer with the same driver.
- The Manager on Linux doesn't use a Broker, it uses a 'driver store'.
- The Manager on NetWare doesn't always bind to the same broker. I didn't know that.
- It is recommended to have only one Broker, or one driver store per tree.
- Novell recommends using DNS rather than IP for your printer-agents, check your manager load scripts.
Labels: brainshare, edir, linux, microsoft, netware, novell, OES, pcounter, printing
Tuesday, March 18, 2008
BrainShare Tuesday
Today started off with a bit of panic, as I hadn't set my alarm. Me being a west-coaster, 7:20 (when I woke up) is an entirely reasonable time to get up as far as my body is concerned. Only, I needed to get dressed and breakfasted before my first session at 8:30. Aie! I had to eat quick, but I got there. Didn't get a chance to check work email, though.
ATT326: Advanced Linux Troubleshooting
An ATT, therefore hard to summarize. But I learned about a few new commands I didn't know about before. Like strace. And vimdiff.
TUT130: Challenges in Storage I/O in Virtualization
Another nice one, but an emergency at work (printing down in a dorm, during finals week) distracted me heavily during the first half of it. Which resulted in the following note in my notes:
TUT216: OES2 SP1 Architectural Overview
There is a LOT of new stuff in SP1.
TUT211: Enhanced Protocol Support in OES2 SP1
This is the session where they went into detail about the AFP and CIFS support. They said that netatalk, the existing AFP stack on Linux, gets really slow once you go over the 20 concurrent users. Whoa! I can soooo understand why Novell felt the need to make a new one.
Tonight was the night formerly known as 'Sponsor Night,' but has a new name now that everyone who gets a booth is no longer a 'sponsor'. Some are sponsors, some are exhibiters. I can't keep track. Anyway, today was their party. "World of Novellcraft!" Homage to vid-gaming.
Lots of Wii, lots of Rock Band, some Halo, lots of women dressed in Renaissance Festival gear getting their pictures taken by the 90%+ male audience. I've blogged before about my ambivalence about Sponsor Night. I lasted until about 7, when I came back to the hotel.
Tomorrow I have an actual LUNCH BREAK in my schedule! Ooo! AndSoul Asylum Soul Coughing Collective Soul plays the concert! I've been listening to two of their CD's for the past two months so I think I may even know a few songs by now.
ATT326: Advanced Linux Troubleshooting
An ATT, therefore hard to summarize. But I learned about a few new commands I didn't know about before. Like strace. And vimdiff.
TUT130: Challenges in Storage I/O in Virtualization
Another nice one, but an emergency at work (printing down in a dorm, during finals week) distracted me heavily during the first half of it. Which resulted in the following note in my notes:
NPIV looks really nifty. Look into it.NPIV being how you can use fibre-channel zoning to zone off VM's, rather than HBA's. Highly useful. I also learned about a neat new thing called Virtual Fabrics. Virtual Fabrics work kind of like VLANS for fabrics. You can segregate your fabrics into fabrics that share hardware but nothing else. Handy if your, say, Solaris admins don't want you mucking about with their zoning, while saving money through consolidated hardware.
TUT216: OES2 SP1 Architectural Overview
There is a LOT of new stuff in SP1.
- It will include eDir 8.8.4 (8.8.3 will ship this summer sometime)
- NCP and eDir will be fully 64-bit
- OES2 SP1 will be based on SLES SP2, which will be releasing about the same time
- AFP Support
- AFP 3.1
- Uses Diffie-Helman 1 for password exchange, meaning the 8-character password problem is solved.
- Fully SMP-safe
- Has cross-protocol locking with NCP. CIFS doesn't have cross-protocol locking yet, but IIRC, Samba does
- Does not need LUM enabled users
- CIFS Support
- NTLMv1, but v2 is a possibility if enough people ask, so file those enhancement requests!!
- CIFS is separate from Samba, therefore can not be used in conjunction with Domain Services for Windows
- As with AFP, fully SMP safe
- EDir 8.8.4
- LDAP auditing enhanced
- "newer auth protocols", but they didn't say what.
TUT211: Enhanced Protocol Support in OES2 SP1
This is the session where they went into detail about the AFP and CIFS support. They said that netatalk, the existing AFP stack on Linux, gets really slow once you go over the 20 concurrent users. Whoa! I can soooo understand why Novell felt the need to make a new one.
- The 8 character password limit has been fixed! They now support DH1 for passing passwords.
- The 'afptcp' daemon can use one password protocol at a time, so you can only use DH1, or one of the other three I can't remember.
- Support for OSX 10.1 and 10.2 is scanty, and 10.5 is limited but users may not notice anyway.
- Passwords will be case sensitive.
- Kerberos will be in a future release
- Performance is faster than NetWare, partly due to the ability to multi-thread
- Can register services by way of SLP
- Only supports NSS for the time being, the other Linux file-systems will be a future feature.
- Can support 500 concurrent users, and 1000+ in the future. This fits our current AFP loads.
- We can configure more about how it works than we could on NetWare, such as how many worker threads to spawn.
- Has meaninful debug logs!
- Has a new command, 'afpstat' that works like 'netstat' for giving a snapshot of afp connections.
Tonight was the night formerly known as 'Sponsor Night,' but has a new name now that everyone who gets a booth is no longer a 'sponsor'. Some are sponsors, some are exhibiters. I can't keep track. Anyway, today was their party. "World of Novellcraft!" Homage to vid-gaming.
Lots of Wii, lots of Rock Band, some Halo, lots of women dressed in Renaissance Festival gear getting their pictures taken by the 90%+ male audience. I've blogged before about my ambivalence about Sponsor Night. I lasted until about 7, when I came back to the hotel.
Tomorrow I have an actual LUNCH BREAK in my schedule! Ooo! And
Labels: brainshare, edir, linux, netware, novell, OES, sysadmin
Monday, March 17, 2008
Today at Brainshare
Monday. Opening day. I had trouble getting to sleep last night due to a poor choice of bed-time reading (don't read action, don't read action, don't read action). And had to get up at 6am body time in order to get breakfast before the morning keynote. There be zombies.
Breakfast was uninspired. As per usual, the hashbrowns had cooled to a gellid mass before I found everything and got a seat.
The Monday keynotes are always the CxO talks about strategy and where we're going. Today a mess of press releases from Novell give a good idea what the talks were about. Hovsepian was first, of course, and was actually funny. He gave some interesting tid-bits of knowledge.
Another thing he mentioned several times in association with Fossa and agility, is mergers and acquisitions. This is not something us Higher Ed types ever have to deal with, but it is an area in .COM land that requires a certain amount of IT agility to accommodate successfully. He mentioned this several times, which suggests that this strategy is aimed squarely at for-profit industry.
Also, SAP has apparently selected SLES as their primary platform for the SMB products.
Pat Hume from SAP also spoke. But as we're on Banner, and it'll take a sub-megaton nuclear strike to get us off of it, I didn't pay attention and used the time to send some emails.
Oh, and Honeywell? They're here because they have hardware that works with IDM. That way the same ID you use for your desktop login can be tied to the RFID card in your pocket that gets you into the datacenter. Spiffy.
ATT375 Advanced Tips & Tricks for Troubleshooting eDir 8.8
A nice session. Hard to summarize. That said, they needed more time as the Laptops with VMWare weren't fast enough for us to get through many of the exercises. They also showed us some nifty iMonitor tricks. And where the high-yield shoot-your-foot-off weapons are kept.
BUS202 Migrating a NetWare Cluster to OES2
Not a good session. The presenter had a short slide deck, and didn't really present anything new to me other than areas where other people have made major mistakes. And to PLAN on having one of the linux migrations go all lost-data on you. He recommended SAN snapshots. It shortly digressed into "Migrating a NetWare Cluster to Linux HA", which is a different session all together. So I left.
TUT215 Integrating Macintosh with Novell
A very good session. The CIO of Novell Canada was presenting it, and he is a skilled speaker. Apparently Novell has written a new AFP stack from scratch for OES2 Sp1, since NETATALK is comparatively dog slow. And, it seems, the AFP stack is currently out performing the NCP stack on OES2 SP1. Whoa! Also, the Banzai GroupWise client for Mac is apparently gorgeous. He also spent quite a long time (18 minutes) on the Kanaka client from Condrey Consulting. The guy who wrote that client was in the back of the room and answered some questions.
Breakfast was uninspired. As per usual, the hashbrowns had cooled to a gellid mass before I found everything and got a seat.
The Monday keynotes are always the CxO talks about strategy and where we're going. Today a mess of press releases from Novell give a good idea what the talks were about. Hovsepian was first, of course, and was actually funny. He gave some interesting tid-bits of knowledge.
- Novell's group of partners is growing, adding a couple hundred new ones since last year. This shows the Novell 'ecosystem' is strong.
- 8700 new customers last year
- Novell press mentions are now only 5% negative.
- High Capacity Computing
- Policy Engines
- Orchestration
- Convergence
- Mobility
Another thing he mentioned several times in association with Fossa and agility, is mergers and acquisitions. This is not something us Higher Ed types ever have to deal with, but it is an area in .COM land that requires a certain amount of IT agility to accommodate successfully. He mentioned this several times, which suggests that this strategy is aimed squarely at for-profit industry.
Also, SAP has apparently selected SLES as their primary platform for the SMB products.
Pat Hume from SAP also spoke. But as we're on Banner, and it'll take a sub-megaton nuclear strike to get us off of it, I didn't pay attention and used the time to send some emails.
Oh, and Honeywell? They're here because they have hardware that works with IDM. That way the same ID you use for your desktop login can be tied to the RFID card in your pocket that gets you into the datacenter. Spiffy.
ATT375 Advanced Tips & Tricks for Troubleshooting eDir 8.8
A nice session. Hard to summarize. That said, they needed more time as the Laptops with VMWare weren't fast enough for us to get through many of the exercises. They also showed us some nifty iMonitor tricks. And where the high-yield shoot-your-foot-off weapons are kept.
BUS202 Migrating a NetWare Cluster to OES2
Not a good session. The presenter had a short slide deck, and didn't really present anything new to me other than areas where other people have made major mistakes. And to PLAN on having one of the linux migrations go all lost-data on you. He recommended SAN snapshots. It shortly digressed into "Migrating a NetWare Cluster to Linux HA", which is a different session all together. So I left.
TUT215 Integrating Macintosh with Novell
A very good session. The CIO of Novell Canada was presenting it, and he is a skilled speaker. Apparently Novell has written a new AFP stack from scratch for OES2 Sp1, since NETATALK is comparatively dog slow. And, it seems, the AFP stack is currently out performing the NCP stack on OES2 SP1. Whoa! Also, the Banzai GroupWise client for Mac is apparently gorgeous. He also spent quite a long time (18 minutes) on the Kanaka client from Condrey Consulting. The guy who wrote that client was in the back of the room and answered some questions.
Labels: brainshare, clustering, netware, novell, OES, sysadmin, virtualization
Thursday, February 14, 2008
OES2-SP1 soon to be in closed beta
Novell just announced that OES2 SP1 is going into closed beta.
"What is in this release of Open Enterprise ServerNovell Open Enterprise Server 2 Support Pack 1 refreshes the SUSE Linux Enterprise Server 10 distribution with SLES10 SP2, fixes defects found since the release of OES2 and also adds in the following functionality:
- Novell engineered CIFS and AFP protocols
- New version of iFolder (3.7)
- Updated iPrint with an accounting API
- 64-bit version of eDirectory
- Enhanced migration tools and migration GUI
- Improved performance of the XEN hypervisor
- Domain Services for Windows
- NetWare 6.5 Support Pack 8
Note that although Domain Services for Windows is part of OES2 SP1, a separate beta program will be run in order to collate DSfW feedback."
Novell engineered CIFS? I soooo want to know what that is. Is is a completely new CIFS stack, or is it Samba with Novell extensions whacked on? I want to know! The other important bit of information:
The beta test program is currently scheduled to begin mid March and run through October.Which means there won't be product for my 2008 upgrade window. Fie. Well, at least we'll have ample time to prototype and test for the 2009 upgrade window.
Update 9/2008: Novell has posted on their beta site that a public beta is 'coming soon'.
Update 10/2008: The public beta for OES2 SP1 has been posted.
Labels: clustering, netware, novell, OES
Monday, February 04, 2008
Today's 18 year olds...
Over the time I've been here there has occasionally been a list posted in the break-room. This list is the, "Incoming freshman today...." list of things they know, experience, or haven't experienced. It contains things like:
Which got me thinking about a few things. One of the items that is frequently put forth about Kids These Days (tm) is that they don't KNOW anything, they just know how to FIND things. There is some debate about this, but it is a common sentiment. I believe that kids these days (KTD) have figured out keyword based searching, and the search engines have gotten good enough at mind-reading that arcane search incantations aren't needed nearly as often as they were in the past.
Before Google, there was AltaVista. This was an era of the internet where boolean search incantations were needed to really narrow down to what you wanted. I didn't switch to Google for a long time because Google didn't have the NEAR search term, which I used on AltaVista as a way to narrow results to be more relevant. I didn't know at the time that Google effectively threw that term in on every search.
Those of us who lived through that era of the internet built up searching skills. I remember some searches I did back then that were pretty complex. I can't remember the exact terms used, but they looked like this:
bootes AND (antaries OR proxima) AND (fulcrum NEAR pinnacle)
I had a logic class in college, so these sorts of parenthetical statements made sense to me. Still do, I just don't end up needing to uncork the boolean logic to find what I need anymore as the search engines have gotten good enough that I don't NEED to do it. I know google allows much of the above, but I haven't had to do it so I don't know the syntax for it.
So I posit that yes, KTD don't know anything, but neither are their search skills robust.
Which brings me to Novell. I got to thinking what a NetWare administrator in 1990 had to know to do their job, and how I could fit into such a hypothetical time.
Right now if I don't know the answer to a problem I have a few methods to figure it out.
As I see it, a NetWare admin of 1990 was on average more knowledgeable about their product than the NetWare admin of 2008. Such administrators avoided the cost of paying for support incidents by having the manuals in hard-copy form, and plonking down real money for CompuServe accounts. If I have a weird problem I'll hit up the Novell KB to see if there is a TID on it, then check the support forums to see if it is mentioned there, before I'll expend an incident on the thing. In time I've managed to teach myself how NetWare works in some very basic ways, simply by troubleshooting oddball problems. This is why I typically end up talking to backline support when I call in, unless the problem is a known issue in the private KB. My skills are probably on par with what was normal 'back in the day'.
I think this holds true for a lot of the tech field. Back then there was a lot of stuff you just had to KNOW. Or failing that, have spent the money to get the backup resources in place (manuals, support contracts). These days a base understanding of how things work is the key to phrasing the right search queries in the online knowledge bases, and less rote memorization (training) can be effective in solving a greater list of problems.
Prosthetic memory! Prosthetic training! The tools of geeks everywhere.
- Were born in 1990
- ...have never known life without cable or satellite TV.
- ...probably have never seen a rotary dial phone.
- ...have had internet access for most of their school life.
Which got me thinking about a few things. One of the items that is frequently put forth about Kids These Days (tm) is that they don't KNOW anything, they just know how to FIND things. There is some debate about this, but it is a common sentiment. I believe that kids these days (KTD) have figured out keyword based searching, and the search engines have gotten good enough at mind-reading that arcane search incantations aren't needed nearly as often as they were in the past.
Before Google, there was AltaVista. This was an era of the internet where boolean search incantations were needed to really narrow down to what you wanted. I didn't switch to Google for a long time because Google didn't have the NEAR search term, which I used on AltaVista as a way to narrow results to be more relevant. I didn't know at the time that Google effectively threw that term in on every search.
Those of us who lived through that era of the internet built up searching skills. I remember some searches I did back then that were pretty complex. I can't remember the exact terms used, but they looked like this:
bootes AND (antaries OR proxima) AND (fulcrum NEAR pinnacle)
I had a logic class in college, so these sorts of parenthetical statements made sense to me. Still do, I just don't end up needing to uncork the boolean logic to find what I need anymore as the search engines have gotten good enough that I don't NEED to do it. I know google allows much of the above, but I haven't had to do it so I don't know the syntax for it.
So I posit that yes, KTD don't know anything, but neither are their search skills robust.
Which brings me to Novell. I got to thinking what a NetWare administrator in 1990 had to know to do their job, and how I could fit into such a hypothetical time.
Right now if I don't know the answer to a problem I have a few methods to figure it out.
- Hit the online Novell Knowledge Base over at novell.com/support
- Hit the peer-support forums over at forums.novell.com (or nntp://forums.novell.com/ if you prefer old-school)
- Pay for a support incident
- Ask around the office
- Hit the peer-support forums over on CompuServe, which required a modem and a CompuServe account.
- See if the problem is mentioned in the book-shelf of manuals, which was a big investment to own.
- Pay for a support incident.
- Ask around the office.
As I see it, a NetWare admin of 1990 was on average more knowledgeable about their product than the NetWare admin of 2008. Such administrators avoided the cost of paying for support incidents by having the manuals in hard-copy form, and plonking down real money for CompuServe accounts. If I have a weird problem I'll hit up the Novell KB to see if there is a TID on it, then check the support forums to see if it is mentioned there, before I'll expend an incident on the thing. In time I've managed to teach myself how NetWare works in some very basic ways, simply by troubleshooting oddball problems. This is why I typically end up talking to backline support when I call in, unless the problem is a known issue in the private KB. My skills are probably on par with what was normal 'back in the day'.
I think this holds true for a lot of the tech field. Back then there was a lot of stuff you just had to KNOW. Or failing that, have spent the money to get the backup resources in place (manuals, support contracts). These days a base understanding of how things work is the key to phrasing the right search queries in the online knowledge bases, and less rote memorization (training) can be effective in solving a greater list of problems.
Prosthetic memory! Prosthetic training! The tools of geeks everywhere.
Wednesday, January 30, 2008
I don't think it works that way
We've been having some abend issues on the cluster lately, something with the network services rather than file serving. It seems to be triggered more by iPrint/NDPS than by MyFiles, but both are associated with it. The abend itself is in WS2_32.NLM, so it's in the network stack. I have a call open with Novell.
I finally, finally managed to get a meaningful packet-capture after it fails, and I found some traffic that... doesn't look right. Take a look:
Note the three packets in the middle. The responding server is tearing down the connection twice for some reason. Compare this with a 'normal' tear-down:
The first example I gave is the last traffic on the wire before the server abends, so is of course highly suspicious. The pattern that leaps right out is that the responding server is issuing the FIN,PSH,ACK and RST,ACK pair, rather than the sending server, and doing so before the sending server can say "I got it" to the connection close packet.
Now I need to catch it in the act again to prove this theory.
I finally, finally managed to get a meaningful packet-capture after it fails, and I found some traffic that... doesn't look right. Take a look:
-> NCP Connection Destroy
<- R OK
<- FIN, PSH, ACK
<- RST, ACK
-> ACK (to R OK)
<- RST
Note the three packets in the middle. The responding server is tearing down the connection twice for some reason. Compare this with a 'normal' tear-down:
-> NCP Connection Destroy
<- R OK
-> FIN, PSH, ACK
<- ACK
-> RST, ACK
The first example I gave is the last traffic on the wire before the server abends, so is of course highly suspicious. The pattern that leaps right out is that the responding server is issuing the FIN,PSH,ACK and RST,ACK pair, rather than the sending server, and doing so before the sending server can say "I got it" to the connection close packet.
Now I need to catch it in the act again to prove this theory.
Friday, January 25, 2008
A needed patch.
Novell has released a patch for the "ConsoleOne sorting problem."
The sorting problem happens when you have eDir 8.8 installed. Suddenly C1 starts sorting things by creation date rather than as you've ever seen it before. This is... confusing. ConsoleOne 1.3h helped some of it for us, but not all. And now, we have a patch!
Let ConsoleOne Sort Correctly!
The sorting problem happens when you have eDir 8.8 installed. Suddenly C1 starts sorting things by creation date rather than as you've ever seen it before. This is... confusing. ConsoleOne 1.3h helped some of it for us, but not all. And now, we have a patch!
Let ConsoleOne Sort Correctly!
Wednesday, January 16, 2008
NetWare library patches
Novell recently split the libc and clib patches for NetWare. For a long time patches like "nwlib6a" included both. Now, they're split.
This just caused me a problem. It turns out that if you have libcsp6b (the LibC patch) applied and not nwlib6k (the CLib patch), there is an abend possibility. It happened yesterday. It turns out that in that case, a badly formed network broadcast can cause an abend. This caused three of my six cluster nodes to fall on their butts at the same time. That was fun. Strange (but good) thing is, I had already applied both patches to these three servers but hadn't gotten around to rebooting them yet. So, by killing themselves they actually fixed the problem.
The abend, key details:
Heh heh heh. Oops.
And now a bit of history. Long time NetWare admins can ignore this part.
Q: Why are there two C libraries?
CLIB is the library NetWare started with. It began life in the dark and misty past, probably in the late 1980's. It is the deepest, darkest bowels of NetWare from the era when Novell was it when it came to office networking. Being so old, its APIs are very mature. Applications developed against CLIB generally speaking just plain work.
CLIB is also depreciated since it is highly proprietary, and doesn't play well with others. "Just plain works" in this instance means an assumption of 8.3 names, with kludging to support long file names if at all possible. CLIB applications have a tendency to have IPX dependencies for no good reason.
LIBC was created, IIRC, around the release of NetWare 5.0 when it became possible for NetWare to operate in a "pure IP" environment. LIBC was designed with the concept of POSIX semantics in mind, which CLIB was not. LIBC was created from scratch with long file name support. By now, as of NetWare 6.5 SP7, most of the NetWare kernel is written against LIBC rather than CLIB.
As an example of LIBC vs CLIB, take the 'MyWeb' service this blog is served by. When I did this the first time, it was on NetWare 6.0, using Apache 1.3. Apache 1.3 was linked against CLIB and was very stable. The service notes for the Apache Modules I needed to run to make it work made it clear that supporting long file-names on remote servers was something that only recently started working.
When the migration to NetWare 6.5 came around, it meant I had to migrate MyWeb to Apache 2.0. Apache 2.0 is linked against LIBC and used a different apache module to make things work. I had troubles. The LibC functions were not nearly as mature as their CLIB counterparts, and it showed. 3.5 years later things are now a lot more stable then back then.
This just caused me a problem. It turns out that if you have libcsp6b (the LibC patch) applied and not nwlib6k (the CLib patch), there is an abend possibility. It happened yesterday. It turns out that in that case, a badly formed network broadcast can cause an abend. This caused three of my six cluster nodes to fall on their butts at the same time. That was fun. Strange (but good) thing is, I had already applied both patches to these three servers but hadn't gotten around to rebooting them yet. So, by killing themselves they actually fixed the problem.
The abend, key details:
EIP in SERVER.NLM at code start +0015FD27hHeh heh heh. Oops.
And now a bit of history. Long time NetWare admins can ignore this part.
Q: Why are there two C libraries?
CLIB is the library NetWare started with. It began life in the dark and misty past, probably in the late 1980's. It is the deepest, darkest bowels of NetWare from the era when Novell was it when it came to office networking. Being so old, its APIs are very mature. Applications developed against CLIB generally speaking just plain work.
CLIB is also depreciated since it is highly proprietary, and doesn't play well with others. "Just plain works" in this instance means an assumption of 8.3 names, with kludging to support long file names if at all possible. CLIB applications have a tendency to have IPX dependencies for no good reason.
LIBC was created, IIRC, around the release of NetWare 5.0 when it became possible for NetWare to operate in a "pure IP" environment. LIBC was designed with the concept of POSIX semantics in mind, which CLIB was not. LIBC was created from scratch with long file name support. By now, as of NetWare 6.5 SP7, most of the NetWare kernel is written against LIBC rather than CLIB.
As an example of LIBC vs CLIB, take the 'MyWeb' service this blog is served by. When I did this the first time, it was on NetWare 6.0, using Apache 1.3. Apache 1.3 was linked against CLIB and was very stable. The service notes for the Apache Modules I needed to run to make it work made it clear that supporting long file-names on remote servers was something that only recently started working.
When the migration to NetWare 6.5 came around, it meant I had to migrate MyWeb to Apache 2.0. Apache 2.0 is linked against LIBC and used a different apache module to make things work. I had troubles. The LibC functions were not nearly as mature as their CLIB counterparts, and it showed. 3.5 years later things are now a lot more stable then back then.
Labels: clustering, netware, novell, sysadmin
Friday, January 11, 2008
Disk-space over time
I've mentioned before that I do SNMP-based queries against NetWare and drop the resulting disk-usage data into a database. The current incarnation of this database went in August of 2004, so I have just over 4 years of data in it now. You can see some real trends in how we manage data in the charts.
To show you what I'm talking about, I'm going to post a chart based on the student-home-directory data. We have three home-directory volumes for students, which run between 7000-8000 home directories on them. We load-balance by number of directories rather than least-size. The chart:
As you can see, I've marked up our quarters. Winter/Spring is one segment on this chart since Spring Break is hard to isolate on these scales. We JUST started Winter 2008, so the last dot on the chart is data from this week. If you squint in (or zoom in like I can) you can see that last dot is elevated from the dot before it, reflecting this week's classes.
There are several sudden jumps on the chart. Fall 2005. Spring 2005. Spring 2007 was a big one. Fall 2007 just as large. These reflect student delete processes. Once a student hasn't been registered for classes for a specified period of time (I don't know what it is off hand, but I think 2 terms) their account goes on the 'ineligible' list and gets purged. We do the purge once a quarter except for Summer. The Fall purge is generally the biggest in terms of numbers, but not always. Sometimes the number of students purged is so small it doesn't show on this chart.
We do get some growth over the summer, which is to be expected. The only time when classes are not in session is generally from the last half of August to the first half of September. Our printing volumes are also w-a-y down during that time.
Because the Winter purge is so tiny, Winter quarter tends to see the biggest net-gain in used disk-space. Fall quarter's net-gain sometimes comes out a wash due to the size of that purge. Yet if you look at the slopes of the lines for Fall, correcting for the purge of course, you see it matches Winter/Spring.
Somewhere in here, and I can't remember where, we increased the default student directory-quota from 200MB to 500MB. We've found Directory Quotas to be a much better method of managing student directory sizes than User Quotas. If I remember my architectures right, directory quotas are only possible because of how NSS is designed.
If you take a look at the "Last Modified Times" chart in the Volume Inventory for one of the student home-directory volumes you get another interesting picture:
We have a big whack of data aged 12 months or newer. That said, we have non-trivial amounts of data aged 12 months or older. This represents where we'd get big savings when we move to OES2 and can use Dynamic Storage Technology (formerly known as 'shadowvolumes'). Because these are students and students only stick around for so long, we don't have a lot of stuff in the "older than 2 years" column that is very present on the Faculty/Staff volumes.
Being the 'slow, cheap,' storage device is a role well suited to the MSA1500 that has been plaguing me. If for some reason we fail to scare up funding to replace our EVA3000 with another EVA less filled-to-capacity, this could buy a couple of years of life on the EVA3000. Unfortunately, we can't go to OES2 until Novell ships an edirectory enabled AFP server for Linux, currently scheduled for late 2008 at the earliest.
Anyway, here is some insight into some of our storage challenges! Hope it has been interesting.
To show you what I'm talking about, I'm going to post a chart based on the student-home-directory data. We have three home-directory volumes for students, which run between 7000-8000 home directories on them. We load-balance by number of directories rather than least-size. The chart:
As you can see, I've marked up our quarters. Winter/Spring is one segment on this chart since Spring Break is hard to isolate on these scales. We JUST started Winter 2008, so the last dot on the chart is data from this week. If you squint in (or zoom in like I can) you can see that last dot is elevated from the dot before it, reflecting this week's classes.
There are several sudden jumps on the chart. Fall 2005. Spring 2005. Spring 2007 was a big one. Fall 2007 just as large. These reflect student delete processes. Once a student hasn't been registered for classes for a specified period of time (I don't know what it is off hand, but I think 2 terms) their account goes on the 'ineligible' list and gets purged. We do the purge once a quarter except for Summer. The Fall purge is generally the biggest in terms of numbers, but not always. Sometimes the number of students purged is so small it doesn't show on this chart.
We do get some growth over the summer, which is to be expected. The only time when classes are not in session is generally from the last half of August to the first half of September. Our printing volumes are also w-a-y down during that time.
Because the Winter purge is so tiny, Winter quarter tends to see the biggest net-gain in used disk-space. Fall quarter's net-gain sometimes comes out a wash due to the size of that purge. Yet if you look at the slopes of the lines for Fall, correcting for the purge of course, you see it matches Winter/Spring.
Somewhere in here, and I can't remember where, we increased the default student directory-quota from 200MB to 500MB. We've found Directory Quotas to be a much better method of managing student directory sizes than User Quotas. If I remember my architectures right, directory quotas are only possible because of how NSS is designed.
If you take a look at the "Last Modified Times" chart in the Volume Inventory for one of the student home-directory volumes you get another interesting picture:
We have a big whack of data aged 12 months or newer. That said, we have non-trivial amounts of data aged 12 months or older. This represents where we'd get big savings when we move to OES2 and can use Dynamic Storage Technology (formerly known as 'shadowvolumes'). Because these are students and students only stick around for so long, we don't have a lot of stuff in the "older than 2 years" column that is very present on the Faculty/Staff volumes.
Being the 'slow, cheap,' storage device is a role well suited to the MSA1500 that has been plaguing me. If for some reason we fail to scare up funding to replace our EVA3000 with another EVA less filled-to-capacity, this could buy a couple of years of life on the EVA3000. Unfortunately, we can't go to OES2 until Novell ships an edirectory enabled AFP server for Linux, currently scheduled for late 2008 at the earliest.
Anyway, here is some insight into some of our storage challenges! Hope it has been interesting.
Monday, January 07, 2008
I/O starvation on NetWare, another update
I've spoken before about my latency problems on the MSA1500cs. Since my last update I've spoken with Novell at length. Their own back-line HP people were thinking firmware issues to, and recommended I open another case with HP support. And if HP again tries to lay the blame on NetWare, to point their techs at the NetWare backline tech. Who will then have a talk about why exactly it is that NetWare isn't the problem in this case.
This time when I opened the case I mentioned that we see performance problems on the backup-to-disk server, which is Windows. Which is true, when the problem occurs B2D speeds drop through the floor; last Friday a 525GB backup that normally completes in 6 hours took about 50 hours. Since I'm seeing problems on more than one operating system, clearly this is a problem with the storage device.
The first line tech agreed, and escalated. The 2nd line tech said (paraphrased):
That may be where we have to go with this. Unfortunately, I don't think we have 16TB of disk-drives available to fully mirror the cluster. That'll be a significant expense. So, I think we have some rethinking to do regarding what we use this device for.
This time when I opened the case I mentioned that we see performance problems on the backup-to-disk server, which is Windows. Which is true, when the problem occurs B2D speeds drop through the floor; last Friday a 525GB backup that normally completes in 6 hours took about 50 hours. Since I'm seeing problems on more than one operating system, clearly this is a problem with the storage device.
The first line tech agreed, and escalated. The 2nd line tech said (paraphrased):
I'm seeing a lot of parity RAID LUNs out there. This sort of RAID uses CPU on the MSA1000 controllers, so the results you're seeing are normal for this storage system.Which, if true, puts the onus of putting up with a badly behaved I/O system onto NetWare again. The tech went on to recommend RAID1 for the LUNs that need high performance when doing array operations that disable the internal cache. Which, as far as I can figure, would work. We're not bottlenecking on I/O to the physical disks, the bottleneck is CPU on the MSA1000 controller that's active. Going RAID1 on the LUNs would keep speeds very fast even when doing array operations.
That may be where we have to go with this. Unfortunately, I don't think we have 16TB of disk-drives available to fully mirror the cluster. That'll be a significant expense. So, I think we have some rethinking to do regarding what we use this device for.
Wednesday, January 02, 2008
Where NetWare Fits
NetWare 6.5 still holds top honors in one server niche. Even though it is a 32-bit operating system. That niche is the "large file-server" segment. I define "large" as, "lots of data, way-lots of concurrent users". Yeah, that's highly scientific. But "way-lots" means "over 1000 concurrent" to my thinking.
We regularly run between 1200-6000 concurrent connections on our cluster nodes. This is a density that just doesn't happen all that often in the market. If you have 6000 users close enough together to all talk to the same file-server at LAN speeds using a protocol designed for file-serving (such as NCP, SMB/CIFS, or AFP), you're a big organization. 6000 is a large corporate campus, a large governmental entity of some kind, or a larger .EDU like us. Nationally, the number of 'large' file-servers like that is peanuts compared to the number of 'workgroup' (i.e. under 300 concurrent users) servers out there.
It is therefore no surprise to me that Novell is not devoting a lot of engineering to supporting the top end of this market. While it may pay well, there just isn't enough revenue coming from these customers to try and handle the hardest-to-test use-case: very high concurrency. I find it disappointing because I AM one of those customers (a larger .EDU), but I understand the business drivers supporting the decision.
For the moment, NetWare 6.5 (32bit) is the top-dog performance wise for our environment. That isn't going to stay true for much longer. It would not surprise me to find out that a Windows Enterprise Server (x86_64) with 16GB of RAM can out-perform a NetWare 6.5 (32bit) server with 4GB of RAM, simply due to the added room for a file-cache. What I don't know is how CPU-bound file-serving I/O is on a Windows Enterprise Server, that's the one area that could keep NetWare 6.5 (32bit) on top. I already know that OES2-Linux out-performs NetWare for NCP traffic, so long as you stay within CPU bounds.
For high-concurrency applications, as far as I know NetWare still wins.
We regularly run between 1200-6000 concurrent connections on our cluster nodes. This is a density that just doesn't happen all that often in the market. If you have 6000 users close enough together to all talk to the same file-server at LAN speeds using a protocol designed for file-serving (such as NCP, SMB/CIFS, or AFP), you're a big organization. 6000 is a large corporate campus, a large governmental entity of some kind, or a larger .EDU like us. Nationally, the number of 'large' file-servers like that is peanuts compared to the number of 'workgroup' (i.e. under 300 concurrent users) servers out there.
It is therefore no surprise to me that Novell is not devoting a lot of engineering to supporting the top end of this market. While it may pay well, there just isn't enough revenue coming from these customers to try and handle the hardest-to-test use-case: very high concurrency. I find it disappointing because I AM one of those customers (a larger .EDU), but I understand the business drivers supporting the decision.
For the moment, NetWare 6.5 (32bit) is the top-dog performance wise for our environment. That isn't going to stay true for much longer. It would not surprise me to find out that a Windows Enterprise Server (x86_64) with 16GB of RAM can out-perform a NetWare 6.5 (32bit) server with 4GB of RAM, simply due to the added room for a file-cache. What I don't know is how CPU-bound file-serving I/O is on a Windows Enterprise Server, that's the one area that could keep NetWare 6.5 (32bit) on top. I already know that OES2-Linux out-performs NetWare for NCP traffic, so long as you stay within CPU bounds.
For high-concurrency applications, as far as I know NetWare still wins.
Friday, December 28, 2007
NetWare and Hyperthreading, again
I talked about this one last year during some benchmarking. I just had a classic case crop up today.
It has long been consensus in the support forums that, (paraphrased) "If you have hyperthreading turned on and get an I/O thread stuck on a logical process, woe be unto you."
I have a server that I've been backing up for a fellow admin in another department. This particular server has 525GB of storage to back up, so it's going to take some time. It has been vexing figuring this one out. Until today, when I finally twigged to the fact that this server has HT turned on. I turn HT off as almost the first thing I do when setting up a server, so I don't think about it when troubleshooting.
Between 1000 and 1215 today, the backup got 882MB of data. Yeah, very crappy.
At 1215 I turned off the logical processors. This is a handy feature NetWare has, and I used it in the article I linked above.
At 1222 when I checked back the backup was up to 4.0GB.
At 1417 it is now up to 71GB backed up.
The only thing that changed was me turning off the logical processors. That's it. At that rate, this server should be backed up in around 15 hours, which is a far cry from the 30+ hours it was doing before.
Turn Hyperthreading off on your NetWare servers. Just do it.
It has long been consensus in the support forums that, (paraphrased) "If you have hyperthreading turned on and get an I/O thread stuck on a logical process, woe be unto you."
I have a server that I've been backing up for a fellow admin in another department. This particular server has 525GB of storage to back up, so it's going to take some time. It has been vexing figuring this one out. Until today, when I finally twigged to the fact that this server has HT turned on. I turn HT off as almost the first thing I do when setting up a server, so I don't think about it when troubleshooting.
Between 1000 and 1215 today, the backup got 882MB of data. Yeah, very crappy.
At 1215 I turned off the logical processors. This is a handy feature NetWare has, and I used it in the article I linked above.
At 1222 when I checked back the backup was up to 4.0GB.
At 1417 it is now up to 71GB backed up.
The only thing that changed was me turning off the logical processors. That's it. At that rate, this server should be backed up in around 15 hours, which is a far cry from the 30+ hours it was doing before.
Turn Hyperthreading off on your NetWare servers. Just do it.
Labels: hyperthreading, netware, novell, storage
Thursday, December 27, 2007
eDirectory certificate server changes
The new eDir 8.8 has introduced some changes into my environment, and from the looks of it some of them were there before I did the upgrade. Specifically to the CA in the tree. In googling around, I found this excerpt from the CA documentation:
With Certificate Server 3.2 and later, in order to completely backup the Certificate Authority, it is necessary to back up the CRL database and the Issued Certificates database. On NetWare, these files are located in the sys:system\certserv directory.
For other platforms, both of these databases are located in the same directory as the eDirectory dib files. The defaults for these locations are as follows:
Windows: c:\novell\nds\dibfiles
Linux/AIX/Solaris: /var/opt/novell/edirectory/data/dib
These defaults can be changed at the time that eDirectory is installed.
The files to back up for the CRL database are crl.db, crl.lck, crl.01 and the crl.rfl directory. The files to back up for the Issue Certificates database are cert.db, cert.lck, cert.01, and the cert.rfl directory.
I didn't know about that directory. I also didn't know that the CA is publishing a certificate-revocation-list to sys:apache2\htdocs\crl\. Time to twiddle the backup jobs.
Thursday, December 20, 2007
eDir 8.8, Priority Sync
One of the things that grabbed by attention with 8.8 is 'priority sync'. The documentation has an overview of it:
Looking at iMonitor I see this:
As 90-95% of our user objects are in either the root container or the students container, those are the statistics I'm interested in. The "maximum ring delta" number is very, very rarely over 30 seconds for these two partitions. With it being intersession right now, we're seeing some higher numbers than usual right now but it is still kept in close sync. As we have 24 hour computer labs, and a simple login causes several user-object attributes to update, we have a continual flow of directory changes. In our case, using Priority Sync would buy us a few seconds at most. We're not under any sort of regulatory mandate to do things 'instantly' so that isn't an issue, and our password-change process is well known to our end users for taking "up to 5 minutes".
Still, I like the idea even if it isn't a good fit for us.
6.0 Priority SyncWhich sounds spiffy. Instant sync of passwords? I'm all for that. Then I remembered, 'wasn't that happening already? That's right, that's the "SYNC_IMMEDIATE" flag in schema.' And that's what's described in this older CoolSolutions article.
Priority Sync is a new feature in Novell® eDirectory 8.8™ that is complimentary to the current synchronization process in eDirectory. Through Priority Sync, you can synchronize the modified critical data, such as passwords, immediately.
You can sync your critical data through Priority Sync when you cannot wait for normal synchronization. The Priority Sync process is faster than the normal synchronization process. Priority Sync is supported only between two or more eDirectory 8.8 or later servers hosting the same partition.
6.1 Need for Priority Sync
Normal synchronization can take some time, during which the modified data would not be available on other servers. For example, suppose that in your setup you have different applications talking to the directory. You change your password on Server1. With normal synchronization, it is some time before this change is synchronized with Server2. Therefore, a user would still be able to authenticate to the directory through an application talking to Server2, using the old password.
In large deployments, when the critical data of an object is modified, changes need to be synchronized immediately. The Priority Sync process resolves this issue.
Looking at iMonitor I see this:
As 90-95% of our user objects are in either the root container or the students container, those are the statistics I'm interested in. The "maximum ring delta" number is very, very rarely over 30 seconds for these two partitions. With it being intersession right now, we're seeing some higher numbers than usual right now but it is still kept in close sync. As we have 24 hour computer labs, and a simple login causes several user-object attributes to update, we have a continual flow of directory changes. In our case, using Priority Sync would buy us a few seconds at most. We're not under any sort of regulatory mandate to do things 'instantly' so that isn't an issue, and our password-change process is well known to our end users for taking "up to 5 minutes".
Still, I like the idea even if it isn't a good fit for us.
Wednesday, December 19, 2007
eDir 8.8 is in
And as far as upgrades go, it was pretty much a non-event.
Whenever you do upgrades like this you always wonder if those balls you're juggling are tennis-balls or grenades. It took about a half hour per server and didn't have any significant hitches. The one problem that did surface is that the OES1-linux server's LDAP server had its certificate change from the one it was using to SSL CertificateDNS. This was not good, as that certificate doesn't have the subject-name we need and this caused some S/LDAP binds to fail due to SSL validation problems. That was an easy fix. The LDAP servers on the NetWare boxes didn't change.
This was a tennis-ball upgrade. So far.
We haven't turned on case-sensitive LDAP binds yet, but soon. Soon.
One unexpected side-effect of getting all three eDir servers upgraded to 8.8 like this, is that the Change Cache is now cleaned of those permanent residents we've had for ages. Woo!
Whenever you do upgrades like this you always wonder if those balls you're juggling are tennis-balls or grenades. It took about a half hour per server and didn't have any significant hitches. The one problem that did surface is that the OES1-linux server's LDAP server had its certificate change from the one it was using to SSL CertificateDNS. This was not good, as that certificate doesn't have the subject-name we need and this caused some S/LDAP binds to fail due to SSL validation problems. That was an easy fix. The LDAP servers on the NetWare boxes didn't change.
This was a tennis-ball upgrade. So far.
We haven't turned on case-sensitive LDAP binds yet, but soon. Soon.
One unexpected side-effect of getting all three eDir servers upgraded to 8.8 like this, is that the Change Cache is now cleaned of those permanent residents we've had for ages. Woo!
Monday, December 17, 2007
Not dead.
Wow, last post was the 30th? Jeez. I was on vacation all last week, which accounts for some of it. And it's looking like I'll be out sick for at least a pair of days with a crud I got while wandering about. Not sharing that with work, nosir.
On my list of things to do during the winter inter-session is to get eDir 8.8 deployed in the production tree. I just need to have ALL the servers in the tree (all, not just replica holders due to backlink updates) up and talking when I do the first one, and that could take some scheduling. This is the first step to OES2, which will be deployed on the eDir servers first.
As soon as I get some new hardware, since they're getting old.
On my list of things to do during the winter inter-session is to get eDir 8.8 deployed in the production tree. I just need to have ALL the servers in the tree (all, not just replica holders due to backlink updates) up and talking when I do the first one, and that could take some scheduling. This is the first step to OES2, which will be deployed on the eDir servers first.
As soon as I get some new hardware, since they're getting old.
Friday, November 30, 2007
OES2 SP1 timing
Novell just posted the third draft of their OES2 Best Practices guide. Which you can locate here. In that guide is this text:
Another 21 months of a 32-bit operating system on the single biggest storage consumer on campus. We'll have at least one hardware refresh before then for some of the nodes, and... boy I hope they have NetWare drivers for that. The very limited testing I did with NetWare-in-Xen was not encouraging from a performance stand-point. If it looks like I'll have to deploy that way for the next servers we get in the cluster, I'll have to do more real testing to characterize the performance hit (if any). The idea of a 64-bit memory space for file-caching makes me drool. Not getting it for 21 months is painful.
That said, if Novell releases the eDirectory enabled AFP server for OES2-Linux outside of the service-pack I could still make the 2008 window. That's our only dependency for SP1.
Update (09/08/08): Looks like 'late October' is the date for SP1's release. Should be in public beta before then.
Update (12/03/08): It's out!
Domain Services for Windows, which is scheduled to ship with OES 2 SP1 (currently scheduled for late 2008), will also offer some clear advantages."Late 2008" means they WILL NOT have SP1 out by August of 2008. This means that the upgrade of our 6 node cluster to OES will have to wait until 2009. Grrarrr!
Another 21 months of a 32-bit operating system on the single biggest storage consumer on campus. We'll have at least one hardware refresh before then for some of the nodes, and... boy I hope they have NetWare drivers for that. The very limited testing I did with NetWare-in-Xen was not encouraging from a performance stand-point. If it looks like I'll have to deploy that way for the next servers we get in the cluster, I'll have to do more real testing to characterize the performance hit (if any). The idea of a 64-bit memory space for file-caching makes me drool. Not getting it for 21 months is painful.
That said, if Novell releases the eDirectory enabled AFP server for OES2-Linux outside of the service-pack I could still make the 2008 window. That's our only dependency for SP1.
Update (09/08/08): Looks like 'late October' is the date for SP1's release. Should be in public beta before then.
Update (12/03/08): It's out!
Wednesday, November 28, 2007
I/O starvation on NetWare, HP update
Last week I talked about a problem we're having with the HP MSA1500cs and our NetWare cluster. The problem is still there, of course. I've opened cases with both HP and Novell to handle this one. HP because I really thing that such command latencies are a defect, and Novell since they're having starvation issues with clusters.
This morning I got a voice-mail from HP, an update for our case. Greatly summarized:
Especially since I don't think I've made back-line on the Novell case yet. They're involved, but I haven't been referred to a new support engineer yet.
This morning I got a voice-mail from HP, an update for our case. Greatly summarized:
The MSA team has determined that your device is working perfectly, and can find no defects. They've referred the case to the NetWare software team.Or...
Working as designed. Fix your software. Talk to Novell.Which I'm doing. Now to see if I can light a fire on the back-channels, or if we've just made HP admit that these sorts of command latencies are part of the design and need to be engineered around in software. Highly frustrating.
Especially since I don't think I've made back-line on the Novell case yet. They're involved, but I haven't been referred to a new support engineer yet.
Labels: clustering, hp, msa, netware, novell, OES, storage, sysadmin
Wednesday, November 21, 2007
I/O starvation on NetWare
The MSA1500cs we've had for a while has shown a bad habit. It is visible when you connect a serial cable to the management port on the MSA1000 controller, and doing a "show perf" after starting performance tracking. The line in question is "Avg Command Latency:", which is a measure of how long it takes to execute an I/O operation. Under normal circumstances this metric stays between 5-30ms. When things go bad, I've seen it as far as 270ms.
This is a problem with our cluster nodes. Our cluster nodes can seen LUNs on both the MSA1500cs and the EVA3000. The EVA is where the cluster has been housed since it started, and the MSA has taken up two low-I/O-volume volumes to make space on the EVA.
IF the MSA is in the high Avg Command Latency state, and
IF a cluster node is doing a large Write to the MSA (such as a DVD ISO image, or B2D operation),
THEN "Concurrent Disk Requests" in Monitor go north of 1000
This is a dangerous state. If this particular cluster node is housing some higher trafficked volumes, such as FacShare:, the laggy I/O is competing with regular (fast) I/O to the EVA. If this sort of mostly-Read I/O is concurrent with the above heavy Write situation it can cause the cluster node to not write to the Cluster Partition on time and trigger a poison-pill from the Split Brain Detector. In short, the storage heart-beat to the EVA (where the Cluster Partition lives) gets starved out in the face of all the writes to the laggy MSA.
Users definitely noticed when the cluster node was in such a heavy usage state. Writes and Reads took a loooong time on the LUNs hosted on the fast EVA. Our help-desk recorded several "unable to map drive" calls when the nodes were in that state, simply because a drive-mapping involves I/O and the server was too busy to do it in the scant seconds it normally does.
This is sub-optimal. This also doesn't seem to happen on Windows, but I'm not sure of that.
This is something that a very new feature in the Linux kernel could help out, that that's to introduce the concept of 'priority I/O' to the storage stack. I/O with a high priority, such as cluster heart-beats, gets serviced faster than I/O of regular priority. That could prevent SBD abends. Unfortunately, as the NetWare kernel is no longer under development and just under Maintenance, this is not likely to be ported to NetWare.
I/O starvation. This shouldn't happen, but neither should 270ms I/O command latencies.
This is a problem with our cluster nodes. Our cluster nodes can seen LUNs on both the MSA1500cs and the EVA3000. The EVA is where the cluster has been housed since it started, and the MSA has taken up two low-I/O-volume volumes to make space on the EVA.
IF the MSA is in the high Avg Command Latency state, and
IF a cluster node is doing a large Write to the MSA (such as a DVD ISO image, or B2D operation),
THEN "Concurrent Disk Requests" in Monitor go north of 1000
This is a dangerous state. If this particular cluster node is housing some higher trafficked volumes, such as FacShare:, the laggy I/O is competing with regular (fast) I/O to the EVA. If this sort of mostly-Read I/O is concurrent with the above heavy Write situation it can cause the cluster node to not write to the Cluster Partition on time and trigger a poison-pill from the Split Brain Detector. In short, the storage heart-beat to the EVA (where the Cluster Partition lives) gets starved out in the face of all the writes to the laggy MSA.
Users definitely noticed when the cluster node was in such a heavy usage state. Writes and Reads took a loooong time on the LUNs hosted on the fast EVA. Our help-desk recorded several "unable to map drive" calls when the nodes were in that state, simply because a drive-mapping involves I/O and the server was too busy to do it in the scant seconds it normally does.
This is sub-optimal. This also doesn't seem to happen on Windows, but I'm not sure of that.
This is something that a very new feature in the Linux kernel could help out, that that's to introduce the concept of 'priority I/O' to the storage stack. I/O with a high priority, such as cluster heart-beats, gets serviced faster than I/O of regular priority. That could prevent SBD abends. Unfortunately, as the NetWare kernel is no longer under development and just under Maintenance, this is not likely to be ported to NetWare.
I/O starvation. This shouldn't happen, but neither should 270ms I/O command latencies.
Labels: clustering, hp, msa, netware, novell, OES, storage, sysadmin
Monday, October 15, 2007
Peer-to-peer sharing
One feature that has shown up in some applications and widgets lately has gained some traction internally. That is the concept of peer to peer sharing of disk space without going through all the pain of getting things approved and formally set up. The general idea is this one.
I want to share U:\SharedStuff\ApacheGroup\ to five other users. U: is my home directory, which is actually map-rooted so I don't see the top level directory. So I go to a web page and tell it I want to share this directory, to these people, for this long. Go.
It struck me that this sort of thing can be engineered with NetWare and OES. The key components are eDirectory, NSS, and NetStorage.
The web server takes the request and translates $Path into a real path by referencing the HomeDirectory attribute of the user who requested the share. Then, using LDAP it creates two objects:
A Group Object
There is a small constellation of maintenance tasks that also need to be created, such as a janitor process to deal with expirations, a helpdesk view to track who has what shares, a historic view to see what shares got deleted recently that suddenly need to be back RIGHT NOW, something to interface this with whatever disk or directory quota systems are in use.
The use of NetStorage allows WebDAV to be used as an access method, which allows the shares to be seen. The really brave may be able to leverage DFS to create actual directory structures reflecting the shares in the actual directories so drive mappings can be used; unfortunately I have no idea if a DFS database that large is a good idea.
Users would love this. No need to go through management to get a directory set up on the shared space. You just set up and go. Great for adhoc groups, or small private gatherings.
Unfortunately, this sort of share model is one that a lot of sys-admins are familiar with. If you've ever had a chance to examine the network of a small business with under 15 users, all of whom call themselves 'not that good with computers', you know what I'm talking about. This model of sharing is the one that Windows for Workgroups was designed for, and is still the default mode for plain old WinXP. Excessive use of peer to peer sharing like that can lead to one unholy mess, especially if a key person leaves (or in the case of the Windows example, one hard drive crashes hard).
If left unchecked, you can get whole business processes designed with the assumption that [username] will never retire. That already happens to an alarming extent, but this would make the dependency more invisible to those of us charged with making it all work again when it breaks. You can have shared spaces that are business critical to the company living 100% inside a user's self-managed space, and vulnerable to deletion on termination of that employee.
This is all part of the balance we as system administrators have to keep between end user functionality, and data protection. Desktop techs fight a constant battle to get users to save data on the server where it is backed up, and Novell puts out things like iFolder to help that whole thing become more invisible. We created shared directories to draw a big line between 'my stuff' and 'us stuff'.
That said, data-access habits are changing all the time. My own boss prefers to email a 150KB Excel spreadsheet to all of us, even though all of us have ready access a shared directory setup just for that. SharePoint integrates with Office to make the web-server look like a file-server. We still have to adapt with the times.
User-directed sharing is something I can see as highly desirable among the student population and faculty as well. Among staff, I'm less sure its a good idea outside of the 'trivial' personal use we're allowed.
I want to share U:\SharedStuff\ApacheGroup\ to five other users. U: is my home directory, which is actually map-rooted so I don't see the top level directory. So I go to a web page and tell it I want to share this directory, to these people, for this long. Go.
It struck me that this sort of thing can be engineered with NetWare and OES. The key components are eDirectory, NSS, and NetStorage.
The web server takes the request and translates $Path into a real path by referencing the HomeDirectory attribute of the user who requested the share. Then, using LDAP it creates two objects:
A Group Object
- Created and named dynamically
- [AuxClass] Attribute with user-defined name
- [AuxClass] Attribute with the creator
- [AuxClass] Attribute with the expiry date
- Since this is eDirectory, group memberships apply immediately rather than taking a logout/login cycle to refresh the access token like in MS networks.
- Created & named dynamically
- Associated to the created group
- Assigned to the specified users
- This allows the share to show up in NetStorage
There is a small constellation of maintenance tasks that also need to be created, such as a janitor process to deal with expirations, a helpdesk view to track who has what shares, a historic view to see what shares got deleted recently that suddenly need to be back RIGHT NOW, something to interface this with whatever disk or directory quota systems are in use.
The use of NetStorage allows WebDAV to be used as an access method, which allows the shares to be seen. The really brave may be able to leverage DFS to create actual directory structures reflecting the shares in the actual directories so drive mappings can be used; unfortunately I have no idea if a DFS database that large is a good idea.
Users would love this. No need to go through management to get a directory set up on the shared space. You just set up and go. Great for adhoc groups, or small private gatherings.
Unfortunately, this sort of share model is one that a lot of sys-admins are familiar with. If you've ever had a chance to examine the network of a small business with under 15 users, all of whom call themselves 'not that good with computers', you know what I'm talking about. This model of sharing is the one that Windows for Workgroups was designed for, and is still the default mode for plain old WinXP. Excessive use of peer to peer sharing like that can lead to one unholy mess, especially if a key person leaves (or in the case of the Windows example, one hard drive crashes hard).
If left unchecked, you can get whole business processes designed with the assumption that [username] will never retire. That already happens to an alarming extent, but this would make the dependency more invisible to those of us charged with making it all work again when it breaks. You can have shared spaces that are business critical to the company living 100% inside a user's self-managed space, and vulnerable to deletion on termination of that employee.
This is all part of the balance we as system administrators have to keep between end user functionality, and data protection. Desktop techs fight a constant battle to get users to save data on the server where it is backed up, and Novell puts out things like iFolder to help that whole thing become more invisible. We created shared directories to draw a big line between 'my stuff' and 'us stuff'.
That said, data-access habits are changing all the time. My own boss prefers to email a 150KB Excel spreadsheet to all of us, even though all of us have ready access a shared directory setup just for that. SharePoint integrates with Office to make the web-server look like a file-server. We still have to adapt with the times.
User-directed sharing is something I can see as highly desirable among the student population and faculty as well. Among staff, I'm less sure its a good idea outside of the 'trivial' personal use we're allowed.
Wednesday, September 26, 2007
OES2 release date
Just got out of the WebCast they had. First, the important stuff:
OES2 will be released on October 5th.
OES2-SP1 is targeted for mid-April, 2008.
AFP integration will be in SP1.
I sooooooooo hope they don't push SP1 past July. If that happens, my main migration of our cluster will have to be pushed to 2009. Ick. We're already running out of effective file-cache in 32-bit memory space. I need 64-bit to really give good performance. Hope hope hope.
A few other minor points:
OES2 will be released on October 5th.
OES2-SP1 is targeted for mid-April, 2008.
AFP integration will be in SP1.
I sooooooooo hope they don't push SP1 past July. If that happens, my main migration of our cluster will have to be pushed to 2009. Ick. We're already running out of effective file-cache in 32-bit memory space. I need 64-bit to really give good performance. Hope hope hope.
A few other minor points:
- Around the release of SP1, Prosoft and Condrey Consulting (Kanaka) will release an NCP client for Mac.
- The clearing of throats next to a mic is a sign of someone who doesn't do a lot of work in front of mics.
- OES2 is fully 64-bit optimized (on Linux)
- They claim EVEN BETTER NSS performance on OES2. I hope to try that out, soon as I can figure out how to get SLES10/OES2-beta5 to talk to my SAN luns. It hates me.
Tuesday, September 25, 2007
OES2 Web-chat tomorrow
This isn't exactly widely spread, but here it is:
Open Enterprise Server 2 Live Webcast
Tomorrow, September 26th at 11AM PDT.
They'll be talking about all the spiffy thats in OES2, and some new info about code releases. I think this is the 'event' they mentioned a while back.
Open Enterprise Server 2 Live Webcast
Tomorrow, September 26th at 11AM PDT.
They'll be talking about all the spiffy thats in OES2, and some new info about code releases. I think this is the 'event' they mentioned a while back.
Monday, September 24, 2007
Mod_edir issues again
As I mentioned last week, I'm seeing a LOT of connections related to mod_edir. Late Friday I updated to the 1.0.13 build, which updates from 1.0.12 that comes with SP6. That doesn't seem to have fixed the problem. On the plus side, it would seem that the mod_edir developers know of this problem. On the down side, I don't see a fix.
Right now I'm suspecting libc, as that's been my problem in the past. Perhaps the connection tear-down code in mod_edir isn't "taking" somehow.
Unfortunately, I'm not sure if I can call in an incident against mod_edir, or if I'll have to work with the devs (somehow) and call in against libc. If I reboot the web-servers every couple of days that causes the connections to close, but that is not a fun solution.
Right now I'm suspecting libc, as that's been my problem in the past. Perhaps the connection tear-down code in mod_edir isn't "taking" somehow.
Unfortunately, I'm not sure if I can call in an incident against mod_edir, or if I'll have to work with the devs (somehow) and call in against libc. If I reboot the web-servers every couple of days that causes the connections to close, but that is not a fun solution.
Thursday, September 20, 2007
That... is a lot of connections!
Oh, NetWare fans, take a look at this:
Yep. Check the Concurrent Connections number. That is a very big number. During term we run between 1500 and 4000 concurrent connections. Yet... that is way above that. What's more, going into the Novell Remote Manager, I find this pair of very interesting numbers:
Connection Slots Allocated: 44000
Connection Slots Being Used: 43982
Looking at the connections shows me what the problem is. All those 'extra' connections are for the user account that allows MyWeb (what you're reading this through ultimately) to work. Somehow... and this is a guess... mod_edir seems to be creating a new connection for each request coming in, rather than reusing the old ones. Or perhaps it isn't cleaning up after itself. Probably since I put SP6 in.
This would explain why this particular server has an unreasonably high memory allocation to CONNMGR. Must Poke More.
Yep. Check the Concurrent Connections number. That is a very big number. During term we run between 1500 and 4000 concurrent connections. Yet... that is way above that. What's more, going into the Novell Remote Manager, I find this pair of very interesting numbers:Connection Slots Allocated: 44000
Connection Slots Being Used: 43982
Looking at the connections shows me what the problem is. All those 'extra' connections are for the user account that allows MyWeb (what you're reading this through ultimately) to work. Somehow... and this is a guess... mod_edir seems to be creating a new connection for each request coming in, rather than reusing the old ones. Or perhaps it isn't cleaning up after itself. Probably since I put SP6 in.
This would explain why this particular server has an unreasonably high memory allocation to CONNMGR. Must Poke More.
Tuesday, September 18, 2007
OES2: clustering
I made a cluster inside Xen! Two NetWare VM's inside a Xen container. I had to use a SAN LUN as the shared device since I couldn't make it work doing it just to a single file. Not sure what's up with that. But, it's a cluster, the volume moves between the two just fine.
Another thing about speeds, now that I have some data to play with. I copied a bunch of user directory data over to the shared LUN. It's a piddly 10GB LUN so it filled quick. That's OK, it should give me some ideas of transfer times. Doing a TSATEST backup from one cluster-node to the other (i.e. inside the Xen bridge) gave me speeds on the order of 1000MB/Min. Doing a TSATEST backup from a server in our production tree to the cluster node (i.e. over the LAN) gave me speeds of about 350MB/Min. Not so good.
For comparison, doing a TSATEST backup from the same host only drawing data from one of the USER volumes on the EVA (highly fragmented, but must faster, storage) gives a rate of 550 MB/Min.
I also discovered the VAST DIFFERENCES between our production eDirectory tree, which has been in existence since 1995 if the creation timestamp on the tree object is to be believed, and the brand new eDir 8.8 tree the OES2 cluster is living in. We have a heckova lot more attributes and classes in the prod tree than in this new one. Whoa. It made for some interesting challenges when importing users into it.
Another thing about speeds, now that I have some data to play with. I copied a bunch of user directory data over to the shared LUN. It's a piddly 10GB LUN so it filled quick. That's OK, it should give me some ideas of transfer times. Doing a TSATEST backup from one cluster-node to the other (i.e. inside the Xen bridge) gave me speeds on the order of 1000MB/Min. Doing a TSATEST backup from a server in our production tree to the cluster node (i.e. over the LAN) gave me speeds of about 350MB/Min. Not so good.
For comparison, doing a TSATEST backup from the same host only drawing data from one of the USER volumes on the EVA (highly fragmented, but must faster, storage) gives a rate of 550 MB/Min.
I also discovered the VAST DIFFERENCES between our production eDirectory tree, which has been in existence since 1995 if the creation timestamp on the tree object is to be believed, and the brand new eDir 8.8 tree the OES2 cluster is living in. We have a heckova lot more attributes and classes in the prod tree than in this new one. Whoa. It made for some interesting challenges when importing users into it.
Labels: clustering, netware, novell, OES, virtualization
OES2-beta progress
As mentioned before, I have the OES2 beta. Right now I have two NetWare servers parked in Xen VM's on SLES10SP1. This is how it is supposed to work!
I haven't gotten very far in my testing, but a few things are showing. I managed to do a TSATEST-based throughput run of a backup of SYS. That's about a gig of data. Throughputs for just one stream to one of the servers was around 500 MB/min, which is passible and within the realm of real performance for slower hardware. The downside of that is that the CPU reported by "xm top" was around 45%, where the CPU reported in MONITOR was closer to 25%. That's way higher than I expected, but could be related to all the disk I/O ops. This I/O was to a file in the file-system, not a physical device like a LUN on the SAN (that comes later).
Now I'm trying to get Novell Cluster Services installed. I want to get a weensy 2-node cluster set up to prove that it can be done. I suspect it can, but actually seeing it will be very nice.
I haven't gotten very far in my testing, but a few things are showing. I managed to do a TSATEST-based throughput run of a backup of SYS. That's about a gig of data. Throughputs for just one stream to one of the servers was around 500 MB/min, which is passible and within the realm of real performance for slower hardware. The downside of that is that the CPU reported by "xm top" was around 45%, where the CPU reported in MONITOR was closer to 25%. That's way higher than I expected, but could be related to all the disk I/O ops. This I/O was to a file in the file-system, not a physical device like a LUN on the SAN (that comes later).
Now I'm trying to get Novell Cluster Services installed. I want to get a weensy 2-node cluster set up to prove that it can be done. I suspect it can, but actually seeing it will be very nice.
Labels: clustering, netware, novell, OES
Thursday, September 13, 2007
OES2: virtualization
I have the beta up and running. I have a pair of OES2-NW servers running in Xen on SLES10SP1. And it loads just spiffy. Haven't done any performance testing on it, kind of hard to really interpret results at this point anyway.
What I HAVE been spending time on is seeing if it is possible to get a cluster set up. Clusters, of course, rely on shared storage. And if it works the way I need it to work, I need multiple Xen machines talking to the same LUNs. It may be doable, but I'm having a hard time figuring it out. The documentation on Xen isn't what you'd call complete. Novell has some in the SLES10SP1 documentation, but the stuff in the OES2 documentation is... decidedly overview-oriented. This is the most annoying thing, as I can't just put my nose to a manual and find it.
So, looking for Xen manual. It has to be around somewhere. Google-foo failed me today.
What I HAVE been spending time on is seeing if it is possible to get a cluster set up. Clusters, of course, rely on shared storage. And if it works the way I need it to work, I need multiple Xen machines talking to the same LUNs. It may be doable, but I'm having a hard time figuring it out. The documentation on Xen isn't what you'd call complete. Novell has some in the SLES10SP1 documentation, but the stuff in the OES2 documentation is... decidedly overview-oriented. This is the most annoying thing, as I can't just put my nose to a manual and find it.
So, looking for Xen manual. It has to be around somewhere. Google-foo failed me today.
Labels: netware, novell, OES, virtualization
Tuesday, September 11, 2007
What's been keeping me up late
Unfortunately, I haven't had time to crack open the OES2 beta. I'd really like to, but no time for that.
Instead what's been getting my attention are dsrepair.log entries like this:
ERROR: Inconsistent: Transitive Vector on partitionID: 00008204, DN: OU=[not that one].O=wwu.T=WWU
=>> Purging: 6 invalid entries from partition: OU=[not that one].O=wwu.T=WWU
=>> on server: CN=HERA
(1)Time stamp: 6-27-2006 3:12:46 pm; rep # = 0004; event = 0001
(2)Time stamp: 7-08-2004 9:00:57 am; rep # = 0005; event = 0001
(3)Time stamp: 5-09-2055 8:31:14 pm; rep # = 13F7; event = 0000
(4)Time stamp: 10-09-2002 1:37:16 pm; rep # = 0006; event = 0000
(5)Time stamp: 2-24-2070 7:43:44 am; rep # = 0070; event = 0034
(6)Time stamp: 8-25-1999 4:03:01 pm; rep # = 0007; event = 0000
=>> Updated: Transitive Vector on partition: OU=[not that one].O=wwu.T=WWU,
=>> with: 3 entries for server: CN=HERA
(1)Time stamp: 8-16-2007 12:02:19 pm; rep # = 0001; event = 0004
(2)Time stamp: 9-10-2007 9:56:11 pm; rep # = 0002; event = 0028
(3)Time stamp: 9-10-2007 10:00:03 pm; rep # = 0003; event = 0003
Those weren't alone. In fact, all of our partitions had some invalid transitive vectors. I get to go on another stomp fest tonight. Maybe this time they'll clear.
Instead what's been getting my attention are dsrepair.log entries like this:
ERROR: Inconsistent: Transitive Vector on partitionID: 00008204, DN: OU=[not that one].O=wwu.T=WWU
=>> Purging: 6 invalid entries from partition: OU=[not that one].O=wwu.T=WWU
=>> on server: CN=HERA
(1)Time stamp: 6-27-2006 3:12:46 pm; rep # = 0004; event = 0001
(2)Time stamp: 7-08-2004 9:00:57 am; rep # = 0005; event = 0001
(3)Time stamp: 5-09-2055 8:31:14 pm; rep # = 13F7; event = 0000
(4)Time stamp: 10-09-2002 1:37:16 pm; rep # = 0006; event = 0000
(5)Time stamp: 2-24-2070 7:43:44 am; rep # = 0070; event = 0034
(6)Time stamp: 8-25-1999 4:03:01 pm; rep # = 0007; event = 0000
=>> Updated: Transitive Vector on partition: OU=[not that one].O=wwu.T=WWU,
=>> with: 3 entries for server: CN=HERA
(1)Time stamp: 8-16-2007 12:02:19 pm; rep # = 0001; event = 0004
(2)Time stamp: 9-10-2007 9:56:11 pm; rep # = 0002; event = 0028
(3)Time stamp: 9-10-2007 10:00:03 pm; rep # = 0003; event = 0003
Those weren't alone. In fact, all of our partitions had some invalid transitive vectors. I get to go on another stomp fest tonight. Maybe this time they'll clear.
Monday, September 10, 2007
OES2 public beta is out
Jason Williams said so.
This looks to be Beta5. They released both the Linux and NetWare parts of it. The NW65SP7 overlay iso is 1.1GB in size. I sooooooooooooooooooooo gotta get DVD drives into my servers.
Rumor has it release is now mid-October. So who knows what's going on with the 'launch' on the 26th.
This looks to be Beta5. They released both the Linux and NetWare parts of it. The NW65SP7 overlay iso is 1.1GB in size. I sooooooooooooooooooooo gotta get DVD drives into my servers.
Rumor has it release is now mid-October. So who knows what's going on with the 'launch' on the 26th.
Friday, September 07, 2007
The mystery of the OES2 release date
Various sources have pointed at evidence that Novell will be launching OES2 on the 26th. As has been pointed out, "Launch" and "Release" are different things. And yet, and the same time rumor has it to "watch for events this Monday".
I don't know what to make of that.
It COULD be that the open beta will be out Monday. I have doubts about that, as that leaves very little time for reports to come back from the field for incorporation into OES2-release, presumably on or about the 26th.
It COULD be that it'll be released Monday, and the major PR push for launch will be two and a half weeks later. I have my doubts about that, Novell will be scooped by the likes of me as we put the new product to the test, but it could happen.
It COULD be that Monday is a red herring and Novell will announce a ship date on the 26th, and the opening of the beta. I put more stock into this possibility. The likes of me will swoop up the beta code, run it through its paces and send feedback about what we manage to break, for a presumed ship of OES in November or so.
Or it could be none of these. I guess we'll find out Monday or something.
I don't know what to make of that.
It COULD be that the open beta will be out Monday. I have doubts about that, as that leaves very little time for reports to come back from the field for incorporation into OES2-release, presumably on or about the 26th.
It COULD be that it'll be released Monday, and the major PR push for launch will be two and a half weeks later. I have my doubts about that, Novell will be scooped by the likes of me as we put the new product to the test, but it could happen.
It COULD be that Monday is a red herring and Novell will announce a ship date on the 26th, and the opening of the beta. I put more stock into this possibility. The likes of me will swoop up the beta code, run it through its paces and send feedback about what we manage to break, for a presumed ship of OES in November or so.
Or it could be none of these. I guess we'll find out Monday or something.
Friday, August 31, 2007
SSL puzzler resolved!
The anonymous commenter on the previous post nailed it. Disabling TLS in firefox & seamonkey removed the delay. Now that they've mentioned it, I knew this! This is one of the reasons why we're discouraging students from using Vista! I just didn't know that the newer Mozilla browsers were also affected by NetWare's inability to do TLSv1.
Here's an interesting thing
Novell is putting together a Best Practices guide for migrating to OES2 from NetWare. Obviously this is OES2-Linux, as there is not much that needs migrating when going from OES-NW to OES2-NW. They're soliciting community input for the guides, and will be offering Cool Solutions reward points for contributions.
This is interesting. I know that the Novell Support Forum Sysops tend to build up their own micro guides based on problems people report in the forums, and this is a way to better formalize that. Some of the sysops have taken to using the Cool Solutions Wiki as a place to park boiler-plate answers and forward questioners to those pages. This is an interesting concept.
More interesting as OES2 isn't out yet, even in an open-beta form. Where are we going to get our experience from, eh? This implies that shortly we'll have at least an open beta to try out. I hope so.
I can't contribute much to this document because my main migration is contingent on AFP being eDir integrated, and they've said that'll not happen until probably SP1. If I do anything it'll be the eDir servers, and those are relatively easy migrations. DFS is the only sticking point for that.
This is interesting. I know that the Novell Support Forum Sysops tend to build up their own micro guides based on problems people report in the forums, and this is a way to better formalize that. Some of the sysops have taken to using the Cool Solutions Wiki as a place to park boiler-plate answers and forward questioners to those pages. This is an interesting concept.
More interesting as OES2 isn't out yet, even in an open-beta form. Where are we going to get our experience from, eh? This implies that shortly we'll have at least an open beta to try out. I hope so.
I can't contribute much to this document because my main migration is contingent on AFP being eDir integrated, and they've said that'll not happen until probably SP1. If I do anything it'll be the eDir servers, and those are relatively easy migrations. DFS is the only sticking point for that.
Thursday, August 30, 2007
An SSL puzzler
One thing I've noticed lately is that hitting NetWare SSL webpages gives me a 20-60 second lag if I hit them with Seamonkey or Firefox. IE6 doesn't give the same lag. In order to see what's happening at the network level I broke out Wireshark.
Weirdly, the IE6 trace has 6 packets until the SSLv3 Server Hello, and the Seamonkey trace is 16 packets (and a big delay) until then. Some other differences in the Seamonkey trace (firefox shows the same delay, so I'm assuming similar reasons):
Another difference in the traces is that the first SSLv3 Client Hello in the Seamonkey trace includes 28 Cipher Suites, to IE's 11. Wireshark can only identify 12 of them (for the curious, most of the identifiable ciphers are different than the IE ones). I can only suppose that the NetWare SSL provider gets this Hello and goes +++OUT OF CHEESE ERROR+++ and waits to get more sensible data.
This is a tricky one. Tomorrow I delve into the Novell KB database and see if I can find anything like it. And if that and delving the support forums fails, a call in.
PS: I'd post some packet traces, but wireshark here on openSUSE 10.2 is crashing hard everytime I try and bring up a 'browse files' window. This makes saving traces difficult.
Weirdly, the IE6 trace has 6 packets until the SSLv3 Server Hello, and the Seamonkey trace is 16 packets (and a big delay) until then. Some other differences in the Seamonkey trace (firefox shows the same delay, so I'm assuming similar reasons):
- Uniformly, packet 6 in the Seamonkey trace is a FIN, ACK from the client
- Packets 7-10 are connection tear-down
- Packets 11-13 are connection setup
- Packet 14 is an SSLv2 Client Hello (it was SSLv3 up there in packet 4)
- Packet 15 is an ACK from the server
- Packet 16 is the SSLv3 Server Hello
Another difference in the traces is that the first SSLv3 Client Hello in the Seamonkey trace includes 28 Cipher Suites, to IE's 11. Wireshark can only identify 12 of them (for the curious, most of the identifiable ciphers are different than the IE ones). I can only suppose that the NetWare SSL provider gets this Hello and goes +++OUT OF CHEESE ERROR+++ and waits to get more sensible data.
This is a tricky one. Tomorrow I delve into the Novell KB database and see if I can find anything like it. And if that and delving the support forums fails, a call in.
PS: I'd post some packet traces, but wireshark here on openSUSE 10.2 is crashing hard everytime I try and bring up a 'browse files' window. This makes saving traces difficult.
Wednesday, August 29, 2007
Dynamic Storage Technology, more data
Two days ago Novell posted an AppNote on Dynamic Storage Technology, formerly known as 'shadow volumes'.
Setting up Dynamic Storage Technology with Open Enterprise Server 2
One thing I noticed right at the top of the article is a little blurb that reads:
Anyway, back to the article. This is a click-by-click guide for setting up DST. This includes screenshots, which are of the new iManager 2.7. Unsurprisingly, Novell re-themed the iManager interface. There is a gotcha on step 17, where you have to edit a local config file on the OES server to get it going, that would probably trip up most people trying to set up DST by going solely on looking at the UI.
This is a very good article describing it all. I recommend it!
Setting up Dynamic Storage Technology with Open Enterprise Server 2
One thing I noticed right at the top of the article is a little blurb that reads:
This article was written for Novell Open Enterprise Server 2. Sign up here to be notified when the Novell Open Enterprise Server 2 open beta becomes available.Which tells me that the public beta is probably pretty near, and that OES2 release will probably not be "end of Q3" like Jason Williams indicated a while back. I could be wrong, of course. As soon as I get the public beta code there is some serious testing I need to do.
Anyway, back to the article. This is a click-by-click guide for setting up DST. This includes screenshots, which are of the new iManager 2.7. Unsurprisingly, Novell re-themed the iManager interface. There is a gotcha on step 17, where you have to edit a local config file on the OES server to get it going, that would probably trip up most people trying to set up DST by going solely on looking at the UI.
This is a very good article describing it all. I recommend it!
Labels: netware, novell, OES, shadowvolumes
Thursday, August 23, 2007
Politics of passwords
It has been a common theme here for some years now to increase our password security. Two years ago (or was it three?) we rolled out Universal Passwords in an effort to gain more flexibility in the passwords we support. We've had password sync between Novell, AD, and Solaris for y-e-a-r-s, so our users have grown used to single sign-on. I've talked a couple of times about password complexity and how it works in a multiple system environment, twice in October (the 16th, and 17th). I've even talked about why Novell had to resort to Universal Passwords, because the NDS Password was too secure.
When we got our new Vice Provost, we got a person who wasn't familiar with the history of our organization. These sorts of things are always a mixed blessing. In this case, he wanted to get password aging going. The previous incumbent had considered it, but the project was on perma-hold while he worked certain political issues. The new guy managed to make a convincing argument to the University leadership, and the fiat to do password aging came down from the very top. And So It Shall Be. And Is. As with our existing password sync systems, this is a system we built from internal components and uses no Novell IDM stuff at all. It works for us.
Yesterday we got asked to make certain that the Novell password was case-sensitive.
I thought it already was, as Universal Passwords are case sensitive. But testing showed that you could set a mixed-case password on an account, and log in to Novell with the lower-case password. It won't allow workstation login on domained PCs as the AD password is mixed-case. In the case of students who only ever login using web-services, they sometimes got a shock when using a lab for the first time and the password they'd been using for months didn't work.
There are two things working against us here.
Looking at what would break if we turn NDS passwords off, I got a large list. We have some older servers in the tree (NetWare 6.0, and one lone NetWare 5.1 out there), and some console utilities would just plain break. Plus, at least one of us is still using ArcServe of an unknown version and I have zero clue if that would break if we remove NDS passwords (I'm guessing so, but I have no proof). Also, all older clients, such as the DOS boot disks used by our desktop group for imaging and any lingering Win9x we have out there, would break. Not Good.
The list of what'll break if we go to eDir 8.8 is shorter. As that allows the continued setting of the NDS Password, the amount of broken things out there is reduced. We'll have to put a specific dsrepair.nlm on all servers in the tree, but that is easier than working around breaking things. So, we're going to go to eDir 8.8.
This is not without its own problems, as some things DO still break. That lone NetWare 5.1 server will have to go. I've been assured that it is redundant and can go, but it'll need to ACTUALLY go. The NetWare 6.0 servers should be fine, as they're all at a DS rev that'll work with 8.8. Some of the 8.7.3 servers are still at 8.7.3.0 and should get updated for safety's sake. Also, all administrative workstations need to have NICI 2.7.x installed on them in order to understand the new eDir dialect, but that's a minor detail.
We won't be able to take advantage of some of the other nifty things eDir 8.8 introduces, as were still 95% NetWare when it comes to replica holders. Encrypted replication and multiple eDir instances will have to wait.
I HOPE to get eDir 8.8 in before class start, as the downtime required for DIB conversion is not trivial, and the first 4 weeks of class are always pretty hard on the DS servers due to updates.
When we got our new Vice Provost, we got a person who wasn't familiar with the history of our organization. These sorts of things are always a mixed blessing. In this case, he wanted to get password aging going. The previous incumbent had considered it, but the project was on perma-hold while he worked certain political issues. The new guy managed to make a convincing argument to the University leadership, and the fiat to do password aging came down from the very top. And So It Shall Be. And Is. As with our existing password sync systems, this is a system we built from internal components and uses no Novell IDM stuff at all. It works for us.
Yesterday we got asked to make certain that the Novell password was case-sensitive.
I thought it already was, as Universal Passwords are case sensitive. But testing showed that you could set a mixed-case password on an account, and log in to Novell with the lower-case password. It won't allow workstation login on domained PCs as the AD password is mixed-case. In the case of students who only ever login using web-services, they sometimes got a shock when using a lab for the first time and the password they'd been using for months didn't work.
There are two things working against us here.
- We did NOT set the "NMAS Authentication = On" setting in the Client we push. This means that while we are setting a universal password, none of our Novell Clients have been told to use them.
- LDAP logins to edir 8.7.3 use the NDS password by default first, and those are caseless. This means that anything using an LDAP bind, all of our web-sites that require authentication, will have a caseless password.
Looking at what would break if we turn NDS passwords off, I got a large list. We have some older servers in the tree (NetWare 6.0, and one lone NetWare 5.1 out there), and some console utilities would just plain break. Plus, at least one of us is still using ArcServe of an unknown version and I have zero clue if that would break if we remove NDS passwords (I'm guessing so, but I have no proof). Also, all older clients, such as the DOS boot disks used by our desktop group for imaging and any lingering Win9x we have out there, would break. Not Good.
The list of what'll break if we go to eDir 8.8 is shorter. As that allows the continued setting of the NDS Password, the amount of broken things out there is reduced. We'll have to put a specific dsrepair.nlm on all servers in the tree, but that is easier than working around breaking things. So, we're going to go to eDir 8.8.
This is not without its own problems, as some things DO still break. That lone NetWare 5.1 server will have to go. I've been assured that it is redundant and can go, but it'll need to ACTUALLY go. The NetWare 6.0 servers should be fine, as they're all at a DS rev that'll work with 8.8. Some of the 8.7.3 servers are still at 8.7.3.0 and should get updated for safety's sake. Also, all administrative workstations need to have NICI 2.7.x installed on them in order to understand the new eDir dialect, but that's a minor detail.
We won't be able to take advantage of some of the other nifty things eDir 8.8 introduces, as were still 95% NetWare when it comes to replica holders. Encrypted replication and multiple eDir instances will have to wait.
I HOPE to get eDir 8.8 in before class start, as the downtime required for DIB conversion is not trivial, and the first 4 weeks of class are always pretty hard on the DS servers due to updates.
Tuesday, August 21, 2007
Events while I was gone
We did some work with the UPS/Generator transfer switch. That caused a spew of SMS messages that I got out in the boonies.
Also, Novell has released the Novell Vista Client. This is the full release, not a beta. It even is fully localized!
The end of classes is nigh, so we're gearing up for the mass of upgrades that'll be going in while we have no classes being taught.
Also, Novell has released the Novell Vista Client. This is the full release, not a beta. It even is fully localized!
The end of classes is nigh, so we're gearing up for the mass of upgrades that'll be going in while we have no classes being taught.
- Get a new VCS version on the EVA
- Upgrade BlackBoard to a newer rev
- Move BlackBoard back-end database to SQL2005
- BIQuery upgrade
- Banner work
- A lot of disaster-recovery testing and setup
Labels: blackboard, netware, novell, sysadmin
Friday, August 03, 2007
NW65SP6 is in
I put it into WUF on Tuesday night. It went well enough, though not hitch free. The third server I did hung hard on a PORTAL.NLM error, so I ended up having to backrev that NLM to the SP5 version (and yes, I did use the post-SP6 PORTAL.NLM, it was the one that hung). I also managed to get all the fibre cards flashed to new BIOS to support an EVA maintenance we'll be doing during intersession.
I must say that the Novell Wiki on NW65SP6 is a fine and wonderful thing. This is a page that the Novell Support Forum sysops keep up to date with their distilled knowledge. Once upon a time the NetWare minimum patch list was useful for this, but that's fallen by the wayside in recent years. This wiki page includes abends that people have run into, the patches that fix known problems with SP6 (biiiig problems with NDPS, by the way), and suggestions learned through other people's hard experience. I recommend it.
I must say that the Novell Wiki on NW65SP6 is a fine and wonderful thing. This is a page that the Novell Support Forum sysops keep up to date with their distilled knowledge. Once upon a time the NetWare minimum patch list was useful for this, but that's fallen by the wayside in recent years. This wiki page includes abends that people have run into, the patches that fix known problems with SP6 (biiiig problems with NDPS, by the way), and suggestions learned through other people's hard experience. I recommend it.
Thursday, July 19, 2007
That darned MSA again
I'm not sure where this problem sits, but I'm having trouble with this MSA1500cs and my NetWare servers. I've found a failure case that is a bit unusual, but things shouldn't fail this way.
The setup:
zERR_WRITE_FAILURE 20204 /* the low level async block WRITE failed*/
Which, you know, shouldn't happen. And this error is consistent across the following changes:
I haven't thrown SP6 on there yet, as this is a WUF cluster node and this isn't intersession ;). This is one of those areas where I'm not sure who to call. Novell or HP? This is a critical error to get fixed as it impacts how we'll be replicating the EVA. It was errors similar to this, and activities similar to this, that caused all that EXCITEMENT about noon last Wednesday. That was not fun to live through, and we really really don't want to have that happen again.
Call Novell
The next step here is to delete these pools and volumes, recreate them, and see if things go Poink in quite the same way. I'm not convinced that'll fix the problem, as the errors being reported are Write errors, not Read errors, and the faulting blocks are different every time. I'm suspecting instability in the Write channel somewhere that is unique to a nss /poolrebuild, as I didn't get these errors when FILLING these volumes. Write channel in this case has a lot of Fibre Channel in it.
The setup:
- NetWare 6.5, SP5 plus patches
- EVA3000 visible
- MSA1500cs visible
- Pool in question hosted on the MSA
- Pool in question has snapshots
- Do a nss /poolrebuild on the pool
7-19-2007 9:48:22 am: COMN-3.24-1092 [nmID=A0025]The block number changes every time, and when it decides to crap out of the rebuild also changes every time. No consistency. The I/O error (20204) decodes to:
NSS-3.00-5001: Pool FACSRV2/USER2DR is being deactivated.
An I/O error (20204(zio.c[2260])) at block 36640253(file block
-36640253)(ZID 1) has compromised pool integrity.
zERR_WRITE_FAILURE 20204 /* the low level async block WRITE failed*/
Which, you know, shouldn't happen. And this error is consistent across the following changes:
- Updating the HAM driver (QL2300.HAM) from version 6.90.08 (a.k.a 6.90h) to 6.90.13 (6.90m).
- Updating the firmware on the card from 1.43 to 1.45 (I needed to do this anyway for the EVA3000 VCS upgrade next month)
- Applying the N65NSS5B patch, I had N65NSS5A on there before
I haven't thrown SP6 on there yet, as this is a WUF cluster node and this isn't intersession ;). This is one of those areas where I'm not sure who to call. Novell or HP? This is a critical error to get fixed as it impacts how we'll be replicating the EVA. It was errors similar to this, and activities similar to this, that caused all that EXCITEMENT about noon last Wednesday. That was not fun to live through, and we really really don't want to have that happen again.
Call Novell
| Good: | Bad: |
|
|
Wednesday, July 18, 2007
The OES2 push, what it means to me.
With the release of OES2 pushed to Christmas, or possibly BrainShare 2008, I'm in a hard spot. The magnitude of this migration means that I have one period a year I can pull that off, and that is the last week in August and the first two weeks of September. If I don't have code in that period, I can't migrate. Period.
As I learned at BrainShare this year, the Apple Filing Protocol stack on OES2-Linux is not eDirectory integrated. This is a project stopper for us, so we need that to be in place before we migrate. They quoted us, "Possibly SP1 timeframe, definitely not first-customer-ship, but don't hold us to it." They learned of the AFP problem at BrainShare and said they'd get right on it to try and get that in. That told me that summer 2008 would be the earliest I could expect to have the eDir integrated AFP stack.
Since I don't think Novell is planning on pushing OES2 ship to summer 2008, I suspect the AFP stack will be in with SP1. I consider it likely that OES2 SP1 will ship about the same time as SLE10 SP2. Which means I have real strong doubts that I'll be doing an OES2-Linux migration during next year's intersession. So we'll probably end up staying on NetWare for file-serving at least until 2009. In 2009 those NetWare servers may very well be in either an ESX or Xen virtual container, but it'll still be the 32-bit NetWare code doing the serving. That said, the web and print services (MyFiles, MyWeb, iprint) may move earlier, as they do not have the same AFP dependency.
Our storage needs on the WUF cluster are already pushing the boundaries of the 32-bit memory space. I'd be a lot happier of I could throw another 2 gigs of RAM at the file-servers in order to keep their cache-levels at a good spot. Can't do that on 32-bit NetWare, at least not while expecting improved performance. In 2009 we'll be managing anywhere from 12 to 18 terabytes of data on WUF, with a good chunk of it active. That is a situation that screams for 64-bit limits to memory space in order to provide zippy performance.
Thus, I am worried. Please, Novell. Ship at Christmas. It'll make my schedules look a LOT less grim.
As I learned at BrainShare this year, the Apple Filing Protocol stack on OES2-Linux is not eDirectory integrated. This is a project stopper for us, so we need that to be in place before we migrate. They quoted us, "Possibly SP1 timeframe, definitely not first-customer-ship, but don't hold us to it." They learned of the AFP problem at BrainShare and said they'd get right on it to try and get that in. That told me that summer 2008 would be the earliest I could expect to have the eDir integrated AFP stack.
Since I don't think Novell is planning on pushing OES2 ship to summer 2008, I suspect the AFP stack will be in with SP1. I consider it likely that OES2 SP1 will ship about the same time as SLE10 SP2. Which means I have real strong doubts that I'll be doing an OES2-Linux migration during next year's intersession. So we'll probably end up staying on NetWare for file-serving at least until 2009. In 2009 those NetWare servers may very well be in either an ESX or Xen virtual container, but it'll still be the 32-bit NetWare code doing the serving. That said, the web and print services (MyFiles, MyWeb, iprint) may move earlier, as they do not have the same AFP dependency.
Our storage needs on the WUF cluster are already pushing the boundaries of the 32-bit memory space. I'd be a lot happier of I could throw another 2 gigs of RAM at the file-servers in order to keep their cache-levels at a good spot. Can't do that on 32-bit NetWare, at least not while expecting improved performance. In 2009 we'll be managing anywhere from 12 to 18 terabytes of data on WUF, with a good chunk of it active. That is a situation that screams for 64-bit limits to memory space in order to provide zippy performance.
Thus, I am worried. Please, Novell. Ship at Christmas. It'll make my schedules look a LOT less grim.
Wednesday, July 11, 2007
Novell Client for Linux beta 2.0 release on openSUSE
I have managed to get the new beta of the NCL 2.0 working on my openSUSE 10.2 workstation. This is very nice. Some nice details can be found here in the Novell support groups. My steps were rather similar.
See that little sliver of an icon between the magnifying glass and the vertical bar? That's the nwtray icon. It's about 2 pixels wide. If I can click on it I get the full NWTRAY menu just fine, but it's hard to hit.
The other problem is the "Novell Services" button in nautilus. When I click that button, it looks like gnome crashes. I haven't been able to find out where the dump traces are going so I don't know what's up with that. If I access the services from the 2-pixel wide NWTRAY things work just fine, though.
Throughput still sucks, though. The Windows client is still better for that. But... throughput is w-a-y better than using a WebDav connection! Progress!
- Install the referenced RPM. I did it with an "rpm -i [rpm-name]". Use the RPM for your processor type. I'm using 64-bit and it worked just fine for me.
- Run the "ncl-install" from the beta download.
See that little sliver of an icon between the magnifying glass and the vertical bar? That's the nwtray icon. It's about 2 pixels wide. If I can click on it I get the full NWTRAY menu just fine, but it's hard to hit.
The other problem is the "Novell Services" button in nautilus. When I click that button, it looks like gnome crashes. I haven't been able to find out where the dump traces are going so I don't know what's up with that. If I access the services from the 2-pixel wide NWTRAY things work just fine, though.
Throughput still sucks, though. The Windows client is still better for that. But... throughput is w-a-y better than using a WebDav connection! Progress!
Monday, July 09, 2007
More fun OES2 tricks
I had an idea while I was googling around a bit ago. This may not work the way I expect as I'm not 100% on the technologies involved. But it sounds feasible.
Lets say you want to create a cluster mirror of a 2-node cluster for disaster recovery purposes. This will need at least four servers to set up. You have shared storage for both cluster pairs. So far so good.
Create the four servers as OES2-Linux servers. Set up the shared storage as needed so everything can see what it should in each site. Use DRBD to create new block-devices that'll be mirrored between the cluster pairs. Then set up NetWare-in-VM on each server, using the DRBD block-devices as the Cluster disk devices. You could even do SYS: on the DRBD block-devices if you want a true cluster-clone. That way when disk I/O happens on the clustered resources it gets replicated asynchronously to the DR site; unlike software RAID1 the I/O is considered committed when it hits local storage, SW RAID1 only considers writes committed when all mirrored LUNs report the commit.
Then, if the primary site ever dies, you can bring up an exact replica of the primary cluster, only on the secondary cluster pair. Key details like how to get the same network in both locations I leave as an exercise for the Cisco engineers. But still, an interesting idea.
Lets say you want to create a cluster mirror of a 2-node cluster for disaster recovery purposes. This will need at least four servers to set up. You have shared storage for both cluster pairs. So far so good.
Create the four servers as OES2-Linux servers. Set up the shared storage as needed so everything can see what it should in each site. Use DRBD to create new block-devices that'll be mirrored between the cluster pairs. Then set up NetWare-in-VM on each server, using the DRBD block-devices as the Cluster disk devices. You could even do SYS: on the DRBD block-devices if you want a true cluster-clone. That way when disk I/O happens on the clustered resources it gets replicated asynchronously to the DR site; unlike software RAID1 the I/O is considered committed when it hits local storage, SW RAID1 only considers writes committed when all mirrored LUNs report the commit.
Then, if the primary site ever dies, you can bring up an exact replica of the primary cluster, only on the secondary cluster pair. Key details like how to get the same network in both locations I leave as an exercise for the Cisco engineers. But still, an interesting idea.
Labels: clustering, netware, novell, OES, virtualization
Friday, July 06, 2007
Dynamic Storage Technology
Novell Connection Magazine has an article up right now that describes DST, formerly known as Shadow Volumes. I've talked about them before, both last year around this time (6/15/07, and 6/26/07) and back at BrainShare (TUT205). So, I've been following this.
As said previously, this'll not work for NetWare, just OES-Linux. From what I understand you can host migration volumes on NetWare, but the server presenting the unified view of the storage has to be OES-linux.
Anyway, on with the article.
As said previously, this'll not work for NetWare, just OES-Linux. From what I understand you can host migration volumes on NetWare, but the server presenting the unified view of the storage has to be OES-linux.
Anyway, on with the article.
Labels: netware, novell, OES, shadowvolumes
Tuesday, July 03, 2007
OES2: pushed several months
A new post up on Cool Blogs shows where OES2 is sitting:
http://www.novell.com/coolblogs/?p=921
To quote from one of the comments by the author:
Crap.
http://www.novell.com/coolblogs/?p=921
To quote from one of the comments by the author:
There will be a public beta. It might take couple of months more for a public beta.This blows my schedule. From the sounds of it, they're looking at a Christmas or possibly BrainShare 2008 release. We'll have to put NetWare inside ESX server instead of a Xen paravirtualization. Due to this delay, and the presumed SP1 schedule, chances are now much worse for Novell to make the summer intersession 2008 migration window.
Crap.
Labels: netware, novell, OES, virtualization
Thursday, June 28, 2007
Novell iPrint and Vista
Novell has a TID:
http://www.novell.com/support/search.do?cmd=displayKC&amp;amp;amp;docType=kc&externalId=3288691&sliceId=SAL_Public&dialogID=39375570&amp;amp;stateId=0%200%2039383497
In short, Vista has native IPP support. Unlike Linux, Vista's native IPP support supports both SSL and Authentication. So out of the box Vista can print to iPrint printers. It can't do the nifty automatic setup through a web page like iPrint allows, but at least it can print.
http://www.novell.com/support/search.do?cmd=displayKC&amp;amp;amp;docType=kc&externalId=3288691&sliceId=SAL_Public&dialogID=39375570&amp;amp;stateId=0%200%2039383497
In short, Vista has native IPP support. Unlike Linux, Vista's native IPP support supports both SSL and Authentication. So out of the box Vista can print to iPrint printers. It can't do the nifty automatic setup through a web page like iPrint allows, but at least it can print.
Novell Client for Vista, in public beta
Announced in Cool Blogs.
On the Beta Page.
Downloads.
Documentation.
Still no word on when OES2 is coming out. This is somewhat disheartening, as I had heard at BrainShare that the OES2 release would be simultanious with the Novell Client for Vista release. At this point, it is looking like an August release for OES2, which soooo blows my schedule.
On the Beta Page.
Downloads.
Documentation.
Still no word on when OES2 is coming out. This is somewhat disheartening, as I had heard at BrainShare that the OES2 release would be simultanious with the Novell Client for Vista release. At this point, it is looking like an August release for OES2, which soooo blows my schedule.
Labels: brainshare, netware, novell, OES
Wednesday, June 13, 2007
Still waiting
Any day now OES2 will come out.
Any day now.
Any day now I'll get a paravirtualizable NetWare and will be able to run it through its paces.
Any day now I'll get to try and figure out how Xen virtualization of NetWare interacts with an HP MSA1500cs.
But not today.
Any day now.
Any day now I'll get a paravirtualizable NetWare and will be able to run it through its paces.
Any day now I'll get to try and figure out how Xen virtualization of NetWare interacts with an HP MSA1500cs.
But not today.
Labels: msa, netware, novell, OES, virtualization
Friday, May 18, 2007
Acessing NetWare from openSUSE
I've been asked how I've been managing a NetWare network if my primary workstation is OpenSUSE. That's a good question. Right now I'm doing it through two methods:
As you can see. In fact, I used the above screen to drop the screen-cap I made into my myweb folder for publication.
That said, there IS a way to get the Novell Client for Linux 1.2 installed to openSUSE, I just haven't done it yet. You can find instructions in the Novell newsgroups, which google handilly archives. I must mention that NCL1.2 will be getting service-packed when SLES10 SP1 comes up within the next month. These instructions may not be valid once SP1 is being published from download.novell.com.
- I have WinXP running in a Xen VM that I use for 90% of it. Network throughput blows chunks, though.
- NetStorage (MyFiles for you WWU people) supports WebDav, and so does Nautilus.
As you can see. In fact, I used the above screen to drop the screen-cap I made into my myweb folder for publication.
That said, there IS a way to get the Novell Client for Linux 1.2 installed to openSUSE, I just haven't done it yet. You can find instructions in the Novell newsgroups, which google handilly archives. I must mention that NCL1.2 will be getting service-packed when SLES10 SP1 comes up within the next month. These instructions may not be valid once SP1 is being published from download.novell.com.
Tuesday, May 01, 2007
Microsoft permissions
It is clear our helpdesks are very used to Novell/NetWare style permissions. Over the years they've gotten used to calling up one of us, having us add someone to a group, and bam they're in! Now that we're doing a bit more with Microsoft permissions, they've run into one of the key differences between how Novell and Microsoft handle permissions.
In MS-land, on login you get an access token with all of your group memberships on it. If you add someone to a group, they have to log out and log in again to refresh that token before they can gain access to the resource.
In Novell-land, every time you access a resource that resource queries the directory service to see if you're in the right groups for access. If you add someone to a group, it takes effect immediately.
Very different expectations! The MS-way may be ultimately more computer resource efficient, but it does come at the cost of user efficiency. This bites us most often when it comes to Exchange. In what we call 'shared mailboxes' we use AD groups to manage access into those. Many times I'll get a call from the helpdesk that a user can't get into a just-created shared mailbox, and this behavior is the reason for it. They're so used to the Novell way of doing it that this seems like an error.
In MS-land, on login you get an access token with all of your group memberships on it. If you add someone to a group, they have to log out and log in again to refresh that token before they can gain access to the resource.
In Novell-land, every time you access a resource that resource queries the directory service to see if you're in the right groups for access. If you add someone to a group, it takes effect immediately.
Very different expectations! The MS-way may be ultimately more computer resource efficient, but it does come at the cost of user efficiency. This bites us most often when it comes to Exchange. In what we call 'shared mailboxes' we use AD groups to manage access into those. Many times I'll get a call from the helpdesk that a user can't get into a just-created shared mailbox, and this behavior is the reason for it. They're so used to the Novell way of doing it that this seems like an error.
Tuesday, April 24, 2007
Migrating a tricky application
We have a line of business application that is used for a small department. Like financial applications, this one is consistently about 4-6 years behind the current trends in application development. Version 9 which was just released a few months ago is finally web based. The version we're running, 8.something, is still based on the same model that Access databases are. I.e., file based databasing.
This particular application is excruciatingly sensitive to oplocks. We've fought this application for years as a result of that. Why is it so sensitive?
Any long time NetWare admin will tell you about PDOXUSRS.NET files. This particular application uses the same kind of access mode. One file is used to mediate who is authorized to access the application as a whole. While users are using the application they keep that file open, and update the file with their application level lock. Okay so far.
The problem comes with oplocks. How it is supposed to work:
One of the things that this vendor has done is decertify NetWare as a valid File Server to store this stuff. This is why I migrated the directory to a Windows 2003 server last night. And even there they had us do a reg-hack to turn off oplock support in Windows. They REALLY do not like oplocks.
Once this app goes web-based, it should help reduce the problems we have with that license file. I hope.
This particular application is excruciatingly sensitive to oplocks. We've fought this application for years as a result of that. Why is it so sensitive?
Any long time NetWare admin will tell you about PDOXUSRS.NET files. This particular application uses the same kind of access mode. One file is used to mediate who is authorized to access the application as a whole. While users are using the application they keep that file open, and update the file with their application level lock. Okay so far.
The problem comes with oplocks. How it is supposed to work:
- Station 100 opens LICENSE.LOG and requests an oplock on it
- Server, seeing no other stations with that file open, grants the oplock.
- Station 100 copies LICENSE.LOG to memory, thus improving access times to it.
- Station 105 opens LICENSE.LOG and requests and oplock on it.
- Server, seeing Station 100 has an oplock on it, tells Station 100 to release its oplock.
- Station 100 writes LICENSE.LOG to the server, and releases its oplock.
- Server tells Station 105 it can open the file, but can't have an oplock.
- Station 105 accesses the file without an oplock.
- Station 100 opens LICENSE.LOG and requests an oplock on it.
- Server, seeing no other stations with that file open, grants the oplock.
- Station 100 copies LICENSE.LOG to memory, thus improving access times to it.
- Station 100 crashes hard. Does not reset its connection to the server, and the Watchdog doesn't scavenge it.
- Station 105 opens LICENSE.LOG and requests an oplock on it.
- Server, seeing station 100 has an oplock on it, tells Station 100 to release its oplock.
- Station 100 is no longer there.
- Server waits until station 100 releases its oplock, which will be never.
- Station 105 doesn't get its lock. Application will not load for anyone else.
- Server Admin hard clears Station 100.
- Application Admin goes into LICENSE.LOG and cleans up bad entries.
- Application Admin tells all stations running Application to log out and log in again.
One of the things that this vendor has done is decertify NetWare as a valid File Server to store this stuff. This is why I migrated the directory to a Windows 2003 server last night. And even there they had us do a reg-hack to turn off oplock support in Windows. They REALLY do not like oplocks.
Once this app goes web-based, it should help reduce the problems we have with that license file. I hope.
Monday, April 09, 2007
OES2, not until 2008
The revelation about AFP in OES2 (how did I miss that?) is the last nail. OES2 will not be rolled out to the WUF cluster until August/September 2008 at the earliest. We'll be staying on NetWare until then. We have a couple of Mac labs and at least one class track that depends on AFP support. CIFS is not an option for many reasons.
So we will be waiting until Novell catches up. In the mean time our 'utility' servers could possibly move, but there aren't many of them. The other two NDS servers, and the server that ATUS hosts their Ghost images on. We're already running OES on one of the NDS servers. The other two are the SLPDAs for our environment, and also house the DFS databases.
So we will be waiting until Novell catches up. In the mean time our 'utility' servers could possibly move, but there aren't many of them. The other two NDS servers, and the server that ATUS hosts their Ghost images on. We're already running OES on one of the NDS servers. The other two are the SLPDAs for our environment, and also house the DFS databases.
Friday, April 06, 2007
OES2 and AFP
If you're an instituion of education like us, chances are real good you have PowerBooks and other Mac hardware desiring access to your NetWare/OES servers. It turns out I missed something while at BrainShare. OES2-Linux does NOT have an eDir integrated AFP stack like NetWare does. Whoa.
Details here: http://www.novell.com/coolblogs/?p=836
That's Jason Williams posting, and he is the Project Manager to OES. I spoke with him for a while during Meet the Experts regarding the concurrency concerns we have with OES in general. He has been on Novell Open Audio several times, so I know his voice. He was run downright ragged during BrainShare, which is very not surprising due to his level of oversight of a major product.
He's asking for people who need AFP to talk to them about it. The details of what he's looking for is in the posting I linked above. I've sent in my own impressions, and I've forwareded it to internal people who are Very Concerned about how Mac interacts with our NetWare servers.
Details here: http://www.novell.com/coolblogs/?p=836
That's Jason Williams posting, and he is the Project Manager to OES. I spoke with him for a while during Meet the Experts regarding the concurrency concerns we have with OES in general. He has been on Novell Open Audio several times, so I know his voice. He was run downright ragged during BrainShare, which is very not surprising due to his level of oversight of a major product.
He's asking for people who need AFP to talk to them about it. The details of what he's looking for is in the posting I linked above. I've sent in my own impressions, and I've forwareded it to internal people who are Very Concerned about how Mac interacts with our NetWare servers.
Labels: coolsolutions, netware, novell, OES
Thursday, March 22, 2007
TUT 202: NetWare cluster migrations to LInux clusters
There is a book on this: "Novell Cluster Services for NetWare and Linux". This session was about OES, not OES2. Again, my notes:
- On linux, cluster nodes are added through YaST
- 120 bytes of meta-data per file on NSS
- iPrint volumes could go on non-NSS volumes
- ext3 on OES2 is indexed, not indexed on OES1. Problem for larger directories.
- Novell Server Consolidation and Migration Tool can migrate Netware to Linux
- While running in mixed mode, can not extend or create NSS pools. Reboots all around to make this take.
- In mixed mode, trustee modifications do NOT transfer to the other OS. Migrate your NetWare volumes to OES-Linux, and leave them there!
- In OES-Linux, trustees are kept in a file, not in the file-system.
- In mixed mode, cluster load/unload scripts are kept in /etc/opt/novell/ncs/
- When out of mixed mode, scripts are promoted into edir
- Cluster licenses are not checked in OES-linux, but still 'count' come audit time. So have them.
- The 'cluster convert' command ends mixed-mode operation
- Clustering inside VMWare ESX server: only 2-node Microsoft clusters are supported. All others are not.
Labels: brainshare, clustering, netware, novell, OES
OES 210: OES, architectural overview
This sounds basic, but it extends on IO102. Again, my notes:
- Probable beta in the next few weeks
- OES2 will not install on SLES10, only on SLES10 sp1
- This was done for Product Certification reasons, as was the fact that OES is an 'add on' to SLES
- Most of OES2 is still 32-bit code. Parts with kernel interaction will be 64-bit.
- Shipping on DVD media, though the OES add-on will be CD.
- It will use Novell Customer Center for updates
- http://www.novell.com/products/openenterpriseserver/partners for AV and Backup partners
- CASA is a new auth package, stores things. Also exists on the client
- NLDAP has been ported to openLDAP, in that the openLDAP community has accepted the patches submitted by Novell.
- The kernel in OES2 will be 2.6.16
- SMS allows backing up of Xen VMs
- eDir 8.8 comes with OES2, no word on eDir 8.7
- pureFTP is edir integrated
- iManager 2.7 comes with JRE1.5
- iManager WILL be ported to NetWare, which means OES2-NetWare will also come with JRE1.5
- Samba new 'passdb' option = NDS_ldapsam
- Allows use of Universal passwords as a Samba password. Nifty.
- Tomcat 5 now, separate OES instance from the default SLES10 instance.
- New migration framework, script based from the looks of it.
- LAS, light auditing framework, new audit API
- NSS is instrumented to use it.
Labels: brainshare, netware, novell, OES
Tuesday, March 20, 2007
TUT211: NetWare virtualization
- Xen 3.0.4+ is the codebase. They wanted 3.0.5, but Xensource didn't get the release out in time for that.
- Server.EXE contains the bulk of the paravirtualization code.
- New loader, XNLOADER.SYS replaces NWLOADER.SYS, if used in Xen.
- New console driver. The old method, writing directly to video memory, won't work in a paravirtualized VM.
- New PSM: XENMP.PSM. Permits SMP in Xen.
- So far, no "P2V" equivalent application, though they promise something by OES2-ship.
- Improved VM management screens.
Labels: brainshare, netware, novell, OES, virtualization
What's new in OES2
A good many things are new in OES2. The high-points:
- 64-bit support (woo!)
- iFolder 3.6
- Dynamic Storage Technology (f.k.a. Shadow Volumes)
- eDir integrated DHCP/DNS & FTP
- Major Samba improvements
- DFS support, including linking to sub-directories
- Make a link to, for example, DATA3:/shared/, rather than making a new volume just for "shared"
- NetWare in a VM, with improved VM management
- Xen 3.0.4+ support
- They wanted 3.0.5, but Xensource didn't make the cut off date. So OES2 will have 3.0.4 heavily patched.
- Service packs for OES will be synchronized with SLES
- OES is going to be an add-on product on top of SLES, choose 'add on product' during install and use the OES CD's.
- The 'Volume Location Database' for DFS is clusterable now
- iManager 2.7 now has support for managing file-system trustees
- OES3 will only have support for NetWare inside of a VM. This is a move that was pushed by the hardware vendors, NOT Novell. The hardware vendors have notified Novell that they'll be discontinuing driver support for NetWare after OES2.
- It has 802.1 support
- New client for SLED10
- No DLU for vista, that will come from Zen
Labels: brainshare, netware, novell, OES
Monday, March 19, 2007
TUT212: Novell Storage Services
Not a new topic, but it contained the updates to NSS that'll be there in OES2.
By far the biggest thing is a 64-bit version of OES. Big big big. How big? Very big.
Remember those benchmarks I ran? The ones that compare the ability of OES to keep up with NetWare? And how I learned that on OES NCP operations are CPU bound w-a-y more than on NetWare? That may be going away on 64-bit platforms.
You see, 64-bit linux allows the Kernel to have all addressable memory as kernel memory. 32-bit linux was limited to the bottom 1GB of RAM. If NSS is allowed to store all of its cache in kernel memory, it'll behave exactly like 32-bit NetWare has done since NSS was introduced with NetWare 5.0. I have very high hopes that 64-bit OES will solve the performance problems I've had with OES.
By far the biggest thing is a 64-bit version of OES. Big big big. How big? Very big.
Remember those benchmarks I ran? The ones that compare the ability of OES to keep up with NetWare? And how I learned that on OES NCP operations are CPU bound w-a-y more than on NetWare? That may be going away on 64-bit platforms.
You see, 64-bit linux allows the Kernel to have all addressable memory as kernel memory. 32-bit linux was limited to the bottom 1GB of RAM. If NSS is allowed to store all of its cache in kernel memory, it'll behave exactly like 32-bit NetWare has done since NSS was introduced with NetWare 5.0. I have very high hopes that 64-bit OES will solve the performance problems I've had with OES.
Labels: benchmarking, brainshare, netware, novell, OES
Monday keynote
Ron Hovespian is a better speaker than Messman was, and was much better about hiding the fact that he was using a teleprompter. All in all the session wasn't terribly informative, but then the Monday session generally aimed at Press Releases rather than gee whiz. That comes Friday.
That said, there was some good stuff in this session:
Update: The picture.
That said, there was some good stuff in this session:
- OES2 public beta will be 'soon'. It will not be released at BrainShare
- AD / eDir federation will be in OES2
- SLES10 SP1 is out
- A new certification: Novell Certified Engineer (NCE), a migration of the old Certified Novell Engineer (CNE) to the new Linux regime. (I have to look in to that)
- Virtualization managers are coming soon. Possibly in Zen for Linux 7.2, releasing "after Q2".
- NetWare SP7 will be OES2
Update: The picture.
Labels: brainshare, netware, novell, OES
Friday, February 09, 2007
NetStorage and gnome davs::
I just figured out how to get NetStorage to work with openSUSE 10.2! You have to set NetStorage to 'cookieless mode'. Now I have access to my NetWare hosted volumes from openSUSE! It'd be better if I could direct mount them, much more efficient transfer protocols, but this will do until Novell releases the next Client for Linux with SLED10 sp1.
DST & NetWare
DST on NetWare itself is simplicity. Two lines in the Autoexec.NCF file. However, NetWare has been shipping with Java for some time now and that doesn't use the local server's time. So, time to update your java!
Novell has posted two patches to address JVM on NetWare:
JVM-1.4.2_13 for NW6.5
Timezone Updater for NW6.0 and NW5.1
Both update files reflect Novell's new linuxy naming scheme for patches. The first one is 'novpatch3880', the second one is 'novpatch3864'. So when you save the file locally, you'll probably want to rename them to something useful. In olden days the first would probably be named 'jvm142sp6a' or somesuch.
Novell has posted two patches to address JVM on NetWare:
JVM-1.4.2_13 for NW6.5
Timezone Updater for NW6.0 and NW5.1
Both update files reflect Novell's new linuxy naming scheme for patches. The first one is 'novpatch3880', the second one is 'novpatch3864'. So when you save the file locally, you'll probably want to rename them to something useful. In olden days the first would probably be named 'jvm142sp6a' or somesuch.
Monday, February 05, 2007
Novell Vista client POSTED
Technology Preview, very beta. But IT IS THERE!
http://download.novell.com/Download?buildid=yIEJzwGwlu0~
And,documentation.
http://download.novell.com/Download?buildid=yIEJzwGwlu0~
And,documentation.
Tuesday, January 30, 2007
Filefinder feedback
Two weeks ago I blogged about Novell getting rid of FileFinder in favor of putting everything on download.novell.com. As you can expect, there has been some pushback. This morning I found a Wiki entry about that, over on Cool Solutions. Don't care for the differences? Want to see what other people are griping about? It's a good read.
Thursday, January 25, 2007
The verbal ticks of old tyme NetWare admins
A conversation I read recently reminded me of this one. One of the things that caused me to grind teeth at my old job was the use of the word 'dsrepair' as a catch-all for anything run inside of dsrepair. This extended to some TIDs, which made the problem more wide-spread than my little corner of NetWare-land. The problem only started abating when it became possible to do some of the dsrepair tasks without actually loading DSREPAIR.NLM.
I'd see documentation like this one (generated internally, though my treacherous memory is telling me that a very early version of the NDS Health Check TID also had the bad phrasing):
The phrase, "run dsrepair until there are no errors," showed up a lot for a few years there. The conversation I read showed a clear case of an Old Tyme NetWare Admin maintaining the sense of dark mystery around dsrepair, which caused their young apprentice to go out into the world with an incomplete understanding of what this tool really does. This is a verbal short-cut that I'm glad has died out.
I'd see documentation like this one (generated internally, though my treacherous memory is telling me that a very early version of the NDS Health Check TID also had the bad phrasing):
What am I doing in DSREPAIR? Unattended health check? Timesync-check and sych-status check? Local trustee check? Check external references? Full database repair? WHAT? DSREPAIR does a lot of things. It is not VREPAIR, where it either works or it doesn't.
- Rconsole to the console
- Run dsrepair until there are no errors
- In NWAdmin...
The phrase, "run dsrepair until there are no errors," showed up a lot for a few years there. The conversation I read showed a clear case of an Old Tyme NetWare Admin maintaining the sense of dark mystery around dsrepair, which caused their young apprentice to go out into the world with an incomplete understanding of what this tool really does. This is a verbal short-cut that I'm glad has died out.
Labels: netware
Wednesday, January 17, 2007
Changing times: Novell patches
Weird. I saw this this morning because I'm slow:
No new patches will be put on ftp.novell.com (ftp://ftp.novell.com/) or FileFinder (http://support.novell.com/filefinder) after January 12, 2007 because patches are being moved to the Novell Downloads website.So, in other words, downloading patches will require a novell.com login from now on. Not that I mind, I have that. But I find such requirements annoying from other vendors.
On the evening of January 16th, all new patches and all patches currently on FileFinder will be made available on the Novell Downloads web site http://download.
novell.com/ instead of through FileFinder and ftp.novell.com.
To allow transition time, patches that are currently on ftp.novell.com will remain there until February 12th at 5:00 PM Mountain Time. At that point in time, all patches in the /pub directory will be taken off ftp.novell.com and will need to be accessed on the Novell Downloads web site.
Wednesday, January 03, 2007
Updated cool-tool
SEG has been updated!
New version, new look.
Coooooooooool.
New version, new look.
Coooooooooool.
