December 2006 Archives

Year in review: web-stats

So I won't have Saturday and Sunday in these stats. Big deal, don't care much. But heck, these are fun!
MyWeb servers
Total sessions 2006
  • Student: 436,048
  • Facstaff: 11,640
Total pageviews 2006
  • Student: 2,927,305
  • Facstaff: 760,077
Total hits 2006
  • Student: 8,178,044
  • Facstaff: 1,282,892
Total bytes transferred 2006
  • Student: 1.74 TB
  • Facstaff: 116.37 GB
Clearly, the Student myweb gets a LOT more traffic. A lot more bigger traffic, as the hits ratio is 8:1 but the data transferred ratio is closer to 10:1. Now, lets look at some averages:
MyWeb servers
Average pageviews per session
  • Student: 6.71
  • Facstaff: 65.30
Average hits per session
  • Student: 18.75
  • Facstaff: 110.21
Average bytes per session
  • Student: 4.18 MB
  • Facstaff: 10.24 MB
Average length of session (HH:MM:SS)
  • Student: 00:04:14
  • Facstaff: 00:42:21
THAT'S a difference! The traffic generated by the FacStaff MyWeb looks to be a lot more browsing, where I'd guess a lot of the traffic coming out of the Student MyWeb is one-off stuff like user-avatars in web-forums, and media files. The average session length is night-and-day, though; average session length of 42 minutes? Clearly there is a lot more 'dwell' on instructor sites than on the student sites.

At a guess, I'd say that students are using the Student MyWeb service more as a link-space than as a web-page host. Since that space is linked in with their home-directory quota, and also isn't bandwidth limited, it provides a much better experience than things like photobucket. Since items like avatars are files that get hit a LOT, icon-hosting is something of a bugabo and having your own host for that is handy.

One 125K JPG file is the number one hit file on the Student MyWeb for fall quarter, and is also the #4 file in terms of bytes transferred. This file is a Friendster background. Clearly, this person has a lot of traffic to their friendster site. Of the top 10 files-by-hit, 7 are clearly icons, avatars, or other 'personalization' images. On the flip side, all but one of the top 10 files-by-transfer are movies; this one lone (big) JPG file is the exception.

Looking at the Facstaff side, and 8 of the top 10 files-by-hit are components of web-pages hosted on One is the feed-file for this blog, and the other is an EXE file hosted by ATUS that seems to get a lot of off-campus requests. The files-by-byes are a bit different, in that four of the top 10 are EXE files from the same page. There is a PDF and three PPT files in the top 10 by byte, and only one movie.

Interesting stuff.

Fine tolerances

1/16th of an inch. That is the amount that this new rack is too narrow. The MSA20 and MSA1500 just plain will not fit between the uprights. Not going to happen. The fibre switch will fit with painful screeching noises, which I don't count as fitting. Clearly, this rack will have to be RMAed.

Wish I'da known that before I anchored it and got all the rails in. Fooy.

And since I've now seen three different people inspect the rack to see if there is any way to move the upright posts, I'll say this now. No. Not without a sledge hammer. Two of those other rack inspectors recommended a hammer independently of the other, the third had it suggested to him. The upright posts are held in place by horizontal braces that are themselves welded to the frame.

One bloody sixteenth. Bah.

Other client news

My earlier message about the open audio podcast also included some information about the Novell Client for Linux. It still isn't quite there yet, and Jason Williams admitted as much in the podcast. However, he did share some news about the next version of the Novell Client for Linux due out with either OES2 or SLED10 sp1.
  • NCL is build using AutoBuild, but a private AutoBuild.
  • The NCL will never be open-sourced because it has to use proprietary third party modules that Novell does not have the rights to open-source. One of the key modules are the RSA modules.
  • The new NCL will be able to auto-detect the kernel being used and will be able to automatically generate the required kernel modules (novfs and others) when kernel version change. Really nifty!
  • Introducing Gnome integration for Novell volumes. They already have the KDE version out. This will allow you to do some of the things you can do right now in Windows Explorer, only on Gnome and KDE.
Some of the key technologies behind the Novell Client on any platform were developed w-a-y back in the day. Like, NetWare 4.0 back in the day. Back when Novell still held the complete rights to Unix(tm). Open Source didn't have nearly the mind-share it does now, so building your infrastructure on proprietary third party components was perfectly OK. The key technology behind how Novell handled NetWare authentication as of the advent NDS was built around RSA-licensed cryptography.

The RSA bits ended up being just a little bit too secure for a heterogeneous environment. For a pure NetWare/NDS environment it was great, arguably the most secure thing around at the time. Since then Novell has realized that password synchronization between different authentication sources is a key technology in and of itself, which required that password encryption be reversible. Which the RSA stuff wasn't. Thus, the introduction of Simple passwords, and ultimately Universal passwords; both of which are less secure than the RSA methods, but still secure enough.

A lot of companies have moved to Universal passwords, but Novell still has to support the RSA-style login methods through the client. Therefore, the RSA-libraries have to be distributed with the client, and that makes it impossible to open-source without RSA's approval. Which isn't coming any time soon. There may be other 3rd party licensed technology in the client, but I can't remember who else may be involved.

It may be possible to create a true open-source client that just doesn't speak the proprietary protocols, though I strongly suspect that doing so will require significant reconfiguration of Open Enterprise Server. If not significant re-engineering. This is a legacy issue.

2GB Exchange mailboxes? Owie.

MS Fights GMail with 2GB Exchange Mailboxes

Yeesh. OldJob was on GroupWise, and we didn't have mail quotas in place. The largest mailbox I saw (not including archives) was about 900MB. These days that'd probably translate to a 2.5GB mailbox. So yeah, they can get that big.

When I started here the standard Exchange mailbox settings were set to start complaining when the 30MB line was crossed. We've upped it to 46MB since then. We manage our large users by having a higher tier quota group with much higher limits. That group is currently set to start warning at 200MB. Our largest mailbox right now is 233MB.

The problem with mailboxes that large is, of course, backing it all up. The article goes on to say that Exchang 2007 will have features that will help mitigate that. What I suspect that means is replication to another site, rather than the mail archive features some folk use backup/recovery for.

Setting the max quota to 2GB will result in a LOT more people using email as a filing cabinet. Right now the total size of our Exchange system is around 310GB, which is a direct result of those mail quotas I mentioned above. Additionally, we're backing up around 100GB of .PST files on the Novell cluster; this of course does not include those PST files located on PCs. Taking the breaks off the mail quotas would expand our mail significantly faster than its expanding now. Those folk who legitimately deal with huge files will be less inclined to delete redundant copies of Monster Attachments.

One of the more annyoing problems with just taking the breaks off is how long it'll take to sanity-check a bad mail database. The last time we did a round of that the data files were in the 28-30GB range, and it took about eight hours per mail-store to clean the database files. Exchange could handle that no problem, but that did result in an extensive downtime. Two servers, four large mail-stores, meant that once we started the repair process it was a minimum of 16 hours before everything was back up.

It'll be interesting to see the Exchange 2007 guidance for designing enterprises with that much storage.
Novell Open Audio did a piece on the client!

Find it here:

I took notes so you don't have to!
  • There will be a Tech Preview of the Vista client released in late January. Both 32-bit and 64-bit versions.
  • The login experience will change by definition, since Microsoft did away with the GINA.
  • There will be no IPX support.
  • The Tech Preview is not UI-complete, so end-user experience may change between the tech-preview and code-ship.
  • They're targeting release in "the OES2 timeframe," which I know from other podcasts to be late May to early June.
  • The Beta program (closed) signup will be in February.
  • 802.1x integration will be a module for both the Vista client and the WinXP 4.91 client. They use Microsoft's EAP hooks, rather than build their own.
  • All the installation modules for the client are fully signed, per Microsoft specification.
  • Dynamic Local Users have been an especially tricky thing to engineer.
  • The Zen Agent for Desktops is not included in the client. That'll come out on the Zen timeline, which was not covered in the interview.
  • Printing may not work the way you want it to in the Tech Preview.
And a transcribed quote from Jason Williams, project manager for OES and the Novell Client, regarding his philosophy for the timeline of releasing client builds:
" know, this is something that doesn't mess with, but interacts with Vista at a very low level. And there is NO WAY I'm going to release something like that, that interacts with a Brand Spanking New operating system, by Microsoft, that's got a new driver model, a new driver signing process, a new API set, a new method of working, a new notification tray, a new GINA, I'm not going to release that thing until I'm certain it's actually ready and our customers are going to be able to use it with confidence. Because if I just ship that thing out there and say, 'hey, we're done,' and it didn't work, then it's just not going to happen." [...] "Same thing with iPrint, and same thing with iFolder."
I've heard a lot of complaining about how Novell not having a client reflects very poorly on Novell as a company. Vista desktops are already on desks in enterprises, they should have had something. Especially since Microsoft had so many betas and RCs to develop against. Jason Williams addressed it this way.

Look at it this way. How many drivers and clients are production ready right now? So far only Intel has offical non-beta drivers. ATI and NVidia both only have beta drivers for their display adaptors. Creative Labs only has beta drivers for their sound cards. Vista is not enterprise ready yet, so Novell not having a beta out to try out is really not that bad.

My view of it comes closer to Mr. Williams. Microsoft has been diddling the security model on Vista all through the Beta and Release Candidate stages. There are bits there that were changed between the final release candidate and RTM that affect the security environment. The Novell client by necessity, through inserting itself into the authentication stream, was affected by that. DLU means that the Novell client has to be able to automatically generate user accounts from eDirectory information, which is a deep in the bowels security action. Password synchronization between eDir and the Vista desktop is another deep in the bowels security action. All of these can be affected by small changes in how Vista behaves with security.

The Novell Client isn't as simple as a display driver, it has to meld itself into a lot of the Vista security environment. This is something that Microsoft spent years making harder to accomplish, and they were tweaking bits of it right up until RTM. That takes time to engineer around.

I just hope that they refresh the technical previews periodically. I know we'll be prototyping distribution methods with those technical previews. After speaking with one of our higher desktop support managers, we don't have the man-power to participate in the Beta program. Therefore, we'll be dependant on technical previews and the open beta when it starts.

More migrations

One of the thornier pieces of migrating to a Linux desktop has been some background utilities. One of them is a Perl-based script that uses SNMP to populate a database about point-in-time free space on each volume. This script is the basis for how we manage storage usage on the SAN, for both NetWare (6.5TB these days) and Windows (the rest of it, less than 2TB). I first talked about it back in 2004. Also mentioned a couple times in 2005.

Well, now I have the same script running on an openSUSE box. I only had to add a single line, and uppercase one string. That's it. That beats the previous migration quite handily. Gotta love platform independent code!

The hard problem was getting the ODBC source set up. It turns out that openSUSE ships with unixODBC. When paired with freeTDS I was able to update the same MS-SQL database I was using on Windows. It wasn't five-clicks-and-wham-it-works, but it was a lot easier than the alternative I was pondering... JDBC. I'm not doing anything fancy with this script, just a bunch of INSERTS, so I don't really need a bullet proof ODBC setup. All of my reporting is still in Windows, so I'm not worried about that yet.

But... one line? And a UCASED string? That's one of the easiest conversions I've done! This script started life dumping to a CSV file, then dumping to an Oracle table, back to CSV once I got to WWU, then MSSQL.

Yummy stats (fall quarter)

Today is the close of fall quarter, so I'm taking a look at Student MyWeb usage for the quarter. First, some quick hits:
  • 235GB were transferred.
  • 609,501 page views.
  • Top file by hit: at around 89,000 hits.
  • Top file by bytes: at 12.71GB. This is specifically interesting, since it got there only over the past week. I should probably take a look at that and see if its something I have to officially notice.
  • #1 browser used (by hit): IE, at 48%
  • #1 browser used (by session): "Mozilla Compatible Agent" at 47%
  • #1 platform (by hit): Windows, at 66% (of which WinXP is 94%). #2 is Macintosh, at 19%.
And now FacWeb!
  • 146GB were transferred
  • 516,866 pageviews.
  • Top file by hit:
  • Top file by bytes: by a long shot. This represented 25% of traffic!
  • #1 browser used (by hit): IE at 61.2%
  • #1 browser used (by session): IE at 38.62%
  • #1 platform (by hit): Windows, at 79% (of which WinXP is 91%). #2 is 'unknown' at 12%, and Macintosh comes in at 7%.
During Spring quarter, the Mac percentage on MyWeb for Students was a whopping 22%. Clearly, we have a lot of PowerBook owners on campus.
I see from logfiles that my post back in March about Novell client for Vista is a frequently hit blog entry. No surprise, as Novell doesn't yet have a client out for Vista and desktops are already showing up with it. By the time Novell will have a Beta out, mass quantaties of early adopters (and more importantly, students with shiny new Vista laptops) will have been on the scene.

Understandably, the question, when is Novell going to release a Client for Vista? is the #1 question in the support forums. So frequent, they've developed boiler plate for it. First off Novell hasn't formally answered the when question. Internal resources give hints.

So far the rumor mill has turned out the following predictions:
  • The closed beta will be opening up real soon, though we don't know when.
  • The public beta will open up sometime between February and April.
  • Release is dependant on the issues the beta comes up with
  • Novell isn't going to release massively buggy code just to get a client out the door. They caught hell with the Novell client for Linux 1.0, after all, and that's not an experience they really want to repeat.
  • Features that Novell needed to leverage to even build a client weren't solidified until Goldmaster (a.k.a. RTM), which has predictable impacts on the development cycle.
  • If a client isn't in open beta by the time BrainShare rolls around, the poor client people in the Tech Lab are going to be royally tired of that question.
In the mean time CIFS/Samba is your only real hope for connecting Vista machines to your NCP servers.

Procedures have been posted here.

There is a chance we might participate in the beta. But that decision needs to include our desktop people to a very great degree, as they're the ones who will have to do 80% of the leg work testing builds.

Odd SLP memory allocations

I've noticed some really odd memory allocations for SLP on the cluster nodes. As in, really odd.

SLP Service Agent Memory


514 MB

That's a lot of memory! By comparison, one of the two SLPDA's has all of 36736 bytes allocated to it. The DA itself is around 1MB. We don't have a lot of objects. What the heck is it doing with all that memory? I'm sure this is a contributing factor to our on-going memory allocation problems. What is clear is that any node that has been up for long enough (looks to be a week) has north of 400MB allocated to it.

I'll probably be opening an incident on this one next week, once finals are over and I can reboot things willy nilly for things like core dumps.

We haven't gone to SP6 yet, so I made a point of checking SP6 for any SLP updates. There are none, so with luck I can deflect the, "upgrade to SP6 and call back," request that will happen when I call in.

openSUSE 10.2 on the ASUS P5B Deluxe

This board in general is not a good Linux board. It was purchased with Windows in mind. Which in the end turned out to be something of a negative, as three of the five people who got them wanted a Linux-something installed. Two of us have gotten it working, and the third may come around once I'm done dorking with my setup.

There are two big areas that cause problems with this board:
  1. The chipset, Intel 965G, has no native PATA support so ASUS attached a JMicron PATA controller. The JMicron PATA controllers are broken in all but the newest kernels (2.6.18 or newer). There is a limp-mode for 2.6.16 and older that can work (google all-generic-ide). This is a b-i-g problem when you consider almost all Optical drives are still PATA.
  2. The kernel included with the Goldmaster media ( locks up the install hard. The cause of which is, as of this writing, unknown [Update: the kernel falsly detects AGP, which is the source of the lock, using the boot option "agp=off" may bypass this detection]. This will probably change when they remaster the install media later on.
There are some BIOS settings that need to be set in order to make the system usable on Linux. Because there are functionally two separate IDE controllers on the board, there are two separate places in BIOS you need to go to in order to configure your IDE environment. This board has the option of RAID, but it is actually a software RAID and is not well supported in Linux yet. I didn't use it.

On the MAIN page in BIOS there is a sub-menu for IDE Configuration. You want to set 'Configure SATA as' to IDE. Don't use ACHI, so far only Vista can use ACHI right there. Linux will eventually support it. (and if you're reading this article two years from now, Linux probably has figured it out. Google harder.)

The second setting is on the ADVANCED menu, in the OnBoard Devices Configuration sub-menu. Set 'JMicron Controller Mode' to ACHI. That's right, ACHI. Not IDE. This allows the PATA optical devices to be detectable. This looks counter intuitive, but this is what seems to work.

Now that you've done that, you have put the shiny DVD into the optical drive and booted from it. You hit 'install from DVD' on the front screen, and.... it bombs, unable to detect the media.

That's because 10.2 isn't quite smart enough yet to detect it. You can force the issue by putting the following line on the "Boot Options" line on the front screen:


This tells the kernel to force-load the JMICRON driver. This allows the kernel to locate the DVD drive with which it'll continue the installation process.

Now comes the harder problem, the kernel shipped with 10.2 Goldmaster will hard lock the P5B. There is a work-around which you can find in Bug 229365.
  1. When the install gets to the 'now we will reboot your machine', click SKIP.
  2. [Control]+[Alt]+[F5] to get to a shell
  3. cd to /mnt/etc/
  4. vi modprobe.conf.local
  5. Append this text to the file "blacklist intel_agp"
  6. Save the file
  7. Reboot
That SHOULD get you in.

[revised 12/21/2006]
Update 4/2/2007:

Turns out there have been some key problems in the Linux kernel relating to this particular P965 chipset. From a bug I reported a few weeks ago:
Looking at the lspci output I get the understanding that
intel-agp shouldn't initialize at all - there's not Intel Integrated Graphics
controller. The code in intel-agp, however, blindly assumes an 845 if there's a
host bridge with no matching IG (in agp_intel_probe()).
Which is a long way of saying that if Intel AGP is loaded, either compiled in or falsely loaded due to a bad detection by the kernel, it'll have problems. The kernel falsely detects an integrated graphics controller and attempts to load agp_gart, which if loaded does Bad Things. It looks like the openSUSE guys are now aware of this problem, so we may have a new kernel in the not too distant future to fix this. Or perhaps this is just on the list for 10.3 when it comes out. Don't know yet.

More on openSUSE 10.2

Reinstalling with LILO as a boot-loader this time caused me to actually have a working boot loader. Unfortunately, Linux still won't boot. It'll load the kernel and boot gets to the line that reads:

agpgart: Detected an Intel 965G Chipset.

Then locks the computer hard. Kinda useless.

On the up side I now can boot the Windows partition so I have access to big hard drives again. Yay. I'm currently re-downloading RC1, since I KNOW that worked. I'll then selectively bootstrap packages from 10.2-Final after I get it in. I'll avoid the kernels since I know those don't work. I'll wait for new kernels to get checked in that hopefully fix my problems.

I have a bug report in on it, but it's ranked 'normal' so won't get special attention.

Or maybe I won't write that cooltip

openSUSE10.2 is out!

Unfortunately, it has a bug somewhere that prevents GRUB from successfully installing onto my system. This is new between RC1 and Final, since RC1 worked pretty well for me. There is an existing bug that comes close to this one, and suggests that perhaps the SATA nature of my boot device may be getting in the way. I know that the JMICRON thingies have been a problem in the past, but the boot drives are all attached to the chipset SATA headers, not the JMicron ones.

Even if openSUSE comes out with a patch, it won't help install much unless they remaster. Upgrading will require me to install 10.1 on, and then doing an 'update installation' from YAST over the Internet with a 10.2 repository. Which will take 12-18 hours right now with everyone and their mother slamming the repositories.


Brainshare time

Novell replaced the old NNTP-based brainshare forum with a web-based one supplied by Leverage Software. It's a social networking thingy designed to get Novell people talking to eachother, and undoubtably generating sales leads in the end. Since I'm deep into their pockets already (and have been saying NO to all kinds of things so I'm in practice), I created an account.

Join Me at Novell BrainShare Connect!

It seems that 'brainshare2007' is appended to each username, kind of annoying.

I note with amusement that BrainShare Connect is driven by ASPX. I know for a fact that BrainShare Connect is outsourced contracted out, so Novell doesn't have as much say about what the application is written in. Perhaps the only LAMP-running competition for the bid were small time apps that couldn't stand up to the task. Whatever. I noticed that it seems rather .NETy, but I'm not holding it against Novell.

That said, if'n you're going (and I know some of you are), you now have a way of finding me! Maybe see you there.

openSUSE 10.2 out real soon

| 1 Comment
OpenSUSE has 'goldmastered', which means that all bugs in bugzilla right now are stuck until after release. During the reconfiguration process of my new workstation I've used Beta2 and RC1-4 with Xen. I've solved a lot of problems over the time. I am this -> <- close to using openSUSE as my primary workstation with a WinXP install in a Xen container for those unavoidable Microsoft-only bits.

Such as Novell Client32, ConsoleOne, rconip, NDPS (for us), pcounter tools, and anything managed through an MMC. I'm still using Outlook since Evolution (even with the Exchange plugin) still feels like the squinty-eyed bastard step-child of the real thing.

[Brainwave: I wonder if the GroupWise client for linux looks any better? Wouldn't that be a kick in the pants]

When openSUSE 10.2 hits the servers I'll download it and refresh my install. THEN, assuming everything doesn't break, I'll try dogfooding for a while. If I can go a few days without going to the old station for anything but the stuff I CAN'T move temporarily, I may just go all the way. Scary thought. I'll still be doing most of my work in that WinXP XEN machine, but most web-browser work and SSH stuff will be in Linux-land.

Depending on how the release looks, I may just submit a Cool Tip on how to get the Nvidia drivers to compile for the Xen kernel. Needed, if I want dual-head support, and was a real fight to figure out how to make it work. Since I don't fully understand exactly what I did, I'm a bit leery of the solution, but I haven't crashed X yet so... maybe it worked out, eh?