Recently in linux Category

My testing for a cheap NAS solution has progressed to the option that costs the most money, Windows 2008 running KernSafe's iStorage. As it happens, it works really well when the iSCSI initiator is Windows but Linux clients don't really want to talk to it. Windows: 30-50 MB/s. Linux: 3-5 MB/s. Biiiig difference there.

Looking at packets I'm noticing a similar pattern on the wire to one I'd seen before. Back when I was troubleshooting exactly why NetWare backups to DataProtector were horrible I came across this problem. It seems that TCP Windowing is fundamentally broken between Server 2008 and NetWare which leads to really bad throughputs, which in turn is very bad for half TB backups. The receiving server seemed to feel the need to ACK after every two packets, which in turn really slowed things down. And that's what the Linux clients are doing for iSCSI to Server 2008.

It has to be something affecting basic TCP services but not complex protocols. Using smbclient to upload a 4GB DVD iso runs at 50MB/s but the iSCSI throughput on the same client is a piddly 3-5MB/s. I'm sure some kind of tuning on either side might be able to jar things loose, heaven knows Linux 2.6.31 is a heck of a lot more current on TCP settings than NetWare 6.5 SP8 is. I just haven't found it yet.

Conversely, Server 2008 talking to a Linux iSCSI client works at line speed pretty much. I'm testing this for completeness's sake. We need something that can serve up to 30TB via both iSCSI and SMB. My findings aren't fully complete yet, but in general:

• OpenFiler: GREAT iSCSI host, completely blows for SMB in our environment.
• OpenSolaris: Great iSCSI host, just can't convince the kernel-mode CIFS to join our domain. Also, worst-of-breed random I/O performance.
• OpenFiler + Windows: OpenFiler for iSCSI, Windows (mounting an iSCSI share) for SMB. Should work GREAT. Current best-best for the future.
• OpenSolaris + Windows: As previous option, but I/O problems make it less attractive.
• Windows + KernSafe: GREAT SMB performance, solid iSCSI for Windows hosts. Linux hosts will take lots of tuning (perhaps, or it could be intractable).
  

I've been playing around with OpenFiler the last week. It seems to fit our need for a free-to-us software package that allows us to serve both CIFS and iSCSI from the same host, in an easy to manage package. I haven't done much serious testing with it, but I have done enough to get a feel for how it works.

One thing is pretty clear, if we domain this thing certain UI elements become unusable due to timeouts building the page. Because we have so many groups in our AD tree, and the fact that it has to list every single group in the system in one big pick-list on the Share Permission screen, that page takes a very very long time to load. Long enough that it won't show the network-based permissions dialog at the bottom of the page, and is critical for enabling CIFS sharing in the first place. Unless I can find a tweak somewhere, that's a pretty serious road-block for CIFSiness.

iSCSI, on the other hand, just flies like a dream. I haven't had a chance to try out real complexity with it, I lack enough servers with GigE NICs capable of an MTU larger than 1500b that can be used for testing, so I can't say how robust it is. But I can say that I can saturate the GigE NIC in the OpenFiler box.

This does suggest a solution, though. We'll need another (%!#$!) server, upon which we'll install a Winders of some flavor and use an iSCSI presentation for the storage. Or, if we feel like we need more hand-holding in our lives, a Linux box of some flavor and hand-roll the Samba config needed.

This thing can also do NFS, but we have limited demand for that. The same for 1980's style FTP. There is also a WebDAV option, but I shiver at the notion of turning that on; the WebDAV setup in our existing file-cluster has already caused enough hair loss thank you.

It can also do snapshots. Since this thing is Linux based, these are LVM-level snapshots. That could be useful.

File-systems are restricted to Ext3 and XFS, which is good to a point. These are not the filesystems you want for multi-million-file shares. However, if all you want is bulk storage for disk images, they're just peachy. Or a departmental share space (hundreds of thousands of files). Neither of these are terribly great at handling the "bajillion files in one directory" problem, but we have few of those as it is.

But if we can't figure out a way to make the CIFS sharing useful, file-system choice is mostly moot.

Anyway, more testing!

I've been looking for a new laptop. I finally found one, Dell had a sale on their Studio 15 line. This is a Core i5 laptop with all the niftyness I wanted. I also purchased a new 320GB hard-drive for it. Why? So I could put the Dell supplied one, with its pre-built Win7 install, into a baggie for later use if I so chose, and so I could get openSUSE onto it without mucking about with resizing partitions and all that crapola.

I did what I could to make sure the components were Linux compatible (Intel wireless not Dell, that kind of thing), but some things just don't work out. This is a brand new laptop with a brand new processor/chipset/GPU architecture, so I planned on having at least one thing require several hours of hacking to get working. This is the price you pay for desktop-linux on brand spanking new hardware. I, at least, am willing to pay it.

And pay I am. I installed OpenSUSE pretty simply, or at least it started there. It got to the first reboot and got a black screen of nothingness. Watching post it was pretty clearly a kernel video handling problem. Some quick googling identified the problem:

OpenSUSE 11.2 uses the 2.6.31 kernel. This laptop uses an intel 4500MHD GPU. Support for which was introduced in 2.6.32, and greatly refined in 2.6.33. What's more, it uses Kernel Mode Setting Direct Rendering Management, support for which was introduced in 2.6.32. All this means is that 2.6.31 simply can't support this video GPU at anything like reasonable speeds.

OpenSUSE 11.3 (currently in a very buggy Milestone 3 release, soon to be M4) has a 2.6.33 series kernel. But I don't want to be buggy. So...

Time to compile a kernel!

Because I've done it before (a LOT), kernel compiles do not scare me. They take time, and generally require multiple runs to get right, so you just have to plan for time to get it right. So I booted to the OpenSUSE 11.2 Rescue System, followed these instructions, and got into my (still half installed) file-system. I plugged it into wired ethernet because that's hella easier to set up from command-line, and used yast to grab the kernel-dev environment. Then downloaded 2.6.33 from kernel.org. I had to grab the /proc/config.gz from a working x86_64 system, so I pulled the one from my 11.2 install here at work, threw it into the kernel-source directory, ran 'make oldconfig', answered a bajillion questions, and tada. make bzImage; make modules; make modules_install ; make install ; mkinitrd -k vmlinuz-2.6.33 -i initrd-2.6.33, and a bit of Yast GRUB work to make certain Grub was set up right, and reboot.

Most of the way there. I had to add this line to /etc/modprobe.d/99-local:

options i915 modeset=1

Which got me enough graphics to finish the openSUSE install, and get me to a desktop. I don't yet know how stable it is, haven't had time to battletest it. I probably need updated x.org software for full stability. I did get it up long enough this morning to find out that the wireless driver needs attention; dmesg suggested it had trouble loading firmware. So that's tonight's task.

Update: Getting the wireless to work involved downloading firmware for the 6000 from here, dropping the files into /lib/firmware, and rebooting. Dead easy. Now, Suspend doesn't work for some reason. That might be intractable, though.

I'm going over some of my older posts and am reposting some of the good stuff that's still relevant. I've been at this a while, so there is a good week's worth of good essays hiding in the archives. 

---

Back in 2005 I posted a story of my first bit of serious Linux hacking. This was back in college, and involved a 1.2.x era kernel. I had a 1.2GB drive, but somehow both DOS and Linux were ignoring the partition table. I figured it out, and this is how I did it.

Linux hacking from way back.

Looking back on it, this would have been a prime opportunity for me to turn into a kernel-hacker. All of my C training was fresh, and back then the barrier to entry for kernel-hackers was a lot lower. But, I didn't.

This post is from back before Blogger supported comments or labels! Old times, man.

I'm going over some of my older posts and am reposting some of the good stuff that's still relevant. I've been at this a while, so there is a good week's worth of good essays hiding in the archives.

---

This one got some inbound links and attracted several readers. The question was asked on ServerFault but I had to echo by reply on my blog since it was too juicy. I've been working with directories since 1996 when I first ran into Novell NDS, so the concepts behind LDAP are engraved on my bones it seems like. So explaining it to someone else is an effort in... restraint of information. They don't need to know every little detail, just enough to get the concepts.

I still get wordy.

Explaining LDAP.

It's time for another openSUSE survey! If you're an openSUSE user (or even a user of SLES/SLED) it's a good idea to take this thing. They set development priorities based on these surveys, so if you have an area that needs buffing up this is the place to tell them. Or if you want to tell them 'works great!' this is where you do it too.

http://www.surveymonkey.com/s/6MJYV7T

Watching traffic on the opensuse-factory mailing list has brought home one of the maxims of Linuxdom that has been true for over a decade: People run Linux on some really old crap. And really, it makes sense. How much hardware do you really need for a router/firewall between your home network and the internet? Shoving packets is not a high-test application if you only have two interfaces. Death and fundamental hardware speed-limits are what kills these beasts off.

This is one feature that Linux shares with NetWare. Because NetWare gets run on some really old crap too, since it just works, and you don't need a lot of hardware for a file-server for only 500 people. Once you get over a 1000 or very large data-sets the problem gets more interesting, but for general office-style documents... you don't need much. This is/was one of the attractions for NetWare, you need not much hardware and it runs for years.

On the factory mailing list people have been lamenting recent changes in the kernel and entire environment that has been somewhat deleterious for really old crap boxes. The debate goes back and forth, but at the end of the day the fact remains that a lot of people throw Linux on hardware they'd otherwise dispose of for being too old. And until recently, it has just worked.

However, the diaspora of hardware over the last 15 years has caught up to Linux. Supporting everything sold in the last 15 years requires a HELL of a lot of drivers. And not only that, but really old drivers need to be revised to keep up with changes in the kernel, and that requires active maintainers with that ancient hardware around for testing. These requirements mean that more and more of these really old, or moderately old but niche, drivers are drifting into abandonware-land. Linux as an ecosystem just can't keep up anymore. The Linux community decries Windows for its obsession with 'backwards compatibility' and how that stifles innovation. And yet they have a 12 year old PII box under the desk happily pushing packets.

NetWare didn't have this problem, even though it's been around longer. The driver interfaces in the NetWare kernel changed a very few times over the last 20 years (such as the DRV to HAM conversion during the NetWare 4.x era, and the introduction of SMP later on) which allowed really old drivers to continue working without revision for a really long time. This is how a 1998 vintage server could be running in 2007, and running well.

However, Linux is not NetWare. NetWare is a special purpose operating system, no matter what Novell tried in the late 90's to make it a general purpose one (NetWare + Apache + MySQL + PHP = a LAMP server that is far more vulnerable to runaway thread based DoS). Linux is a general purpose operating system. This key difference between the two means that Linux got exposed to a lot more weird hardware than NetWare ever did. SCSI attached scanners made no sense on NetWare, but they did on Linux 10 years ago. Putting any kind of high-test graphics card into a NetWare server is a complete waste, but on Linux it'll give you those awesome wibbly-windows.

There comes a time when an open source project has to cut away the old stuff. Figuring this out is hard, especially when the really old crap is running under desks or in closets entirely forgotten. It is for this reason that Smolt was born. To create a database of hardware that is running Linux, as a way to figure out driver development priorities. Both in creating new, missing drivers, and keeping up old but still frequently used drivers.

If you're running a Pentium 2-233 machine as your network's NTP server, you need to let the Linux community know about it so your platform maintains supportability. It is no longer good enough to assume that if it worked in Linux once, it'll always work in Linux.

I like reading kernel chagelogs. There is usually at least one NEAT thing in there. This time (2.6.32) that's a memory de-duplication technology that will be of great benefit for VM environments.

The result is a dramatic decrease in memory usage in virtualization environments. In a virtualization server, Red Hat found that thanks to KSM, KVM can run as many as 52 Windows XP VMs with 1 GB of RAM each on a server with just 16 GB of RAM. Because KSM works transparently to userspace apps, it can be adopted very easily, and provides huge memory savings for free to current production systems. It was originally developed for use with KVM, but it can be also used with any other virtualization system - or even in non virtualization workloads, for example applications that for some reason have several processes using lots of memory that could be shared.

So. Cool. And there is more:

To make easier a local configuration, a new build target has been added - make localmodconfig. It runs "lsmod" to find all the modules loaded on the current running system. It will read all the Makefiles to map which CONFIG enables a module. It will read the Kconfig files to find the dependencies and selects that may be needed to support a CONFIG. Finally, it reads the .config file and removes any module "=m" that is not needed to enable the currently loaded modules. With this tool, you can strip a distro .config of all the unuseful drivers that are not needed in our machine, and it will take much less time to build the kernel. There's an additional "make localyesconfig" target, in case you don't want to use modules and/or initrds.

If you want to build a monolithic kernel for some reason, they've just made that a LOT easier. I made one a long while back in the 2.4 era and it took many tries to be sure I got the right drivers compiled in. Turn off module support and you've just made a kernel that's harder to rootkit. I don't see this improvement being as widely useful as the previous, but it is still nifty.

There is an ongoing thread on opensuse-factory right now about the announcement to ditch SaX2 from the distro. The reason for this is pretty well laid out in the mail message. Xorg now has vastly better auto-detection capabilities, so a tool that generates a static config isn't nearly as useful as it once was. Especially if its on a laptop that can encounter any number of conference room projectors.

This is something of the ultimate fate of Xwindows on Linux. Windows has been display-plug-n-play (and working plug-n-play at that) for many, many years now. The fact that XWindows still needed a text file to work right was an anachronism. So long as it works, I'm glad to see it. As it happens, it doesn't work for me right now, but that's beside the point. Display properties, and especially display properties in an LCD world, should be plug-n-play. 

As one list-member mentioned, the old way of doing it was a lot worse. What is the old way? Well... when I first started with Linux, in order to get my Xwindows working, which was the XFree86 version of Xwin by the way, I needed my monitor manual, the manual for my graphics card, a ruler, and a calculator. I kid you not. This was the only way to get truly accurate ModeLines, and it is the ModeLines that tell Xwindows what resolution, refresh rate, and bit-depth combinations it can get away with.

Early tools had databases of graphics cards and monitors that you could sort through to pick defaults. But fine tuning, or getting a resolution that the defaults didn't think was achievable, required hand hacking. In large part this was due to the analog nature of CRTs. Now that displays have all gone digital, the sheer breadth of display options is now greatly reduced (in the CRT days you really could program an 800x800x32bit display into Xwin on a regular old 4:3 monitor, and it might even have looked good. You can't do that on an LCD.).

In addition, user-space display tools in both KDE and Gnome have advanced to the point that that's what most users will ever need. While I have done the gonzo-computing of a hand-edited xwindows config file, I do not miss it. I am glad that Xwindows has gotten to this point.

Unfortunately, it seems that auto-detect on Xwindows is about as reliable as Windows ME was about that. Which is to say, most of the time. But there are enough edge cases out there where it doesn't work right to make it feel iffy. It doesn't help that people tend to run Linux on their crappiest box in the house, boxes with old CRTs that don't report their information right. So I believe SaX2 still has some life in it, until the 8 year old crap dies off in sufficient numbers that this tool that dates from that era won't be needed any more.

Yesterday I upgraded my work desktop to openSUSE 11.2 from 11.0. I'd skipped 11.1 because upgrading (rather than reformat/reinstall) is always a lot of work. This proved to be no exception. Even though I was involved with the milestone builds for openSUSE, I didn't have the ability to test my guaranteed-to-have-problems desktop.

I had two problems:

1. The XWindows hot-plug support for input devices just plain doesn't work on my system. This was introduced in 11.1, so that would have bit me then.
2. VMWare Workstation just wouldn't compile the modules. The 11.2 kernel is too new.

I've managed to solve both problems. The first was solved by putting my old (backups are your friend!!) xorg.conf file into play and adding one line in the ServerFlags section to tell it to not bother auto-adding input devices. The second involved installing 6.5.3 instead of 6.5.2, and using a bit of crowbar-fu to make it fit. 6.5.2 just wouldn't work, where 6.5.3 required some special handling.

I don't know why the 'evdev' stuff in xorg is failing on me. It's on my list to figure out, but as I have a working config at the moment that's kind of low priority. But that needs fixing, since 11.3 will most assuredly reintroduce that fault.

VMWare.... reportedly version 7 works without a hitch. But I can't have that yet, so I had to get 6.5 in somehow. The crowbar I mentioned was killing certain processes the vmware installer spawns to compile the kernel modules it requires. Once those were ruthelessly murdered, the installer completed and it would work. Clearly, I have to be leery of kernel updates until I can use 7.

Other than some theme tweaks that I needed to redo, this was the cleanest upgrade I've had. All the Gnome stuff worked right out of the gate, which is a first. In the past I've had to blow away most of my Gnome configs and restart from scratch. I'm glad those bugs have been ironed out.