Friday evening that is.
Right before I left for the day I noticed my computer lost network. Seeing as it's directly connected to a switch, this was surprising. When I bipped into the utility room to see what was going on, I found the switch in reboot mode and a fellow employee behind the rack doing perfectly legitimate business things.
Perfectly legitimate business things that over the course of a year or so had managed to work the power cable out of the Ethernet switch. We didn't get one with redundant power supplies, it's just the office network, not critical like our actual revenue systems, so this caused a switch reboot.
It didn't come up. Crap.
Very, very happily I'd already figured out what combination of serial cable and minicom settings I needed to talk to this switch over the console port so I was able to plug in and see WTF was going wrong.Â
Error, /cfa0/boot.ini corrupted; please reboot to console and repair.
Bugger.
Happily, I already had software images on that laptop so I proceeded to set my baud rate to 115,200 and uploaded a new one via XModem. Since I was not doing this at 9600 baud, this only took about 10 minutes for an 8.2MB file.
Software image corrupted.
Bugger. Looking around the file-system I saw a strange directory in there:
/cfa0/.Trash-nebeker/
Huh. A ".Trash-$Username" folder is dropped by Gnome2 on removable media if something is deleted on it. How in blazes did that get onto a factory firmware image? A bit of Googling brought me to a certain HP Customer Advisory. Yep, looks like from 2009 to May of 2010 that directory was indeed baked into switches, and was definitely causing problems.
Since my switch was in the switch has already failed to reboot or failed on software update state, I had to follow that workflow. Running the given lshw command and removing the bad boot.ini file did allow the switch to boot into its normal state. I tried updating to "a switch software version which automatically removes the extraneous files", but no matter how I tried to update the firmware I got
Software image corrupted.
USB, TFTP, even another XModem upload. Same thing, every time, from fresh downloads even. Clearly, this option wasn't going to work for me, so I had to go to the "show tech custom" script they mention.
Frought with peril.
First of all, the link to download the show-tech file didn't work anymore. The password had been changed or something, I don't know. It didn't work.
Google to the rescue!
Some happy soul had posted it to a spiceworks page. Looking at it, I saw nothing that would screw my switch over, so I downloaded it and tried to TFTP it to my switch.
While firmware updates TFTPed in just fine, for some reason the show-tech script just Would. Not. Download. Ever. I spent far too long on that one, and even set up another tftp server just in case the one I had been using was futzy in some way. Nope. Not happening.
Grasping at straws I booted back into Console Mode to maybe, I don't know, upload the show tech script that way or something. Straws, grasping of. It turns out you can't do that from the console. Nor can you hand-execute the commands in the show-tech file.
However, I noticed there already was a show-tech file on the filesystem next to all of those other bad files. Maybe that's getting in the way?
- rm /cfa0/show-tech
- lshw cfa0 btm.swi secondary.swi boot.ini cfg/ autorun/ ssh/ rbtcnt iflags ssl/ mgrinfo.txt btm.swi_$TmP$ secondary.swi_$TmP$
- rm /cfa0/boot.ini
- jp 1
Which got me booted at least. THEN the TFTP worked and I could run the bloody file. Yay!
I tried to update directly to the latest version we're running, but apparently you can't go directly from W.14.30 to W.15.08 (or at least, this switch couldn't). I had to detour through W.14.70 first:
Image          Size(Bytes)  Date  Version
-----Â Â Â Â Â Â Â Â Â Â ----------Â -------- -------
Primary Image  : 8457322  08/05/09 W.14.30
Secondary Image : 8649497Â Â 12/13/11 W.14.70
And then onto W.15.08.
Image          Size(Bytes)  Date  Version
-----Â Â Â Â Â Â Â Â Â Â ----------Â -------- -------
Primary Image  : 9395197  07/20/12 W.15.08
Secondary Image : 8649497Â Â 12/13/11 W.14.70
It was a lot of updating and waiting, especially while I updated the Secondary firmware image. But in the end, this switch is now at the same version as our other switches of that type!
It took not quite four hours for all of that.