A Knoppix Rescue
Where I work, we’re just a bit paranoid, so it’s not enough to just mirror your bootdisk and have it backed up to tape, we also create an extra bootable volume, just in case. The extra bootdisk comes in very handy for doing OS upgrades, allowing us to install a new OS on the extra bootdisk with a very easy fallback to the original OS.
I’d been running one server on the (former) extra bootdisk and it was time to restart the cron job that keeps the extra bootdisk up-to-date. The cron job ran fine and I checked out the disk, everything looked fine. It was then time to check out the (new) extra bootdisk, so I picked a slow time, hopped on the console and issued the shutdown. BIOS checks went as expected, the screen cleared and I waited for the grub menu to appear. And I waited and all it said was “GRUB GRUB”. Ahhhhh what is happening?
What I’d forgotten was that the new extra bootdisk was actually the first disk in the system and it had the bootable partition on it. And it had the grub bootloader on it. And that the cron job recreated the ext3 filesystem using mke2fs, which ended up trashing the bootloader.
I ended up grabbing a Knoppix CD that I had at the house and ran into the office so I could attempt to recover. Knoppix can boot up the Compaq ML-380 my system is on, but for some reason, it didn’t automatically make the devices for the CCISS drives. Looking at the lsmod output, I found that it did contain the cciss.o module, so I just had to figure out what the major and minor numbers and do the appropriate mknod commands. (The cciss devices are major number 104 and c0d0 has minor number 0, c0d0p1 minor 1, c0d0p2 minor 2, c0d1 minor 16, c0d1p1 minor 17, etc.) I could then mount the partition, do a grub-install and reboot successfully.
A couple of Knoppix cheat codes also came in handy, booting with knoppix-txt instead of knoppix (to keep from going into graphics mode, just in case the kvm setup can’t handle it) and using ‘2’ so that it just drops straight to a root bash prompt (no need to go fully graphical when the command line is all you need).
I’ve also updated the cron job so that it always puts the grub bootloader on the extra bootdisk, no matter if it’s the first disk in the system or not :-)
Friday December 10, 2004 · Permalink