This isn't my most elegant hack, but it is the one I'm most proud of given both the stakes involved and the amount of learning I had to due under pressure.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
This is the story of how early in my career (~15 years ago) I saved the jobs of 100+ employees along with $20M+ of accounted revenue...
I had built an application that had become the core of a large and prestigious business (can't describe the software or business sector due to NDA). One night I get an alert that one of the drives in the server hosting the application was failing and needed to be replaced -- not a big deal since the system was running RAID-5 so it it could take a hit. So I contact the IT company providing service for the business and they sent out a tech...
.. who was a little green...
When they reconnected the drives to the motherboard they were out of order and the RAID controller went to work using the new (zero'd out drive) as part of the main array. The tech freaked out when the system wouldn't mount and then proceeded to swap the cables around several times while rebooting!!
This effectively "encrypted" the drives!
I get a call much later that night from a crying tech informing me that the machine is trashed and that the backups had failed!!
I immediately drive to the site and collect the hard drives to bring back to my home office, first thing in the morning I purchase several large drives to duplicate on to and return the original drives to the business so they can send them out to a pro data recovery company.
Upon examining the drives I learn that there are three different partition schemes across the four drives which are using RAID-5 with LVM, but with unknown RAID parameters...
So I start hitting the books, and by that I mean reading the source code for ext3, LVM, and MDADM -- at the time the relevant block level device driver code for MDADM was extremely terse, almost un-commented, and in German, which I do not speak O_O
I took several 16-20 hour days of pouring through sector by sector hex dumps of the drives, but I was eventually able to line up and decode the LVM header blocks and work backwards to find the ext3 superblocks and the start of the filesystem. I then wrote my own RAID-5 virtual loopback device driver that knew about the permutations in the data blocks caused by the drive shuffling and was able to extract a complete filesystem. After hand patching the superblocks I was able to mount it and recoverd all of the mission critical data!
Prior to this I only had an abstract sysadmin-level knowledge of Linux filesystems :)
Also, the pro data recovery firm wasn't able to recovery anything and tried to lie saying that the drives were physically damaged! The CEO and CFO had me sit in on the phone call and jump in to correct them, the tech from the recovery company was speechless and then accused us of trying to set them up! We (the C-levels and myself) had a good laugh and then went out to have a couple drinks ;)