Unix Recovery Legend (1986)

> Well, for one thing, you must always remember the immortal words, DON'T PANIC

So true. A colleague of mine managed - on his second or third day on job - to delete every single user account in our Active Directory. After an hour, we gave up trying to restore the AD (it was an SBS2008, so no AD recycle bin) and simply restored the entire DC (at the time, our domain only had the one DC) from backup. Surprisingly, most of our users took it very well and used the time to get some paperwork done or clean up their desks or something like that. Still, it was one of the most stressful days of my life. So we kind of panicked. In restrospect, I think another hour or so of research might have saved us the eight hours of restoring that server (did I mention that our backup infrastructure really, really sucked at the time?).

In smaller desasters, I've found the ability to remain calm most valuable, though. Having your boss breathing down your neck impatiently can instill a deep desire to simply do something just to show that you are working on the problem. But if you don't understand what's wrong, at best you are wasting time, and possibly making the problem even worse.

Back in the days when there were bad proprietary IDE controller chipsets that enumerated differently under Linux than they did windows...

Well, as a kid I clobbered my windows install while trying to destructively check the validity of a disk. Checks to see if a filesystem is mounted won't protect you if it isn't.

Since then I've been -extremely- careful for any destructive operation on block devices, and generally careful otherwise.

Also, I don't know if blkid existed at the time, but it's /very/ useful in avoiding those types of mistakes.

That's why I freakin' love Slackware. The ability to repair stuff by hand, or even to get libc6 binaries running on a libc5 system (by putting the appropriate .sos in /lib), was a godsend during my early Linuxing, and completely unthinkable under Windows. It's very nearly unthinkable under Ubuntu or Fedora, but Slackware just keeps chugging the best it can no matter what shape you bash it into.

I'm curious to see what your tcl code looks like. Did you keep it around for future disasters or was it just a quick and dirty one-off?

6-7 years ago, a co-worker was doing some work on a server that we were phasing out. It wasn't being used for anything of critical importance, so a Jr level person was permitted to have root. He was actually my colleague and equal on the org-chart but he had less experience with *NIX administration.

He was copying directories to the new server and deleting them when he was done. Well...

I don't remember the sub-directory he had just finished copying but when he went to delete it he typed in "rm -rf / SOMEDIR/SOMESUBDIR" and hit enter.

He almost immediately realized what he had done and hit CTRL-C but by that point the damage was done. Our boss had access to the previous week's backups and since the server wasn't critical, he just rebooted it and restored but we had a good laugh about it.

The next day, I made a paper airplane and on the side I wrote in red Sharpie "Linux Air" and then "rm -rf" inside of a red circle with a line through it and taped it to the top of his monitor.

He was a good sport about it and left it up for a couple of months.