> Well, for one thing, you must always remember the immortal words, DON'T PANIC So true. A colleague of mine managed - on his second or third day on job - to delete every single user account in our Active Directory. After an hour, we gave up trying to restore the AD (it was an SBS2008, so no AD recycle bin) and simply restored the entire DC (at the time, our domain only had the one DC) from backup. Surprisingly, most…
The proactive fix was simple. Backup, restore the table in a two hour outage window. I asked for 5 hours, as the previous DBA never tested a restore. "Unacceptable", said the SVP.
Fast forward two weeks. We hit the limit, and the entire company is essentially down. Between lost revenue, SLA fines and payroll, they were losing something like $5k/minute. Recovery at this point required 30 hours of full database restore, including journal recoveries from slooooow DAT tapes.
There were three of us, everyone stayed calm, provided regular updates and handled a few hours of direct observation by the CEO and a board member for a few hours.