So the post is not on the front page anymore but I guess confessing feels good juging by all the people that contributed.
My screwup was at my first "real" job, fresh out of college. I was asked to free up some space from the production server at $BIGCOMPANY, because it was already at 99% capacity (it managed to get to a 100% for a few minutes before I "solved" the problem). The thing is, at this $BIGCOMPANY, for some reason the budget for disk drives was non-existant, and this meant that whenever the disk usage was at or below 95%, we were happy because we still had free space... figure that.
So here I come, armed with the most dangerous tool a newbie can wield... root access and the drive to impress your boss. I said to myself, "I've used root at my home machines plenty of times and nothing bad happened because I've been using Linux for several years by now and I know I need to be careful... so I don't get why everyone says you should never log in as root". Oh boy, how I learned the hard way.
To continue my story, it turns out that the easiest/fastest way to free up some space was to delete the log files for pretty much everything(except the last 5 or 10 logs... because we were "careful", in case we ever needed them). We usually deleted things under certain directories known to hold "useless" logs. So here comes Mr. Newbie-guy-with-the-need-to-shine, and I thought to myself, "why keep deleting the logs from the same directories over and over if that only buys us about 1 or 2 percentage points, instead of cleaning as much logs as possible for the system and freeing up a lot more space?"
After thinking about it for like 10 seconds the most genius thought of my career materializes: do an rm -rf *.log on the topmost level directory of where we used to store everything (webserver, webservices, databases, etc). I happily pressed enter, and a couple of minutes later, hooray! I got the disk usage down to a whooping 90%! I was a hero! that meant we had bought enough time to keep on working without worrying about the disk space for at least another one or one and a half months. This was a clear victory and an testament to my superb sysadmin skills.
Fast forward 4 hours, and the phone starts ringing like crazy as every other employee (non-IT ones) started wondering and then calling us to try to figure out why was their data gone. They did not understand how come they have been working A-OK so far, and then suddenly ALL data from sales team, admin team, the bosses, etc was gone. And then a few minutes later... the whole intranet came down crashing and burning.... then a full stop... nothing was working.
So we went to the logs directory... oops... no logs there!. Ok, let's try to ping the DB. Dead. It's not running and it's responding with an unknown error. When I tried to connect to it would do so, but then some cryptic ORA-xxxxx error came up. No problem says I, I'll just google it and fix it.
Not so fast young grasshopper. That error meant that the DB was out of sync with its own files used for, ironically, data corruption prevention and rollback (or something like that... to this day I still don't fully undersand what those files were used for).
As far as I can remember those logs where a sort of pre-commit place, where all changes would be stored on those files and every X amount of hours the changes would get commited to the actual DB tables. It was some functionality that supposedly was used to correct corrupted entries and to recover (figure that.. ) and rollback data when lost, or something like that. And unfortunately bringing the system back in-sync was way out my league (did I failed to mention that I was by no means a DBA?).
However a struck of good luck came down on me, as the company had a support contract with Oracle and it was the Platinum-covered-diamonds level or something. That meant that after creating a support ticket at like 1AM, I got a call from one of the support guys like less than 20-30 minutes later. This guy seems calm and tells me I should not panick, it was just as easy as doing $crypticOracleStep1, $crypticOracleStep2, $crypticOracleStep3 and voilá! all would be good again. Except for the fact that I had NO IDEA what those steps actually required me to do. Almost in tears I ask the rep to pretty please SPELL every command I needed to execute, letter by letter. I did not want to screw up again.
So there I was, at close to 2AM, with my boss breathing down my neck asking me what every frigging letter of the command I was typing did (which I had no idea...), all the while trying to keep up with this supper friendly guy that was patient enough to spell everything two times.
After a couple of commands later, behold! the DB could be brought up again! oh boy, did I felt relieved. I was jumping up and down because I had fixed my stupid mistake... or so I thought. After almost causing the support guy to go deaf due to my loud cheering, he says "however...". wait... what? there's a "however"?!?. Then he continues saying, "since you deleted the pre-commit file of the last day, the DB is back in-sync... up to yesterday". My jaw dropped to the floor. That meant that the ENTIRE previous day was utterly lost.... sales data, contracts, customer's info, etc.
I thanked the guy for his help, hanged up the phone and turned to my boss telling him that I was ready to turn in my resignation letter just after helping capture what availabe data was actually there (in papers, by calling customers and asking them again, etc).
My boss then turns to me and says, don't worry. We've all been through this at least once in our careers. Even I made a mistake that is terribly similar... however when I brought down the database, it took us one full week instead of one day... and rest assured that as I learned my lesson, you did as well. And I need guys like you, that have the initiative to solve things... and the ability to learn from mistakes. So don't worry, you are not losing your job. However you can't go home until you help everyone get as much data back as you can.
Aw shoot... well.. I guess it could've been worse. So after having lunch with my boss and the other teammates at like 6-7 AM, I went to the sales dept and started asking around how could I help them get their data back.
Those were the longest 38 continous work hours I've ever had to resist. I did not go back home until more than a full day and a half later. I was tired as hell to say the least... but to this day I think it was a blessing that I got to learn such a hard lesson but being backed up by a boss that was very cool and progressive about it.
Lessons learned:
0) Never ever ever ever use root, especially for deleting files and ESPECIALLY with the -f flag.
1) Do not assume that something you know will hold. Confirm it in the particular system you are going to be working with. (i.e. do not assume .log files are always log files because in your laptop that holds true)
2) Be ready and willing to assume the consequences of your actions. Most of the time if you assume responsability for your mistakes, people will forgive you and even give you a piece of advice.
3) Never ever ever ever use root.