I now never execute ! commands as root. Actually, nowadays I simply use CTRL-r.
Ask HN: What's the worst you've ever screwed up at work?
121–130 of 322 posts
Re: Ask HN: What's the worst you've ever screwed up at work?
#122I once revoked my bosses e-mail and VPN access because his password was 'password123'. It was my job to keep things safe after all and I had asked him nicely a few times. EDIT: I proposed a new password of: @$tevezA$$ignedPwD@# (Steve's Assigned Password) He said no to that one.
The screw-up is the boss in this one.
Re: Ask HN: What's the worst you've ever screwed up at work?
#123Most servers had those hot swap drive bays for convenient access from the front while the server was running. You only had to make sure no write operation occurred while you pulled the drive out of the bay.
So, I had to exchange a backup disk on a database server running quite a few rather large forums. The server had two disk bays: One for the live hard disk and one for the backup disk. I was absolutely sure at that time which one was the backup disk so I didn't bother to shut down the database server and incur a minimal downtime. Of course, I was wrong and blithely yanked the live disk from the drive bay.
I spent the rest of the night and most of the following day running various MySQL database table repair magic. It worked out surprisingly well but having to admit this error to our forum users was embarrassing, nonetheless.
Lesson: Appropriately label your servers and devices.
Re: Ask HN: What's the worst you've ever screwed up at work?
#124# cd /etc # emacs inetd.conf # ls ... ... inetd.conf ... inetd.conf~ ... # rm * ~ # ls # ls
Re: Ask HN: What's the worst you've ever screwed up at work?
#1251. First day at a job. I need to get familiar with a legacy system and get a SQL dump from it to create a local copy of the database. After some SSHing and MySQLing, I confuse my two split terminal panes and end up importing my local dump to production server. Of course database names and users were the same so I end up dropping the database. No biggie. Backups were available from previous day.
2. Similar story to the first one. I got a new shiny Zend Studio IDE. Want to set up sync with remote server (just a static company website with no version control). Fill all the settings, press the sync button - and what happens? Zend Studio somehow figured that I want to force sync my local folder, which is empty, to the remote site, and it just deletes everything on the web root and uploads my empty folder. Wat. Should have read the settings twice.
Re: Ask HN: What's the worst you've ever screwed up at work?
#126Earlier quoted context omitted.
Sure there's plenty of blame to spread around, but I still would have felt terrible if someone had been hurt or killed. What system would you put in place to prevent this? The issue was that I connected to prod when I thought I was connecting to a test DB. We each had different credentials for prod vs everything else, but the SQL client remembered my username and password. Anyone with prod access could have made the…
In the past, I've set up big MOTD style messages that say "PROD" in fancy ASCII graffiti when I ssh/connect a DB client/whatever to production. I think I will set one of those up now for my current setup. Also, sort of related, I'm using MacOS, and in the back of my head I've wanted to create a tool that will change the color of the menu bar (at the top of the screen) to, say, bright yellow, when I'm connected to the…
Re: Ask HN: What's the worst you've ever screwed up at work?
#127I left that job about 3 years later when the metaphorical train stopped at a nicer place. My name is still known in certain circles for this ["Oh bah, how could I forget?" one former manager recently stated], but I don't plan to go back there at this time.
I learned that life's too short for assholes and working in an environment you don't like. If you don't screw up, your soul will die and you'll become that former coworker you hated so much and who hated you in return. It's worth picking and choosing where you work.
Re: Ask HN: What's the worst you've ever screwed up at work?
#128Oh yea, I run a proprietary trading firm (still at the same spot), as a result of that bug we went down and lost about $250k over the next few hours. Testing is important in automated trading :)
Re: Ask HN: What's the worst you've ever screwed up at work?
#129I've only cried literal tears once in the last ten years, over business. Due to inattention while coding during an apartment move, I pushed a change to Appointment Reminder which was poorly considered. It didn't cause any immediate problems and passed my test suites, but the upshot is it was a time bomb that would inevitably bring down the site's queue worker processes and keep them down. Lesson #1: Don't code when y…
Lesson #1: Don't push code on Friday afternoon.
Lesson #2: Beer, Code and Commit is totally fine. Just don't push! Wait until next day to review and push/deploy it..
Re: Ask HN: What's the worst you've ever screwed up at work?
#130It wouldn't be a big deal, wasn't for the fact it was an EC2 instance, and back then halting the instance was equivalent to deleting it permanently. We then spent the night at the office recovering and testing the server. I think we left 3:00 AM that day.
Lesson #1: it's never a good idea to "shutdown -h now" on a shell. any shell.
Lesson #2: have the process to spin up a new production server fully automated and tested