Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

121–130 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#121
A long time ago while working on a *nix box logged in as root, I executed a simple "!find". Basically execute the last find. In root's history, the last find command was something like "find ... -exec rm ...". The command was run at the root of the content directory of a CMS, deleting all the content (major media website). CMS was down while backups were restored.

I now never execute ! commands as root. Actually, nowadays I simply use CTRL-r.

Re: Ask HN: What's the worst you've ever screwed up at work?

#122
post #89

I once revoked my bosses e-mail and VPN access because his password was 'password123'. It was my job to keep things safe after all and I had asked him nicely a few times. EDIT: I proposed a new password of: @$tevezA$$ignedPwD@# (Steve's Assigned Password) He said no to that one.

While awesome this is not a screw-up.

The screw-up is the boss in this one.

Re: Ask HN: What's the worst you've ever screwed up at work?

#123
Around 2000 my team was responsible for installing and maintaining a larger amount of servers in 19" racks in a data centre.

Most servers had those hot swap drive bays for convenient access from the front while the server was running. You only had to make sure no write operation occurred while you pulled the drive out of the bay.

So, I had to exchange a backup disk on a database server running quite a few rather large forums. The server had two disk bays: One for the live hard disk and one for the backup disk. I was absolutely sure at that time which one was the backup disk so I didn't bother to shut down the database server and incur a minimal downtime. Of course, I was wrong and blithely yanked the live disk from the drive bay.

I spent the rest of the night and most of the following day running various MySQL database table repair magic. It worked out surprisingly well but having to admit this error to our forum users was embarrassing, nonetheless.

Lesson: Appropriately label your servers and devices.

Re: Ask HN: What's the worst you've ever screwed up at work?

#125
Two screw ups come to mind.

1. First day at a job. I need to get familiar with a legacy system and get a SQL dump from it to create a local copy of the database. After some SSHing and MySQLing, I confuse my two split terminal panes and end up importing my local dump to production server. Of course database names and users were the same so I end up dropping the database. No biggie. Backups were available from previous day.

2. Similar story to the first one. I got a new shiny Zend Studio IDE. Want to set up sync with remote server (just a static company website with no version control). Fill all the settings, press the sync button - and what happens? Zend Studio somehow figured that I want to force sync my local folder, which is empty, to the remote site, and it just deletes everything on the web root and uploads my empty folder. Wat. Should have read the settings twice.

Re: Ask HN: What's the worst you've ever screwed up at work?

#126
post #79
post #58

Earlier quoted context omitted.

Sure there's plenty of blame to spread around, but I still would have felt terrible if someone had been hurt or killed. What system would you put in place to prevent this? The issue was that I connected to prod when I thought I was connecting to a test DB. We each had different credentials for prod vs everything else, but the SQL client remembered my username and password. Anyone with prod access could have made the…

In the past, I've set up big MOTD style messages that say "PROD" in fancy ASCII graffiti when I ssh/connect a DB client/whatever to production. I think I will set one of those up now for my current setup. Also, sort of related, I'm using MacOS, and in the back of my head I've wanted to create a tool that will change the color of the menu bar (at the top of the screen) to, say, bright yellow, when I'm connected to the…

Most VPNs can be configured to only route certain subnets over them (so all work related networks, for example) instead of everything. This is very simple to do with OpenVPN; can't speak as to the Mac builtin solution.

Re: Ask HN: What's the worst you've ever screwed up at work?

#127
I dropped two units of equipment, ~$1.5Mil a piece. Each unit was dropped in separate incidents. No damage at all, but management didn't care. I blamed myself despite mitigating factors such as impossible schedules, vicious multi-tasking "to compensate", and less-than-ideal support equipment. At the time, I didn't handle it very well but I ended up living through it -- first job//assignment ever in the worst environment I've ever had before or since with the worst coworker I've ever had before or since, and I mess up in the millions of dollars. "Lasting Impressions", tonight at 8/7 Central.

I left that job about 3 years later when the metaphorical train stopped at a nicer place. My name is still known in certain circles for this ["Oh bah, how could I forget?" one former manager recently stated], but I don't plan to go back there at this time.

I learned that life's too short for assholes and working in an environment you don't like. If you don't screw up, your soul will die and you'll become that former coworker you hated so much and who hated you in return. It's worth picking and choosing where you work.

Re: Ask HN: What's the worst you've ever screwed up at work?

#128
Added some additional logging for an edge case, rolled it out to production and then went camping in the remote wilderness for a week. Two days in, the edge case got hit and the logging wasn't sufficiently tested. It logged as intended... and kept logging and logging... until out of disk space :(

Oh yea, I run a proprietary trading firm (still at the same spot), as a result of that bug we went down and lost about $250k over the next few hours. Testing is important in automated trading :)

Re: Ask HN: What's the worst you've ever screwed up at work?

#129
post #93

I've only cried literal tears once in the last ten years, over business. Due to inattention while coding during an apartment move, I pushed a change to Appointment Reminder which was poorly considered. It didn't cause any immediate problems and passed my test suites, but the upshot is it was a time bomb that would inevitably bring down the site's queue worker processes and keep them down. Lesson #1: Don't code when y…

Can't agree more with Patrick, mine is formulated a bit differently:

Lesson #1: Don't push code on Friday afternoon.

Lesson #2: Beer, Code and Commit is totally fine. Just don't push! Wait until next day to review and push/deploy it..

Re: Ask HN: What's the worst you've ever screwed up at work?

#130
Happened to a colleague: it was the end of the day, and we were packing up to leave. He used Ubuntu on his notebook, so he typed "shutdown -h now" on his shell prior to closing the lid. Seconds later he's groaning, having noticed it was a SSH session to the production server...

It wouldn't be a big deal, wasn't for the fact it was an EC2 instance, and back then halting the instance was equivalent to deleting it permanently. We then spent the night at the office recovering and testing the server. I think we left 3:00 AM that day.

Lesson #1: it's never a good idea to "shutdown -h now" on a shell. any shell.

Lesson #2: have the process to spin up a new production server fully automated and tested

Post reply on HN