Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

41–50 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#41
I added some products to a system on a Thursday, not remembering we added some new columns to the product definitions, and the columns were nullable.

I was off Friday, so I come in Monday morning to see that ~20k customers have been getting free stuff since Thursday lunchtime.

Lost something like $200k because of two nullable columns :(

Re: Ask HN: What's the worst you've ever screwed up at work?

#42
post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

In this case though, wouldn't you have to COMMIT before the actual update happens ? Usually in production, it is not a good idea to have auto COMMIT on.

Re: Ask HN: What's the worst you've ever screwed up at work?

#43
Ticked a debug output flag on prod for a specific IP (Proprietary CMS, couldn't replicate the issue on test even with a full codebase and db sync), brought down the entire server for an hour.

edit: This was after I asked for permission to do this.

Lesson learned: Don't EVER use Coldfusion as a web server.

Re: Ask HN: What's the worst you've ever screwed up at work?

#44
I once worked for a company that schedules advertising before films. This wasn't in the US and the company had a monopoly over all of the ads shown across the country. It was my first programming job and done during university holidays, so I was there for a couple of months and then back to university. Toward the end of the following year I get a phone call: something was wrong with the system, it was allowing agents to overbook advertising slots. I diagnosed the problem over the phone and they put a fix in but management decided it was too late for the company to go back and cancel all of the ads that were already booked. This was not surprising as it was the most money they'd ever made. Conveniently, the parent company owned the cinemas so they did a deal where they just showed all of the ads that were booked.

Because of me, one December, everyone in the country who went to the cinema got to watch anywhere between 30 and 45 minutes of ads before the main presentation started.

Lesson learned: write more tests, monitor everything.

Re: Ask HN: What's the worst you've ever screwed up at work?

#45
At Ticketmaster, I once lazily used a N^2 instead of N in a place. I didn't think N was large. It caused the system to boot slow and my boss asked what I had done! It was a simple fix. It was in the module string table initialization.

At Graphic Technologies, I set-up a computer for our client, the Bank of England printing building. My boss asked what the root password was. I told him. He said, "That's our password!" I was never clear what our business was-- who was our company, who was other people. I did not give a fuck or they didn't bother to tell me shit.

God says... do_you_like_it what's_it_to_you God pow vengeance fer_sure commanded spoiled_brat nut_job biggot face_palm vermin abnormal horrendous jealousy smile after_a_break

Re: Ask HN: What's the worst you've ever screwed up at work?

#47
We shipped an Android app that didn't like the way we had our HTTPS certs configured, so I had logic in there to accept the connection if the cert matched the one we had.

Two months later, the certs were expiring soon and we changed our configuration to something Android liked by default. The bad news was that our production Android app rejected the new configuration and only wanted to accept the current certs.

We ended up quickly shipping a hotfix that accepted the current and upcoming configuration a few days before the certs expired. There technically wasn't any 'downtime' as long as users updated the app, but this all took place right before 'holiday vacations', and the QA team had to test the fix while all the devs were away.

Re: Ask HN: What's the worst you've ever screwed up at work?

#49
I bet > 66% of these are something to do with databases. :-)

My story (though I wasn't directly responsible): we were delivering our software to an obscure government agency. Based on our recommendation, they had ordered a couple of SGI boxes. I wrote the installation script, which copied stuff off the CD, etc. Being a tcsh afficianado, I decided to write it in tcsh with the shebang line

   #!/usr/local/bin/tcsh
Anyways: we send them the CD. Some dude on the other side logs in as root, mounts the CD, and tries to run "installme.csh". "command not found" comes the response. So he peeks at the script, and sees that it's a shell script. He knows enough of unix that "shell == bash". So he runs "bash installme.csh" . A few minutes go by, and lots of errors. So he reboots; now the system won't come up. The genius that he is, he decides to try the CD on the second SGI box. Same results.

In the script, the first few lines were something like:

    set HOME = "/some/location"
    /bin/rm -rf $HOME/*
Hint: IRIX didn't ship with /usr/local/bin/tcsh. And guess what's the value of "HOME" in bash?

Re: Ask HN: What's the worst you've ever screwed up at work?

#50
post #22

One summer in college, I got an internship at a company that made health information systems. After fixing bugs in PHP scripts for a couple weeks, I was granted access to their production DB. (Hey, they were short on talent.) This database stored all kinds of stuff, including the operating room schedules for various hospitals. It included who was being operated on, when, what operation they were scheduled for, and im…

You didn't screw up here. The entire infrastructure, org chart, and policies that allowed you to accidentally modify a production database containing critical medical information screwed up.

Blaming yourself here is like blaming yourself for being hurt after being told to drive a car with no seatbelt or brakes.

Post reply on HN