Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#81

Post by rmc »

When trying to put our webserver-cum-database-server onto nagios, I tried to apt-get install nagios-plugins. For some reason when installing that, apt wanted to remove mysql-server. I just pressed "Y" without thinking (because, hey, it's like 99.9999999% the right thing to do). So apt dutifuly stopped and uninstalled MySQL in the middle of the day.

Within about 2 minutes CTO strolls in asking about the flood of exception emails due to each request being unable to connect to the database.

Thankfully, I was able to apt-get install mysql-server, all the data was still there, and things were back to normal within 5 minutes.

Re: Ask HN: What's the worst you've ever screwed up at work?

#82

Post by pc86 »

munificent wrote: post #50
ggreer wrote: post #22

One summer in college, I got an internship at a company that made health information systems. After fixing bugs in PHP scripts for a couple weeks, I was granted access to their production DB. (Hey, they were short on talent.) This database stored all kinds of stuff, including the operating room schedules for various hospitals. It included who was being operated on, when, what operation they were scheduled for, and im…

You didn't screw up here. The entire infrastructure, org chart, and policies that allowed you to accidentally modify a production database containing critical medical information screwed up. Blaming yourself here is like blaming yourself for being hurt after being told to drive a car with no seatbelt or brakes.

He's not entirely innocent. It's more like blaming himself for being hurt after being told to drive a car with no seatbelt or brakes, while knowing that the car has neither, fully understanding what could happen, agreeing to it anyway, then driving 80 mph down a residential street while still groggy from waking up.

Regardless, I am very glad they are now out of business. :)

Re: Ask HN: What's the worst you've ever screwed up at work?

#83

Post by smartician »

ggreer wrote: post #22

One summer in college, I got an internship at a company that made health information systems. After fixing bugs in PHP scripts for a couple weeks, I was granted access to their production DB. (Hey, they were short on talent.) This database stored all kinds of stuff, including the operating room schedules for various hospitals. It included who was being operated on, when, what operation they were scheduled for, and im…

Did a similar thing, but in a less critical domain (warehouse management). Updated the status of all packages to "NEW", which would have meant that everyone who ever ordered something from that company would have gotten another delivery for free, provided the articles were in stock.

We were able to restore the data pretty quickly, but we had to interrupt warehouse workflow for a few minutes. They were surprisingly accommodating, almost amused by my mistake.

Re: Ask HN: What's the worst you've ever screwed up at work?

#85

Post by earino »

krisdol wrote: post #67
earino wrote: post #15

easy: me: "unix definitely won't just let me cat /dev/urandom > /dev/sda" other: "sure it will" me: what I learned? unix will absolutely let you hang yourself. 1998, production server for a fortune 5 company.

Why in heaven's name did you try that in production?

it was 1998. i was young and foolish. no, seriously, i was 19. i was also SUPER convinced that it wouldn't work :) I have since learned to be waaaay less convinced since then.

Re: Ask HN: What's the worst you've ever screwed up at work?

#86

Post by snikch »

Sigh, I cringe even remembering this one.

We were storing payment details sent from a PHP system into a Ruby system, I was responsible for the sending and receiving endpoints. Everything was heavily tested on the Ruby end but the PHP end was a legacy system with no testing framework. Since the details were encrypted on the Ruby end, I didn't do a full test from end to end AND unencrypt the stored results.

Turns out for two months we were storing the string '[Array]' as peoples payment details.

Takeaway: If you're doing an end to end test, make sure you go all the way to the end.

Re: Ask HN: What's the worst you've ever screwed up at work?

#87

Post by ixmatus »

ggreer wrote: post #58
munificent wrote: post #50

Earlier quoted context omitted.

You didn't screw up here. The entire infrastructure, org chart, and policies that allowed you to accidentally modify a production database containing critical medical information screwed up. Blaming yourself here is like blaming yourself for being hurt after being told to drive a car with no seatbelt or brakes.

Sure there's plenty of blame to spread around, but I still would have felt terrible if someone had been hurt or killed. What system would you put in place to prevent this? The issue was that I connected to prod when I thought I was connecting to a test DB. We each had different credentials for prod vs everything else, but the SQL client remembered my username and password. Anyone with prod access could have made the…

He could have easily revoked any UPDATE and DELETE commands from your privs list. INSERT, CREATE, SELECT is (usually) plenty fine and any database migrations that need to happen should typically be reviewed by him then run by him.

Re: Ask HN: What's the worst you've ever screwed up at work?

#88

Post by peg_leg »

This was some time ago. I've learned a lot since. I mkfs'd the main disk on our email server. There was no redundancy. There was a new volume that needed to be formatted, my superior told me to do it. I protested that I didn't quite know how. He pushed it. So, I would up wiping it by mistake. since then I've made it a mission to make the entire stack at that place resilient and redundant. Now, it's virtualized, failover file and DB systems, NLB web servers, redundant storage, proper backups. It would take a hell of a lot more than what I did to make the same mistake again.

Re: Ask HN: What's the worst you've ever screwed up at work?

#89

Post by meifun »

I once revoked my bosses e-mail and VPN access because his password was 'password123'. It was my job to keep things safe after all and I had asked him nicely a few times.

EDIT: I proposed a new password of: @$tevezA$$ignedPwD@# (Steve's Assigned Password)

He said no to that one.

Re: Ask HN: What's the worst you've ever screwed up at work?

#90

Post by miahi »

glogla wrote: post #76
michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

This is why, while I hate Oracle and everything they represent as a company, I kind of like their database because of the flashback feature. You can do SELECT * FROM table AS OF TIMESTAMP some_timestamp; and that is pretty practical. It works online, no restore, no nothing, and while it only works as long as the old data are in logs, on a production system, you should have the spare space to have some history. Theres…

I usually allocate a large part of the free space to FRA. In the production system I use right now (about 2TB of data, 50GB changes/day), I can go back a couple of weeks if needed. Fortunately everything is stable now, but that flashback was quite useful a few times.
Post reply on HN