Ask HN: What's the worst you've ever screwed up at work? • Hacker News Board

Re: Ask HN: What's the worst you've ever screwed up at work?

#31

Post by yen223 » Wed, Jan 29, 2014, 5:52 PM UTC

I wrote a piece of code controlling an assembly line machine. These machines require manual operation, and would come with a light curtain, which detects when someone places their hand near the moving parts, and should temporarily stop the machine.

A relatively minor bug in the software that I wrote caused the safety curtain to stop triggering when a certain condition was met. We discovered this bug after an operator was injured by one of these machines. Her hand needed something like 14 stitches.

Lessons learnt:

1. Event-driven code is hard.

2. There's no difference between a 'relatively minor' bug and a major one. The damage is still the same.

Re: Ask HN: What's the worst you've ever screwed up at work?

#32

Post by Davertron » Wed, Jan 29, 2014, 5:52 PM UTC

michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

I've done similar and now I almost always write a select first and then only after I've verified I'm getting the rows that I expect do I update my query to an update/delete.

Re: Ask HN: What's the worst you've ever screwed up at work?

#33

Post by [deleted] » Wed, Jan 29, 2014, 5:53 PM UTC

[deleted]

Re: Ask HN: What's the worst you've ever screwed up at work?

#34

Post by ambiate » Wed, Jan 29, 2014, 5:55 PM UTC

joshbaptiste wrote: post #7

In 2001 my first IT tech job as help desk analyst I heard beeping in the server room on one of the Solaris/Oracle machines and pressed the power off/power on button on the chassis. DBA came running in and I promptly left saying "oh I think it rebooted itself". The company went bankrupt shortly after so no huge lashing came my way but all my more experienced friends where like "wtf never do that again!"

Was probably a program running beep codes, haha. I have also had servers executing beep codes that brought great anxiety to me for endless hours. Turns out it was just some debug alert calling the motherboard speaker beep.

Re: Ask HN: What's the worst you've ever screwed up at work?

#35

Post by frogpelt » Wed, Jan 29, 2014, 5:55 PM UTC

Non-tech-related:

I was doing HVAC work while I was in college and we were removing an old air handler from underneath a house. Just inside the crawl space, under the access door was a water pipe. My boss told me to make sure I held it down while we slid the air handler out through the hole. I lost my grip on the pipe and the air handle snapped it in two, at which point gallons of water began to gush into the crawl space.

I ran for all I was worth to the road, which in this case was about 600 feet away, to turn off the water at the water meter. I ran up and down the road in front of the house and never found the water meter. So I ran back to the house and inside and told the homeowner who promptly informed me that they used well water. She called her husband and he told us where to turn off the well pump.

It wasn't really that bad in the grand scheme of things but letting the homeowner's water gush under the house for about 15 minutes does not bode well when you are supposed to be there to fix problems not create them.

Re: Ask HN: What's the worst you've ever screwed up at work?

#36

Post by Volundr » Wed, Jan 29, 2014, 5:56 PM UTC

michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

Reading all of these makes me think, the admin tool for your database of choice should probably put you inside a transaction by default, and require you to explicitly commit changes. For the madmen, it could still have an auto-commit mode, but should be opt-in rather than the default.

Re: Ask HN: What's the worst you've ever screwed up at work?

#37

Post by jmspring » Wed, Jan 29, 2014, 5:57 PM UTC

Early on in the implementation of one of the PKCS "standards" while at a browser company many years ago, due to an improper interpretation of a spec that was still in flux. There wasn't enough testing and "release bits" went live.

I had to quickly get a patch in for the improper code and had to maintain that buggy implementation. In addition, the "standard" itself got a rather scathing write up from Peter Gutmann, which is completely valid:

https://www.cs.auckland.ac.nz/~pgut001/pubs/pfx.html

This is a critique on the "standard" itself, the process was just as ugly.

Re: Ask HN: What's the worst you've ever screwed up at work?

#38

Post by cunac » Wed, Jan 29, 2014, 5:57 PM UTC

yan wrote: post #21

Not the worst at all, but probably one I found most amusing. One of my jobs included some sys admin tasks (this wasn't the position, but we all did dev ops), among my other responsibilities. I spent half a day going through everything with the person responsible for most of the admin tasks at the time. She was an extremely dilligent and competent admin, did absolutely everything through configuration management and k…

not so bad 47 minutes in first day :-)

Re: Ask HN: What's the worst you've ever screwed up at work?

#39

Post by byoung2 » Wed, Jan 29, 2014, 5:57 PM UTC

When I worked at ClearChannel back in 2010, we rebuilt Rush Limbaugh's site. When migrating over the billing system, I realized a flaw that granted at least 20,000 people free access to the audio archive ($7.95/month). The billing provider processed the subscriptions, but their system would only sync with our authentication database once a week with a diff of accounts added or removed in the past 7 days. You got the first 7 days free for this reason. If this process failed (e.g. due to a connectivity issue, timeout, or SQL error), all accounts after the error would not be updated. Anyone with a free trial or people who cancelled during a week with an error would get a permanent free trial. I rewrote the code to handle errors and retry on failure so that errors wouldn't happen in the future, but my downfall was running a script that updated all accounts to the correct status. Imagine angry Rush Limbaugh fans used to getting something for free now getting cut off (even though it shouldn't have been free). Management quickly made the decision to give them free access anyway, so I rolled back the change.

Re: Ask HN: What's the worst you've ever screwed up at work?

#40

Post by dougbarrett » Wed, Jan 29, 2014, 5:58 PM UTC

I used to work at Fry's Electronics right before iPhones were released and MP3 players were seeing better days. Creative had come out with a nice, $300 MP3 player and I was in charge of creating the sign tags in my department because I was the only one who could get it done the quickest. I would do hundreds a day, and sometimes there would be slight slip ups, in this case I forgot a 0, so there was a lucky customer that got a $300 MP3 player for $30 that day.

Luckily, there was no slap on the wrist or anything, the store manager knew that after doing thousands of these cards this was only one of a few slip ups I've made so they just brushed it off and moved on.