Ask HN: What's the worst you've ever screwed up at work? • Hacker News Board

Re: Ask HN: What's the worst you've ever screwed up at work?

#21

Post by yan » Wed, Jan 29, 2014, 5:47 PM UTC

Not the worst at all, but probably one I found most amusing. One of my jobs included some sys admin tasks (this wasn't the position, but we all did dev ops), among my other responsibilities. I spent half a day going through everything with the person responsible for most of the admin tasks at the time. She was an extremely dilligent and competent admin, did absolutely everything through configuration management and kept very thorough personal logs and documentation on the entire network. One of my first tasks was to change backup frequency (or other singular change) and going by how I usually did things at the time, just sudid a vi session, changed the frequency and restarted the service.

She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows and gave me a look. She quickly reverted what I did, updated our config management tool, tested it, then deployed it, while explaining why this was the right way to do things. I slowly came around to doing things the right way and haven't thought much about the initial incident until we found her personal logs that she archived and left on our public network share for future reference.

In the entries for the day that I started, we saw the following two lines:

    [*] 2007/09/09 09:58 - yan started. gave sudo privs and initial hire forms.
    [*] 2007/09/09 10:45 - revoked yan's sudo privs.

Re: Ask HN: What's the worst you've ever screwed up at work?

#22

Post by ggreer » Wed, Jan 29, 2014, 5:48 PM UTC

One summer in college, I got an internship at a company that made health information systems. After fixing bugs in PHP scripts for a couple weeks, I was granted access to their production DB. (Hey, they were short on talent.) This database stored all kinds of stuff, including the operating room schedules for various hospitals. It included who was being operated on, when, what operation they were scheduled for, and important information such as patient allergies, malignant hyperthermia, etc.

I was a little sleepy one morning and accidentally connected to prod instead of testing. I thought, "That's weird, this UPDATE shouldn't have taken so long-oh shit." I'd managed to clear all allergy and malignant hyperthermia fields. For all I knew, some anesthesiologist would kill a patient because of my mistake. I was shaking. I immediately found the technical lead, pulled him from a meeting, and told him what happened. He'd been smart enough to set up hourly DB snapshots and query logs. It only took five minutes to restore from a snapshot and replay all the logs, not including my UPDATE.

Afterwards, my access to prod was not revoked. We both agreed I'd learned a valuable lesson, and that I was unlikely to repeat that mistake. The tech lead explained the incident to the higher-ups, who decided to avoid mentioning anything to the affected hospitals.

If it's any consolation, the company is no longer in business.

Just remember when you screw things up: Your mistake probably won't get anyone killed, so don't panic too much.

Re: Ask HN: What's the worst you've ever screwed up at work?

#23

Post by sugerman » Wed, Jan 29, 2014, 5:48 PM UTC

michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

I did this once and have since always typed the WHERE first in an UPDATE.

Re: Ask HN: What's the worst you've ever screwed up at work?

#24

Post by Volundr » Wed, Jan 29, 2014, 5:48 PM UTC

michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

Been there done that. Usually I always work inside a transaction, and carefully examine the results before typing that all important 'commit'. But a "simple" change at 4:55 and me in a hurry to get home....

Re: Ask HN: What's the worst you've ever screwed up at work?

#25

Post by dboyd » Wed, Jan 29, 2014, 5:49 PM UTC

michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

Exactly the same...

I was hired by my college to build a grade management system in my second-to-last year there. I was in a hurry due to a lunch meeting with other IT staff at the University, forgot to add the where clause, and suddenly every single student was a Computational-Science major (mine).

Funny part of the story was that the moment it happened I uttered "oh shit." My boss, who sat beside me, said "what'd you do?", and about 15 IT staff from other departments walked into the office to go out for lunch. I'm sure I was an interesting shade of red.

I had to explain what I did in front of all these people. My boss laughed out loud, brought the system offline, and simply said: "well, after lunch we get to test our backup process." We went for lunch.

Two valuable lessons I learned...

People make mistakes, that isn't a problem, it's how they respond that's important.

Don't try and solve hard problems when emotions are running high. If shit is going down in production, the most important thing to do is to breathe, and get a glass of water. That little bit of time helps a lot.

Re: Ask HN: What's the worst you've ever screwed up at work?

#26

Post by marvvelous » Wed, Jan 29, 2014, 5:50 PM UTC

michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

I did something similar once. Now my query writing goes:

> UPDATE

> UPDATE SET url='new_url' WHERE source_id IN (etc)

> UPDATE sources SET url='new_url' WHERE source_id IN (etc)

Re: Ask HN: What's the worst you've ever screwed up at work?

#27

Post by gargarplex » Wed, Jan 29, 2014, 5:51 PM UTC

michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

I always add a LIMIT even when not necessary.

Why doesn't MySQL have a version control baked in? Even if it preserves just the last n hours of state..

Re: Ask HN: What's the worst you've ever screwed up at work?

#28

Post by Davertron » Wed, Jan 29, 2014, 5:51 PM UTC

I wrote an update script for a database table not realizing I had the key wrong (I'm kind of fuzzy on the details, but essentially I think it was a composite key but I was only using one of the columns in my WHERE clause...) and accidentally updated all customers addresses in our database to the addresses of one account.

Luckily we had backups from that morning so we only lost any address updates people would have done that day, but it made for some interesting customer service calls for awhile...

Re: Ask HN: What's the worst you've ever screwed up at work?

#29

Post by ambiate » Wed, Jan 29, 2014, 5:51 PM UTC

zimpenfish wrote: post #2

Many years ago, when I was but a fresh faced idiot, the partition that contained the mSQL database which had All The Data filled up. I moved it into /tmp because there was plenty of space. On a Solaris box. Hilarity ensued when we next rebooted it.

For those who don't know, solaris uses tmpfs for /tmp. It is a virtual memory/swap based file system. Anything in /tmp is actually temporary if the machine reboots/powers off.

Re: Ask HN: What's the worst you've ever screwed up at work?

#30

Post by TheCapn » Wed, Jan 29, 2014, 5:51 PM UTC

I may, and/or may not have caused a production site's PLC to go into STOP mode during daily operations while making network updates remotely.

Possible outcomes of unplanned system haults include plugged machinery that would need to be manually cleared, mixed products which would become immediate net losses for the company and damaged motors.

Thankfully no product was being run at the time. I have also implemented changes across the board to our client sites that prevent this type of shit from ever happening again. You know when you look at a system and go "this is going to bite us in the ass eventually?" This was one of those systems, they just needed a new hire to give them the push.