Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

211–220 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#211
Still feeling my way around in the new job, I was fiddling with a backup script, got distracted, turned back, and dropped the production database. Two minutes later "Hey, is the website down"? Then I look at the prompt...

I run around like a headless chicken trying to find who knows the right backup to use and so forth, and I can't figure out why everyone is so calm and collected about it. Production was down/shit, I hope I still have a job. Turns out we had no active clients at the time - no-one was accessing the site. We'd finished one run and were in 'dead time' before the next. My next project involved implementing coloured prompts and I no longer leave production ssh sessions lying around when I've finished with them.

My CTO still has me listed as "database [vacri]" in his phone...

Re: Ask HN: What's the worst you've ever screwed up at work?

#212
This wasn't me, but a coworker.

We were rearranging the layout of the office. Coworker was moving in to his new space, setting up his desk. He boots up his computer, wonders why he has no network. Looks around, discovers the ethernet cable isn't plugged in. Plugs it in to the wall, still has no network.

A few minutes pass, and the entire office is running around wondering why the hell the network isn't working. Maybe an hour passes, the network guys are losing their shit trying to hunt down what is wrong. I'll give you a hint: the router was lit up like a Christmas tree, and the aforementioned coworker had both ends of his ethernet cable plugged in--but neither end was attached to his computer.

Re: Ask HN: What's the worst you've ever screwed up at work?

#213

Long ago when I was, I think, a sophomore in college and worked for the university IT group, I was trying to add an external drive to an early NeXT machine [1]. I wanted to try out their fancy GUI development stuff, you see. I was at best a modestly competent Unix admin, and this was circa NextStep 1.0, so the OS was... rough. It was in the dark days of SCSI terminators, so just telling if the drive was properly conn…

"When some young idiot fucks up, check to make sure that they are sufficiently freaked out. If they are, no need to yell at them. Instead support them in solving the problem."

My first job was as a Marine avionics technician and I had the misfortune to work for bosses who reassured themselves by blaming you at full volume and making you feel like shit. Usually with the 'you could have killed somebody you useless idiot' which is actually not that often true. Not a great environment.

Last week I'm just starting my second year at this company as a real proper electrical engineer and I neglected to run all the steps of the pre-tests I said I ran and I therefore missed a (in retrospect, really cool) bug that showed up in front of the certifying examiner on a Friday afternoon. Said examiner costs a lot of money per hour, it's a government agency, etc. etc. I spent a weekend and Monday finding the bug and squashing it. It took til Friday before the new code was released and all the paperwork up to the certifying agency was fixed.

The whole time my boss was there and clearly frustrated but also very clearly keeping quiet. I definitely went and thanked him on Friday afternoon when it was all over for being so chill about it. He said it was no problem as he could see I had sufficiently embarrassed myself and was working hard to fix it. It's so true. I was keeping my head down, being calm but intent on the problem, and I don't know how I could've solved anything if he'd lost his shit.

Fear and care are definitely with me now. As is proper testing and better logging of my own day-to-day dev process. (Note to self: power cycle in the test when it tells you to power cycle. You're also testing the non-obvious functionality.)

Re: Ask HN: What's the worst you've ever screwed up at work?

#214
rm -rf .

yup. that really happened. it was 4-5am in the morning and I'd been working all night. I was on the server trying to set something up and was trying to blow away a folder ... I did a normal rm and that didn't work (obviously) because there was crap in the folder. So I pulled out my nuclear weapon to nuke the folder but left off the preceding ./ (which still wasn't that smart anyway) ... I sat there for a second wondering why the deletion was taking so long ... then another 30 then a minute ... then I looked at what I'd just typed again ... then I realized what had happened.

ctrl-c'ed (or d, can't remember now) out of it. then tried to find root folders

cd /etc => folder not found

cd /var => folder not found

I'm from a third world country where we laugh at Americans (sorry) for throwing up when they're nervous or having panic attacks, but at that moment, I had a full blown panic attack. I'll never forget it.

The work was a subcontract for a client who was doing work for Nike, and it was a decently sized project that was critical to the success of the firm, and I'd just blown away their live production server ...

Afer freaking out and almost crying for 5 minutes. I decided to call media temple support (we were using one of their vps servers) ... and by the biggest absolute stroke of luck they'd just backed up the entire server ... not even 2 hours prior to my madness. $100 for a full restore (I don't recall why) and would I like to do that?

HECK YES I WOULD!

so they restored the server for me. I wrote an email to the head of the small company I was doing all the work for, explaining what I had happened and telling him I'd sent over a check for $100 to cover the backup because it was my fault. He was obviously very relieved and never cashed the check I sent.

I still get chills thinking about that exact moment when I thought I'd fucked up my career and reputation for good.

Re: Ask HN: What's the worst you've ever screwed up at work?

#215
The following was not actually me, but worth sharing.

They had ASIC design runs for research purposes once every three months, yielding your design on Silicon as ten 6" wafers. It gives enough parts for testing the first revision of your design. The person was carrying the wafers to a vendor for cutting into separate ICs and packaging or something. Gets to the parking lot, and where are the keys. Puts the wafers on the top of the car, finds the keys in his pockets and starts driving. Boom, the box of wafers was still on the top of the car, now on the ground. All broken. Some $100K in wafers + three months lost + bad face before the customer + ... Lesson: Don't put stuff on the top of the car!

Re: Ask HN: What's the worst you've ever screwed up at work?

#216
The following was not actually me, but worth sharing.

They had ASIC design runs for research purposes once every three months, yielding your design on Silicon as ten 6" wafers. It gives enough parts for testing the first revision of your design. The person was carrying the wafers to a vendor for cutting into separate ICs and packaging or something. Gets to the parking lot, and where are the keys. Puts the wafers on the top of the car, finds the keys in his pockets and starts driving. Boom, the box of wafers was still on the top of the car, now on the ground. All broken. Some $100K in wafers + three months lost + bad face before the customer + ... Lesson: Don't put stuff on the top of the car!

Re: Ask HN: What's the worst you've ever screwed up at work?

#217
post #55

I run Correlated.org, which is the basis for the upcoming book "Correlated: Surprising Connections Between Seemingly Unrelated Things" (July 2014, Perigee). I had had some test tables sitting around in the database for a while and decided to clean them up. I stupidly forgot to check the status of my backups; because of an earlier error, they were not being correctly saved. So, I had a bunch of tables with similar nam…

Ha, nice one. Whenever I start a new job, the first thing I do is create a backup of the database, because I have done something similar before. Backup everything the first day, onto your own machine.

Re: Ask HN: What's the worst you've ever screwed up at work?

#218

I work at a newspaper as a programmer for the website. Mostly my job is backend programming, some HTML and CSS work (mostly left to designers). I run our local computer infrastructure as well as manage a cluster for our online presence and assist in technology related journalism as well as assisting our CEO in managing the IT budget. I inherited a mess of an architecture and am finally getting around to rewriting our…

> ... and my new cluster goes live day after tomorrow. On a Friday? :-)

End-of-month.

Re: Ask HN: What's the worst you've ever screwed up at work?

#219

Demonstrated SQL injection to a colleague on the live website. Bringing a sample URL up into the address bar, I explain, "You see, that ASP script takes the value of ?urlparameter and updates the record - but what if I modify urlparameter so that instead of 1, it is... (types) semicolon dash dash DROP TABLE usermaster (presses enter)" "Shit. Well, as I have just demonstrated, it becomes possible to wipe out a million…

The problem was the service being susceptible to injection in the first place.

This wasn't a mistake; just a hilariously successful penetration test!

Post reply on HN