Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

Post reply on HN
Print view View original

231–240 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#231

Post by kenesom1 »

michaelochurch wrote: post #17

Tried to prevent a massive product failure. It failed anyway, but I wasn't around when it did and there would have been no "I told you so" credit even if I were. One of those "big company" lessons, but probably applicable to startups (which have an even higher ego density).

Bummer, they should've listened to you...

Some companies are using internal prediction markets, where employees can speculate on various initiatives:

http://en.wikipedia.org/wiki/Prediction_market#Use_by_corpor...

Open allocation seems like the ultimate prediction market - people deciding on what initiatives to work on and invest their time in is a stronger signal than dysfunctional internal politics. And people are likely to be more motivated working on projects they think have value.

Re: Ask HN: What's the worst you've ever screwed up at work?

#232

Post by Tloewald »

In terms of feeling bad, I once had a client who wanted to demo a multimedia project that we currently had in alpha on his Windows 3.11 laptop, but the sound drivers weren't working properly (everything else was fine). He had about an hour before he had to leave for the airport. I started monkeying with the four horsemen of the apocalypse (Windows.ini, System.ini, Autoexec.bat, and Config.sys) as I had many times before but I screwed up saving backups, bricked his machine, and couldn't fix it). In the end it was more embarrassing than anything else, but it was a facepalm stupid mistake.

The lesson from this is pretty obvious. Backup. Make sure your backup is good and safe.

My worst work-related mistake was getting into business with a friend. It cost me the friendship, a very valuable client, and a good portion of my retirement savings. I'm not sure how related it was, but a few years later my (former) friend killed himself.

And the lesson here is not to go into business with friends. Or at least to set up the business as if you're not friends.

Re: Ask HN: What's the worst you've ever screwed up at work?

#234

Post by FireBeyond »

peg_leg wrote: post #88

This was some time ago. I've learned a lot since. I mkfs'd the main disk on our email server. There was no redundancy. There was a new volume that needed to be formatted, my superior told me to do it. I protested that I didn't quite know how. He pushed it. So, I would up wiping it by mistake. since then I've made it a mission to make the entire stack at that place resilient and redundant. Now, it's virtualized, failo…

You could ask your boss if you can test it via re-running that fateful command ...

Re: Ask HN: What's the worst you've ever screwed up at work?

#235

Post by bradbatt »

patio11 wrote: post #93

I've only cried literal tears once in the last ten years, over business. Due to inattention while coding during an apartment move, I pushed a change to Appointment Reminder which was poorly considered. It didn't cause any immediate problems and passed my test suites, but the upshot is it was a time bomb that would inevitably bring down the site's queue worker processes and keep them down. Lesson #1: Don't code when y…

Thanks for sharing—some good lessons learned in there! I love that you posted this as a comment on HN…it is worthy of it's own blog post.

Re: Ask HN: What's the worst you've ever screwed up at work?

#236

Post by brey »

hcarvalhoalves wrote: post #130

Happened to a colleague: it was the end of the day, and we were packing up to leave. He used Ubuntu on his notebook, so he typed "shutdown -h now" on his shell prior to closing the lid. Seconds later he's groaning, having noticed it was a SSH session to the production server... It wouldn't be a big deal, wasn't for the fact it was an EC2 instance, and back then halting the instance was equivalent to deleting it perma…

molly-guard is an good way to prevent this - forces you to type in the hostname of the machine you are trying to halt / restart etc.

http://manpages.ubuntu.com/manpages/trusty/man8/molly-guard....

Re: Ask HN: What's the worst you've ever screwed up at work?

#237

Post by ClayFerguson »

I didn't do this, but was the one who figured out what happened. I guy wrote an installation utility for internal use to automate certain software setups. Part of the program had to clear out a certain directory, where you had to enter the name of the directory. Problem was, if you leave that field blank (the default), it converted into c:\ and people would run it, and it would wipe out their hard drive. After finding the problem, I told only the guy who did it, and no one else. I didn't have the heart to destroy his reputation by telling everyone what had done it. I SHOULD HAVE let the chips fall where they may, because I needed to be sure NO ONE ever ran that EXE "utility" again. They figured it out pretty quick, but nobody really knew the true problem but me and the guy who wrote the bug!

Re: Ask HN: What's the worst you've ever screwed up at work?

#238

Post by bradbatt »

ggreer wrote: post #22

One summer in college, I got an internship at a company that made health information systems. After fixing bugs in PHP scripts for a couple weeks, I was granted access to their production DB. (Hey, they were short on talent.) This database stored all kinds of stuff, including the operating room schedules for various hospitals. It included who was being operated on, when, what operation they were scheduled for, and im…

"That's weird, this UPDATE shouldn't have taken so long-oh shit."

That right there is one of the worst feelings in the world. I imagine that everyone on HN just felt it with you as they were reading it!

Totally agree that the higher-ups were responsible for not putting better roadblocks in place to prevent this type of thing from happening.

Re: Ask HN: What's the worst you've ever screwed up at work?

#239

Post by malka »

alexcroox wrote: post #229
jwdunne wrote: post #163

Earlier quoted context omitted.

Lesson 1 rings a mad bell. I distinctively remember a text from a colleague saying "yeah, I'm never launching anything on a Friday". As a general rule, the best time to launch is first thing on a Tuesday. Why Tuesday morning? We are also well aware of "Monday morning madness" too - pre-planning stress for Monday morning is akin to masochism.

Agreed, we have a strict no deployments on Fridays or after 3pm on any day at work. Allows us to go home with a clear head and enjoy our weekends.

Never really understood person that do otherwise. What's the point ? ruining a perfectly fine night / week end ?

Re: Ask HN: What's the worst you've ever screwed up at work?

#240

Post by rjbwork »

[deleted] wrote: post #70

[deleted]

Virtual goods? Fired why? Ambien is known to do these kinds of things. Was it because of the aderall and ambien combined made them thing you were a risk due to drug abuse? I don't get the firing over your description of the situation to be honest.
Post reply on HN