Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

Post reply on HN
Print view View original

221–230 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#221

Post by jpatokal »

michh wrote: post #13

Classic forgetting the full WHERE-part of a manual UPDATE-query on a production system. The worst part is you know you fucked up the nanosecond you hit enter, but it's already too late. Lesson learned? Avoid doing things manually even if a non-technical co-worker insists something needs to be changed right away. And if you do: wrap it in a transaction so you can rollback, leave in a syntax error that you'll only remo…

This is why you have SET SQL_SAFE_UPDATES=1; (or equivalent) in your DB shell startup. It only takes one UPDATE users SET password='foo'; to learn why...

Re: Ask HN: What's the worst you've ever screwed up at work?

#222

Post by 10feet »

alok-g wrote: post #215

The following was not actually me, but worth sharing. They had ASIC design runs for research purposes once every three months, yielding your design on Silicon as ten 6" wafers. It gives enough parts for testing the first revision of your design. The person was carrying the wafers to a vendor for cutting into separate ICs and packaging or something. Gets to the parking lot, and where are the keys. Puts the wafers on t…

I don't think that is the lesson, I think the lesson is that it is clearly a two person job. One person to carry the wafers, the other person to remove and hazards/ open doors/ double check everything.

Re: Ask HN: What's the worst you've ever screwed up at work?

#223

Post by Zikes »

quackerhacker wrote: post #195

I messed up epically on an interview. It was a 3 part interview for a JS/RoR coder. 1. I passed the resume and chat portion 2. I passed the telephone questionnaire and got along great with the interviewer 3. (Fail) I scheduled my interview on a Friday at 4:30pm and there is a 30 min travel time. I left 1hr early...still it was Memorial Day weekend, so I thought the streets would be quicker than the freeway since it w…

I think most interviewers would be understanding if you called to reschedule in that situation.

Re: Ask HN: What's the worst you've ever screwed up at work?

#224

Post by taf2 »

I hacked our development machines using a rooted rpm, we only had access to the sudo rpm command so I decided to deploy our rails app using capistrano. to work around the sudo rpm only access I decided to add some install scripts to the rpm because these run as root. This allowed me to re-configure sshd making it possible to do a local capistrano deploy. I was smart about it by reverting the ssh changes back after the deploy completed - bash has a kind of ensure that allows you to roll things back like a transaction. The cool thing about the whole thing was that our ops team was on the ball and detected the changes to the sshd configuration even though I restored them. Mind you this was all in a staging development environment. The issue was just how immature it was of me to go this far to cap deploy instead of rpm install our rails app. For me, I looked at it then like a good learning experience in hacking rpms and in security. When you run sudo rpm -Uhv package.rpm - you better trust package.rpm it can execute any shell scripts it wants as root. Also, in the future I would walk away from a company like this much sooner. I enjoyed everyone there I worked with and would work with any of them again, but just would not want to work in such a stress filled environment for so long again.

Re: Ask HN: What's the worst you've ever screwed up at work?

#225

Post by itwasme »

mcintyre1994 wrote: post #57
itwasme wrote: post #44

I once worked for a company that schedules advertising before films. This wasn't in the US and the company had a monopoly over all of the ads shown across the country. It was my first programming job and done during university holidays, so I was there for a couple of months and then back to university. Toward the end of the following year I get a phone call: something was wrong with the system, it was allowing agents…

That time isn't normal where you are? How much did you make selling your system to the UK cinema industry? :)

Haha. It was quite a long time ago otherwise I would have remembered the usual maximum booking time. I wouldn't be surprised if they exported the bug, given its success.

Re: Ask HN: What's the worst you've ever screwed up at work?

#227

Post by FireBeyond »

edw519 wrote: post #77

Boss: We have thousands of bad orders that must be fixed now! Me: No we don't. We have 121 bad orders. Boss: There are thousands of them! Me: No there aren't. There are exactly 121 of them. I'm sure. Boss: I'm not going to argue with you! Me: Good. Because you'd lose. I fixed 121 orders that night. The next day my login & password wouldn't work.

Tangential to this, in the middle of the dot-com boom, a downsizing. Company meeting: "Here's what's happening, layoffs, etc, etc. When you go back to your desk, if you find your login is disabled, take a moment to pack your desk and come back to the Conference Room."

That's how they "notified" people. You got to play a lucky jackpot game of logging into your desktop after the meeting to see if you still had a job. Straight out of Dilbert.

Re: Ask HN: What's the worst you've ever screwed up at work?

#228

Post by jrabone »

georgemcbay wrote: post #157
wpietri wrote: post #146

Long ago when I was, I think, a sophomore in college and worked for the university IT group, I was trying to add an external drive to an early NeXT machine [1]. I wanted to try out their fancy GUI development stuff, you see. I was at best a modestly competent Unix admin, and this was circa NextStep 1.0, so the OS was... rough. It was in the dark days of SCSI terminators, so just telling if the drive was properly conn…

root shell, plus rm with any sort of wildcard matching, plus a bit too much of a delay before you get your shell prompt back results in a very specific kind of panicked anxiety that almost anyone who has been programming or sysadmining for a while can easily relate to.

I'm sure zfs on Linux has a random sleep built in just to increase my anxiety levels...

Re: Ask HN: What's the worst you've ever screwed up at work?

#229

Post by alexcroox »

jwdunne wrote: post #163
d0m wrote: post #129

Earlier quoted context omitted.

Can't agree more with Patrick, mine is formulated a bit differently: Lesson #1: Don't push code on Friday afternoon. Lesson #2: Beer, Code and Commit is totally fine. Just don't push! Wait until next day to review and push/deploy it..

Lesson 1 rings a mad bell. I distinctively remember a text from a colleague saying "yeah, I'm never launching anything on a Friday". As a general rule, the best time to launch is first thing on a Tuesday. Why Tuesday morning? We are also well aware of "Monday morning madness" too - pre-planning stress for Monday morning is akin to masochism.

Agreed, we have a strict no deployments on Fridays or after 3pm on any day at work. Allows us to go home with a clear head and enjoy our weekends.

Re: Ask HN: What's the worst you've ever screwed up at work?

#230

Post by alttab »

steveklabnik wrote: post #188
jwdunne wrote: post #163

Earlier quoted context omitted.

Lesson 1 rings a mad bell. I distinctively remember a text from a colleague saying "yeah, I'm never launching anything on a Friday". As a general rule, the best time to launch is first thing on a Tuesday. Why Tuesday morning? We are also well aware of "Monday morning madness" too - pre-planning stress for Monday morning is akin to masochism.

I wonder if this is the rationale for Patch Tuesday.

No the rationale for patch Tuesday is shoddy development while continually introducing bugs with your fixes.
Post reply on HN