Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

251–260 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#251

Earlier quoted context omitted.

Who packs their cell phone in a box when moving???

Who ships a product release while in no position to monitor the results? Who ships a product release immediately after a commit?

"People who are shipping actual products instead of talking about them on message boards", perhaps. Running someone else's product company from a message board is a little like playing Jeopardy! from your couch, right?

Re: Ask HN: What's the worst you've ever screwed up at work?

#252
post #129
post #93

I've only cried literal tears once in the last ten years, over business. Due to inattention while coding during an apartment move, I pushed a change to Appointment Reminder which was poorly considered. It didn't cause any immediate problems and passed my test suites, but the upshot is it was a time bomb that would inevitably bring down the site's queue worker processes and keep them down. Lesson #1: Don't code when y…

Can't agree more with Patrick, mine is formulated a bit differently: Lesson #1: Don't push code on Friday afternoon. Lesson #2: Beer, Code and Commit is totally fine. Just don't push! Wait until next day to review and push/deploy it..

There is a great lesson in one of pg's essays. If you push you better be there for few hours to monitor what is happening. You do not just push and leave. I hope we all learn from this thread.

Re: Ask HN: What's the worst you've ever screwed up at work?

#253
post #22

One summer in college, I got an internship at a company that made health information systems. After fixing bugs in PHP scripts for a couple weeks, I was granted access to their production DB. (Hey, they were short on talent.) This database stored all kinds of stuff, including the operating room schedules for various hospitals. It included who was being operated on, when, what operation they were scheduled for, and im…

You didn't screw up here. The entire infrastructure, org chart, and policies that allowed you to accidentally modify a production database containing critical medical information screwed up. Blaming yourself here is like blaming yourself for being hurt after being told to drive a car with no seatbelt or brakes.

To be fair, it's still a little bit your fault if you end up crashing the car.

Re: Ask HN: What's the worst you've ever screwed up at work?

#254
post #252
post #129

Earlier quoted context omitted.

Can't agree more with Patrick, mine is formulated a bit differently: Lesson #1: Don't push code on Friday afternoon. Lesson #2: Beer, Code and Commit is totally fine. Just don't push! Wait until next day to review and push/deploy it..

There is a great lesson in one of pg's essays. If you push you better be there for few hours to monitor what is happening. You do not just push and leave. I hope we all learn from this thread.

In the IT services industry, we'll often talk about running a "war room" for the hours following a significant change, and then follow with a period of a nominated on-call person who has the authority to wake the project team up, tell them to drop everything and fix the issue.

Funnily enough, change control actually runs a lot more smoothly in terms of getting past senior non-technical managers when you include elements like that

Re: Ask HN: What's the worst you've ever screwed up at work?

#256
my first day ever using unix. Left with a root shell. Trying things out, learning, made a few junk files somewhere or other. I was done with that and decided to delete them... "delete everything from that directory" I think: rm * /path/towhateveritwas

now on to my tasks.. had some files to print out. Where did they g...... FUCK.

I found a box of tapes and some sunos manuals. Spent the next several hours figuring out how tar and tape drives worked. Got everything back. Never told a soul.

1992. I've never done anything so careless since.

Re: Ask HN: What's the worst you've ever screwed up at work?

#257
post #129

Earlier quoted context omitted.

Can't agree more with Patrick, mine is formulated a bit differently: Lesson #1: Don't push code on Friday afternoon. Lesson #2: Beer, Code and Commit is totally fine. Just don't push! Wait until next day to review and push/deploy it..

I work for a Forex (have USD? Want JPY?) service provider that runs a 24-5 service (At any time of day during a weekday an exchange will be open somewhere, on some continent. They all close on weekends) We exclusively push code on Fridays.

I've worked with a global services industry organisation based in the UK but with MAJOR work happening in the middle east, India and Asia-Pacific regions. To add to that, because so much of the service industry's core activities tend to be based around the weekend and start of the week, we can't simply push at the weekend. Add in regular scheduled downtime and backup periods, having to work with people in the US, and many, many dozens of systems...

Re: Ask HN: What's the worst you've ever screwed up at work?

#258

Earlier quoted context omitted.

Lesson 1 rings a mad bell. I distinctively remember a text from a colleague saying "yeah, I'm never launching anything on a Friday". As a general rule, the best time to launch is first thing on a Tuesday. Why Tuesday morning? We are also well aware of "Monday morning madness" too - pre-planning stress for Monday morning is akin to masochism.

I wonder if this is the rationale for Patch Tuesday.

Pretty much. It's arbitrary, but is fixed, not the first week or last week of the month (often reporting periods), not Monday, and not the weekend. If you then assume that organisations test the patches, that means they can deploy them for thursday morning

Re: Ask HN: What's the worst you've ever screwed up at work?

#259

Not the worst, but certainly most infamous thing I've done: I was testing a condition in a frontend template which, if met, left a comment in the header HTML of all the sites we served. Unfortunately the condition was always met and I pushed the change without thinking. This was back in the day when bandwidth was precious and extraneous HTML was seriously frowned upon. We didn't realize it was in production for a wee…

Haha! I had to go out and check, and sure enough, there it is in CNET.com's source.

Re: Ask HN: What's the worst you've ever screwed up at work?

#260

Earlier quoted context omitted.

Who ships a product release while in no position to monitor the results? Who ships a product release immediately after a commit?

"People who are shipping actual products instead of talking about them on message boards", perhaps. Running someone else's product company from a message board is a little like playing Jeopardy! from your couch, right?

Yeah. This was... my biggest takeaway from running my own company. I struck out on my own because I thought the business people who ran the companies I worked for were unethical idiots, and being not-an-idiot, I could do better.

Turns out? Nope.

I also find that the arguments I had with my bosses come around again, from my employees.

I still don't really understand how business works - but I've learned enough to understand that I don't understand how business works.

Post reply on HN