Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What's the worst you've ever screwed up at work?

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 322 posts

Re: Ask HN: What's the worst you've ever screwed up at work?

#161

Post by bluedino »

canadev wrote: post #79
ggreer wrote: post #58

Earlier quoted context omitted.

Sure there's plenty of blame to spread around, but I still would have felt terrible if someone had been hurt or killed. What system would you put in place to prevent this? The issue was that I connected to prod when I thought I was connecting to a test DB. We each had different credentials for prod vs everything else, but the SQL client remembered my username and password. Anyone with prod access could have made the…

In the past, I've set up big MOTD style messages that say "PROD" in fancy ASCII graffiti when I ssh/connect a DB client/whatever to production. I think I will set one of those up now for my current setup. Also, sort of related, I'm using MacOS, and in the back of my head I've wanted to create a tool that will change the color of the menu bar (at the top of the screen) to, say, bright yellow, when I'm connected to the…

It doesn't really solve anything, but I've done bright blue prompts for staging, red for production, and green for development.

Re: Ask HN: What's the worst you've ever screwed up at work?

#162

Post by benburleson »

bengarvey wrote: post #48

I poured gasoline into the tractor's radiator instead of the gas tank. Thankfully, someone stopped me before I turned it on.

Is this really that bad? IANAMechanic, but if the coolant system is sealed, shouldn't the gas just work similar to antifreeze?

Re: Ask HN: What's the worst you've ever screwed up at work?

#163

Post by jwdunne »

d0m wrote: post #129
patio11 wrote: post #93

I've only cried literal tears once in the last ten years, over business. Due to inattention while coding during an apartment move, I pushed a change to Appointment Reminder which was poorly considered. It didn't cause any immediate problems and passed my test suites, but the upshot is it was a time bomb that would inevitably bring down the site's queue worker processes and keep them down. Lesson #1: Don't code when y…

Can't agree more with Patrick, mine is formulated a bit differently: Lesson #1: Don't push code on Friday afternoon. Lesson #2: Beer, Code and Commit is totally fine. Just don't push! Wait until next day to review and push/deploy it..

Lesson 1 rings a mad bell. I distinctively remember a text from a colleague saying "yeah, I'm never launching anything on a Friday".

As a general rule, the best time to launch is first thing on a Tuesday. Why Tuesday morning? We are also well aware of "Monday morning madness" too - pre-planning stress for Monday morning is akin to masochism.

Re: Ask HN: What's the worst you've ever screwed up at work?

#164

Post by jlgaddis »

Beltiras wrote: post #134

I work at a newspaper as a programmer for the website. Mostly my job is backend programming, some HTML and CSS work (mostly left to designers). I run our local computer infrastructure as well as manage a cluster for our online presence and assist in technology related journalism as well as assisting our CEO in managing the IT budget. I inherited a mess of an architecture and am finally getting around to rewriting our…

> ... and my new cluster goes live day after tomorrow.

On a Friday? :-)

Re: Ask HN: What's the worst you've ever screwed up at work?

#165

Post by ejk314 »

jawns wrote: post #55

I run Correlated.org, which is the basis for the upcoming book "Correlated: Surprising Connections Between Seemingly Unrelated Things" (July 2014, Perigee). I had had some test tables sitting around in the database for a while and decided to clean them up. I stupidly forgot to check the status of my backups; because of an earlier error, they were not being correctly saved. So, I had a bunch of tables with similar nam…

Neat solution.

Re: Ask HN: What's the worst you've ever screwed up at work?

#166

Post by asdasf »

sentenza wrote: post #80
sillysaurus2 wrote: post #69

Earlier quoted context omitted.

She found out about it pretty quickly due to having syslog be a constant presence in one of her gnu screen windows I'm amazed that this is possible. How would I set something like that up? A realtime log of only the most significant events of a remote system? In fact, I'd like to take this opportunity of ignorance-admitting to ask the community for general linux/bsd sysadmin resources. What books should I read, or wh…

What position are you starting from? My old workplace was a university group where we (admins) were recruted from the available pool of PhD students. So I'm used to guiding people from "no knowledge" to "enough knowledge to be dangerous". The first step was to force the prospective admins to run a specific system on their "productive machine" and keep it in such conditions that _everything_ works. This way, a complet…

>If they were using Arch or Gentoo at home,

If they are using gentoo, you should be finding someone else. Gentoo users are typically the most dangerous combination of profoundly ignorant, yet absurdly overconfident in their abilities. Seeing a bunch of autotools and gcc output scroll by does not teach you anything. But the mistaken reputation as an "advanced" distro makes people think that by using gentoo, they are therefore "advanced".

Re: Ask HN: What's the worst you've ever screwed up at work?

#167

Post by _mikelcelestial »

I accidentally deleted all data from live database, thought it was our beta database server. Good thing it is synchronized on our beta servers so was able to bring it back in no time. The moment I clicked that delete button I was like face palming myself all over. I learned from thereon to double check every time especially when working on between production and test servers.

Re: Ask HN: What's the worst you've ever screwed up at work?

#168

Post by jonathanjaeger »

I often turn on my performance-based ad campaigns before going into the office, as they are very predictable at the beginning (slow ramp up to spend). However this time the CTR was through the roof for something new and spent $15,000 by the time I could turn them off when I got to the office. It only brought in about $5,000 in revenue. Not the end of the world in the grand scheme of the monthly P&L, but still not something to replicate.

Re: Ask HN: What's the worst you've ever screwed up at work?

#169

Post by seanhandley »

benburleson wrote: post #162
bengarvey wrote: post #48

I poured gasoline into the tractor's radiator instead of the gas tank. Thankfully, someone stopped me before I turned it on.

Is this really that bad? IANAMechanic, but if the coolant system is sealed, shouldn't the gas just work similar to antifreeze?

Not if it's diesel - it'll be heated up (so it'll partially vaporize) and it'll be under pressure. There'll be air in there as well. Could quite happily combust if it got warm enough!

Re: Ask HN: What's the worst you've ever screwed up at work?

#170

Post by ShaneOG »

msh wrote: post #103
edw519 wrote: post #77

Boss: We have thousands of bad orders that must be fixed now! Me: No we don't. We have 121 bad orders. Boss: There are thousands of them! Me: No there aren't. There are exactly 121 of them. I'm sure. Boss: I'm not going to argue with you! Me: Good. Because you'd lose. I fixed 121 orders that night. The next day my login & password wouldn't work.

I might be dense, but I dont get it.

I think the "screw up" was disagreeing with his boss, and thus losing his job.
Post reply on HN