Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#131

Post by lnjarroyo »

notatoad wrote: post #116
mansoon wrote: post #115

Earlier quoted context omitted.

I’m confused by the last part of your post. Sounds like you appreciated that your boss gave you space to learn, and understood that you made an honest mistake, but you’d fire someone who made this mistake if they were working for you? How do you square those two things internally?

It's not good to punish people for making mistakes in the course of their work (especially if that work is meant to be educational) It is good to punish people who give access to production databases to people who shouldn't have it. And the guy learning MySQL should not be given that access. Taking down prod is always a symptom of a systemic failure. The person responsible for the systemic failure should see the cons…

Well, one could argue that "giving access to production databases to people who shouldn't have it" was just "making mistakes in the course of their work" for this dude.

Re: Ask HN: Best “I brought down production” story?

#132

Post by mikeywazowski »

maxk42 wrote: post #108

Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…

Wait, this happened twice? Weren't you at great pains to avoid it reoccurring after the first time?

Re: Ask HN: Best “I brought down production” story?

#133

Post by peeters »

notatoad wrote: post #116
mansoon wrote: post #115

Earlier quoted context omitted.

I’m confused by the last part of your post. Sounds like you appreciated that your boss gave you space to learn, and understood that you made an honest mistake, but you’d fire someone who made this mistake if they were working for you? How do you square those two things internally?

It's not good to punish people for making mistakes in the course of their work (especially if that work is meant to be educational) It is good to punish people who give access to production databases to people who shouldn't have it. And the guy learning MySQL should not be given that access. Taking down prod is always a symptom of a systemic failure. The person responsible for the systemic failure should see the cons…

> The person responsible for the systemic failure should see the consequences

You don't see the contradiction of terms there? A systemic failure is by definition not the responsibility of one person. You're saying people should be able to make mistakes. But not those people.

Re: Ask HN: Best “I brought down production” story?

#134

Post by yjftsjthsd-h »

bch wrote: post #120
jhugo wrote: post #65

Earlier quoted context omitted.

`ssh $host /bin/sh` (or another shell) should work?

On a Linux box isn’t that just a link to bash?

It depends on the distro; on RHEL yes it is BASH, but Debian uses dash, and Alpine uses busybox sh.

Re: Ask HN: Best “I brought down production” story?

#135

Post by _f9cu »

lamontcg wrote: post #122

I was a system engineer at Amazon from 2001-2006. Sometime around 2004/2005 or so there was a development team working on the "a9 search engine" (meant to complete with google) down in SF. They were sort of an official "shadow IT" offshoot and asked for special treatment and they got me assigned specifically to them to build out the first of their two webservers. They did the usual mistake of wanting to jettison all…

Hahaha

Re: Ask HN: Best “I brought down production” story?

#136

Post by exikyut »

ackbar03 wrote: post #67

I'm a guy in two person startup team, mostly handle the technical stuff. Woke up to see some error on our main work machine, which also hosts some of the services, after a sudo apt update. Wasn't the first time and was usually able to fix by just googling what the error is as on SO and running it. Did it again, told me to uninstall nvidia drivers, proceeded to do so, and bricked my hard disk. Was completely gut wrenc…

Very curious what you mean by "bricked the hard disk"?

Re: Ask HN: Best “I brought down production” story?

#137

Post by erik_seaberg »

I was using -1 as a dummy timestamp in an ETL job, for some reason I can’t remember. Well, the MySQL version in prod had a bug handling queries that do that. My job failed because database shard number one had instantly crashed. For about ten minutes, nobody at Harvard could use Facebook.

This being the height of “move fast and break things,” they asked me to do it again to confirm our diagnosis.

Re: Ask HN: Best “I brought down production” story?

#138

Post by exikyut »

mrleinad wrote: post #77
mikewarot wrote: post #56

It was 1985, I was in a VAX computer lab with about 40 other people typing on the VT100 terminals... I ran a program to compute something, and forgot that I had bumped my priority wayyy up... everything in the room stopped, even the line printer, everyone went ohhhhhh. 10 seconds later, my program finished... and everything snapped back to life. Another time, I walked into a different, bigger lab, with 100 terminals.…

Wait, what? So the system was purposefully hindering everyone's productivity? Sounds more like a "I brought up production" story...

My guesstimated translation: some large number of those users were running compile jobs, which were running as background tasks instead of foreground tasks.

Re: Ask HN: Best “I brought down production” story?

#139

Post by exikyut »

D13Fd wrote: post #26

This was long ago and the details are hazy. Back in 2003 or so, I was in tech support for a company that used desktop computers running java applets to connect to a mainframe via Telnet (IBM Host-on-Demand IIRC). Most of the core business processes were handled by mainframe apps, which the company largely developed. I used to hang out in the data center with the mainframe guys who coded in COBOL all day. On a Friday…

Yayyy.

If you remember, how did this end up getting fixed? Did users have to re-input their connection info (meep) or were you able to re-fill everything in through some heuristic?

Re: Ask HN: Best “I brought down production” story?

#140

Post by mathgladiator »

markus_zhang wrote: post #112
mathgladiator wrote: post #93

I ran a bunch of scripts on some aux boxes, and Tokyo stopped working. Can't talk details... sadly

You mean the city stops functioning?

No, that's how I think about it. Internet had loss of availability in that region.
Post reply on HN