Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

241–250 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#241

Post by mrunkel »

Around 1999/2000 I worked at Universal Studios in Burbank. Shortly after I started working there in the networking group, a fellow employee asked me to reset a port on the core token ring switch. Being a helpful type, I clicked "disable" on the network management tool for the port and promptly everyone lost connectivity. To everything. A quick walk over to the NOC confirmed that yes, the entire campus was down, as well as a few satellite campuses.

I explained what I had done (which was met with incredulity, I mean, surely I had fucked something up and didn't just disable the described port)..

After the longest five minutes of my life, the network came back alive. And sure enough, only the port that I had disabled was disabled. After some troubleshooting with the vendor, it was determined that there was a software bug in the switch firmware. If the device was under heavy load, disabling a port caused the switch to reboot.

So that's how I took down the entire Universal Studios corporate network.

Re: Ask HN: Best “I brought down production” story?

#243

Post by avh02 »

samus wrote: post #236
avh02 wrote: post #229

When i was working for one of the biggest regional websites: We had blue/green deployments using AWS Elastic Beanstalk - our actual deployment process was still manual and we had an informal (i.e: not written down but everybody knew) checklist to follow - one of the steps involved checking that capacities matched. Well, everything looked good and I switched environments, as I was checking that the website was still u…

Not too bad. Unless you have a contract or regulations that says otherwise, 5 mins downtime once in a while is really nothing to lose your sleep about. Did you even have to report it?

I honestly don't remember, but i definitely remember the checklist became more formal as we made a lot of errors like this.

I left soon afterwards but the stack was also a transition-to-AWS stack so was later moved to something more suited.

5 mins downtime was definitely small for that place - we regularly had larger issues. I learned a phenomenal amount in my time there though.

Re: Ask HN: Best “I brought down production” story?

#244

Post by sodality2 »

rrauenza wrote: post #121
chasd00 wrote: post #92

Earlier quoted context omitted.

I had a DBA call me on my way back from lunch absolutely sobbing after fat fingering a semicolon before the where clause while logged in to Prod as root. we got things restored and back online in a couple of hours. I let her go home afterwards, heh she had suffered enough.

I was taught to also use a transaction and then check how many rows were affected before committing.

As was Tom Scott: https://youtu.be/X6NJkWbM1xk

Re: Ask HN: Best “I brought down production” story?

#245

Post by andrewinardeer »

go_prodev wrote: post #212

My colleague asked me to run crontab -l, which I misheard as crontab -r (a switch I was unaware of, which instantly removes all Cron jobs) I still wonder why crontab has that switch? Who needs that? We had to piece together all scheduled commands from run logs.

Done this one fat fingered on production. Wanted crobtab -e, did crontab -r. That was a fun night.

Re: Ask HN: Best “I brought down production” story?

#246

Post by asim »

exikyut wrote: post #208
asim wrote: post #204

$ rm -rf /var/mysql/data *Thinks for 5 seconds...is this prod? ....@$1£!? ... ctrl + c....should I tell anyone that just happened....walks over to line manager...* I did not taste anything that day.

Hopefully there was a backup...?

That was a live read-only replica which we were able to switch to within 20-30 mins of the issue. Then snapshotted that and restored to the master later that night. The CTO was on holiday at the time so writing that email to him was brutal.

Re: Ask HN: Best “I brought down production” story?

#247

Post by chasd00 »

rrauenza wrote: post #121
chasd00 wrote: post #92

Earlier quoted context omitted.

I had a DBA call me on my way back from lunch absolutely sobbing after fat fingering a semicolon before the where clause while logged in to Prod as root. we got things restored and back online in a couple of hours. I let her go home afterwards, heh she had suffered enough.

I was taught to also use a transaction and then check how many rows were affected before committing.

Many mistakes were made prior in order to get her in the situation to begin with. The semicolon was just the icing on the cake.

Re: Ask HN: Best “I brought down production” story?

#248

Post by teknopaul »

trefgin wrote: post #15

Slightly off topic but I think in most cases responsibility for accidentally bringing down production likes with management. However I’ve only ever heard stories where management lays the blame.

Where I have worked, its devs and ops who bork the systems then have to set about fixing it while management run off to face the customer. I wouldn't change place with management for love nor money.

You see a lot of mea culpa outage stories on HN, many written by management. If you have never read one you should. The outcome of a mistake is a learnin.

Altho it is also fun to share fail stories just for the scale of the fsck up. ;)

Re: Ask HN: Best “I brought down production” story?

#249

Post by gorbachev »

lamontcg wrote: post #122

I was a system engineer at Amazon from 2001-2006. Sometime around 2004/2005 or so there was a development team working on the "a9 search engine" (meant to complete with google) down in SF. They were sort of an official "shadow IT" offshoot and asked for special treatment and they got me assigned specifically to them to build out the first of their two webservers. They did the usual mistake of wanting to jettison all…

> This is why you always keep it professional, kids...

In the late 90s someone in a company I worked put a placeholder HTML file on a client's production web server that instead of the usual lorem ipsum stuff had something like "this shit is beneath me", except much worse. The placeholder was never removed, and while it wasn't the index page and if I remember correctly wasn't even directly linked from anywhere, someone still found it. The incident found itself on the national newspapers.

Re: Ask HN: Best “I brought down production” story?

#250

Post by mvolfik »

ramoz wrote: post #27

Nothing crazy, but something I always laugh at. I was so excited to meet a legit/professional dev team the first day of my career. I was paired with a Sr dev and sat in his cubicle so he could show me some backend update to a prod app with 20K internal users... "normally I'd run this on the dev server, but, its quick & easy so Ill just do it on prod" ...watched him crash the whole thing & struggle the rest of the day…

I had to reread this to check if you're really not talking about Ingress servers, because Niantic really seems to be doing changes directly in production all the time

also I completely understand how invested one can get in explaining Ingress

Post reply on HN