Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

171–180 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#171

Post by TheDauthi »

I went to a residential high school in a very rural state. Our internet went through a local college, then the state university.

This is 1998. I was setting up my new main desktop with linux that year. When it got to network services, the installer asked me if I wanted dhcp and dns. "Of course I want dhcp and dns, I don't have a static IP here."

It was asking if I wanted to install DHCP and DNS servers on this machine. I can only guess as to what kind of configuration allowed this to spread as far as it did, but for about 2 days the state university shut down the entire lower college's network because my machine was apparently responsible for DHCP for everyone for just a little while.

Re: Ask HN: Best “I brought down production” story?

#173

Post by cheese123 »

cheese123 wrote: post #8

Doom on a single PC was fun. But the same game on our network with all those broadcasts running through our net, which hosted a university's PCs. That was bad. And I started it several times until I noticed that "I" was the cause for network slowdown.

Best part was that a sysop entered the room and within a second knew that I was the problem and that I was playing doom. He just told me "if you wanna play games instead of studying than play doom 2. This does not broadcast"

Re: Ask HN: Best “I brought down production” story?

#174

Post by joddystreet »

I was sitting in an escalation-call, called by one of our most important customer and India’s largest ISP. There were 13 of them, 3 of us, and only 1 technical (that’s me). Things were super intense, extremely heated debates were going on, I was fixing a few issues, while answering the questions coming my way, in parallel I was also chatting with my teammates, who were back in the office and were on support calls with the same client’s ops team. And, then it happened - rm -rf /var/lib/mysql (i was already sudo su)

Escalation got further escalated - “Dashboard is not opening”

For a moment, I was shell-shocked, couldn’t hear a thing, just sitting there frozen.

Then i remembered the backup, i, against my usual style, took and the mysql replica i hosted, just as an insincere effort to calm the people in the room, i got to work. Restored the database, re-ram a few etl jobs, and we were back up online.

I was relieved, and actually quite happy, started interacting with the people in the room again, and showed them the system robustness even under a disaster, two of the 13 caught my bluff, but were smiling, They winked, turned back to face the others, and the heated debate continued.

The escalation went up till the CIO, but the guys who caught my bluff, never gave my deed away (in return i had to code a few more features and reports, just for them).

Don’t multitask, especially with a sudo access.

Re: Ask HN: Best “I brought down production” story?

#175

Post by puchatek »

mansoon wrote: post #115
maxk42 wrote: post #108

Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…

I’m confused by the last part of your post. Sounds like you appreciated that your boss gave you space to learn, and understood that you made an honest mistake, but you’d fire someone who made this mistake if they were working for you? How do you square those two things internally?

I think the hint is in "after the second time I did this". I would also wonder if to keep them on at that point.

Re: Ask HN: Best “I brought down production” story?

#177

Post by Agentlien »

Here's a good reminder of the dangers of scale.

When Need for Speed (2015) came out I was one of the software engineers in the war room, monitoring crashes and usage statistics.

At one point we saw a big drop in active users and it turned out it was because servers kept crashing. That was a big deal since a server crash was usually rare and naturally meant disconnecting all players on it.

After a bit of searching I found the responsible code. It was a client side bug which crashed the server. The line even had a comment mentioning that it wasn't entirely correct, but that the probability of a crash was about one in a million.

Well, we had millions of players.

Re: Ask HN: Best “I brought down production” story?

#178

Post by axegon_ »

Best one I've seen was a former manager of mine who was testing the performance of an API which was a descision tree underneath but insanely poorly implemented. He ran wrk against it and... Yeah, the rest you can imagine.

Mine was an unfortunate chain of events. Connecting to a databases was done through a specific server with port forwarging. A sysadmin changed the ports for whatever reason without notifying us so the port that was supposed to point to an alpha environment suddenly became production environment. We had done changes to an authentication system and the old wan was being depricated so I was going to get rid of it in the alpha environment. Soooo... Drop table sessions on production. Luckily that was an internal system so all that happened was that ~800 people got logged out.

Re: Ask HN: Best “I brought down production” story?

#179

Post by brutal_chaos_ »

monkeydreams wrote: post #97
cvrjk wrote: post #91

Sent in a SQL command deleting multiple million records (intended) wrapped in a single transaction (not intended). The replication queues could not keep up and failed, bringing down most of the replicas. Master server kept trying to recover and maxed out all connections - no DBA could log in to perform manual recovery. We had to hard reboot not knowing what state the system is in and how long it'll take to fully reco…

> TBH, my team was very gracious about it and the RCA focused purely on the events that occurred and how to never let if happen it again. No blame game at all. Which is how a PIR, PER or PCR should be. If you don't understand why someone makes a mistake, you can't avoid future mistakes.

I understand SQL, DBA and TBH, but what do RCA, PIR, PER, and PCR stand for?

Re: Ask HN: Best “I brought down production” story?

#180

Post by brutal_chaos_ »

truetraveller wrote: post #106
bombcar wrote: post #90

Earlier quoted context omitted.

And why it’s often best to mark a record deleted and then have a reaper remove the records at a later point. But you must make sure all normal queries don’t see deleted items.

That's over engineering. At this point, just rely on PITR. FWIW, postgres does have a "reaper" via vacuum, but not for the purpose of safety. but rather to allow for mvcc.

Hi, if I may, what does PITR stand for?
Post reply on HN