Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

271–280 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#271

Post by mianas »

tasuki wrote: post #265
Agentlien wrote: post #177

Here's a good reminder of the dangers of scale. When Need for Speed (2015) came out I was one of the software engineers in the war room, monitoring crashes and usage statistics. At one point we saw a big drop in active users and it turned out it was because servers kept crashing. That was a big deal since a server crash was usually rare and naturally meant disconnecting all players on it. After a bit of searching I f…

A client side bug was crashing the server ? I always thought the client depended on the server, not vice versa.

I have no idea, but I'd guess it was the client generating a random number to use as a unique ID, with insufficient length.

AFAIK, anything UUID-sized or bigger doesn't have this problem on Earth.

Re: Ask HN: Best “I brought down production” story?

#272

Post by mianas »

alfiedotwtf wrote: post #53
breakfastduck wrote: post #23

Earlier quoted context omitted.

Lack of validation? I.e if no criteria, it could be sending a DELETE message with no where clause in SQL land.

i remember sitting next to someone who screamed after realising he just did a ’DELETE FROM company' without a WHERE clause. Our database was way too big to backup, so we only had production, but luckily I had rolled out database logging in the inyerface that recorded all UPDATE and DELETEs a few weeks before the event.

I got myself into the habit of opening transactions before doing anything that could alter data by hand. Huge panic-saver.

Even earlier, I got into the habit of typing the where clause before typing the from clause. Not much of a change, but it feels more fail-safe.

Re: Ask HN: Best “I brought down production” story?

#273

Post by jhanschoo »

mikewarot wrote: post #153
exikyut wrote: post #138

Earlier quoted context omitted.

My guesstimated translation: some large number of those users were running compile jobs, which were running as background tasks instead of foreground tasks.

Yes, because the system administrator, in his great wisdom made the compiler a batch job, but left the default priority way too low, below all interactive task. Which meant the users had to wait around, so the labs would always end up full of logged in users, frustrated that the system was so darn slow.

resource starvation with a human element!

Re: Ask HN: Best “I brought down production” story?

#274

Post by mattlockyer »

Not mainnet but production testnet used by hundreds of people on a blockchain I worked for. I noticed a certain type of transaction returned me funds, so I set up a script to see how far it could go, next stop was mainnet. Eventually my script was blocked by the testnet relayer, so I inquired if we had a blacklist set up.

The respond was, "No dude, but you just blacklisted everyone."

Re: Ask HN: Best “I brought down production” story?

#275

Post by Vaslo »

Most of these are way better and I didn’t quite bring down production. I am a neophyte DB user (working in Finance, not IT.). I once wrote a query to try and abstract a simple table of about 8 years of dates by calendar month, fiscal month, holiday, etc (fiscal was different than calendar.). I ran a fairly open query in our oracle 12 dB. I waited about 20 mins and had no results so I figured I’d let it run overnight after I left for the day.

When I got in the next morning, it was still running and people were having issues with getting their sales data and much of the sales reports didn’t run overnight.

A few hours later I got an email from a DB admin I worked with closely on custom pulls and she was like wtf. Apparently the way I wrote the query I really killed the performance of the database and it broke lots on of processes.

My badge of honor from that situation is the email from the DB admin that was the forward of a long email trail of the IT folks trying to figure out what happened with the subject “Nasty Query”. I saved it and occasionally share it with people.

Re: Ask HN: Best “I brought down production” story?

#276

Post by charsi »

Changed the outbound firewall rules for our production server. And suddenly it couldn't speak to MngoDB Atlas. We definitely had rules to allow mongo traffic but still somehow it wasn't working . We noticed every time a connection to mongo was being made some connections were also going to a CDN on port 80. Looked pretty suspicious until we discovered it was just OCSP which verifies cert revocation status. So ultimately it was going to the cert issuer - letsencrypt who had issued mongo's cert.

Re: Ask HN: Best “I brought down production” story?

#277

Post by thedanbob »

BalancedThought wrote: post #47

2 days before I got married, I dropped the production database by accident from a GUI tool where “right-clicking” can be destructive if you click to fast. The application scheduled radio and television commercials and within 48 US states for a large international advertising group. The bigger problem was that the DBA had only been doing incremental backups and didn’t have a full back against which to run the incremen…

> a GUI tool where “right-clicking” can be destructive if you click to fast

Fun fact, there’s a current bug in VS Code where right clicking on a file in the sidebar immediately triggers the menu entry that appears under the cursor, which happens to be delete. I’ve had it happen randomly for several months now.

Re: Ask HN: Best “I brought down production” story?

#278

Post by iJohnDoe »

mikeywazowski wrote: post #132
maxk42 wrote: post #108

Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…

Wait, this happened twice? Weren't you at great pains to avoid it reoccurring after the first time?

Yeah, about firing him for making the same mistake twice that took down production both times.

Sounds like the boss was cool.

Re: Ask HN: Best “I brought down production” story?

#279

Post by shoo »

yjftsjthsd-h wrote: post #141
Animats wrote: post #51

Back when I was working on proof of correctness, when that was a very new thing, I was using the Boyer-Moore theorem prover remotely on a large time-shared mainframe at SRI International. At the time, you needed a mainframe to run LISP. I was working on proofs of basic numeric functions for bounded arithmetic. So I was writing theorems with numbers such as 65536. This caused the mainframe to run out of memory, page o…

> Boyer and Moore explained to me that the internal representation of numbers was exactly that of their constructive mathematics theory. 2 was (ADD1 (ADD1 (ZERO))). 65536 was a very long string of CONS cells. I was told that most of their theorems involved numbers like 1. Can we have a moment for the folks who managed to turn numerical purity into integers being O(n) ? It's unbearably beautiful... And I do mean unbea…

could have been much worse: https://en.wikipedia.org/wiki/Dedekind_cut

Re: Ask HN: Best “I brought down production” story?

#280

Post by maxk42 »

mansoon wrote: post #115
maxk42 wrote: post #108

Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…

I’m confused by the last part of your post. Sounds like you appreciated that your boss gave you space to learn, and understood that you made an honest mistake, but you’d fire someone who made this mistake if they were working for you? How do you square those two things internally?

> How do you square those two things internally?

Easy: (1) He wasn't my boss. (2) He allowed a person not associated his team or even the tech department to conduct potentially harmful operations on the production database without supervision. (3) Those actions resulted in millions of dollars of lost revenue and make-goods. (4) He did not coach the person who brought the database down. (5) He repeated the mistake.

Post reply on HN