Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

261–270 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#261

Post by andrewstuart2 »

lights0123 wrote: post #94
bentcorner wrote: post #69

Earlier quoted context omitted.

Now I'm curious how you managed to recover. I only know enough of my way around a shell to be dangerous and I'd be SoL if I ended up in this situation.

Recovery disk, then either copy the disk's copy of bash (if it doesn't depend on a later glibc version), copy another shell to /bin/bash (as the system probably doesn't depend on bash-specific commands to boot), chroot and use the package manager, or use the package manager with an explicit sysroot (e.g. pacman --sysroot). The first two steps are very easy compared to the latter two, but should be followed by a reins…

You should also usually be able to just reboot and as long as another shell is installed, choose that shell as your startup command in the bootloader.

Re: Ask HN: Best “I brought down production” story?

#262

Post by bpodgursky »

laurent92 wrote: post #214
bpodgursky wrote: post #159

As a new junior, remote, part-time hire, I got frustrated that my test suite wasn't passing on my laptop. So I checked out the codebase on a production server and ran my tests there. The test suite initialized by truncating all tables and loading fixtures instead. Using production database credentials. Oops.

Well… did the tests pass in production, at least?

No, because the calls to format the Hadoop filesystem failed... (luckily).

Re: Ask HN: Best “I brought down production” story?

#263

Post by etaioinshrdlu »

bostik wrote: post #192
etaioinshrdlu wrote: post #6

I worked in online advertising and pushed infinite loops that froze browsers to millions of unsuspecting victims. On another occasion I had a division operation happen on integers instead of floats, and the code was running on some hardware that steered antennas for radios on airplanes. Much time was spent by pilots flying in circles over LA while I gathered data and found the "oops". It was fixed by adding a period…

> On another occasion my machine learning demo API failed due to heavy load, but only when India's prime minister was looking at it. Okay, I'll bite. You can't just drop a war story headline like that and walk off. Please expand.

We deployed some machine learning demos via API and they got used in some exhibits for some powers-that-be. But we ran it in the cloud on shared infrastructure like fools, and the code was pretty bad, so it failed due to heavy load while Modi was looking at it. There was an outpouring of anger afterward. Here is Modi staring at one loading spinner. https://imgur.com/a/kLklVZV

Re: Ask HN: Best “I brought down production” story?

#264

Post by r00f »

breakfastduck wrote: post #23
throwaway6734 wrote: post #21

Earlier quoted context omitted.

>One night in bed I realized that if someone hit submit on the delete screen without filling in any criteria it would just delete the whole database. how

Lack of validation? I.e if no criteria, it could be sending a DELETE message with no where clause in SQL land.

in one web shop i worked with, developers who were fanaticslly using constructions "WHERE user LIKE '%$user%' "

we've had some really heated debates, but nobody would listen to me. and then one day DELETE happened where $user was empty. they spent half day restoring the database, but nobody made any conclusions and nothing changed...

Re: Ask HN: Best “I brought down production” story?

#265

Post by tasuki »

Agentlien wrote: post #177

Here's a good reminder of the dangers of scale. When Need for Speed (2015) came out I was one of the software engineers in the war room, monitoring crashes and usage statistics. At one point we saw a big drop in active users and it turned out it was because servers kept crashing. That was a big deal since a server crash was usually rare and naturally meant disconnecting all players on it. After a bit of searching I f…

A client side bug was crashing the server? I always thought the client depended on the server, not vice versa.

Re: Ask HN: Best “I brought down production” story?

#266

Post by rlonn »

Once, I was going to make a quick backup copy of /usr on a busy multi-user system. I did:

$ (cd /usr; tar cf - *) | (cd /mnt; tar xvpf -)

Only I typoed ’cd /mnt’ which made the changedir fail and my current working dir was /usr so I had one tar-process archiving /usr then streaming the data to another tar process that used it to overwrite everything under /usr. After a few seconds of nothing appearing under /mnt I got really nervous I’d done something stupid. Then I realised what was happening and had a couple more seconds of hoping it’d be fine (given that I was overwriting /usr with its own data). Then the system log lit up like a christmas tree with error messages and a few seconds later the machine froze. Can’t recommend doing backups this way.

Re: Ask HN: Best “I brought down production” story?

#267

Post by klysm »

GordonS wrote: post #240
klysm wrote: post #231

Earlier quoted context omitted.

Yeah that’s really not supposed to happen - was one of the MACs redefined or was it a manufacturer error?

Manufacturer error! IIRC, the NICs were the same make & model.

Wow very surprising! That really shouldn’t happen lol

Re: Ask HN: Best “I brought down production” story?

#268

Post by suhailpatel »

SOLAR_FIELDS wrote: post #163
suhailpatel wrote: post #43

I'm in the fortunate position of having been able to tell our story in detail on our blog after a major outage involving Cassandra and bootstrap behaviour that we didn't fully understand. This is a story of how I bought down the bank for two hours. https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on... In summary, we were scaling up our production Cassandra data store and we didn't migrate/backfill the data…

I have a Cassandra story as well. At a previous employer our org used a database that was a custom wrapper around Cassandra. This was a fairly large organization and this particular database was the keystone to the vast majority of operations of this particular organization. Well, one day I was giving a demo to some junior devs on how to use the REST API for the database which just so happened to take in raw Solr que…

After having spent many years working with it and interacting with it deeply, I would strongly recommend folks stay far away from Cassandra if you remotely care about your data. It provides way too many footguns to lose or corrupt or outright ruin your data.

Unless you work at Apple or Netflix or Spotify, finding Cassandra experts is going to be nigh on impossible and the community just isn't there unfortunately.

Re: Ask HN: Best “I brought down production” story?

#269

Post by suhailpatel »

xref wrote: post #117
suhailpatel wrote: post #43

I'm in the fortunate position of having been able to tell our story in detail on our blog after a major outage involving Cassandra and bootstrap behaviour that we didn't fully understand. This is a story of how I bought down the bank for two hours. https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on... In summary, we were scaling up our production Cassandra data store and we didn't migrate/backfill the data…

That’s a great writeup, thanks for all the detail! I was always worried about something like this happening so only ever provisioned (via ansible) one server at a time. When the logs showed it was fully synced, we provisioned the next node. It could take two days to add 10 nodes but I always felt much safer

This is the standard approach and the one we have detailed runbooks for. We've scaled the cluster fine one at a time after this experience. It also prompted us to get a much better understanding of all the other flags that have been changed beyond the defaults.

Re: Ask HN: Best “I brought down production” story?

#270

Post by monkeydreams »

robmobz wrote: post #194
brutal_chaos_ wrote: post #179

Earlier quoted context omitted.

I understand SQL, DBA and TBH, but what do RCA, PIR, PER, and PCR stand for?

RCA is "Root Cause Analysis" and I assume PIR is "Post Incident Review". I don't know PER or PCR.

"Post Event Review" and "Post Change Review".
Post reply on HN