Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

191–200 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#191
I was taking over a server that had software, that was not in version control.

I created a git repository where I wanted to start versioning, started picking the important non-cache files with individual git add X, but I accidentally added a folder I didn't want.

I quickly typed git reset --hard (as I would do working on the main production app, in some scenarios) and deleted most of the production system...

Thankfully I had a full backup on S3 (a hand-rolled backup tool), but I didn't have software to restore the files.

I turned to my right, where the CEO was sitting, and I said "I just accidentally deleted X, but it'll be back up in 30 minutes."

45 minutes later and after the fastest I've ever thrown something together I had the site restored.

Re: Ask HN: Best “I brought down production” story?

#192

I worked in online advertising and pushed infinite loops that froze browsers to millions of unsuspecting victims. On another occasion I had a division operation happen on integers instead of floats, and the code was running on some hardware that steered antennas for radios on airplanes. Much time was spent by pilots flying in circles over LA while I gathered data and found the "oops". It was fixed by adding a period…

> On another occasion my machine learning demo API failed due to heavy load, but only when India's prime minister was looking at it.

Okay, I'll bite.

You can't just drop a war story headline like that and walk off. Please expand.

Re: Ask HN: Best “I brought down production” story?

#193
So, I was working for a direct competitor of Expedia in EU.

I was in the team handling flight research and I was adding some special rules for the markup engine.

While adding this new feature I noticed there was an instance of class getting initialized directly in a method, no dependency injection, and it was really hard to test this way since you could not fake this object.

The purpose of this object was to generate an hash for the request.

So I naturally let the IDE extract the dependency like I did many many times.

It was a really small feature and the part of the code was really old so nobody really knew all the secrets. Code review went quite ok so I just deployed it on our 65 servers.

And that's when I actually learned what a lock does on your CPU.

The flight search engine was usually hit by 350 search/s and on our dashboard I could see our CPU being constant 100%.

And that's how I learned that that class included a library that, a creation creates one write lock, and whenever you get the call to generate the hash you will use that lock.

So 350 req/s on a server were sharing only one lock.

Luckily we liked observability as a concept so production was down for less than 5 minutes and that's how I discovered why that class wasn't injected.

Eventually I had to remove that class because conceptually, without sharing the lock you would not get unique hashes but turns out that there was a workaround in place by using the name of the server and added to the hash...

Anyway that's when I caused my company to get silenced by flight metasearch engines across the world because we didn't respond for 5 minutes.

The next two hours our flight business people needed to call all our partners to unlock us.

It would have naturally happened in 6 hours but that was a lot of lost money.

Re: Ask HN: Best “I brought down production” story?

#194

Earlier quoted context omitted.

> TBH, my team was very gracious about it and the RCA focused purely on the events that occurred and how to never let if happen it again. No blame game at all. Which is how a PIR, PER or PCR should be. If you don't understand why someone makes a mistake, you can't avoid future mistakes.

I understand SQL, DBA and TBH, but what do RCA, PIR, PER, and PCR stand for?

RCA is "Root Cause Analysis" and I assume PIR is "Post Incident Review". I don't know PER or PCR.

Re: Ask HN: Best “I brought down production” story?

#195
post #117

I'm in the fortunate position of having been able to tell our story in detail on our blog after a major outage involving Cassandra and bootstrap behaviour that we didn't fully understand. This is a story of how I bought down the bank for two hours. https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on... In summary, we were scaling up our production Cassandra data store and we didn't migrate/backfill the data…

That’s a great writeup, thanks for all the detail! I was always worried about something like this happening so only ever provisioned (via ansible) one server at a time. When the logs showed it was fully synced, we provisioned the next node. It could take two days to add 10 nodes but I always felt much safer

On the cloud, it is likely simpler and faster to just spin up a new cassandra datacenter, and then do a rebuild from the old datacenter to the new datacenter, either all nodes at once in parallel or in smaller batches. This procedure works fine regardless of using static tokens allocation or vnodes, and adds very little load to the old datacenter which is still serving traffic.

Re: Ask HN: Best “I brought down production” story?

#196
post #85
post #80

A friend of mine ran a large and relatively popular (as in at least 30 users online at any given time ...) PvP MUD on a server of mine back in the (late?) 90s. I didn't play muds and my experience was mostly limited to helping him fix C programming bugs from time to time and fielding an occasional irate phone call from users who got my number off the whois data. But because of the programming help I had some kind of…

See also: https://en.wikipedia.org/wiki/Corrupted_Blood_incident

I was there for it, but I literally walked into Ironforge right after the first wave hit, and it was just corpses everywhere. I had no idea what was going on at the time!

Re: Ask HN: Best “I brought down production” story?

#197
EVE Online.

I designed and wrote most of the code for the hacking minigame in EVE. Sadly and unknown to us there was a small memory leak that happened once per game. This was basically unnoticeable until it hit production. Our little game was part of a bigger rework of Exploration in EVE and that plus the game being an entertaining way to make money meant there were upwards of 150k game instances being played a day. EVE does away with the Python GC as well so the memory leak caused us to have to restart nodes every three hours IIRC. My tech lead and I had to comb through and find it which he eventually did and sanity was restored very quickly.

I don’t think it even cracks the top ten of production fuck ups at CCP though.

Re: Ask HN: Best “I brought down production” story?

#198
post #65

Not me, but a colleague - he wanted to look around the system as the `uwsgi` user, so he ran `sudo -u wsgi -s /bin/bash`. Except that he typoed, and instead ran `sudo -c wsgi -s /bin/bash`. What that does is instead of launching the (-s)hell as the uwsgi (-u)ser, it interprets the rest as a (-c)ommand. Now, `wsgi` is also a binary, and unfortunately, it does support a `-s` switch. It tries to open a socket at that ad…

`ssh $host /bin/sh` (or another shell) should work?

That won't work because sshd runs the command using the user's login shell. From https://man.openbsd.org/sshd#LOGIN_PROCESS:

> When a user successfully logs in, sshd does the following:

> ...

> 9. Runs user's shell or command. All commands are run under the user's login shell as specified in the system password database.

Re: Ask HN: Best “I brought down production” story?

#199

I was a system engineer at Amazon from 2001-2006. Sometime around 2004/2005 or so there was a development team working on the "a9 search engine" (meant to complete with google) down in SF. They were sort of an official "shadow IT" offshoot and asked for special treatment and they got me assigned specifically to them to build out the first of their two webservers. They did the usual mistake of wanting to jettison all…

Awesome.

Sadly, it seems that the Web Archive didn't happen to grab any pages from Amazon during the (presumably-small) window this was live.

Specifically, I ran the CDX query hxxp://web.archive-dot-org/cdx/search/cdx?url=amazon.com&matchType=domain, and then grepped through the 174MB of results (1,274,038 lines) for response lengths 9999 bytes and less (ie, [0-9]{1,4}), on the assumption this should find every tiny response. The only such responses are 30x redirects and a couple of 503s. :(

(That's a normal URL above - s/xx/tt/ and s/-dot-/./ - but since it spits out 174MB of text I figured I'd save IA the bandwidth from crawlers fetching everything they see on HN etc.)

Re: Ask HN: Best “I brought down production” story?

#200

Most of these stories seem to be from 20+ years ago. Do newer sysadmins just not make mistakes anymore? Or is 20 years the timespan that is needed to get over the embarassment? :-)

This is actually a great question.

I suspect there are multiple reasons, but the increased reliance on cloud DevOps is likely one of them. Truth is that far fewer companies roll their own critical infrastructure these days.

Post reply on HN