Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Best “I brought down production” story?

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 301 posts

Re: Ask HN: Best “I brought down production” story?

#141

Post by yjftsjthsd-h »

Animats wrote: post #51

Back when I was working on proof of correctness, when that was a very new thing, I was using the Boyer-Moore theorem prover remotely on a large time-shared mainframe at SRI International. At the time, you needed a mainframe to run LISP. I was working on proofs of basic numeric functions for bounded arithmetic. So I was writing theorems with numbers such as 65536. This caused the mainframe to run out of memory, page o…

> Boyer and Moore explained to me that the internal representation of numbers was exactly that of their constructive mathematics theory. 2 was (ADD1 (ADD1 (ZERO))). 65536 was a very long string of CONS cells. I was told that most of their theorems involved numbers like 1.

Can we have a moment for the folks who managed to turn numerical purity into integers being O(n)? It's unbearably beautiful... And I do mean unbearably...

Re: Ask HN: Best “I brought down production” story?

#142

Post by xtracto »

whoisjohnkid wrote: post #32

heard a few years back, someone accidentally deleted everything across all AWS accounts at our company; had to reach out to AWS and they helped recover everything. Took 6-12 hours to recover

A similar thing happened on a previous team: One of the engineer's was experimenting with Ansible, and apparently setup some command that Terminated all instances in a VPC... he ran it for production. Suddenly I start getting alerts of unavailable instances and whatnot. Fortunately, most of the important servers had "termination protection".

Another time, one of the Engineers was doing some analysis in the MongoDB production Database, and found it easier to setup a tunnel to 27017 in his computer to connect to the DB directly. While multitasking, he decided to run the test suite of some changes he was doing to one of the programs.... but that test had as a first step to delete all tables from your local Mongo database. Next morning when I come back Ops people tell me that "the system is empty". My first thought was "What do they mean the system is empty? maybe one of the dashboards failed to update or something". My surprise when I login and see what really happened...oh boy. We spent like 5 or 6 hours getting everything back. Fortunately a combination of Backups and MongoDB replication logs allowed us to recover everything.

Re: Ask HN: Best “I brought down production” story?

#143

Post by andrewstuart2 »

peeters wrote: post #133
notatoad wrote: post #116

Earlier quoted context omitted.

It's not good to punish people for making mistakes in the course of their work (especially if that work is meant to be educational) It is good to punish people who give access to production databases to people who shouldn't have it. And the guy learning MySQL should not be given that access. Taking down prod is always a symptom of a systemic failure. The person responsible for the systemic failure should see the cons…

> The person responsible for the systemic failure should see the consequences You don't see the contradiction of terms there? A systemic failure is by definition not the responsibility of one person. You're saying people should be able to make mistakes. But not those people.

Having worked in a very large, bureaucratic company I can say that I strongly suspect that just ignoring systemic failures as learning opportunities is also not a sustainable. Too many times I’ve had to yield on something and say “I guess they’ll learn when this fails” only to see them easily move on or get promoted before the failure occurs. They don’t learn their lessons.

I suspect the solution is to find a way to make sure the consequences of the decision are fed back properly into the system directed at the right person. How to do that, I have no idea.

Re: Ask HN: Best “I brought down production” story?

#144

Post by [deleted] »

RHSeeger wrote: post #128
sneak wrote: post #85

Earlier quoted context omitted.

See also: https://en.wikipedia.org/wiki/Corrupted_Blood_incident

That was an annoying time in the game. Not horrible, but you pretty much had to avoid town because people were purposefully trying to spread the plague. Certainly memorable though.

[deleted]

Re: Ask HN: Best “I brought down production” story?

#145

Post by ratcline »

RHSeeger wrote: post #128
sneak wrote: post #85

Earlier quoted context omitted.

See also: https://en.wikipedia.org/wiki/Corrupted_Blood_incident

That was an annoying time in the game. Not horrible, but you pretty much had to avoid town because people were purposefully trying to spread the plague. Certainly memorable though.

Little did we know how realistic that would end up being.

Re: Ask HN: Best “I brought down production” story?

#146

Post by exikyut »

cvrjk wrote: post #91

Sent in a SQL command deleting multiple million records (intended) wrapped in a single transaction (not intended). The replication queues could not keep up and failed, bringing down most of the replicas. Master server kept trying to recover and maxed out all connections - no DBA could log in to perform manual recovery. We had to hard reboot not knowing what state the system is in and how long it'll take to fully reco…

Hmmm. I can't help but wonder if maybe the database engine should have caught the queue overflow (even if the event occurred over the network) and failed the transaction.

Re: Ask HN: Best “I brought down production” story?

#147

Post by gumby »

yjftsjthsd-h wrote: post #141
Animats wrote: post #51

Back when I was working on proof of correctness, when that was a very new thing, I was using the Boyer-Moore theorem prover remotely on a large time-shared mainframe at SRI International. At the time, you needed a mainframe to run LISP. I was working on proofs of basic numeric functions for bounded arithmetic. So I was writing theorems with numbers such as 65536. This caused the mainframe to run out of memory, page o…

> Boyer and Moore explained to me that the internal representation of numbers was exactly that of their constructive mathematics theory. 2 was (ADD1 (ADD1 (ZERO))). 65536 was a very long string of CONS cells. I was told that most of their theorems involved numbers like 1. Can we have a moment for the folks who managed to turn numerical purity into integers being O(n) ? It's unbearably beautiful... And I do mean unbea…

> Can we have a moment for the folks who managed to turn numerical purity into integers being O(n)?

It’s not an unreasonable approach given the application was automatic theorem proving: https://en.wikipedia.org/wiki/Peano_axioms

Both Boyer and Moore are quite sharp so I wouldn’t jump to the conclusion that they didn’t know what they were doing.

Re: Ask HN: Best “I brought down production” story?

#149

Post by ketzo »

peeters wrote: post #133
notatoad wrote: post #116

Earlier quoted context omitted.

It's not good to punish people for making mistakes in the course of their work (especially if that work is meant to be educational) It is good to punish people who give access to production databases to people who shouldn't have it. And the guy learning MySQL should not be given that access. Taking down prod is always a symptom of a systemic failure. The person responsible for the systemic failure should see the cons…

> The person responsible for the systemic failure should see the consequences You don't see the contradiction of terms there? A systemic failure is by definition not the responsibility of one person. You're saying people should be able to make mistakes. But not those people.

A piece of the system (a junior developer) is allowed to make mistakes. The person responsible for architecting and protecting the system (the CTO)... less so.

Re: Ask HN: Best “I brought down production” story?

#150

Post by fulafel »

linsomniac wrote: post #107
hamburglar wrote: post #62

Earlier quoted context omitted.

Does the proc entry for a running process still link to the now-deleted file in that situation? If so, you might be able to save yourself from a running bash shell by doing a “cat /proc/$$/exe > /bin/bash”

Probably not if it was overwritten (": >/bin/bash") rather than removed and recreated ("rm -f /bin/bash; : >/bin/bash"). The former will cause all processes to see the empty file, the latter would leave processes with access to the old contents. In this case if you noticed and still had a shell, you could just copy another shell over ("cp /bin/sh /bin/bash"), to at least get back to probably able to login, until you…

In the wsgi case of replacing a regular file with an AF_UNIX socket, the destination would have needed to to be unlinked first - the bind() call will otherwise fail.
Post reply on HN