Ask HN: Best “I brought down production” story?

Long ago this one (close to 2000), but we were hosting some of our clients on machines in our office because it was (a lot) cheaper for our startup to do so. We had 3 rather large (for our company) clients running at the time in the server room in the office. The servers where hooked up to 2nd hand APCs so power failures went unnoticed if they were short. One Friday afternoon, we had some drinks (tgif) and a bunch of us (I was the cto....) were fooling around in one of the rooms throwing tennis balls; I threw one straight into the firealarm. This was just a glass with a button behind it: if you pressed the glass it would trigger for the entire (20 story) building. This cut the power and switched on the emergency lights. There was no fire obviously, but, for bureaucracy rules, the firebrigade had to pull up, inspect the building, we had to sign docs etc and then they switched off the alarm and on the power. Too late for our APCs: everything was down and (like said: long time ago for Linux etc) we had to run fsck and basically spent a large portion of the evening getting it all back. We moved to xs4all co location after that incident...

Early on in my career, I worked for a secret unit of a secret government law enforcement group that handled surveillance. Being young, full of verve, and not nearly as smart as I thought I was, I was always trying to improve things and tinker. Knowing nothing about networking, I plugged a switch into itself. Due to the configuration it knocked the entire surveillance network offline and everyone was freaking out. I was cool as a cucumber, because it couldn’t have been me. Must have been a coincidence right?

Right?

The sense of dread the dawned on me as the former Navy Seal turned Network Engineer (and later doctor) started sniffing around the switch I had just touched was palpable. Luckily for me, He kept my mistake quiet and fixed it quickly.

Slightly off topic but I think in most cases responsibility for accidentally bringing down production likes with management.

However I’ve only ever heard stories where management lays the blame.

Can It be a story I was involved in but I didn't do it?

I used to work for a major university as a student systems admin. The only thing that was "student" about it was the pay-- I had a whole lab of Sun and SGI servers/desktops, including an INCREIDBLE 1TB of storage-- we had 7xSun A1000's (an array of arrays) if memory serves.

Our user directories were about 100GB at the time. I had sourced this special tape drive that could do that, but it was fidgety (which is not something you want in a backup drive admittedly). The backups worked, I'd say, 3/4ths of the time. I think the hardware was buggy, but the vendor could never figure it out. Also, before you lecture me, we were very constrained with finances, I couldn't just order something else.

So I graduated, and as such had to find a new admin. We interviewed two people, one was very sharp and wore black jeans and a black shirt-- it was obvious he couldn't afford a suit which would have been the correct thing to wear. The other candidate had suit, and he was punching below his weight. Over my objections, suit guy gets hired.

Friday night, my last day of employment I throw tapes into the machine and start a full L0 backup which would take all weekend to complete.

Monday morning I get a panicked phone calls from my former colleagues. "The new guy deleted the home directories!"

The suit guy literally, had in his first few hours destroyed the entire labs research. All of it. Anyways, I said something to the effect of, "Is the light on the AIT array green or amber?"

"Green."

"You're some lucky sons of bitches. I'll be down in an hour and we'll straighten it out."

I wrote some software to handle charging customer's credit cards. It worked fine in the dev environment so a week later we deployed it since we hadn't been charging customers at all until that point. We would run bills once a day until all the unprocessed billing was caught up.

Well, in dev, the database was refreshed with prod data every night at midnight, so we never saw the bug in my code. I had a sign error in updating the customer's balance so instead of lowering their balance by the payment amount, my code increased their balance. Geometric growth is an amazing thing. A few days later we had calls from angry customers because we had maxed out their credit cards. Miraculously, I was not fired. In retrospect, I think that it might have been because the manager would have then had to explain why he had not made sure there was not adequate testing on something so central to the business.