Ask HN: Best “I brought down production” story?

I oversaw a Solaris machine for a very short period, and never got any reason to use it. But if you can change the signal, sending SIGHUP into everything looks like reasonable thing to do.

Still, it's not something common enough to deserve it's own program.

One Tuesday morning, my code brought down 67 restaurants piloting the new point-of-sale (POS) system.

I'd written the code to reformat the mainframe database of menu items, prices, etc, to the format used by the store systems. I hadn't accounted for the notion that the mainframe would run out of disk space. When the communications jobs ran, a flock of 0-byte files were downloaded to the stores. When the POS systems booted with their 0-byte files, they were... confused. As were the restaurant managers. As were the level 1, level 2, vendor, and executive teams back at headquarters. Once we figured it out, we re-spun the files, sent them out, and the stores were back in business. I added a disk space check, and have done much better with checking my return codes ever since.

That's over engineering. At this point, just rely on PITR. FWIW, postgres does have a "reaper" via vacuum, but not for the purpose of safety. but rather to allow for mvcc.

Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily.

I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an office outside Los Angeles.

MyISAM uses a read lock on every SELECT query. I did not know this at the time. I was running a number of queries that were trying to pull historical performance data for all our ads across all time. They were taking a long time so I let them run in the background while working on a spreadsheet somewhere else.

A little while later I hear some murmuring. Apparently the whole network was down. The engineering team was frantically trying to find the cause of the problem. Eventually, the CTO approaches my desk. "Were you running some queries on the database?" "Yes." "The query you ran was trying to generate billions of rows of results and locked up the entire database. Roughly three quarters of the ads in the world have been gone for almost two hours."

After the second time I did this, he showed me the MySQL EXPLAIN command and I finally twigged that some kinds of JOINs can go exponential.

Kudos to him for never revoking my access and letting me learn things the hard way. Also, if he worked for me I would have fired him.

I did the reverse. Using DOS batch scripts I wrote I imported orders twice for a 100 person generic drug company. It was a heavy set of orders as it was. Everyone in the warehouse had to work late that day. Sales guys loved it because it was near the end of the month. No one at the company seemed to mind the mistake. I was mystified that I did not get in trouble. They had tight relationships with thier customers and just shipped less over the next month.