Incremental backups and full backups on weekends. Operators doing the backups didn’t know it was incremental. With time, there were only two tapes dedicated to those. This was just an ad newspaper and they lost 2 days of ads. As it was only a local failure, they just ran at a loss for to
two days and the business didn’t suffer.
Ask HN: Best “I brought down production” story?
211–220 of 301 posts
Re: Ask HN: Best “I brought down production” story?
#212My colleague asked me to run crontab -l, which I misheard as crontab -r (a switch I was unaware of, which instantly removes all Cron jobs)
I still wonder why crontab has that switch? Who needs that?
We had to piece together all scheduled commands from run logs.
Re: Ask HN: Best “I brought down production” story?
#213Doom on a single PC was fun. But the same game on our network with all those broadcasts running through our net, which hosted a university's PCs. That was bad. And I started it several times until I noticed that "I" was the cause for network slowdown.
Best part was that a sysop entered the room and within a second knew that I was the problem and that I was playing doom. He just told me "if you wanna play games instead of studying than play doom 2. This does not broadcast"
He knows, he plays at night ;)
Re: Ask HN: Best “I brought down production” story?
#214As a new junior, remote, part-time hire, I got frustrated that my test suite wasn't passing on my laptop. So I checked out the codebase on a production server and ran my tests there. The test suite initialized by truncating all tables and loading fixtures instead. Using production database credentials. Oops.
Well… did the tests pass in production, at least?
Re: Ask HN: Best “I brought down production” story?
#215I bought down one of Australasia's biggest banks for 8 hours when I requested a backup of the production database for a test cycle we were about to start.
Unfortunately the database structure had changed and the backup script hadn't been updated and it locked all the tables it tried to access, while also waiting for incomplete transactions to finish... which never happened.
$8 million dollars of deposits and withdrawals vanished into thin air.
Luckily for me, there were perimeter logs for all transactions so a team of people replayed the lost transactions and fixed everything. I had to read the incident reports (which were pretty cringing - people put money into their personal account and there was no record of it ever happening) and write a response which said what went wrong and what I would do differently next time.
No repercussions for me though... I made another request for the database and it worked the next day.
Re: Ask HN: Best “I brought down production” story?
#216Earlier quoted context omitted.
Oh man. MUD's in the late 90s. Just randomly reminded me of all the time I spent in Medivia (I'm fairly sure that was the one). Those were good times. Surprised the name even came to me since I haven't thought about those in... 20ish years.
Medivia? Judging from screenshots that looks a lot like Tibia ( https://www.tibia.com )
Never heard of Medivia before, but looking at screenshots and their site, that is absolutely some sort of fork/copy of Tibia. Even the world map is basically same shape, with locations shifted slightly and renamed, and the game interface is the same. I think they may even be using the old sprites from before a UI revamp a bit over a decade ago, from when I played Tibia.
Edit: Found this [0] which states Medivia originally started as a private Tibia server that eventually evolved into its own game. People were only just starting to make private Tibia servers when I left (or maybe they were only just starting to become popular, not sure anymore), which explains why I hadn't heard of this.
Tibia was created in 1997, I played roughly 2002 - 2006. Looks like Medivia was first created in 2009 [1], I would guess probably still just a private server at the time.
[0] https://www.reddit.com/r/MMORPG/comments/i480jd/medivia_4_ol...
[1] https://www.reddit.com/r/MediviaMMO/ (right sidebar)
Re: Ask HN: Best “I brought down production” story?
#217I had to code a feature in one of the services that essentially made a simple SQL call every x minutes to check up on something. The amount was supposed to be configurable. I put 20 minutes as the default config and everything seemed fine.
Next week I found out that I took down prod because the service made calls every 20 milliseconds instead of every 20 minutes. I remembered to convert "20" to an actual time representation but I accidentally used the unconverted value anyway and it turned out there's an another constructor for the method I used to execute the SQL query and it accepts integer values as well and treats them as milliseconds. What's weird is that this version of the service was running for a week in our test environment and absolutely no one noticed the degraded performance. A temporary workaround was to change the value in the config to 1200000. It taught me the importance of having configurable values in the application as we'd have to build and redeploy it again if the value was hardcoded.
Re: Ask HN: Best “I brought down production” story?
#218In early 00's I got my first gig as a web developer for an oil company. They had no other developers.
There was some kind of security incident, and the PHB (literally) wanted to beef up security. For some reason, infosec was a kind of hobby for me, and I suggested (and volunteered) to install Snort as a company-wide wire tap.
So, during a maintenance period I installed the tap and server where the cable came in.
And it worked great!
A month or so later, all network activity stopped working after a new server was plugged in. It took days to figure out that the cause was a duplicate MAC on the NICs of my Snort box and this new box - a real one in a million thing that I've never heard of since.
Re: Ask HN: Best “I brought down production” story?
#219Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…
Not as bad as yours, but MySQL and also blocking prod table: on my first job after graduate, I once run a delete commands on about 20 rows on a quite large table (maybe 500M+ rows), but was causing deadlock because of gap lock, it has been 6 years so I don't really remember the details.
I was not expert but knew about MySQL optimization at that time, but it looks like sometime you just do things and not think through.
15 minutes later, sysad team PM me and ask WTF am I doing, and I realized what happen.
Re: Ask HN: Best “I brought down production” story?
#220So during my last job, we were getting a new warehouse up and running, which involved a lot of mass inventory updates to get things moved around as quickly as possible; my boss and I were tasked with doing these updates in the database directly in order to bypass some really annoying checks in our warehouse management system that'd slow the process down to a crawl. I wrote up a T-SQL script that'd swap the inventories of two of our "pallet drop" locations, and each of us would accompany someone from the warehouse management team running the pallet jack to do the physical swaps (during which we'd then run the script with the relevant location IDs). During this, the new warehouse was already operational, so as we were swapping things around, warehouse pickers would be picking things from the very locations we're finagling (and part of the script I wrote handled updating the pickers' pick paths to ensure they didn't get sent on crazy goose chases).
So my boss and I are both running this script, and it goes well enough for a few minutes, until all of a sudden we start hearing people shouting "system's down!" throughout the warehouse. Everything's at a standstill, and at this point I start immediately combing through my script, wondering what the hell I could've fucked up. We also get on the phone with the WMS vendor; this being very much a "warranty void" situation, my boss and I didn't expect them to be able to help us, and the whole warehouse being down meant the clock was ticking to get this fixed.
At some point, the WMS vendor's resident SQL expert noticed that there were a bunch of uncommitted transactions that were deadlocking the DB. Turns out that when my boss copy-pasted the script I wrote out of my Slack message to him, he forgot to include the last line: "COMMIT TRANSACTION". We repeatedly COMMIT'd in his session for a few minutes until there was nothing left to commit, and then everything was working again.
So lesson learned: if you're gonna copy and paste some script, make sure you copy and paste the whole thing, lol