Earlier quoted context omitted.
Hi, if I may, what does PITR stand for?
https://en.wikipedia.org/wiki/Point-in-time_recovery
Thanks!
Ask HN: Best “I brought down production” story?
201–210 of 301 posts
Re: Ask HN: Best “I brought down production” story?
#2022 days before I got married, I dropped the production database by accident from a GUI tool where “right-clicking” can be destructive if you click to fast. The application scheduled radio and television commercials and within 48 US states for a large international advertising group. The bigger problem was that the DBA had only been doing incremental backups and didn’t have a full back against which to run the incremen…
> where “right-clicking” can be destructive if you click to fast
That’s pretty amazing in the most terrible way possible
Re: Ask HN: Best “I brought down production” story?
#203> rewriting a repository to use a ORM, jOOQ in this particular case with Java8
> spend two weeks on this, write extremely meticulous tests, I am a junior/mid level engineer at this point in my career and this is a game changing ticket
> day of the switch approaches, sweating bullets
> switch happens, so far so good
> This was for a food delivery company so the volume of orders changes throughout the day (lunch, dinner, evening etc)
> at lunch time orders suddenly start disappearing, but eventually over time things go back to normal
> goes on like this for hours, at least a few
> tech team is confused ... why are these orders disappearing into nowhere
> senior engineer is suspicious and reviews my PR from the night before
> I forgot to remove `LIMIT 100` on the GET /order query when I was testing
Still makes me chuckle to this day
Re: Ask HN: Best “I brought down production” story?
#204$ rm -rf /var/mysql/data
*Thinks for 5 seconds...is this prod? ....@$1£!? ... ctrl + c....should I tell anyone that just happened....walks over to line manager...*
I did not taste anything that day.
Re: Ask HN: Best “I brought down production” story?
#205Go-live of a migration of three web apps and their DBs from AWS to Azure resulted in one of the hot customer facing APIs grinding to a halt during peak hours. Compute and DB SKU specs were exactly the same but as it turns out, Azure DB PaaS performance is nowhere near that of AWS RDS. pg_bench reported about 4x more transactions completed in AWS, even though in Azure I had upgraded to postgres 11 enabling parallel query plans.
Another fun time was when a reboot of an ubuntu 14.04 VM previously migrated up to Azure from an on-prem VMware esxi decided it wanted to sit waiting for user input in grub. We had no extension installed to use the serial console for user input. Since it was hosting a production postgres db, I received a call at 12am and was up til 4am transplanting /var/lib/postgresql from the old OS disk onto a fresh ubuntu VM's disk.
Re: Ask HN: Best “I brought down production” story?
#206One Tuesday morning, my code brought down 67 restaurants piloting the new point-of-sale (POS) system. I'd written the code to reformat the mainframe database of menu items, prices, etc, to the format used by the store systems. I hadn't accounted for the notion that the mainframe would run out of disk space. When the communications jobs ran, a flock of 0-byte files were downloaded to the stores. When the POS systems b…
Huh, one doesn't hear about mainframes in the context of restaurants much. But I'm honestly not sure how far back to guesstimate - 20 years is probably reasonable, but it could even be 15...
Re: Ask HN: Best “I brought down production” story?
#207Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…
> some kinds of JOINs can go exponential.
How? AFAIK a single join can be at most quadratic, and multiple joins should at most be polynomial, where the exponent is the number of joins. To go exponential, you'd need some kind of recursion or self reference, but I know no way to express such a thing as an ordinary join statement.
(of course quadratic performance is already prohibitively slow on large tables, so there is no need to go exponential in order to take "forever")
Re: Ask HN: Best “I brought down production” story?
#208$ rm -rf /var/mysql/data *Thinks for 5 seconds...is this prod? ....@$1£!? ... ctrl + c....should I tell anyone that just happened....walks over to line manager...* I did not taste anything that day.
Hopefully there was a backup...?
Re: Ask HN: Best “I brought down production” story?
#209Here is what I did to GCE: https://status.cloud.google.com/incident/compute/15046
Can’t say more than the provided link, but that was definitely a messy weekend which is also my boss’ birthday… it is the only time so far that I chatted and discussed tech issues directly with so many SVP/VPs in Google. Hmm, miss it :P
Re: Ask HN: Best “I brought down production” story?
#210Back in the days of MyISAM and before Google had their own ad network I worked for the world's largest advertising network. It had a global reach of 75%, meaning 3 / 4s of people saw at least one of our ads daily. I was trying to learn MySQL and the CTO made the mistake of giving me access to the prod database. This huge network that served most of the ads in the world ran off of only two huge servers running in an o…
In our software, a minority codepath sometimes reported database deadlocks. Nothing critical but it littered the ops error logs and probably displayed error messages to a few customers. So I added a pessimistic exclusive lock to a query which basically solved the deadlock problem (not a great solution but it worked). However, what I missed was the query, even though in a minority code path, was touching another table used basically in all hot-path queries. So basically the code seemed to work fine until deployed to all servers when all operations of the whole cluster got basically serialized through this single lock.
So, yeah, database locks can bite you hard!