I just finished reading the Roblox outage postmortem (trending on HN: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/). I learned a lot reading about it and found myself thinking "surely there's other well-written postmortems that are a doozy". Does anyone know a good resource where I can find a compilation of such outages? If they're classified/tagged then even better :)
Famous outages along with deep postmortems?
1–10 of 13 posts
Re: Famous outages along with deep postmortems?
#2Not a compilation, but here are a couple of interesting postmortems I recall from my time at PagerDuty:
https://www.pagerduty.com/blog/the-discovery-of-apache-zooke...
https://status.pagerduty.com/incidents/v2vrgccbtgxn
The second one was a 30+ hour outage that forced us to really up our incident management game.
Re: Famous outages along with deep postmortems?
#3Re: Famous outages along with deep postmortems?
#4There are links on this page [0] to proper reports on the cause of the 2003 blackout in eastern North America (apologies to the maritimes and Quebec)
0: https://www.ieso.ca/en/Corporate-IESO/Media/Also-of-Interest...
Re: Famous outages along with deep postmortems?
#5If you'd like to go beyond technical postmortems, I've found the IAEA reports on radiological accidents etc to make very interesting technical reading. Abstractly there is a often something to be learned about having a good playbook to deal with specific events.
Re: Famous outages along with deep postmortems?
#6Re: Famous outages along with deep postmortems?
#7I did start a site where I was logging what I could at outagereports.net, but I let it lapse when cpanel fees shot up. Dan Luu's site referenced below is a good source [1]. I think also there is a big list on github somewhere of all k8s-caused outages.
Re: Famous outages along with deep postmortems?
#8Check out anything from the NTSB!
OUTLANDISHLY-GOOD reporting: https://www.ntsb.gov/investigations/AccidentReports/Pages/Re...
Re: Famous outages along with deep postmortems?
#9Project Zero's exploit reporting is fantastically in-depth: https://googleprojectzero.blogspot.com
Re: Famous outages along with deep postmortems?
#10Citizen Lab is known to have good reporting: https://citizenlab.ca/2022/01/cross-country-exposure-analysi...