A collection of postmortems

#1

Post by luu » Sat, Aug 08, 2015, 7:15 PM UTC

Re: A collection of postmortems

#2

Post by dewey » Sat, Aug 08, 2015, 8:29 PM UTC

I think it'd be easier to browse if it'd be organized in a directory structure like:

    Microsoft/
        Outage 1
        Outage 2

In case anyone feels like contributing:

https://hn.algolia.com/?query=postmortem&sort=byPopularity&p...

https://hn.algolia.com/?query=post-mortem&sort=byPopularity&...

Re: A collection of postmortems

#3

Post by luxpir » Sat, Aug 08, 2015, 8:35 PM UTC

Really interesting repo and kind of a relief that mishaps happen to everyone, more or less. We can try to mitigate as best as possible, but these are just the high profile cases. Imagine the millions of internal post mortems carried out daily by all manner of successful companies.

My outage anxiety is reducing, even if contingencies are in place.

Re: A collection of postmortems

#4

Post by josemagana » Sat, Aug 08, 2015, 10:20 PM UTC

I like how the healthcare.gov one is the only one without a description

Re: A collection of postmortems

#5

Post by [deleted] » Sat, Aug 08, 2015, 10:31 PM UTC

josemagana wrote: post #4

I like how the healthcare.gov one is the only one without a description

[deleted]

Re: A collection of postmortems

#6

Post by 0x400614 » Sun, Aug 09, 2015, 12:40 AM UTC

I like how HealthCare.gov doesn't have a description

Re: A collection of postmortems

#7

Post by leothekim » Sun, Aug 09, 2015, 12:52 AM UTC

Relevant: Foursquare's post-mortem with mongodb. https://groups.google.com/forum/m/#!topic/mongodb-user/UoqU8...

Re: A collection of postmortems

#8

Post by avz » Sun, Aug 09, 2015, 1:50 AM UTC

dewey wrote: post #2

I think it'd be easier to browse if it'd be organized in a directory structure like: Microsoft/ Outage 1 Outage 2 In case anyone feels like contributing: https://hn.algolia.com/?query=postmortem&sort=byPopularity&p... https://hn.algolia.com/?query=post-mortem&sort=byPopularity&...

I'd rather have them arranged by root cause category (e.g. bad service config, bad network config, units mismatch, inconsistent binary version, software bug, etc) than by which company it affected.

Re: A collection of postmortems

#9

Post by avitzurel » Sun, Aug 09, 2015, 3:24 AM UTC

When we have a horrible bug or an outage, I always share a post-mortem email company wide, not just engineering.

I think post-mortems are a huge opportunity to learn about resiliency and common system and engineer errors, I know I grew as an engineer with each of those.

Thanks for sharing!

Re: A collection of postmortems

#10

Post by yuvipanda » Sun, Aug 09, 2015, 3:39 AM UTC

https://wikitech.wikimedia.org/wiki/Incident_documentation has a fairly complete list of Wikimedia's Post Mortems.