I can't say that I've ever been a fan of Atlassian or their products, but this blog post makes it sounds like they've at least learned the right lessons from this: 1. Establish universal "soft deletes" across all systems. 2. Better DR for multi-site, multi-product incidents. 3. Fix their incident-management process for large-scale incidents. 4. Fix their incident communications. Regarding #4 in particular: "Rather th…
Post-incident review on the Atlassian April 2022 outage
31–40 of 157 posts
Re: Post-incident review on the Atlassian April 2022 outage
#32I can't say that I've ever been a fan of Atlassian or their products, but this blog post makes it sounds like they've at least learned the right lessons from this: 1. Establish universal "soft deletes" across all systems. 2. Better DR for multi-site, multi-product incidents. 3. Fix their incident-management process for large-scale incidents. 4. Fix their incident communications. Regarding #4 in particular: "Rather th…
I tend to agree. However, there's one point that makes me skeptical: there are no organizational changes, or changes to leadership, or anything in that direction. This sounds like "the tech guys screwed up, culture and management is fine here". Which it might be, or it might not. I would have loved to see 5. We will stop pushing customers so hard towards using our cloud for one, but that wouldn't be convenient for At…
So you can’t say it’s slow or unperforming.
Re: Post-incident review on the Atlassian April 2022 outage
#33Re: Post-incident review on the Atlassian April 2022 outage
#34Just this week, I changed a spec in one of our proposed endpoints that did exactly that. We passed in ids of various types of objects to perform actions, and I changed the api so that it would be forced to pass in a struct that contained an object type and object id. Explicitness is so much safer in the long run, especially in enterprise apps.
Re: Post-incident review on the Atlassian April 2022 outage
#35Soft deletion always feels at odds with privacy-related "right to have data deleted" laws. Would be super interested in a technical writeup on how they do this.
Re: Post-incident review on the Atlassian April 2022 outage
#36Seriously, where were the -24hr backups that could be rolled back to once its clear the script is fubar, or using it on just 10% of the estate first? ...
It is never that simple. Say the backup existed, and was global. By the time you get everyone briefed on how fubar it is and get agreement to load the backups, there are hours of changes from the unaffected customers that will be wiped by the restore, or have to be reconciled by hand for months. Sure, you can concoct the perfect antidote with hindsight, but their retro and next steps are sound.
Re: Post-incident review on the Atlassian April 2022 outage
#37Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just you…
An intuitive way to think of bitemporality is it's like MVCC, but with 4 timestamps per row version. One pair describes a range of time in "outside" or "valid" time, ie whatever semantic domain the database is modeling, the other pair describes a range of "system" time, which is when this record was present in the database. This lets you capture and reason about the distinction between when a fact the database models was true in the real world, and when the database was updated to reflect this fact (some people call this "as of" vs "as at"... the terms here aren't fully settled but the basic distinction is). So you can revise history, do complex time travel queries, all sorts of stuff. It's a very useful model that directly aligns with what sort of questions businesses need to answer in the context of a court case or revising their ground source of truth due to past bug or error.
The downside is your database balloons with row versions, and many queries become far more complicated, perhaps needing addition joins, etc. Also from the perspective of database implementors there's a ton more complexity in the code. So that's why it's not widely supported despite the standard.
There's also a niche of databases built around this model from the ground up, usually based on Datalog instead of SQL. There's also overlapping work with RDF and Semantic Web thinking (as awry as all that went).
In practice how most organizations address this is operationally, by keeping generational and incremental backups that let them restore previous database states as needed. Though as the original post we're here for proves, that kind of operational solution can bite back hard when it goes wrong.
Re: Post-incident review on the Atlassian April 2022 outage
#38I can't say that I've ever been a fan of Atlassian or their products, but this blog post makes it sounds like they've at least learned the right lessons from this: 1. Establish universal "soft deletes" across all systems. 2. Better DR for multi-site, multi-product incidents. 3. Fix their incident-management process for large-scale incidents. 4. Fix their incident communications. Regarding #4 in particular: "Rather th…
I am actually pleasantly surprised at the openness of this response, and their taking responsibility of mistakes and detailing what will change in the future. It's not just corporate speak. I think that speaks well for the company and it improved my view of them.
Re: Post-incident review on the Atlassian April 2022 outage
#39Soft deletion always feels at odds with privacy-related "right to have data deleted" laws. Would be super interested in a technical writeup on how they do this.
A simple technical solution is to store all data with per user encryption keys, and then just delete the key. This obviously doesn't let you prove to anyone else that you've deleted all copies of the key, but you can use it as a way to have higher confidence you don't inadvertently leak it.
Re: Post-incident review on the Atlassian April 2022 outage
#40Soft deletion always feels at odds with privacy-related "right to have data deleted" laws. Would be super interested in a technical writeup on how they do this.
Of course, this means trusting Atlassian to actually delete the key on request, but there's not much reason for them not to.