Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-incident review on the Atlassian April 2022 outage

atlassian.com

Post reply on HN
Print view View original

41–50 of 157 posts

Re: Post-incident review on the Atlassian April 2022 outage

#41

Post by largbae »

baskethead wrote: post #36
largbae wrote: post #18

Earlier quoted context omitted.

It is never that simple. Say the backup existed, and was global. By the time you get everyone briefed on how fubar it is and get agreement to load the backups, there are hours of changes from the unaffected customers that will be wiped by the restore, or have to be reconciled by hand for months. Sure, you can concoct the perfect antidote with hindsight, but their retro and next steps are sound.

What it means is that they never tested their disaster recovery system, because this would have been found right away. Or, someone would have reported it and an upper level exec would have signed off on it being okay to take 14 days to restore a small subset of users.

Again, not that simple. The customer restore procedure was almost certainly tested(and in active use as customers blow up their own data often enough). It was _not_ tested on 800 customer stacks simultaneously, as that was considered a sitewide disaster by whoever dreamed up the failure modes to test for. Meanwhile the actual whole site disaster restore plan may or may not have been tested, but it was useless for this case since some customers were unaffected and would be damaged by the whole site plan.

Re: Post-incident review on the Atlassian April 2022 outage

#42

Post by rob_c »

largbae wrote: post #18
rob_c wrote: post #14

Seriously, where were the -24hr backups that could be rolled back to once its clear the script is fubar, or using it on just 10% of the estate first? ...

It is never that simple. Say the backup existed, and was global. By the time you get everyone briefed on how fubar it is and get agreement to load the backups, there are hours of changes from the unaffected customers that will be wiped by the restore, or have to be reconciled by hand for months. Sure, you can concoct the perfect antidote with hindsight, but their retro and next steps are sound.

Build it properly. By definiton of following best practice IT IS or always should be not far from being able to follow this. If its not someone is to blame

Re: Post-incident review on the Atlassian April 2022 outage

#43

Post by geerlingguy »

eastbound wrote: post #32
perlgeek wrote: post #21

Earlier quoted context omitted.

I tend to agree. However, there's one point that makes me skeptical: there are no organizational changes, or changes to leadership, or anything in that direction. This sounds like "the tech guys screwed up, culture and management is fine here". Which it might be, or it might not. I would have loved to see 5. We will stop pushing customers so hard towards using our cloud for one, but that wouldn't be convenient for At…

Note that the ToS also forbid Cloud users from “disseminating information” about “the performance of the products”. So you can’t say it’s slow or unperforming.

I thought, in terms of Jira and Confluence at least, it was just accepted that being slow and underperforming was the status quo and if it was running at a speed you'd consider normal, that's an exception (and cause for alarm... like "did that actually save or is there a silent JS error not being displayed?").

Re: Post-incident review on the Atlassian April 2022 outage

#44

Post by andrewstuart »

For anyone wanting to attack:

* hindsight is 20/20 - it's easy to lecture after the fact about how mistakes could have been avoided

* modern software systems are very complex

* have you never made a mistake?

It is however noteworthy that they have done the wise thing from a publicity perspective, which is post this on a Friday, hoping that by the next tech cycle more interesting things will have happened for the press to report than a rehash of this outage. That's politics.

Re: Post-incident review on the Atlassian April 2022 outage

#45

Post by stubish »

seanwilson wrote: post #15

Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just you…

Generally a database that allows you to revert only allows you to revert the entire database. Bits of information are often related to other bits of information, and if you only undo some of the changes, then you end up with inconsistent data (dangling references) and destroyed data (overriding information more recent than what you are restoring). So soft deletes tends to become a domain specific problem rather than a problem that can be solved with better tech.

I'd argue that SQL has some pretty big guardrails. 'DELETE FROM CUSTOMER WHERE' will generally fail, because there is lots of data referring to the CUSTOMER table and the system will insist the data remain consistent.

Re: Post-incident review on the Atlassian April 2022 outage

#46

Post by posnet »

baskethead wrote: post #34

Oooof. Passing in Application IDs will delete applications and passing in site IDs will delete sites. That's a really really bad design. I'm bookmarking this so that I can use it as a showcase going forward. Just this week, I changed a spec in one of our proposed endpoints that did exactly that. We passed in ids of various types of objects to perform actions, and I changed the api so that it would be forced to pass i…

And when it comes to user interfaces for deleting, have it repeat back to the person how many, and what is being deleted.

And have them type out both the number and they thing they are deleting.

https://rachelbythebay.com/w/2020/10/26/num/

Re: Post-incident review on the Atlassian April 2022 outage

#47

Post by lucb1e »

Andugal wrote: post #6

If I understand correctly, since they deleted only a small subset of all their customers, they could not restore a clean backup of those customers without losing data from other customers not impacted by the outage. So how would one have a clean "partial backup" strategy if something similar would happen in his company?

(or her company)

Depends a lot on the situation and technologies already in use. For example, if you lost ticket sales, you might monkey patch it by having two systems at the entrance: one with the main data, one with the restored backup. If the person isn't in main, you can check backup. That could be deployed more quickly than trying to consolidate the two states.

In another situation, you could isolate customers so that a delete and a restore simply affects everything at that customer and such a partial delete is not a problem. You could still have trouble if there is a partial delete within a customer system, but restoring part of 1 company is a lot less work than restoring parts of hundreds of companies.

Re: Post-incident review on the Atlassian April 2022 outage

#49

Post by AceJohnny2 »

eastbound wrote: post #32
perlgeek wrote: post #21

Earlier quoted context omitted.

I tend to agree. However, there's one point that makes me skeptical: there are no organizational changes, or changes to leadership, or anything in that direction. This sounds like "the tech guys screwed up, culture and management is fine here". Which it might be, or it might not. I would have loved to see 5. We will stop pushing customers so hard towards using our cloud for one, but that wouldn't be convenient for At…

Note that the ToS also forbid Cloud users from “disseminating information” about “the performance of the products”. So you can’t say it’s slow or unperforming.

Well that's some Oracle-tier shit.

(ref: https://danluu.com/anon-benchmark/)

Re: Post-incident review on the Atlassian April 2022 outage

#50

Post by stubish »

pokoleo wrote: post #8

Soft deletion always feels at odds with privacy-related "right to have data deleted" laws. Would be super interested in a technical writeup on how they do this.

Restoring data from backup is the most common data recovery technique. Lots of information there to start from if you are interested in how data recovery relates to privacy laws.
Post reply on HN