Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-incident review on the Atlassian April 2022 outage

atlassian.com

Post reply on HN
Print view View original

21–30 of 157 posts

Re: Post-incident review on the Atlassian April 2022 outage

#21

Post by perlgeek »

justinludwig wrote: post #5

I can't say that I've ever been a fan of Atlassian or their products, but this blog post makes it sounds like they've at least learned the right lessons from this: 1. Establish universal "soft deletes" across all systems. 2. Better DR for multi-site, multi-product incidents. 3. Fix their incident-management process for large-scale incidents. 4. Fix their incident communications. Regarding #4 in particular: "Rather th…

I tend to agree.

However, there's one point that makes me skeptical: there are no organizational changes, or changes to leadership, or anything in that direction.

This sounds like "the tech guys screwed up, culture and management is fine here". Which it might be, or it might not.

I would have loved to see

5. We will stop pushing customers so hard towards using our cloud

for one, but that wouldn't be convenient for Atlassian.

Re: Post-incident review on the Atlassian April 2022 outage

#22

Post by tyre »

seanwilson wrote: post #15

Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just you…

Datomic treats all data as immutable, so it can wind back to any version.

When new data is written, the entire block is copied and rewritten rather than changing the data in-place.

Re: Post-incident review on the Atlassian April 2022 outage

#23

Post by kwertyoowiyop »

largbae wrote: post #18
rob_c wrote: post #14

Seriously, where were the -24hr backups that could be rolled back to once its clear the script is fubar, or using it on just 10% of the estate first? ...

It is never that simple. Say the backup existed, and was global. By the time you get everyone briefed on how fubar it is and get agreement to load the backups, there are hours of changes from the unaffected customers that will be wiped by the restore, or have to be reconciled by hand for months. Sure, you can concoct the perfect antidote with hindsight, but their retro and next steps are sound.

That lesson really stuck out for me also. My definition of “restore” has been too simplistic.

Re: Post-incident review on the Atlassian April 2022 outage

#25

Post by candiddevmike »

seanwilson wrote: post #15

Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just you…

WAL archiving / point in time recovery can help with this.

Re: Post-incident review on the Atlassian April 2022 outage

#26

Post by mdavidn »

tyre wrote: post #22
seanwilson wrote: post #15

Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just you…

Datomic treats all data as immutable, so it can wind back to any version. When new data is written, the entire block is copied and rewritten rather than changing the data in-place.

This is similar to SQL checkpoints in that a rollback is all or nothing. It wouldn’t be segmented by tenant unless the each tenant has its own transactors.

Re: Post-incident review on the Atlassian April 2022 outage

#28

Post by wolf550e »

seanwilson wrote: post #15

Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just you…

In MVCC systems like PostgreSQL, if you don't vacuum (garbage collect the old tuples), your database is append-only and you can query as if your transaction was started at some time in the past. I don't know how to set auto-vacuum to have a fixed delay, e.g. keeping 24h of changes, but I bet it can be added if it's not built-in.

Re: Post-incident review on the Atlassian April 2022 outage

#30

Post by codeflo »

ilayn wrote: post #24

I am very curious if they used a Jira board during this crisis for issue tracking. Because then they would have more than 4 lessons learned.

What you're basically suggesting is that feature development at Atlassian moves at such a glacial speed because of course they're using Jira to manage it. This kind of blows my mind right now.
Post reply on HN