Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-incident review on the Atlassian April 2022 outage

atlassian.com

Post reply on HN
Print view View original

151–157 of 157 posts

Re: Post-incident review on the Atlassian April 2022 outage

#151

Post by closeparen »

yawaramin wrote: post #86
tyre wrote: post #22

Earlier quoted context omitted.

Datomic treats all data as immutable, so it can wind back to any version. When new data is written, the entire block is copied and rewritten rather than changing the data in-place.

That would be difficult to use given requirements like GDPR's user data deletion.

Real world immutable systems usually support the eventual garbage collection of soft-deleted or superseded records. No one actually wants disk usage to grow unbounded forever. It just means you are not going to support "overwrite immediately" semantics in ordinary application code paths.

Datomic describes its capability here: https://docs.datomic.com/on-prem/reference/excision.html

Re: Post-incident review on the Atlassian April 2022 outage

#152

Post by baskethead »

largbae wrote: post #41
baskethead wrote: post #36

Earlier quoted context omitted.

What it means is that they never tested their disaster recovery system, because this would have been found right away. Or, someone would have reported it and an upper level exec would have signed off on it being okay to take 14 days to restore a small subset of users.

Again, not that simple. The customer restore procedure was almost certainly tested(and in active use as customers blow up their own data often enough). It was _not_ tested on 800 customer stacks simultaneously, as that was considered a sitewide disaster by whoever dreamed up the failure modes to test for. Meanwhile the actual whole site disaster restore plan may or may not have been tested, but it was useless for thi…

Then we are in agreement. Even if it were the case they had to do 800 separate recoveries (which I vaguely remember reading they couldn't do it individually), it means they never tested a large scale recovery situation and had no idea that 800 recoveries would take 2 weeks. That's a significant issue that they should have tested.

Re: Post-incident review on the Atlassian April 2022 outage

#153

Post by throwdbaaway »

Aeolun wrote: post #93
ripper1138 wrote: post #65

Earlier quoted context omitted.

So many opportunities missed to avoid this! Look at one ID and ensure it is what you expect it to be. Run the script in a dryrun mode. Run the script for 1 customer. Probably more!

And don’t make a “universal delete” script in the first place…

So, what do you think of terraform?

It's puzzling to me that we, as software developers, spend so much efforts trying to automate such one off deletion tasks, and the automation would inevitably go wrong and result in data loss.

Re: Post-incident review on the Atlassian April 2022 outage

#154

Post by ripper1138 »

Aeolun wrote: post #93
ripper1138 wrote: post #65

Earlier quoted context omitted.

So many opportunities missed to avoid this! Look at one ID and ensure it is what you expect it to be. Run the script in a dryrun mode. Run the script for 1 customer. Probably more!

And don’t make a “universal delete” script in the first place…

After GDPR almost all companies need to have something like a “universal delete”. There are safer ways to deal with data retention policies but I can understand why such a script exists.

Re: Post-incident review on the Atlassian April 2022 outage

#155

Post by larodi »

therealcamino wrote: post #145
nreilly wrote: post #67

Earlier quoted context omitted.

From the article: "To manage the restoration progress we created a new Jira project, SITE, and a workflow to track restorations on a site-by-site basis across multiple teams (engineering, program management, support, etc). This approach empowered all teams to easily identify and track issues related to any individual site restoration."

I read "empowered" there and thought, what a ridiculous misstep. They can't even stop themselves from selling their product in a document that's supposed to be a mea culpa, where you can guarantee that many of the readers are pissed off and absolutely not receptive to your marketing.

As I said in me other down-voted comment and yours is proof for the same - these post-mortems are first of all PR and second - everything else. And lots of people pointed out why they don’t constitute invaluable knowledge - because they lack the really important stuff or substitute it for marketing messages…

Re: Post-incident review on the Atlassian April 2022 outage

#156

Post by rob_c »

kwertyoowiyop wrote: post #23
largbae wrote: post #18

Earlier quoted context omitted.

It is never that simple. Say the backup existed, and was global. By the time you get everyone briefed on how fubar it is and get agreement to load the backups, there are hours of changes from the unaffected customers that will be wiped by the restore, or have to be reconciled by hand for months. Sure, you can concoct the perfect antidote with hindsight, but their retro and next steps are sound.

That lesson really stuck out for me also. My definition of “restore” has been too simplistic.

Not really. They managed closer to "undo" the break, which still makes it look like their disaster recovery might also be un-tested.

I'm sure there are many companies who would lose 24hr of tracking to get back online in a few hr vs 14days. In the real world with deadlines and commitments this is madness.

Re: Post-incident review on the Atlassian April 2022 outage

#157

Post by johnzabroski »

giraffe_lady wrote: post #85
throwaway787544 wrote: post #80

Earlier quoted context omitted.

It should be shouted from the rooftops: don't switch services just to save money if the result is potentially worse business outcomes. Why save a tiny bit of cash if it puts your business at risk?

Presumably no one who made that decision is fucking stupid and thought they were putting their business at risk. Good lord. I wasn't affected by this in the slightest and just found out opsgenie exists from the parent comment but even I can understand that this decision would almost certainly be driven by things like "we're already using atlassian for everything else and will benefit from the interop" and "we already…

In my experience, the decision is driven by a VP who wants a bullet point on their year-end review, namely "notional cost savings achieved".
Post reply on HN