Live data from Hacker News

Post-incident review on the Atlassian April 2022 outage

atlassian.com

11–20 of 157 posts

Re: Post-incident review on the Atlassian April 2022 outage

#11

I can only imagine the way the person who pulled the trigger on the deletion script felt the moment they realized what had happened. I’ve been there with much less significant incidents when a “routine” change turned into a potentially resume generating event. It’s not fun. Ultimately the responsibility is with the organization that made it possible for an event of that scale to happen rather than the individual pers…

That moment when you see DELETE 18388272773 0

Re: Post-incident review on the Atlassian April 2022 outage

#12
post #8

Soft deletion always feels at odds with privacy-related "right to have data deleted" laws. Would be super interested in a technical writeup on how they do this.

"Right to have data deleted" can be 'circumvented' if the data is critical part of the system or is needed for legal purpose (for example it can be mandatory to keep 1 year of IP logs and data associated with it)

In previous companies I have worked for, we did instant soft-delete, then hard anonymisation after 15-30days and then hard delete after a year. That means the data was not recoverable for customer but could still be recovered for legal purpose.

Re: Post-incident review on the Atlassian April 2022 outage

#13

I can only imagine the way the person who pulled the trigger on the deletion script felt the moment they realized what had happened. I’ve been there with much less significant incidents when a “routine” change turned into a potentially resume generating event. It’s not fun. Ultimately the responsibility is with the organization that made it possible for an event of that scale to happen rather than the individual pers…

Thomas J. Watson famously said:

Recently, I was asked if I was going to fire an employee who made a mistake that cost the company $600,000. No, I replied, I just spent $600,000 training him. Why would I want somebody to hire his experience?

Re: Post-incident review on the Atlassian April 2022 outage

#15
Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just your own fault if you're not extra careful, compared to say using Git with code or using "restore from trash" with filesystems.

Re: Post-incident review on the Atlassian April 2022 outage

#16

I can only imagine the way the person who pulled the trigger on the deletion script felt the moment they realized what had happened. I’ve been there with much less significant incidents when a “routine” change turned into a potentially resume generating event. It’s not fun. Ultimately the responsibility is with the organization that made it possible for an event of that scale to happen rather than the individual pers…

I think in this instance it’s the person who sent the IDs that feels worse. The deleters were provided IDs that had 30 correct app IDs and the rest were site IDs.

Like they mentioned they had a delete script that worked for all types of unique IDs so that also can dilute the feeling of ”it’s all my fault” hopefully.

Re: Post-incident review on the Atlassian April 2022 outage

#17

Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just you…

Snowflake has time travel[1] keeping the original data for specified period of time.

1: https://docs.snowflake.com/en/user-guide/data-availability.h...

Re: Post-incident review on the Atlassian April 2022 outage

#18
post #14

Seriously, where were the -24hr backups that could be rolled back to once its clear the script is fubar, or using it on just 10% of the estate first? ...

It is never that simple. Say the backup existed, and was global. By the time you get everyone briefed on how fubar it is and get agreement to load the backups, there are hours of changes from the unaffected customers that will be wiped by the restore, or have to be reconciled by hand for months. Sure, you can concoct the perfect antidote with hindsight, but their retro and next steps are sound.

Re: Post-incident review on the Atlassian April 2022 outage

#19

Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just you…

Quite a few databases support time travel queries, in particular Oracle has for years and CockroachDB has them also. We can query the state of a table as it was at any point in the last 72hrs.
Post reply on HN