Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-incident review on the Atlassian April 2022 outage

atlassian.com

Post reply on HN
Print view View original

101–110 of 157 posts

Re: Post-incident review on the Atlassian April 2022 outage

#101

Post by max-hoffman »

seanwilson wrote: post #15

Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just you…

Dolt is a SQL database that implements Git primitives at the storage layer, including commit, diff, and revert. We wrote a blog the other week summarizing how these features can serve as guardrails:

https://www.dolthub.com/blog/2022-04-14-atlassian-outage-pre...

Re: Post-incident review on the Atlassian April 2022 outage

#102

Post by doubled112 »

dotopotoro wrote: post #54
eastbound wrote: post #32

Earlier quoted context omitted.

Note that the ToS also forbid Cloud users from “disseminating information” about “the performance of the products”. So you can’t say it’s slow or unperforming.

Is that real? That is so bad. Queue up next “Shall not disclose or notice our incompetence” clause.

https://www.atlassian.com/legal/cloud-terms-of-service

> 3.3. Restrictions. Except as otherwise expressly permitted in these Terms, you will not:

> (i) publicly disseminate information regarding the performance of the Cloud Products; or (j) encourage or assist any third party to do any of the foregoing.

Re: Post-incident review on the Atlassian April 2022 outage

#103

Post by na85 »

Aeolun wrote: post #90
gregdoesit wrote: post #75

While the post-mortem is thorough, it misses key details on what companies experienced who were unlucky enough to be caught out by this outage. For example, it fails to mention how impacted customers lost access to certain Atlassian services for up to ~2 weeks: JIRA, Confluence, OpsGenie. But not others like Trello or BitBucket. Of these, losing access to OpsGenie for this long was a massive problem, dwarfing most ot…

But you don’t need a paging system if your services don’t go down. Isn’t the fact you can’t deal without one for two weeks an indictment of your own practices?

Sure and you don't need emergency locator transmitters if your aircraft doesn't crash.

When you're ready to prove that your services "don't go down" send me an email and I'll come work for you.

Re: Post-incident review on the Atlassian April 2022 outage

#104

Post by bradleyjg »

perlgeek wrote: post #21
justinludwig wrote: post #5

I can't say that I've ever been a fan of Atlassian or their products, but this blog post makes it sounds like they've at least learned the right lessons from this: 1. Establish universal "soft deletes" across all systems. 2. Better DR for multi-site, multi-product incidents. 3. Fix their incident-management process for large-scale incidents. 4. Fix their incident communications. Regarding #4 in particular: "Rather th…

I tend to agree. However, there's one point that makes me skeptical: there are no organizational changes, or changes to leadership, or anything in that direction. This sounds like "the tech guys screwed up, culture and management is fine here". Which it might be, or it might not. I would have loved to see 5. We will stop pushing customers so hard towards using our cloud for one, but that wouldn't be convenient for At…

I wouldn’t expect the larger question of what organizational and management issues lead to the problem in the larger sense to be made public. At least not while the issue is still fresh. Maybe down the road as a business school case study after some of the players retire.

Re: Post-incident review on the Atlassian April 2022 outage

#106

Post by thayne »

aunty_helen wrote: post #81
yen223 wrote: post #73

Earlier quoted context omitted.

Soft deletes are great, but are probably not sufficient to meet the GDPR's "right to be forgotten"

Does this apply for B2B companies like Atlassian?

Yes. As I understand it, GDPR does not care if customer data belongs to a business or an individual. And even if it did, business data will likely include PII for employees, which would need to be deleted.

GDPR does give you a grace period , so you can soft delete immediately and then hard delete after some period shorter than the GDPR deadline. However, actually implenting such a system can be rather difficult and potentially expensive.

Re: Post-incident review on the Atlassian April 2022 outage

#107

Post by thayne »

lolinder wrote: post #74
yen223 wrote: post #73

Earlier quoted context omitted.

Soft deletes are great, but are probably not sufficient to meet the GDPR's "right to be forgotten"

There's no reason you can't have both. Use soft deletes for everything except for a formal GDPR right to be forgotten request (or any other compliance situation).

> except for a formal GDPR right to be forgotten request (or any other compliance situation)

That "any other compliance situation" includes things you probably want to have a soft delete for, like deleting/closing an account. Customers accidentally deleting their accounts, then wanting it restored happens more frequently than one would hope.

Re: Post-incident review on the Atlassian April 2022 outage

#108

Post by pianoben »

jsmeaton wrote: post #77
ripper1138 wrote: post #65

Earlier quoted context omitted.

So many opportunities missed to avoid this! Look at one ID and ensure it is what you expect it to be. Run the script in a dryrun mode. Run the script for 1 customer. Probably more!

This was addressed in the write up (it’s very long, so missing it is easy). They ran the script against 30 accounts first to verify it worked, and it did, because the list of 30 ids they tested against came from a different source than the other 750ish. It’s a shitty mistake to make but I’m certain I’ve made similar ones.

One of the favorite tricks tricks I've ever seen is how Twilio uses human-readable prefixes[0] on their various identifiers - you will never mistake a device (HSxxxxxxx) for an account (ACxxxxxxxx). It's prevented us (Twilio customer) from making similar mistakes in the past.

[0]: https://www.twilio.com/docs/glossary/what-is-a-sid#common-si...

Re: Post-incident review on the Atlassian April 2022 outage

#109

Post by cryptonector »

mouly wrote: post #84

> Prior to this incident, our cloud has consistently delivered 99.9% uptime and exceeded uptime SLAs. But it had the risk of catastrophic failure the whole time. I wish we had better ways to measure and communicate risk.

That risk is always there.

Re: Post-incident review on the Atlassian April 2022 outage

#110

Post by cryptonector »

wolf550e wrote: post #28
seanwilson wrote: post #15

Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just you…

In MVCC systems like PostgreSQL, if you don't vacuum (garbage collect the old tuples), your database is append-only and you can query as if your transaction was started at some time in the past. I don't know how to set auto-vacuum to have a fixed delay, e.g. keeping 24h of changes, but I bet it can be added if it's not built-in.

How do you set the time back for a query? How do you specify what txid to use for a point in time query?
Post reply on HN