Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Post-incident review on the Atlassian April 2022 outage

atlassian.com

Post reply on HN
Print view View original

121–130 of 157 posts

Re: Post-incident review on the Atlassian April 2022 outage

#121

Post by helmholtz »

Griffinsauce wrote: post #116
helmholtz wrote: post #94

Earlier quoted context omitted.

I'm curious about your take on the cheapening or dilution of words in language nowadays. One isnt hurt but traumatized, one isn't "once bitten twice shy" but suffering from PTSD. It's assault if one gets in a fight and so on. I sometimes feel we're running out of language to distinguish the truly extreme from the quotidian. Same thing with how GP phrased it. A hard week of work comes across as having gone to battle a…

> nowadays Any argument that uses this qualification should be accompanied with data. You haven't specified anything, not even a rough time frame. Your statement is so vague nearly anything can be projected onto it. Also a sidenote: if you're going to make an argument about language and definitions it seems like a good idea to actually know what PTSD is before using it as an example.

My comment was vague because my feelings on the topic itself are vague. I'm not married to the position and was only trying to spark a line of thought.

That said, nowadays would be social media mass adoption. Circa 2000 for the sake of thought experiment.

Also, what is the definition of PTSD and how did I use it incorrectly? For context, I once dropped a heavy duty suitcase on my foot when the handle gave. Now I always think picture the worst when carrying it. Is that PTSD? I would suggest not, because the extent of the trauma is much lower than a war veteran. That was what I was trying to get at.

Re: Post-incident review on the Atlassian April 2022 outage

#122

Post by AlfeG »

martin_drapeau wrote: post #63

Implementing soft deletes is a lessons every developer learns early in their career. The fact that Atlassian did not implement that in their cloud, is mind boggling. Great case study of a monumental fuck up!

For large systems adding Soft Delete would be more a PR move, then actual work. In some projects I know implementing soft delete is as hard as writing project from scratch. Having scripts and manual restoration for particular records is the way those businesses communicate with customers.

Re: Post-incident review on the Atlassian April 2022 outage

#123

Post by kgeist »

BFLpL0QNek wrote: post #82
yen223 wrote: post #73

Earlier quoted context omitted.

Soft deletes are great, but are probably not sufficient to meet the GDPR's "right to be forgotten"

You can do a soft delete, setting a flag (expire date), have apps ignore records with an expire date then a scheduled job to delete all items with the expire date set with a value less than current time. Alternatively move the deleted data to a temporary location and then delete the temporary location after a short period of time. Or better combine both patterns where expired rows get moved to a temporary location be…

That's exactly what we do: a soft delete followed by a scheduled hard delete 7 days later. The customer is also notified that they won't be able to restore their account after that. Soft deletes are usually automatic and well-tested (for example, a subscription expired) but sometimes there are requests by management or customers to delete accounts manually and this approach really helped avoid disastrous situations like the one at Atlassian because you have a whole week to realize you deleted the wrong accounts (customers will most likely complain much sooner).

Yes, GDPR has a grace period of 30 days or so, it's never been a problem in practice.

Re: Post-incident review on the Atlassian April 2022 outage

#124

Post by jpalomaki »

dotopotoro wrote: post #54
eastbound wrote: post #32

Earlier quoted context omitted.

Note that the ToS also forbid Cloud users from “disseminating information” about “the performance of the products”. So you can’t say it’s slow or unperforming.

Is that real? That is so bad. Queue up next “Shall not disclose or notice our incompetence” clause.

I believe these clauses are usually there to prevent competitors from using (sometimes misleading) benchmarks in their advertising.

Used to be common to see this kind of comparisons between databased for example.

Re: Post-incident review on the Atlassian April 2022 outage

#125

Post by Too »

wolf550e wrote: post #28
seanwilson wrote: post #15

Do any databases have something like native support for soft deletes or ability to undo (other than SQL transactions rollbacks where you're having to specify the undo checkpoint)? Something like what Git does where it keeps a history of edits? If this isn't common, is this a neglected area that should be addressed or it's just too hard of a problem? It feels like with SQL, there's minimal guardrails and it's just you…

In MVCC systems like PostgreSQL, if you don't vacuum (garbage collect the old tuples), your database is append-only and you can query as if your transaction was started at some time in the past. I don't know how to set auto-vacuum to have a fixed delay, e.g. keeping 24h of changes, but I bet it can be added if it's not built-in.

This would just be equivalent to rollback to a checkpoint in time of the whole table. I think the question was more on a row-level.

If you are just interested in global time traveling, there are many solutions, such as replaying the oplog from snapshots in time or delayed replication.

Re: Post-incident review on the Atlassian April 2022 outage

#126

Post by eyelidlessness »

helmholtz wrote: post #121
Griffinsauce wrote: post #116

Earlier quoted context omitted.

> nowadays Any argument that uses this qualification should be accompanied with data. You haven't specified anything, not even a rough time frame. Your statement is so vague nearly anything can be projected onto it. Also a sidenote: if you're going to make an argument about language and definitions it seems like a good idea to actually know what PTSD is before using it as an example.

My comment was vague because my feelings on the topic itself are vague. I'm not married to the position and was only trying to spark a line of thought. That said, nowadays would be social media mass adoption. Circa 2000 for the sake of thought experiment. Also, what is the definition of PTSD and how did I use it incorrectly? For context, I once dropped a heavy duty suitcase on my foot when the handle gave. Now I alwa…

> Is that PTSD?

No one here who’s qualified to answer that is going to give you a definitive answer here. At best you’ll get pattern recognition from either patients or practitioners nudging you towards consulting a professional.

> I would suggest not, because the extent of the trauma is much lower than a war veteran. That was what I was trying to get at.

You probably shouldn’t speculate about PTSD. You seem to have preconceived notions about what qualifies that aren’t consistent with actual people who experience it. I won’t speak for the people in my life who do, but few of them have ever been to war.

If you want to know more I sincerely encourage you to speak to a professional. They’ll have much better insight than any questions on HN.

Re: Post-incident review on the Atlassian April 2022 outage

#127

Post by Too »

pianoben wrote: post #108
jsmeaton wrote: post #77

Earlier quoted context omitted.

This was addressed in the write up (it’s very long, so missing it is easy). They ran the script against 30 accounts first to verify it worked, and it did, because the list of 30 ids they tested against came from a different source than the other 750ish. It’s a shitty mistake to make but I’m certain I’ve made similar ones.

One of the favorite tricks tricks I've ever seen is how Twilio uses human-readable prefixes[0] on their various identifiers - you will never mistake a device (HSxxxxxxx) for an account (ACxxxxxxxx). It's prevented us (Twilio customer) from making similar mistakes in the past. [0]: https://www.twilio.com/docs/glossary/what-is-a-sid#common-si...

I like the idea of human readable identifiers. But generally feels like this class of error could be prevented with more type safety in the api and data model? Like deleteDevice(123) and deleteAccount(123), rather than delete(123). This is how REST is designed, the type of resource is already baked into the url.

Re: Post-incident review on the Atlassian April 2022 outage

#128

Post by doctor_eval »

Too wrote: post #125
wolf550e wrote: post #28

Earlier quoted context omitted.

In MVCC systems like PostgreSQL, if you don't vacuum (garbage collect the old tuples), your database is append-only and you can query as if your transaction was started at some time in the past. I don't know how to set auto-vacuum to have a fixed delay, e.g. keeping 24h of changes, but I bet it can be added if it's not built-in.

This would just be equivalent to rollback to a checkpoint in time of the whole table. I think the question was more on a row-level. If you are just interested in global time traveling, there are many solutions, such as replaying the oplog from snapshots in time or delayed replication.

Maybe I'm misunderstanding but if you know the txid at the time you're looking for then you can find the value of any specific row at any point in time using xmin and xmax at the row-level (if you're not running vacuum, as the parent suggested).

Am I mistaken?

The only problem is that you need to keep a map of timestamps to txids so you can find the txid that was valid at a particular moment in time. This doesn't sound to me like a significantly difficult problem, but maybe I'm mistaken. That said, it's not like you need super high time resolution for the use case in question.

Re: Post-incident review on the Atlassian April 2022 outage

#129

Post by eyelidlessness »

Aeolun wrote: post #91
nomilk wrote: post #70

Earlier quoted context omitted.

As someone who's worked on similar situations, I don't expect any pat on the back, but see it as my responsibility to make sure it doesn't happen in the first place, and when it does, my responsibility to fix it without complaint. Some Atlassian customers might have had much more severe (mental health and other) problems than the Atlassian staff, so I think it can be perceived as mildly solipsistic to be praising the…

I find it hard to imagine mental health being affected negatively by atlassian services disappearing.

I find it exceedingly hard to imagine how it wouldn’t. Think of how people use their products, and the cascading effect of that use being disrupted with long term interrupted access to information that was probably stored only there. Think of whole organizations scrambling to recreate knowledge that’s only partially in their heads.

Re: Post-incident review on the Atlassian April 2022 outage

#130

Post by aidos »

doctor_eval wrote: post #128
Too wrote: post #125

Earlier quoted context omitted.

This would just be equivalent to rollback to a checkpoint in time of the whole table. I think the question was more on a row-level. If you are just interested in global time traveling, there are many solutions, such as replaying the oplog from snapshots in time or delayed replication.

Maybe I'm misunderstanding but if you know the txid at the time you're looking for then you can find the value of any specific row at any point in time using xmin and xmax at the row-level (if you're not running vacuum, as the parent suggested). Am I mistaken? The only problem is that you need to keep a map of timestamps to txids so you can find the txid that was valid at a particular moment in time. This doesn't sou…

Lots of people have updated_at time stamps on all tables, so you could probably inspect those to find your way back. I’ve never tried to query the history implicitly hidden in Postgres tables so I’m not sure how possible (or sensible) any of this is.
Post reply on HN