Inside the longest Atlassian outage

Post by madmulita » Wed, Apr 13, 2022, 10:37 PM UTC

kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

Post by enra » Wed, Apr 13, 2022, 10:41 PM UTC

function_seven wrote: post #467

bzxcvbn wrote: post #452

Earlier quoted context omitted.

Not every business can afford to go one month without income. What's the best thing for customers? Have the business go bankrupt and irremediably lose access to the service?

It's 400 clients, not all their user base. They can handle the lost income from a small slice of their customers for one month. And if they can't sustain that, then it's even more imperative that those customers migrate away.

Post by sizzle » Wed, Apr 13, 2022, 10:45 PM UTC

miketria wrote: post #396

bluedino wrote: post #145

Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had over almost 40 hours. At the end of the second day, it was decided to restore from backup. We had done this before, as a test. It took about 12 hours to restore the data and another 12 hours to import the data and get back up and running. One small thing was diffe…

Hi, I'm Mike and I work in Engineering at Atlassian. Here's our approach to backup and data management: https://www.atlassian.com/trust/security/data-management - we certainly have the backups and have a restore process that we keep to. However, this incident stressed our ability to do this at scale, which has led to the very long times to restore.

Post by voxic11 » Wed, Apr 13, 2022, 10:46 PM UTC

notatoad wrote: post #465

a2800276 wrote: post #424

Earlier quoted context omitted.

"permanently delete" strongly suggests to me that it was the "medical and financial data" kind of compliance. If data can be restored, it's not permanently deleted. But this was a statement from the CEO, so words can have arbitrary meaning :)

"permanently delete" does not mean the same thing as "immediately delete". deleting from the live database is the first step of a permanent deletion, as long as the data exists somewhere the deletion process is still in-progress. there's a whole lot of people in here who are way too quick to assume that just because one part of a permanent deletion process was inadvertently triggered and then caught while they still…

Post by mypalmike » Wed, Apr 13, 2022, 10:53 PM UTC

seanwilson wrote: post #353

inopinatus wrote: post #290

Earlier quoted context omitted.

* makes it much harder to distribute your tables by any other factor, for whatever reason (usually performance, sometimes archival) * disaggregates data that the SaaS might be interested in querying/updating as an aggregate * not all ORM frameworks handle this case well, if at all * dumps are more than a single trivial command basically all your data operations gain an additional dimension of complexity, and you may…

Would it be fair to estimate that the majority of SaaS companies aren't sharding like this then? Seems like a lot of downsides that impact everything often except for backups, which you'd restore rarely.