Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

621–630 of 772 posts

Re: Inside the longest Atlassian outage

#621

Earlier quoted context omitted.

How do you implement per tenant backups? Not every db system cleanly separates where each tenant’s data is stored.

I don't have a ton of experience doing this, but with a lot of multitenant you just give each tenant their own database in the first place. It solves multiple classes of problems. On a single Postgres instance you can (at least theoretically) have 4 billion databases per instance.

Most of the multi-tenant SaaS products I've worked with do NOT have per tenant databases. I'm sure some do, but the bulk of the multi-tenant products use one (or several) larger databases.

Re: Inside the longest Atlassian outage

#622

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

> Actually reading a bit more, it seems like their customer team was partying in Las Vegas instead of taking care of business: https://www.linkedin.com/mwlite/feed/hashtag/atlassianteam22 This was a conference they hosted , not just some Atlassian team members partying in Vegas: https://events.atlassian.com/team22 Thousands of Atlassian customers bought tickets, flights, and hotels for the event. It's unreasonable to…

Fair enough, but anyone who’s been around the block on crisis communications knows that you’re supposed to hit the big red “stop all regular communications until we get a fucking handle on this” button.

Re: Inside the longest Atlassian outage

#623
post #606

Earlier quoted context omitted.

Unless you know, with certainty, who is responsible for customer-facing communications , how can you publicly call on a specific person to be fired? Even if you did know, and are correct in your implication, how do you know that she didn't want to do the right thing but was overruled by people above her in the chain? Overall, this is a pretty irresponsible comment, and reflects the kind of mob mentality that I would…

Completely agree. Can we please keep the pitchforks on Reddit. It's so incredibly easily to point the finger when you really have no idea. I'm all for calling out their lack of communication, but picking someone you deem to be responsible and calling for their head is pretty medieval.

Why do you think "pitchforks" are on or should be on Reddit? That's a very odd thing to say on Hacker News. And if you don't like Reddit, just don't go there!

Re: Inside the longest Atlassian outage

#624

Earlier quoted context omitted.

Good faith would be to lose all of that money to people who are already your customers. Business-wise would be to stay in their good graces and keep those customers by offering the refund, but you don't lose any money to those who either don't care or won't move to a competitor.

25 years ago the clutch in my beater truck was slipping. I was 16 years old, making $50 a week and had very little in savings. I took that truck to a shop within walking distance of my job. 2 hours later I walked back to see what they found. I figured it would be several hundred dollars for a new clutch, and I'd have to borrow money or something to get it done. I talked to the owner who told be it was an adjustment o…

Your anecdote is nice, and sure it can be good advertising to give stuff away for free, But it doesn't really apply here.

If you were charged $123.98 and you said, "hey, I told you where the problem was, why am I being charged a diagnostics and driving fee?" and they corrected it by telling you the whole thing is on the house, is that not good business sense?

Even by your own admission, you would have gladly paid that $123.98 with no issues and you wouldn't have been mad about it. So from a business perspective, if they can provide a service, get paid for it, and the customer has no qualms or issues with the transaction whatsoever, in what way is that hurting the brand or being cynical? I think that's a much more business-wise action to take than to give away your services.

Re: Inside the longest Atlassian outage

#625
post #450

Earlier quoted context omitted.

Reducing the impact analysis within a long running relationship to a single transaction is too narrow. People observe how other people are treated and draw their conclusions even if not impacted. People may tolerate some abuse but it moves them closer to leaving next time. Money lost in the outage may provide for a budget creation to look for an alternative.

A lot of people making those decisions don’t care about a refund because it’s other people’s money anyway. In my experience only small companies care about that. Focussing on communicating open and honestly allows them to explain the crap they’re going through because of your mistakes to their bosses, so in fact you can help them save their asses, and they’ll save your ass in return. This is much more important and v…

I've seen cases where it was actually _more_ work for a business to process a refund. That money has to go all the way back through accounting/financing, be re-added to budgets for the appropriate groups, etc. It's not something done all the time so it takes extra time for those working on it. It's not like a Visa credit card getting a refund for a wrong coffee order.

Re: Inside the longest Atlassian outage

#626

A few years ago we didn't renew our subscription on time because we got the email over Christmas break, and iirc they deleted all of our data in less than two weeks. They were eventually able to manually restore it from backups, but they restored it incorrectly so there was a bunch of stuff broken. This whole thing isn't even remotely surprising to me.

You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons . The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for app…

Generally user data deletion happens in multiple phases for large companies that care about both compliance and user experience.

For example, if you delete an email or document on Google it moves to the "Trash" folder for 30 days.

When you manually empty the trash or the time window expires, most likely the next step would be a soft deletion for a few days where the data is still on hard drives but hidden from the application. Soft deletion is mainly protection against coding errors, since soft deletion is easy to undo if you've caused an incident but hard deletion (removing the data from disk) is not.

Then most likely a garbage collection process comes by a few days later and hard deletes the data from disk, leaving it only on tape backups

Finally, maybe a month or two later it disappears from the tape backups as they get rotated or otherwise disposed of

This addresses the needs of:

- Giving a good user experience (user "oops I made a mistake" undelete)

- Protecting against incidents due to coding errors (software engineer "oops I made a mistake" undelete)

- Making sure data disappears from both disk and backups within a certain time window, like maybe 30 or 60 days (comply with regulation and user expectations of data being cleared)

Re: Inside the longest Atlassian outage

#627
post #626

Earlier quoted context omitted.

You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons . The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for app…

Generally user data deletion happens in multiple phases for large companies that care about both compliance and user experience. For example, if you delete an email or document on Google it moves to the "Trash" folder for 30 days. When you manually empty the trash or the time window expires, most likely the next step would be a soft deletion for a few days where the data is still on hard drives but hidden from the ap…

Another option for clearing tape backups is to throw away the encryption keys, as https://www.youtube.com/watch?v=ejBncCrlAqc mentions

Re: Inside the longest Atlassian outage

#628
post #411

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…

> Atlassian - MUST DO BETTER.

it’s not like people will stop using jira and confluence, lol

they basically have a monopoly there

Re: Inside the longest Atlassian outage

#629

Earlier quoted context omitted.

Sure human failure is probabilistic. But you can design around that by stacking reliability-enhancing approaches together. Let’s say there’s a 10% chance of any given feature being broken. Write a test, (which has another 10% chance of being broken) and now it’s only broken if the test and the code are broken, and broken in the same way. So we’re down to Then add a backup / redundancy system. That has a 10% chance of…

They had backup restore process. The trouble was the restore would set back everyone’s data to that point in time, whereas only some customers data was impacted.

I wonder if in retrospect that would have been better. If they had rolled back to a snapshot 30 minutes after they realized they had a problem, everyone loses 30 minutes of updates (and maybe transaction logs can be copied before the rollback and then replayed to reduce that to even less). Everyone experiences a little bit of pain instead of some customers being down for a week plus. Easy to speculate about from the cheap seats though.

Re: Inside the longest Atlassian outage

#630

Wow would you look at that, a complete Atlassian puff piece got published in the WSJ just hours ago. How peculiar that the biggest active outage in the history of this company is not mentioned once in this "article". I'm left to assume that PR teams can plant whatever they see fit in the WSJ at a moment's notice. I guess that's what passes for journalism these days. https://www.wsj.com/articles/atlassian-puts-easy-to…

> Whatever else changes for CIOs, troubleshooting may never be far behind. Ms. Rao’s remarks came as Atlassian races to restore cloud-based software applications to roughly 400 companies, after a service outage last week caused by a routine maintenance glitch.

> As of Wednesday, she said, services were back online for just under half of the companies hit by the outage, which may take up to two weeks to fully repair.

Source: https://www.wsj.com/articles/atlassian-puts-easy-to-use-codi...

Post reply on HN