Earlier quoted context omitted.
How do you implement per tenant backups? Not every db system cleanly separates where each tenant’s data is stored.
I don't have a ton of experience doing this, but with a lot of multitenant you just give each tenant their own database in the first place. It solves multiple classes of problems. On a single Postgres instance you can (at least theoretically) have 4 billion databases per instance.
Inside the longest Atlassian outage
621–630 of 772 posts
Re: Inside the longest Atlassian outage
#622Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…
> Actually reading a bit more, it seems like their customer team was partying in Las Vegas instead of taking care of business: https://www.linkedin.com/mwlite/feed/hashtag/atlassianteam22 This was a conference they hosted , not just some Atlassian team members partying in Vegas: https://events.atlassian.com/team22 Thousands of Atlassian customers bought tickets, flights, and hotels for the event. It's unreasonable to…
Re: Inside the longest Atlassian outage
#623Earlier quoted context omitted.
Unless you know, with certainty, who is responsible for customer-facing communications , how can you publicly call on a specific person to be fired? Even if you did know, and are correct in your implication, how do you know that she didn't want to do the right thing but was overruled by people above her in the chain? Overall, this is a pretty irresponsible comment, and reflects the kind of mob mentality that I would…
Completely agree. Can we please keep the pitchforks on Reddit. It's so incredibly easily to point the finger when you really have no idea. I'm all for calling out their lack of communication, but picking someone you deem to be responsible and calling for their head is pretty medieval.
Re: Inside the longest Atlassian outage
#624Earlier quoted context omitted.
Good faith would be to lose all of that money to people who are already your customers. Business-wise would be to stay in their good graces and keep those customers by offering the refund, but you don't lose any money to those who either don't care or won't move to a competitor.
25 years ago the clutch in my beater truck was slipping. I was 16 years old, making $50 a week and had very little in savings. I took that truck to a shop within walking distance of my job. 2 hours later I walked back to see what they found. I figured it would be several hundred dollars for a new clutch, and I'd have to borrow money or something to get it done. I talked to the owner who told be it was an adjustment o…
If you were charged $123.98 and you said, "hey, I told you where the problem was, why am I being charged a diagnostics and driving fee?" and they corrected it by telling you the whole thing is on the house, is that not good business sense?
Even by your own admission, you would have gladly paid that $123.98 with no issues and you wouldn't have been mad about it. So from a business perspective, if they can provide a service, get paid for it, and the customer has no qualms or issues with the transaction whatsoever, in what way is that hurting the brand or being cynical? I think that's a much more business-wise action to take than to give away your services.
Re: Inside the longest Atlassian outage
#625Earlier quoted context omitted.
Reducing the impact analysis within a long running relationship to a single transaction is too narrow. People observe how other people are treated and draw their conclusions even if not impacted. People may tolerate some abuse but it moves them closer to leaving next time. Money lost in the outage may provide for a budget creation to look for an alternative.
A lot of people making those decisions don’t care about a refund because it’s other people’s money anyway. In my experience only small companies care about that. Focussing on communicating open and honestly allows them to explain the crap they’re going through because of your mistakes to their bosses, so in fact you can help them save their asses, and they’ll save your ass in return. This is much more important and v…
Re: Inside the longest Atlassian outage
#626A few years ago we didn't renew our subscription on time because we got the email over Christmas break, and iirc they deleted all of our data in less than two weeks. They were eventually able to manually restore it from backups, but they restored it incorrectly so there was a bunch of stuff broken. This whole thing isn't even remotely surprising to me.
You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons . The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for app…
For example, if you delete an email or document on Google it moves to the "Trash" folder for 30 days.
When you manually empty the trash or the time window expires, most likely the next step would be a soft deletion for a few days where the data is still on hard drives but hidden from the application. Soft deletion is mainly protection against coding errors, since soft deletion is easy to undo if you've caused an incident but hard deletion (removing the data from disk) is not.
Then most likely a garbage collection process comes by a few days later and hard deletes the data from disk, leaving it only on tape backups
Finally, maybe a month or two later it disappears from the tape backups as they get rotated or otherwise disposed of
This addresses the needs of:
- Giving a good user experience (user "oops I made a mistake" undelete)
- Protecting against incidents due to coding errors (software engineer "oops I made a mistake" undelete)
- Making sure data disappears from both disk and backups within a certain time window, like maybe 30 or 60 days (comply with regulation and user expectations of data being cleared)
Re: Inside the longest Atlassian outage
#627Earlier quoted context omitted.
You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons . The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for app…
Generally user data deletion happens in multiple phases for large companies that care about both compliance and user experience. For example, if you delete an email or document on Google it moves to the "Trash" folder for 30 days. When you manually empty the trash or the time window expires, most likely the next step would be a soft deletion for a few days where the data is still on hard drives but hidden from the ap…
Re: Inside the longest Atlassian outage
#628This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…
it’s not like people will stop using jira and confluence, lol
they basically have a monopoly there
Re: Inside the longest Atlassian outage
#629Earlier quoted context omitted.
Sure human failure is probabilistic. But you can design around that by stacking reliability-enhancing approaches together. Let’s say there’s a 10% chance of any given feature being broken. Write a test, (which has another 10% chance of being broken) and now it’s only broken if the test and the code are broken, and broken in the same way. So we’re down to Then add a backup / redundancy system. That has a 10% chance of…
They had backup restore process. The trouble was the restore would set back everyone’s data to that point in time, whereas only some customers data was impacted.
Re: Inside the longest Atlassian outage
#630Wow would you look at that, a complete Atlassian puff piece got published in the WSJ just hours ago. How peculiar that the biggest active outage in the history of this company is not mentioned once in this "article". I'm left to assume that PR teams can plant whatever they see fit in the WSJ at a moment's notice. I guess that's what passes for journalism these days. https://www.wsj.com/articles/atlassian-puts-easy-to…
> As of Wednesday, she said, services were back online for just under half of the companies hit by the outage, which may take up to two weeks to fully repair.
Source: https://www.wsj.com/articles/atlassian-puts-easy-to-use-codi...