Selectively restoring data only for certain rows is super hard. But the communications by Atlassian has been the worst I have ever seen in the industry.
As someone who has never had to perform this kind of recovery: why is it so hard?
Inside the longest Atlassian outage
121–130 of 772 posts
Re: Inside the longest Atlassian outage
#122> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…
Personally, given the multi-day outage, I think I would just restore everything to a separate system, and then only point the affected customers to this new system. Take the hit of having two fully separated system initially, and work on reconciling them after without having to worry about the on-going outage.
I wonder if they're not doing this due to some tech limitations, to avoid taking the financial cost of running two systems, or to avoid having to reconcile the systems.
Re: Inside the longest Atlassian outage
#123i hate deleting things. prefer flags that hide things instead (like a boolean deleted flag in an rdbms table). prevents data integrity issues in relational databases, makes debugging easier and prevents disasters. ideally also include a timestamp, both for bookkeeping and safe tools that only remove things that have been soft deleted for some time and are safe to delete without compromising integrity of anything that…
The GDPR and various things have made companies more skittish in doing things this way, because they get scared. Perhaps an effective measure would be to create a key that encrypts a customer's data, and give them a copy of the key, and let them know that after a certain point your copy of the key will be deleted, and if they want a restore past that point they'll need to provide the key.
that's an interesting question, i've given a little thought to this multi tenant saas stuff...
not sure if the right way forward is some sort of innovation in operating system and software design where people write and run apps that feel like single tenant apps attached to dedicated per tenant datastores where os and framework magic handle per tenant encryption and segmentation (tenant id as an os level concept)
or... if it makes more sense to encrypt at the record level with keys that only the customers hold using (assuming it's up to the task) homomorphic encryption for things like searches and other backend functions.
either way, for now, soft deleting and following up with an automatic daily hard delete of things soft deleted more than x days ago is a totally reasonable approach.
ops scripts should require typing "yes i know what i'm doing" if someone attempts to hard delete things that have not yet been soft deleted.
Re: Inside the longest Atlassian outage
#124A few years ago we didn't renew our subscription on time because we got the email over Christmas break, and iirc they deleted all of our data in less than two weeks. They were eventually able to manually restore it from backups, but they restored it incorrectly so there was a bunch of stuff broken. This whole thing isn't even remotely surprising to me.
Did you continue as their customer after that?
Re: Inside the longest Atlassian outage
#125> it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.
> Are SLAs even real? Tommy: Here's the way I see it, Ted. Guy puts a fancy guarantee on a box 'cause he wants you to fell all warm and toasty inside. Ted Nelson: Yeah, makes a man feel good. Ted Nelson: But why do they put a guarantee on the box? Tommy: Because they know all they sold ya was a guaranteed piece of shit. That's all it is, isn't it? Hey, if you want me to take a dump in a box and mark it guaranteed, I…
Haha I needed this, thank you.
Re: Inside the longest Atlassian outage
#126Selectively restoring data only for certain rows is super hard. But the communications by Atlassian has been the worst I have ever seen in the industry.
It is, but between 'hard' and 'impossible' there is the nagging question of whether you actually really still have that data.
Re: Inside the longest Atlassian outage
#127Earlier quoted context omitted.
Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?
I have built multiple multi-tenancy platforms and I never create separate databases for each customer. If you have separate databases, it's almost impossible to run meaningful queries across all of them. That architectural choice creates far more headaches than it solves. Usually people end up with the split-database architecture when they want a quick retrofit for a system that wasn't designed with multiple tenants.…
It's likely a mixture of all these factors, the brain drain could absolutely be responsible.
At least it would not be the first time in history that a company has lost the engineering spirit. And instead the business people have taken over, so that details like disaster plans become less of a priority.
A business person and an engineer will always view risk differently, better disaster plans is a kind of insurance that is a lot harder to sell when too many business people run the company.
Re: Inside the longest Atlassian outage
#128What blow's my mind is that Atlassian stock has barely taken a hit...
It was a long time ago individual stocks represented anything grounded in reality. People talk about "fundamentals" and so on, but that's not what the price is based on. I don't think anyone know why the prices move as they do anymore, as there are so many algorithms involved today, both manual and automatic ones.
Re: Inside the longest Atlassian outage
#129Selectively restoring data only for certain rows is super hard. But the communications by Atlassian has been the worst I have ever seen in the industry.
As someone who has never had to perform this kind of recovery: why is it so hard?
Because it is very difficult to maintain relational integrity during a restore like that.
Re: Inside the longest Atlassian outage
#130so this is the end of Atlassian as a company right?
Why would it ? On our end everything works fine. If you’re not one of the 400 companies, there's no difference
Even those 400, especially Jira is crazy popular with a lot of scrum masters and the scrum crowd in general. I could see some of those 400 stick with Jira even after this shit show if only to avoid losing all their scrum masters.