Earlier quoted context omitted.
> May be Microsoft, Is there a Jira replacement/offering in the Microsoft 365 suite?
Azure DevOps is really underrated: https://www.thoughtworks.com/radar/platforms/azure-devops
Inside the longest Atlassian outage
591–600 of 772 posts
Re: Inside the longest Atlassian outage
#592Earlier quoted context omitted.
They are literally the head of "customer success". The buck should probably not stop ONLY with them, but obviously they're one of the people it needs to stop with
You seem to be implying that customer success is a customer support role. It isn't. Customer success is about helping customers get the greatest business value from your product. They are not going to be munging databases and wrangling backups. Support engineers (among others) do that work.
Re: Inside the longest Atlassian outage
#593Earlier quoted context omitted.
The deletion of customer data was engineering mistake, that is not what I was talking about The Negative fall out was not due to the deletion of customer data, as the Story and multiple customers have stated the negative fall out was the SILENCE / lack of communications, which is Sales / Customer Service not engineering As the comment I was replying to noted while engineering was trying to recover from what might pos…
Ok I agree with these failures, but don't you think that its a PR people problem? Perhaps executives and upper management? Sales people are just doing what they are supposed to do. Sell Atlassian products.
Re: Inside the longest Atlassian outage
#594Earlier quoted context omitted.
They are literally the head of "customer success". The buck should probably not stop ONLY with them, but obviously they're one of the people it needs to stop with
You seem to be implying that customer success is a customer support role. It isn't. Customer success is about helping customers get the greatest business value from your product. They are not going to be munging databases and wrangling backups. Support engineers (among others) do that work.
These are the kind of incidents where parties and celebrations are put on hold and everyone does what they can to help, regardless of department or title.
Re: Inside the longest Atlassian outage
#595Earlier quoted context omitted.
it seems that it worked as intended, thus they have a QA process. The problem was in the wrong IDs provided and I doubt that at their scale they have a staging environment that duplicates the customer data.
Would it be bad practice to append values to a GUID type of ID that would help a human recognize them? For instance, in this specific case they wanted app IDs as APP-XXXXX-XXXX-blahblah and CLOUD-XXXXX-blahblah. I'm not looking to help their specific problems, but this is more from a general question I've thought of doing but never have done just because I'm sure I'd get laughed at for blazing my own trail
This might be customer id and customer name, article number and article description, invoice id and invoice number etc.
Then it is usually very clear to the recipient what they've been handed.
Also, for internal autoinc-type id's, we mostly use sequence generators with non-overlapping "series". That is, we'll start first one at 1 million, second at 2 million or similar. Not perfect but can be useful.
Re: Inside the longest Atlassian outage
#596Earlier quoted context omitted.
Well, their explanation makes sense. These are multi-tenant environments where not every tenant was affected; sensibly, the backups appear divided by environment, not tenant. You can’t blindly revert to an environment’s last backup in this scenario, although you’d think they would have done it before.
Not having per tenant backups is sensible? Seems like a bit of an oversight. Doesn't really matter if the hosting is multi-tenant or not.
Re: Inside the longest Atlassian outage
#597What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.
Re: Inside the longest Atlassian outage
#598Earlier quoted context omitted.
You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants; i.e. any cross-tenant primary keys which will have shifted since one tenant's last backup. Separating the backups wouldn't help with the restoration.
Why not restore the tenants to a different environment that is not otherwise active? At Atlassian's scale you would expect them 1) not to be running all the things on one server anyway, 2) have some existing ability to move tenants between environments for legal or performance reasons, 3) have the ability to backup/restore single tenants, and so on. I am not arguing that the predicament they are in now is real, for t…
But the idea of doing a full restore to a new environment, then only enabling accounts for the impacted tenants, is a good one.
Re: Inside the longest Atlassian outage
#599Re: Inside the longest Atlassian outage
#600Earlier quoted context omitted.
I think this is obviously incorrect. Human error is probabilistic, and the probability of making an error cannot be zero. On the flip side, it’s infeasible to use only provably correct systems; not lazy, but literally not a practical option due to compute costs, developer time, what formal techniques can even be applied to the problem at hand, etc…
Sure human failure is probabilistic. But you can design around that by stacking reliability-enhancing approaches together. Let’s say there’s a 10% chance of any given feature being broken. Write a test, (which has another 10% chance of being broken) and now it’s only broken if the test and the code are broken, and broken in the same way. So we’re down to Then add a backup / redundancy system. That has a 10% chance of…
The trouble was the restore would set back everyone’s data to that point in time, whereas only some customers data was impacted.