Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

591–600 of 772 posts

Re: Inside the longest Atlassian outage

#591

Earlier quoted context omitted.

> May be Microsoft, Is there a Jira replacement/offering in the Microsoft 365 suite?

Azure DevOps is really underrated: https://www.thoughtworks.com/radar/platforms/azure-devops

It’s been a self-hosted products for over a decade in the form of Visual SourceSafe and then TFS (wonky TFVC not withstanding; Git support was added a while ago as well), now living on as Azure DevOps Server.

Re: Inside the longest Atlassian outage

#592
post #571

Earlier quoted context omitted.

They are literally the head of "customer success". The buck should probably not stop ONLY with them, but obviously they're one of the people it needs to stop with

You seem to be implying that customer success is a customer support role. It isn't. Customer success is about helping customers get the greatest business value from your product. They are not going to be munging databases and wrangling backups. Support engineers (among others) do that work.

Well right now customers are getting zero value. It’s a bad look to be partying during an all hands on deck emergency.

Re: Inside the longest Atlassian outage

#593
post #521

Earlier quoted context omitted.

The deletion of customer data was engineering mistake, that is not what I was talking about The Negative fall out was not due to the deletion of customer data, as the Story and multiple customers have stated the negative fall out was the SILENCE / lack of communications, which is Sales / Customer Service not engineering As the comment I was replying to noted while engineering was trying to recover from what might pos…

Ok I agree with these failures, but don't you think that its a PR people problem? Perhaps executives and upper management? Sales people are just doing what they are supposed to do. Sell Atlassian products.

You and the person you're replying to are using the word "Sales" differently. GP is using it as "Sales Representative", a la Jim Halpert, whereas you're using it as "Outbound Sales", like Glengarry Glen Ross.

Re: Inside the longest Atlassian outage

#594
post #571

Earlier quoted context omitted.

They are literally the head of "customer success". The buck should probably not stop ONLY with them, but obviously they're one of the people it needs to stop with

You seem to be implying that customer success is a customer support role. It isn't. Customer success is about helping customers get the greatest business value from your product. They are not going to be munging databases and wrangling backups. Support engineers (among others) do that work.

Customers unable to access the product they paid for, for over a week with very little communication, are not experiencing success.

These are the kind of incidents where parties and celebrations are put on hold and everyone does what they can to help, regardless of department or title.

Re: Inside the longest Atlassian outage

#595
post #70

Earlier quoted context omitted.

it seems that it worked as intended, thus they have a QA process. The problem was in the wrong IDs provided and I doubt that at their scale they have a staging environment that duplicates the customer data.

Would it be bad practice to append values to a GUID type of ID that would help a human recognize them? For instance, in this specific case they wanted app IDs as APP-XXXXX-XXXX-blahblah and CLOUD-XXXXX-blahblah. I'm not looking to help their specific problems, but this is more from a general question I've thought of doing but never have done just because I'm sure I'd get laughed at for blazing my own trail

While we don't do exactly that, when pulling out lists of ID's like that for someone else, internal or external, we strive to include a description column as well.

This might be customer id and customer name, article number and article description, invoice id and invoice number etc.

Then it is usually very clear to the recipient what they've been handed.

Also, for internal autoinc-type id's, we mostly use sequence generators with non-overlapping "series". That is, we'll start first one at 1 million, second at 2 million or similar. Not perfect but can be useful.

Re: Inside the longest Atlassian outage

#596

Earlier quoted context omitted.

Well, their explanation makes sense. These are multi-tenant environments where not every tenant was affected; sensibly, the backups appear divided by environment, not tenant. You can’t blindly revert to an environment’s last backup in this scenario, although you’d think they would have done it before.

Not having per tenant backups is sensible? Seems like a bit of an oversight. Doesn't really matter if the hosting is multi-tenant or not.

How do you implement per tenant backups? Not every db system cleanly separates where each tenant’s data is stored.

Re: Inside the longest Atlassian outage

#597

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

Gitea is nice. It's a self-hosted Github-like. Not sure how well it scales to big teams and projects though. Probably can't be worse than Jira...

Re: Inside the longest Atlassian outage

#598

Earlier quoted context omitted.

You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants; i.e. any cross-tenant primary keys which will have shifted since one tenant's last backup. Separating the backups wouldn't help with the restoration.

Why not restore the tenants to a different environment that is not otherwise active? At Atlassian's scale you would expect them 1) not to be running all the things on one server anyway, 2) have some existing ability to move tenants between environments for legal or performance reasons, 3) have the ability to backup/restore single tenants, and so on. I am not arguing that the predicament they are in now is real, for t…

Depending on the size of the tenant’s data, moving to a new environment could take a while.

But the idea of doing a full restore to a new environment, then only enabling accounts for the impacted tenants, is a good one.

Re: Inside the longest Atlassian outage

#600

Earlier quoted context omitted.

I think this is obviously incorrect. Human error is probabilistic, and the probability of making an error cannot be zero. On the flip side, it’s infeasible to use only provably correct systems; not lazy, but literally not a practical option due to compute costs, developer time, what formal techniques can even be applied to the problem at hand, etc…

Sure human failure is probabilistic. But you can design around that by stacking reliability-enhancing approaches together. Let’s say there’s a 10% chance of any given feature being broken. Write a test, (which has another 10% chance of being broken) and now it’s only broken if the test and the code are broken, and broken in the same way. So we’re down to Then add a backup / redundancy system. That has a 10% chance of…

They had backup restore process.

The trouble was the restore would set back everyone’s data to that point in time, whereas only some customers data was impacted.

Post reply on HN