Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

31–40 of 772 posts

Re: Inside the longest Atlassian outage

#31
> it takes between 4 and 5 elapsed days to hand a site back to a customer.

Atlassian's SLA page says, Premium Cloud Products 99.9%

That's 43 minutes of downtime per month.

That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real?

I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.

Re: Inside the longest Atlassian outage

#32

so this is the end of Atlassian as a company right?

I had the same initial thought. Surely a weekslong outage would drive customers away permanently, right?

Nope. From TFA:

> I asked customers if they would offboard Atlassian as a result of the outage. Most of them said they won’t leave the Atlassian stack, as long as they don’t lose data. This is because moving is complex and they don’t see a move would mitigate a risk of a cloud provider going down.

Re: Inside the longest Atlassian outage

#33

Earlier quoted context omitted.

Most likely the database tables themselves are just a mixture of everyone's data. There's no true multitenancy. So they have to load the backups into a separate database. Then just go through and individually select/insert into the old database. And then you have to worry about things like foreign key constraints complicating the bulk data loading. Are you going to disable constraint enforcement while you bulk load t…

I can't believe that they would intermix the data in that way... but if they did, godspeed to them, they're likely still overpromising what can be done in this time frame.

How else do you run a multitenancy platform?

Re: Inside the longest Atlassian outage

#35
post #10

so this is the end of Atlassian as a company right?

Why would it ? On our end everything works fine. If you’re not one of the 400 companies, there's no difference

Yep. The vast majority of users don't follow these outages (aka don't browse forums like Hacker News or r/sysadmin), and thus aren't aware of them.

Many of these users are decision-makers who decide what tools to use, and will continue to use Atlassian out of inertia due to lots of existing documentation on the tool (this is compounded by not knowing about the outages, or not knowing the severity of the outages), and also because large, professional companies use their tools too.

I don't necessarily agree with the perspective to stay with it, but it uses a lot of political capital/innovation tokens/goodwill/etc. to change systems, when there are usually higher-priority things to do (than to get buy-in to switch).

Re: Inside the longest Atlassian outage

#36

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks their boxes and it has such a huge list of features that nominally it appears to tick the product development boxes (ignoring more subjective concerns like "quality", "performance", and "usability").

I would really like to try working in an organization that uses something simpler, like Trello (although now that this is also an Atlassian property, maybe not exactly Trello?).

Re: Inside the longest Atlassian outage

#37
>"The outage is its 9th day, having started on Monday, 4th of April." >"It took until Day 9 for executives at the company to acknowledge the outage."

Just to put this in perspective. These executives would have left on a Friday afternoon to start their weekends without bothering to publicly address an ongoing outage that was by then 5 days old.

This is mind boggling. Like did some C-level exec say something like "Let's just park this whole outage communication discussion until Monday, have a good weekend everyone."?

Re: Inside the longest Atlassian outage

#38
post #19

Earlier quoted context omitted.

> The fact it's been so long and they still haven't revealed and explained the root cause of the outage They did last night: https://www.atlassian.com/engineering/april-2022-outage-upda...

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 cu…

Right? The way this reads it seems like one person set a flag incorrectly, something I'm sure we've all done numerous times. And there were no checks down the line to catch it.

Re: Inside the longest Atlassian outage

#39
post #19

Earlier quoted context omitted.

> The fact it's been so long and they still haven't revealed and explained the root cause of the outage They did last night: https://www.atlassian.com/engineering/april-2022-outage-upda...

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 cu…

I suppose that’s why you don’t combine a tazer and gun into 1 device with 2 triggers.

Re: Inside the longest Atlassian outage

#40

Earlier quoted context omitted.

I can't believe that they would intermix the data in that way... but if they did, godspeed to them, they're likely still overpromising what can be done in this time frame.

How else do you run a multitenancy platform?

Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here.

Is it not a good idea to spin up separate db instances for each client/company?

Post reply on HN