Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

21–30 of 772 posts

Re: Inside the longest Atlassian outage

#21

Post by VWWHFSfQ »

h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

Most likely the database tables themselves are just a mixture of everyone's data. There's no true multitenancy. So they have to load the backups into a separate database. Then just go through and individually select/insert into the old database. And then you have to worry about things like foreign key constraints complicating the bulk data loading. Are you going to disable constraint enforcement while you bulk load the data? How does that affect existing and new data from customers using the database? Just a guess. But this sounds like a nightmare honestly.

Re: Inside the longest Atlassian outage

#23

Post by mkl95 »

pgwhalen wrote: post #14
mkl95 wrote: post #7

The fact it's been so long and they still haven't revealed and explained the root cause of the outage is going to make it hard to regain trust on their buggy, slow tools. The bright side of the incident is that competitors that somewhat care about users have a unique opportunity to stand out.

> The fact it's been so long and they still haven't revealed and explained the root cause of the outage They did last night: https://www.atlassian.com/engineering/april-2022-outage-upda...

> Communication gap. First, there was a communication gap between the team that requested the deactivation and the team that ran the deactivation. Instead of providing the IDs of the intended app being marked for deactivation, the team provided the IDs of the entire cloud site where the apps were to be deactivated.

So what they are saying is that they are not testing scripts at some staging server before running them in production. It's wild that they've managed to scale their products so much before something like this happened.

I hope they've learnt their lesson and they set up some QA process for that stuff.

Re: Inside the longest Atlassian outage

#24

Post by gmfawcett »

asah wrote: post #12
case0x wrote: post #10

Earlier quoted context omitted.

Why would it ? On our end everything works fine. If you’re not one of the 400 companies, there's no difference

"first they came for ..."

Poor taste, buddy. Comparing the Atlassian mess-up to the Holocaust diminishes the Holocaust.

Re: Inside the longest Atlassian outage

#25

Post by brianwawok »

R0ger wrote: post #13

I guess this is wake call for the people rushing to SaaS solutions.

Is it?

We use JIRA. Not impacted.

If this had hit us.. we would just switch to excel or something for a week/month?

But maybe we are a very light user of JIRA. Nothing in there can't be replaced. It's "nice" to be able to go look up a 3 year old bug and which client reported it, but not really crucial for day to day ops.

Re: Inside the longest Atlassian outage

#26

Post by er4hn »

rmbyrro wrote: post #16

Are Confluence pages and Jira tickets build like a GPT-3 300 Terabyte model? I mean, I thought they were text. 5 days to restore text? They must be generated by a huge complex deep learning voodoo. Atlassian is working on the bleeding edge of technology. This outage is understandable...

I suppose if they recover what they can and restore the rest using GPT-3 that may make the process easier.

Re: Inside the longest Atlassian outage

#27

Post by katbyte »

rmbyrro wrote: post #16

Are Confluence pages and Jira tickets build like a GPT-3 300 Terabyte model? I mean, I thought they were text. 5 days to restore text? They must be generated by a huge complex deep learning voodoo. Atlassian is working on the bleeding edge of technology. This outage is understandable...

images and other files can be attached to issues or embedded in pages so a single instance can use a lot of storage.

Re: Inside the longest Atlassian outage

#28

Post by radicaldreamer »

VWWHFSfQ wrote: post #21
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

Most likely the database tables themselves are just a mixture of everyone's data. There's no true multitenancy. So they have to load the backups into a separate database. Then just go through and individually select/insert into the old database. And then you have to worry about things like foreign key constraints complicating the bulk data loading. Are you going to disable constraint enforcement while you bulk load t…

I can't believe that they would intermix the data in that way... but if they did, godspeed to them, they're likely still overpromising what can be done in this time frame.

Re: Inside the longest Atlassian outage

#29

Post by tempest_ »

pgwhalen wrote: post #14
mkl95 wrote: post #7

The fact it's been so long and they still haven't revealed and explained the root cause of the outage is going to make it hard to regain trust on their buggy, slow tools. The bright side of the incident is that competitors that somewhat care about users have a unique opportunity to stand out.

> The fact it's been so long and they still haven't revealed and explained the root cause of the outage They did last night: https://www.atlassian.com/engineering/april-2022-outage-upda...

Is it just me or is highlighting on that site broken,

Perhaps my ad blocker is causing that stupid highlight to tweet js they are using to break.

Post reply on HN