Inside the longest Atlassian outage • Hacker News Board

Re: Inside the longest Atlassian outage

#11

Post by hsnewman » Wed, Apr 13, 2022, 4:12 PM UTC

Sounds like the continuity planners at Atlassian (the fall guys) will be looking for a new job.

Re: Inside the longest Atlassian outage

#12

Post by asah » Wed, Apr 13, 2022, 4:13 PM UTC

case0x wrote: post #10

luckydata wrote: post #5

so this is the end of Atlassian as a company right?

Why would it ? On our end everything works fine. If you’re not one of the 400 companies, there's no difference

"first they came for ..."

Re: Inside the longest Atlassian outage

#13

Post by R0ger » Wed, Apr 13, 2022, 4:14 PM UTC

I guess this is wake call for the people rushing to SaaS solutions.

Re: Inside the longest Atlassian outage

#14

Post by pgwhalen » Wed, Apr 13, 2022, 4:14 PM UTC

mkl95 wrote: post #7

The fact it's been so long and they still haven't revealed and explained the root cause of the outage is going to make it hard to regain trust on their buggy, slow tools. The bright side of the incident is that competitors that somewhat care about users have a unique opportunity to stand out.

> The fact it's been so long and they still haven't revealed and explained the root cause of the outage

They did last night: https://www.atlassian.com/engineering/april-2022-outage-upda...

Re: Inside the longest Atlassian outage

#15

Post by farseer » Wed, Apr 13, 2022, 4:15 PM UTC

They have recently killed off on premise offerings, it's cloud only now. And this makes it harder to trust both the security and integrity of your data.

Re: Inside the longest Atlassian outage

#16

Post by rmbyrro » Wed, Apr 13, 2022, 4:15 PM UTC

Are Confluence pages and Jira tickets build like a GPT-3 300 Terabyte model?

I mean, I thought they were text.

5 days to restore text?

They must be generated by a huge complex deep learning voodoo.

Atlassian is working on the bleeding edge of technology. This outage is understandable...

Re: Inside the longest Atlassian outage

#17

Post by raincom » Wed, Apr 13, 2022, 4:15 PM UTC

nitinagg wrote: post #2

Selectively restoring data only for certain rows is super hard. But the communications by Atlassian has been the worst I have ever seen in the industry.

So, it must be a bad idea to shove the data of multiple customers in a single table controlled by some column name ('tenant').

Re: Inside the longest Atlassian outage

#18

Post by JoBrad » Wed, Apr 13, 2022, 4:17 PM UTC

h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

It sounds like that’s what they are doing, but it’s manual.

Re: Inside the longest Atlassian outage

#19

Post by hu3 » Wed, Apr 13, 2022, 4:20 PM UTC

pgwhalen wrote: post #14

mkl95 wrote: post #7

The fact it's been so long and they still haven't revealed and explained the root cause of the outage is going to make it hard to regain trust on their buggy, slow tools. The bright side of the incident is that competitors that somewhat care about users have a unique opportunity to stand out.

> The fact it's been so long and they still haven't revealed and explained the root cause of the outage They did last night: https://www.atlassian.com/engineering/april-2022-outage-upda...

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 customers were improperly deleted.

Ouch. I hope no one person got the blame. This is a systemic failure. Regardless, my regards to the engineers involved.

Re: Inside the longest Atlassian outage

#20

Post by omoikane » Wed, Apr 13, 2022, 4:20 PM UTC

h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

Not being able to selectively restore data for a subset of users might also indicate that users are not fully isolated from each other, which is worrying for technical and nontechnical reasons.