Inside the longest Atlassian outage • Hacker News Board

Re: Inside the longest Atlassian outage

#51

Post by BlargMcLarg » Wed, Apr 13, 2022, 4:36 PM UTC

throwaway894345 wrote: post #36

h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

Trickle down and first mover. JIRA was there first being "decently ok", enough people adapted it and now others do the same. Then couple with that what you write, the people in charge of deciding the software are generally the ones who can justify wasting half their day on it.

To this day I still don't know what JIRA does so much better that other products don't which big corps are willing to waste months worth of manhours over. It's biggest selling point is integration with the remainder of the Atlassian stack, not exactly known for being great either.

Re: Inside the longest Atlassian outage

#52

Post by nemothekid » Wed, Apr 13, 2022, 4:37 PM UTC

radicaldreamer wrote: post #40

dabeeeenster wrote: post #33

Earlier quoted context omitted.

How else do you run a multitenancy platform?

Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?

I believe you can sign up an account for free or incredibly cheap ($5/user). You would potentially have tens of thousands of databases. Imagine trying to do something like a database migration to add a column. I believe the day to day operations would be a nightmare as no RDBMS has probably had that kind of feature stress tested.

Re: Inside the longest Atlassian outage

#53

Post by teh_klev » Wed, Apr 13, 2022, 4:37 PM UTC

mkl95 wrote: post #7

The fact it's been so long and they still haven't revealed and explained the root cause of the outage is going to make it hard to regain trust on their buggy, slow tools. The bright side of the incident is that competitors that somewhat care about users have a unique opportunity to stand out.

> and they still haven't revealed and explained the root cause of the outage

They did, this post by Atlassian from yesterday is referenced in the article.

https://www.atlassian.com/engineering/april-2022-outage-upda...

Still doesn't excuse them for the time taken to come clean.

Re: Inside the longest Atlassian outage

#54

Post by drc500free » Wed, Apr 13, 2022, 4:38 PM UTC

pgwhalen wrote: post #14

mkl95 wrote: post #7

The fact it's been so long and they still haven't revealed and explained the root cause of the outage is going to make it hard to regain trust on their buggy, slow tools. The bright side of the incident is that competitors that somewhat care about users have a unique opportunity to stand out.

> The fact it's been so long and they still haven't revealed and explained the root cause of the outage They did last night: https://www.atlassian.com/engineering/april-2022-outage-upda...

Highlighting the text in any of their lists breaks the page in interesting ways, apparently due to some twitter-sharing functionality.

Re: Inside the longest Atlassian outage

#55

Post by z58 » Wed, Apr 13, 2022, 4:39 PM UTC

nemothekid wrote: post #45

> Most of them said they won’t leave the Atlassian stack, as long as they don’t lose data. This is because moving is complex and they don’t see a move would mitigate a risk of a cloud provider going down. I still don't understand the strangehold JIRA has on some clients. I can't quickly think of another SaaS product that could be down for almost 2 weeks and not have most customers leave.

I imagine most people used something like Google Sheets during the downtime

Re: Inside the longest Atlassian outage

#56

Post by some-guy » Wed, Apr 13, 2022, 4:39 PM UTC

radicaldreamer wrote: post #40

dabeeeenster wrote: post #33

Earlier quoted context omitted.

How else do you run a multitenancy platform?

Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?

The company I work at (Workday) does this, but it's for business / liability reasons.

Re: Inside the longest Atlassian outage

#57

Post by chupchap » Wed, Apr 13, 2022, 4:40 PM UTC

nemothekid wrote: post #45

> Most of them said they won’t leave the Atlassian stack, as long as they don’t lose data. This is because moving is complex and they don’t see a move would mitigate a risk of a cloud provider going down. I still don't understand the strangehold JIRA has on some clients. I can't quickly think of another SaaS product that could be down for almost 2 weeks and not have most customers leave.

A lot of companies have integrations to atlassian suite which might not be easy to shift from.

Secondly, there are a lot of individual competitors to Jira, Confluence and Bitbucket but which competitor can offer all three under a single invoice? May be Microsoft, can't think of anyone else.

Also for such an extended downtime the customers are entitled to a discount or a credit note which a lot of CXOs consider in their decision making.

Re: Inside the longest Atlassian outage

#58

Post by dangus » Wed, Apr 13, 2022, 4:40 PM UTC

throwaway894345 wrote: post #36

h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

The answer is medium to large companies. Jira is a tool that can satisfy hundreds of different teams’ work management needs without having to buy dozens of different products.

The fact that it’s so feature packed and customizable is the point.

I think the complainers are not really investing the time in to change project settings to fit their needs.

My only complaint about the Atlassian suite is the performance of Jira and Confluence. The overall page load speed is too slow.

Re: Inside the longest Atlassian outage

#59

Post by anecd0te » Wed, Apr 13, 2022, 4:40 PM UTC

throwaway894345 wrote: post #36

h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

ime people pick Jira because they've used Jira and have been promoted via the peter principle to the level at which they make purchasing decisions.

Re: Inside the longest Atlassian outage

#60

Post by andyjohnson0 » Wed, Apr 13, 2022, 4:41 PM UTC

radicaldreamer wrote: post #40

dabeeeenster wrote: post #33

Earlier quoted context omitted.

How else do you run a multitenancy platform?

Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?

> Is it not a good idea to spin up separate db instances for each client/company?

It depends, really. There is a trade-off in terms of software and operational complexity vs scalability/perf and isolation. And probably a bunch of other factors.

If you have separate databases for each customer, schema migrations can be staged over time. But that means your software backend needs to be able to work with different schemas concurrently. You can also benefit from resilience and isolation guarantees provided by the dbms. On the other hand, having a dbms manage lots of databases can affect perf. Linking between databases can be a minefield, especially w/r/t foreign keys and distributed transactions.

https://docs.microsoft.com/en-us/azure/azure-sql/database/sa...