Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

101–110 of 772 posts

Re: Inside the longest Atlassian outage

#101

Post by judge2020 »

chupchap wrote: post #57
nemothekid wrote: post #45

> Most of them said they won’t leave the Atlassian stack, as long as they don’t lose data. This is because moving is complex and they don’t see a move would mitigate a risk of a cloud provider going down. I still don't understand the strangehold JIRA has on some clients. I can't quickly think of another SaaS product that could be down for almost 2 weeks and not have most customers leave.

A lot of companies have integrations to atlassian suite which might not be easy to shift from. Secondly, there are a lot of individual competitors to Jira, Confluence and Bitbucket but which competitor can offer all three under a single invoice? May be Microsoft, can't think of anyone else. Also for such an extended downtime the customers are entitled to a discount or a credit note which a lot of CXOs consider in the…

> May be Microsoft,

Is there a Jira replacement/offering in the Microsoft 365 suite?

Re: Inside the longest Atlassian outage

#102

Post by echelon »

bborud wrote: post #42
politelemon wrote: post #31

> it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.

Think of SLAs as "this is how hard we'll scramble when shit hits the fan". Except...I don't even believe that.

In some industries, three nines isn't exactly stellar. Every service I've worked on recently has demanded five nines of uptime and tons of reporting on latency and even seconds-long outages.

I've been on-call during a total infrastructure outage whose root cause was a service my team owned [1]. Our CEO was aware of it. Customers and business partners were aware of it. Other CEOs were aware of it. The media, you name it.

Some outages can be "business ending" or "business damaging". That's why we made a practice and process of performing regular disaster recovery exercises, had exceptionally well documented runbooks, had monitoring attached to everything, and engineered for resilience.

Though I'm not familiar with how Atlassian runs, I think this is an "engineering culture" thing or can be mitigated with a proper approach.

[1] The company has only had a few of these in total, and no member of our team was culpable for the complicated failure.

Re: Inside the longest Atlassian outage

#103

Post by stickfigure »

radicaldreamer wrote: post #40
dabeeeenster wrote: post #33

Earlier quoted context omitted.

How else do you run a multitenancy platform?

Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?

I have built multiple multi-tenancy platforms and I never create separate databases for each customer. If you have separate databases, it's almost impossible to run meaningful queries across all of them. That architectural choice creates far more headaches than it solves. Usually people end up with the split-database architecture when they want a quick retrofit for a system that wasn't designed with multiple tenants.

I've also had to restore partial data from backups on a few occasions when customers fat-fingered some data and asked pretty-please to undo. If someone on staff understands the system well, it's not hard. I suspect Atlassian suffers from a complicated schema and a post-IPO brain drain.

Re: Inside the longest Atlassian outage

#105

Post by richardw »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

I’m just a user but totally happy with all our Atlassian apps. Confluence is a huge win across our multi-thousand person company and the best teams use it very well. I like the integration between Jira and Bitbucket. We don’t over complicate things and it works fine.

It’s like my taste in wine. I don’t want an overdeveloped sense of taste where only a $400 bottle will do. I’m fine with what we have because the work is what excites me and if people are documenting projects and managing workloads and committing code, we’re 90% of the way there.

Re: Inside the longest Atlassian outage

#106

Post by indymike »

radicaldreamer wrote: post #40
dabeeeenster wrote: post #33

Earlier quoted context omitted.

How else do you run a multitenancy platform?

Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?

Answer: it depends on the application. For example big social app is not going to provision a new db for every user, or for every customer that runs an ad. Likewise, a lot of enterprise software fits a model where each customer getting it's own db makes sense. So, really, just a design decision.

Re: Inside the longest Atlassian outage

#107

Post by brimble »

nemothekid wrote: post #45

> Most of them said they won’t leave the Atlassian stack, as long as they don’t lose data. This is because moving is complex and they don’t see a move would mitigate a risk of a cloud provider going down. I still don't understand the strangehold JIRA has on some clients. I can't quickly think of another SaaS product that could be down for almost 2 weeks and not have most customers leave.

If they don't lose data, two weeks of downtime every few years might be cheaper than the cost of switching. Plus, it's not like you know the thing you switch to will be any better, if it's another SaaS.

Re: Inside the longest Atlassian outage

#108

Post by x0x0 »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

I made the decision, unfortunately. The rationale was literally that I hated pivotal tracker -- what a garbage app that is -- and I'd heard of jira, needed something to track bugs / work items, and signed up. It crucially had a zendesk -> jira sync, so all our zendesk requests could end up in jira.

In the beginning, with me plus 2 engineers, I noticed it was slow but since I used it for 20 minutes a week, that didn't really matter. By the time I started using it for an hour a day, we had 10 engineers on 2 teams using it. I got to see a friend using linear, and I had some spare time that I was going to use to switch, but I couldn't get in the beta. By the time they let me in, the opportunity was over and I was too busy.

Re: Inside the longest Atlassian outage

#109

Post by vinay_ys »

luckydata wrote: post #5

so this is the end of Atlassian as a company right?

Depends. Are there strong alternate products to which customers can easily migrate in next 6-12 months? If yes, and they choose to move away, then Atlassian will be in serious trouble. I wonder how many of their customers have long-term locked-in contracts and if they have performance clauses that allows them to exit such contracts.
Post reply on HN