Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

261–270 of 772 posts

Re: Inside the longest Atlassian outage

#261

Post by mh- »

mdoms wrote: post #234
ZetaZero wrote: post #81

Earlier quoted context omitted.

This. It would be an impossible nightmare for every account to have their own DB. Hundreds of thousands of accounts and databases....

This is the case. I won't comment on your "hundreds of thousands" figure because the number of Cloud customers was a closely guarded secret at least when I worked there, but yes one DB per tenant, dozens to hundreds of DBs per server, and some complicated shuffling of tenant DBs when you run into noisy neighbours.

That makes this prolonged restore process all the more confusing, then.

I (and many others) assumed they had to graft in data from backups since a full restore would clobber newer changes from unaffected customers.

If they're all isolated in their own logical per-tenant DBs, I'm really at a loss for what is making restoration take 3 weeks for 400 tenants.

I understand if you'd rather not venture into it, but care to offer any speculation?

Re: Inside the longest Atlassian outage

#262

Post by user22 »

brimble wrote: post #107
nemothekid wrote: post #45

> Most of them said they won’t leave the Atlassian stack, as long as they don’t lose data. This is because moving is complex and they don’t see a move would mitigate a risk of a cloud provider going down. I still don't understand the strangehold JIRA has on some clients. I can't quickly think of another SaaS product that could be down for almost 2 weeks and not have most customers leave.

If they don't lose data, two weeks of downtime every few years might be cheaper than the cost of switching. Plus, it's not like you know the thing you switch to will be any better, if it's another SaaS.

Let's say we have an announced release schedule on may 1st. With the tools down, there is no way to meet that date. For a 4 billion dollar company, this can make a huge difference in revenue. For a public company, the stock will definitely drop when it's announced the revenue goals were missed because the tools were down.

For companies of size, the cost of tools being down for 3 weeks can easily be in the multi-millions of dollars.

Re: Inside the longest Atlassian outage

#263

Post by dzikimarian »

originalvichy wrote: post #199
thesh4d0w wrote: post #156

Earlier quoted context omitted.

Atlassian has EOL'ed their non-cloud products https://www.atlassian.com/migration/assess/journey-to-cloud

I have had to correct this too many times already. Server is the name of the deployment type of their on-prem. It means single node non-clustered. Data center is their deployment that supports clustering to multiple nodes (and used to support a few extra features). They are retiring the Server deployment type licenses and pushing everyone to data center or cloud. So no, they aren’t EOLing their on-prem.

Our instance is half the size of minimal Data center license. For us and for many customers this is effectively EOL.

Re: Inside the longest Atlassian outage

#264

Post by hhmc »

throwaway894345 wrote: post #89
systemvoltage wrote: post #78

Earlier quoted context omitted.

What's wrong with C++? Seems unfair to compare it with JIRA.

I was a C++ programmer in a past life and I sorta like it. C++ and JIRA seem to have the same philosophy with respect to choosing which features to admit: "yes". The idea is that by supporting the largest number of features possible, they'll surely build something that everyone likes because it will tick everyone's boxes. What people frequently fail to realize is that the absence of misfeatures or redundant features…

> The idea is that by supporting the largest number of features possible, they'll surely build something that everyone likes because it will tick everyone's boxes.

The idea that the C++ committee are unthinking people pleasers it patently false.

C++ does have a lot of cruft, but mostly because it aims to: i) support new features ii) maintain pretty strong backward compatibility guarantees

In general the new features are actually pretty well liked, but in conjunction with (ii) it creates a big language. There's a reasonably decent subset that can be carved out, but it's also clear why newcomers without legacy baggage (e.g. rust) are making inroads.

Re: Inside the longest Atlassian outage

#265

Post by Crabber »

napolux wrote: post #147

Yeah, let's centralize the Internet (born decentralized). This is what the Internet has become.

How do we solve this problem? In other industries based on physical products there is a big incentive to buy goods as locally as possible because of reduced shipping costs, shorter shipping time, no import taxes etc.

But with software it costs nothing to spin up new instances, costs nothing to deliver half way across the world, and has no delivery time. How can you convince a manager to use a software solution provided by a local company when a company in a completely different country 600 miles away offers similar software with 5 extra features?

It seems like the internet is now perfectly set up to create, for each software type, a single company that has a global monopoly.

Re: Inside the longest Atlassian outage

#266

Post by ordiel »

All I can say as an Attlassian Server products user is that the moment they say it was Cloud or nothing, I choose nothing.

I much rather running Gittea on a raspberry pi that I CONTROL than having to have the impotence of doing nothing for more than a week. + having work at cloud companies and having been requested to "collect customer data" to hand it over to the government I would NEVER move critical pieces to anyone else's infa...

(Note: I am not supporting crime, but I rather to have privacy and criminals than living on an authoritarian regime where a dictator who knows everything abot everyone keeps "peace".... Yes I am looking at you China!)

If mistakes will be made, at least I wont pay others to do them for me....

Re: Inside the longest Atlassian outage

#267

Post by hnlmorg »

akie wrote: post #132
doliveira wrote: post #95

Earlier quoted context omitted.

By segregating as much as you can. Definitely not by putting everything in a single table. At the very least separate databases/schemas with proper permissions so there's not any chance of data intermiBy segregating as much as you can. Definitely not by putting everything in a single table. At the very least separate databases/schemas with proper permissions so there's no chance of data intermixing. The best would be…

Now every time you run a database migration, you have to adjust N tables - and in Atlassian's case, N is 200000. Is that better? It depends. There is no "best" way of doing multitenancy.

That’s just an automation issue. It’s not like you have to write a bespoke database migration script per DB.

Re: Inside the longest Atlassian outage

#268

Post by prescriptivist »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

This reminds me of one of my favorite HN comments of all time: https://news.ycombinator.com/item?id=16424423

Re: Inside the longest Atlassian outage

#269

Post by drjasonharrison »

qiskit wrote: post #94
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point How would they lose committed data? Even after restoring the backups can't they run the logs so that everyone is caught up?

Are you assuming that they record the events in a way that can be played back?

Re: Inside the longest Atlassian outage

#270

Post by oauea »

taeric wrote: post #87
ZetaZero wrote: post #81

Earlier quoted context omitted.

This. It would be an impossible nightmare for every account to have their own DB. Hundreds of thousands of accounts and databases....

Wait. Why? This sounds like something that feels hard, if you are used to the giant DBs of old. But you can probably get many many instances of the smaller databases without much trouble. Would still be some maintenance, don't get me wrong. But far from impossible.

You'll quickly run into limitations of how many tcp connections you can hold open. Unless you also want to run separate app servers for each customer, which will cost a lot of $$$

Oh, and just forget about allowing your customers to share their data with each other, which most enterprises want in one way or another.

Post reply on HN