Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

131–140 of 772 posts

Re: Inside the longest Atlassian outage

#131

Post by jacquesm »

a-dub wrote: post #85

i hate deleting things. prefer flags that hide things instead (like a boolean deleted flag in an rdbms table). prevents data integrity issues in relational databases, makes debugging easier and prevents disasters. ideally also include a timestamp, both for bookkeeping and safe tools that only remove things that have been soft deleted for some time and are safe to delete without compromising integrity of anything that…

Better still: a field that registers at what date a record was supposedly marked as deleted. Because otherwise you still can't bulk recover from an error.

Re: Inside the longest Atlassian outage

#132

Post by akie »

doliveira wrote: post #95
dabeeeenster wrote: post #33

Earlier quoted context omitted.

How else do you run a multitenancy platform?

By segregating as much as you can. Definitely not by putting everything in a single table. At the very least separate databases/schemas with proper permissions so there's not any chance of data intermiBy segregating as much as you can. Definitely not by putting everything in a single table. At the very least separate databases/schemas with proper permissions so there's no chance of data intermixing. The best would be…

Now every time you run a database migration, you have to adjust N tables - and in Atlassian's case, N is 200000. Is that better? It depends. There is no "best" way of doing multitenancy.

Re: Inside the longest Atlassian outage

#135

Post by eropple »

brightball wrote: post #98
radicaldreamer wrote: post #40

Earlier quoted context omitted.

Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?

There are basically two options for multi-tenancy with their own tradeoffs. 1. An account/tenant_id field for each table 2. A schema for each tenant wrapping all of the tables Option 2 gives you cleaner separation but complicates your deployment process because now you have to run every database change across every schema every time you deploy. This gets more complicated as your code is deploying in case the code its…

The secret bomb in option 1 is that you generally have to have smarter primary keys that fully embrace multitenancy and while Atlassian hires smart folks and I'm sure they at some level know this--that's a relatively hard retrofit to work into a system.

Re: Inside the longest Atlassian outage

#136

Post by closeparen »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

JIRA is a framework for making assembly lines out of knowledge workers. When you're a middle manager at a decent sized company, a major problem you face is that the mass of knowledge workers beneath you are opaque: you have no way of knowing whether they're working or not. Another problem you face is that they're uppity: people who went to college and got used to managing their own time now have all kinds of idiosyncratic ideas about how to manage their own time and arrange their own working lives. Since you are a middle manager you despise local differences. Since you are a manager you're pretty sure that only you and your lieutenants can be trusted with this kind of decision making power. Adopting JIRA is a powerful level to put people back in their place as work item churning machines. Constraints such as only certain people can create or assign tickets, only certain people can mark them completed, only certain states are valid transitions from other states, etc. implement a level of domination over white-collar workforces that managers would be otherwise uncomfortable asserting face to face.

Other ticketing systems do not work nearly as well for this purpose because they are designed mainly as external brains or communication platforms for workers, and they assume a level of worker autonomy in moving tasks through their lifecycle. In Trello you cannot make it so that a PM has to sign off before a card is moved to the in-progress column, or that only in-progress cards can have code reviews associated with them. JIRA eats these kinds of requirements for breakfast.

EDIT: This is not to say you can't use JIRA in a workflow-neutral way, or that everyone uses it for this reason, but I would submit that it's JIRA's differentiated advantage.

Re: Inside the longest Atlassian outage

#137

Post by ollien »

jacquesm wrote: post #129
ollien wrote: post #121

Earlier quoted context omitted.

As someone who has never had to perform this kind of recovery: why is it so hard?

Because it is very difficult to maintain relational integrity during a restore like that.

Gotcha. I guess you could be heavy-handed and disable foreign key checks, but who knows what other bugs that would bring into the mix.

Re: Inside the longest Atlassian outage

#138

Post by dangus »

ocdtrekkie wrote: post #43
farseer wrote: post #15

They have recently killed off on premise offerings, it's cloud only now. And this makes it harder to trust both the security and integrity of your data.

The fact that a single bad script could delete 400 of their customers should be absolute proof they do not have the processes in place to be a steward of your data in the cloud. On-prem or bust.

On-premise just means that your overworked IT person is going to spend 5% of their time keeping your service maintained, at no point gaining any more than baseline familiarity with the product.

On-premise isn’t a magic pill guaranteeing 100% uptime and 0 data loss.

While on-premise may be a good choice in many cases, it’s not like running on-premise business tools has no risk associated with that choice.

Remember that the goal of a company is to sell the most product possible (output) with the lowest cost possible (input).

Any Joe off the street starting their own business can pay Atlassian $0/month for up to a 10 users. On-prem doesn’t compete with that.

Re: Inside the longest Atlassian outage

#139

Post by krinchan »

bborud wrote: post #42
politelemon wrote: post #31

> it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.

Think of SLAs as "this is how hard we'll scramble when shit hits the fan". Except...I don't even believe that.

Per the article, if you experience < 95% uptime in any 30 day window you qualify for a 50% discount. On a month or your next year or ... ? it doesn't say.
Post reply on HN