Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

281–290 of 772 posts

Re: Inside the longest Atlassian outage

#281

Post by iso1631 »

nh2 wrote: post #192
politelemon wrote: post #31

> it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.

> it's just a threshold after which I think you're entitled to some money back for that month That is exactly what SLAs are. There are just a lot of people applying the wishful thinking that SLAs are a goal or metric of uptime. Consider the AWS S3 page on the topic: https://aws.amazon.com/s3/sla/ "Reasonable efforts"; if not met, you get some fraction of the money back. S3 has worse uptime than my desktop PC over the…

> S3 has worse uptime than my desktop PC over the last years

That's sacrilege on HN

Re: Inside the longest Atlassian outage

#282

Post by zrail »

ornornor wrote: post #244
mistrial9 wrote: post #225

Earlier quoted context omitted.

I was told by a lifetime manager turned successful consultant, that roughly fifty percent of engineering firms govern their engineers basically using fear.

> using fear Could you elaborate? What kind of fear? “You’re fired”? I wonder how effective it actually is because of the current job market and also because I (and others) react very poorly to this kind of tactics: “you want me to fear getting fired? Joke’s on you, please DO fire me, I dare you”

The perf process is basically one big exercise in fear-based control.

Re: Inside the longest Atlassian outage

#283

Post by mh- »

qiskit wrote: post #94
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point How would they lose committed data? Even after restoring the backups can't they run the logs so that everyone is caught up?

(There's a tacit assumption here that the data across tenants is commingled in tables, and that's being disputed elsewhere in the thread, but playing along..)

You wouldn't be able to do that without forcing downtime for all customers, for the duration it takes to restore the snapshot and then replay the logs. Not to mention the risks of the process failing somehow

You could narrow the window to just the "replay" portion, if you were able to stand up an extra database/infra, to switch over to when it was ready. But at some point you'd probably still have to go read only to checkpoint the logs and begin the replay.

It's of course possible to do something more complicated here and stream the changes then eventually enact a failover, but this would all be too complex and error prone to introduce in their current crisis mode. It's something I'd suggest considering when architecting their DR/BCP, but it's too late for that kind of elegance (and complexity) now.

Re: Inside the longest Atlassian outage

#285

Post by kikki »

dabeeeenster wrote: post #33
radicaldreamer wrote: post #28

Earlier quoted context omitted.

I can't believe that they would intermix the data in that way... but if they did, godspeed to them, they're likely still overpromising what can be done in this time frame.

How else do you run a multitenancy platform?

Not quite the same but at Fandom (Wikia), every wiki has its own DB (over 300,000 wikis), and they are clustered across a bunch of servers (usually balanced by traffic). It works well - but we don't ever really need to query across databases. There's a bunch of logic around instance/db selection but that's about as complex as it gets.

Re: Inside the longest Atlassian outage

#287

Post by biomcgary »

sjtindell wrote: post #202
codycraven wrote: post #185

Earlier quoted context omitted.

It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates. However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system that manages the data about the work that has been completed, is being worked, and is planned to be worked on . Part of that system might be defining…

Agreed. This is a problem of scale in my opinion. When we have 10 engineers, it is easy to check in with everyone and know what they are working on and get a status update. When we have 500 engineers, making sure all their tasks are aligning (organizations are one big race condition) is not just hard but impossible without some sort of tracking system. We all want to grow big. To do so, your processes need to change…

Are they unique because their problem domain allows it or because the leadership is uniquely ideologically driven (and competent) to implement efficient, flat systems?

Re: Inside the longest Atlassian outage

#288

Post by fnord123 »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

> like Trello

Maybe Asana or Monday would work for you.

Re: Inside the longest Atlassian outage

#289

Post by x0x0 »

rectang wrote: post #172
x0x0 wrote: post #114

Earlier quoted context omitted.

I can't believe anyone would do separate databases. Just wait until a migration doesn't run on 2 of your 400+ customer databases. Or multi-hour migrations.

When all customer data lives in the unified database: Just wait until a bug in a query exposes the data of customers to each other, creating instant regulatory and privacy nightmares for everyone.

With an orm and customer objects to create scoped queries, I haven't found this to be a problem. It's also very easy to check in code reviews. And not a painful issue from, well, the lack of this happening given it's an extremely common app design.

Re: Inside the longest Atlassian outage

#290

Post by inopinatus »

seanwilson wrote: post #277
inopinatus wrote: post #242

Earlier quoted context omitted.

in theory, shard your customer databases 1:1, job done. alas, in practice, many SaaS compromise this two ways: a) overwhelmed by creeping featuritis, each customer's data has relationships to global tables, and b) they backup their entire database cluster in one snapshot and there maybe other gotchas for restoration, like relying on denormalized views and caches that have to be rebuilt. they may also have erroneously…

> shard your customer databases 1:1 What are the downsides to this?

* makes it much harder to distribute your tables by any other factor, for whatever reason (usually performance, sometimes archival)

* disaggregates data that the SaaS might be interested in querying/updating as an aggregate

* not all ORM frameworks handle this case well, if at all

* dumps are more than a single trivial command

basically all your data operations gain an additional dimension of complexity, and you may not perceive the benefits until much later

Post reply on HN