Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

181–190 of 772 posts

Re: Inside the longest Atlassian outage

#181

Post by tetha »

VWWHFSfQ wrote: post #21
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

Most likely the database tables themselves are just a mixture of everyone's data. There's no true multitenancy. So they have to load the backups into a separate database. Then just go through and individually select/insert into the old database. And then you have to worry about things like foreign key constraints complicating the bulk data loading. Are you going to disable constraint enforcement while you bulk load t…

Yup. The database schema of one of our products uses a tenant_id in most tables to separate customers logically.

I've eventually gotten a tenant exporter to work. Practically, this requires some deep and nasty digging through the information_schema to build a graph of tables and foreign key constraints. Once it had that, it generates selects with a simple where clause for tables with the tenant_id, and selects with weird joins all over the place for other tables to dump the tenant data.

All of that sounds complex, but that part took a day or two to hammer together to 90% completion, since it's just some graph handling. The other 10% were getting some weird date formatting questions right to produce a properly importable sql dump. And interestingly enough, it's working for more than just that one product.

But that's just where the journey started. After that, it took a weeks and months to sort out legacy tables, old tables, tables without indexes, tables no one knew about, tables that were important (but not), tables with inconsistent data, .... And it's just handling a single relational database. And compared to \copy in psql, it's slow. And at times, weird things happen if you import huge chunks of sql into a postgres with deferred foreign keys (because our schema has cyclical references).

Point is, I know how painful it can be to handle that kind of database schema, at a ridiculously smaller scale. I'm kind of happy to not work there.

Re: Inside the longest Atlassian outage

#182

Post by hinkley »

politelemon wrote: post #31

> it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.

Basically not counting lost sales their income for this year went down 2%, which is not as big a deal to them as it is to their customers.

Re: Inside the longest Atlassian outage

#183

Post by deathanatos »

leeoniya wrote: post #66
politelemon wrote: post #31

> it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.

> Atlassian's SLA page says, Premium Cloud Products 99.9% > That's 43 minutes of downtime per month. we need a better default way to communicate SLOs than "number of 9s", which are more human. how the status quo has stayed this way can only be attributed to intentional dark patterns, imho.

… honestly, even the "number of 9s" concept is a struggle for some companies. I've seen a number of SLAs that fail to correctly state a unit: it's %/, and I see the "unit of time" get dropped every now and then, and the resulting thing is meaningless absurdity.

Re: Inside the longest Atlassian outage

#184

Post by aunty_helen »

dxf wrote: post #62
politelemon wrote: post #31

> it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.

>Are SLAs even real? SLI: Some metric you use to measure a thing (e.g. uptime, latency, etc.) SLO: Some objective you try to hit, as measured by the SLI (e.g. "99.99% of requests are processed within 3 seconds) SLA: A promise to a customer that they will meet some SLO, and consequences if they don't. If there aren't consequences for not meeting the SLO, then measuring and tracking the metrics is a pointless exercise.…

You could use it as a material breach of the contract and possibly get out of any arrangement you have with Atlassian.

Re: Inside the longest Atlassian outage

#185

Post by codycraven »

closeparen wrote: post #136
throwaway894345 wrote: post #36

Earlier quoted context omitted.

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

JIRA is a framework for making assembly lines out of knowledge workers. When you're a middle manager at a decent sized company, a major problem you face is that the mass of knowledge workers beneath you are opaque : you have no way of knowing whether they're working or not. Another problem you face is that they're uppity : people who went to college and got used to managing their own time now have all kinds of idiosy…

It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates.

However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system that manages the data about the work that has been completed, is being worked, and is planned to be worked on .

Part of that system might be defining restrictive workflows for some teams, not for control, but to ensure the agreed upon process is followed for quality or consistency.

One of the many problems Jira has is that if you don't have a Jira admin on your team, it's impossible to build an effective and efficient workflow for your team. Coupled with Jira making many things global by default (it takes a lot of care to make a change that only affects specific Jira projects) most configurations end up being a pile of garbage automatically inherited from changes an admin(that is not part of the team) made when intending to change something for another specific team.

Re: Inside the longest Atlassian outage

#186

Post by flaviotsf »

I recommend doing disaster recovery steps for your personal data as well, such as Gmail. At one point recently I was creating filters to delete bulk messages and - when the filter got created, it somehow missed the from:@xyz.com domain part and I ended up deleting => delete forever all emails. I noticed the issue right away but it was enough to wipe 2-3 months worth of emails (all of them, even Sent ones).

Re: Inside the longest Atlassian outage

#187

Post by NineStarPoint »

jmondi wrote: post #110

What blow's my mind is that Atlassian stock has barely taken a hit...

Unless their revenue takes a long term hit over the outage, no reason for the stock market to care. There isn't news of people actually planning to stop using Atlassian products over this. The only direct consequence is going to be the one time payment of SLA credits. So I guess the part I find surprising is how little impact this looks like it will have on people using their products more so than I am that the stock market doesn't care much about this.

Re: Inside the longest Atlassian outage

#188

Post by chaosite »

Viliam1234 wrote: post #177
Rimbo wrote: post #157

Earlier quoted context omitted.

Oh, nonsense. People buy Atlassisn because the licensing is cheap, not because it's particularly good at what it does or designed with any particular workflow in mind.

Cheaper than whatever is the open-source alternative?

Sure, if you host it yourself you have to pay someone to admin it (usually significantly more expensive than a license), and if you use a hosted solution you have to pay the host.

Re: Inside the longest Atlassian outage

#189

Post by hinkley »

Alex3917 wrote: post #141
jacquesm wrote: post #124

Earlier quoted context omitted.

Did you continue as their customer after that?

Nope. I exported our data after they restored the backup and then we cancelled less than a month later. Like I obviously understand suspending our logins, but why would you ever delete someone's data when it's literally only 160 KB of text? The whole thing made zero sense.

Someone somewhere got a promotion sooner because they lowered the slope of a line a little bit.

Re: Inside the longest Atlassian outage

#190

Post by outworlder »

h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

> OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production

This seems to be exactly what they are doing, as described in the article. They don't have automated tools to do this.

Post reply on HN