Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

91–100 of 772 posts

Re: Inside the longest Atlassian outage

#91

Post by TheCoelacanth »

dylan604 wrote: post #64
chrsig wrote: post #61

Earlier quoted context omitted.

It's more "this is our contractual obligation, if we're down more than this, then we might not charge you"

Lawyers are involved, so I'd assume some text about "excluding acts of god, sabotage,etc" to weasel their way out of things. They might even be able to get away with "acts of incompetence" how ever a lawyer might phrase that to allow their client to weasel.

SLA credits are a thing that actually happen in the industry. I wouldn't automatically assume that they will be able to weasel out of it.

They are typically limited to the amount that you actually paid, though, so basically they don't charge you for the time when you couldn't use the product. You usually won't get more than that.

Re: Inside the longest Atlassian outage

#92

Post by Iolaum »

radicaldreamer wrote: post #75
1970-01-01 wrote: post #63

Interesting note: Atlassian stock (NASDAQ: TEAM) is up 4% as of noon today.

It might be a good short opportunity... I imagine a lot of customers are kicking off their own internal process for migrating away from JIRA. By the time they actually do, it'll be at least a couple of quarters from now, which is when the customer hit will start materializing in quarterly results for the company. Maybe time to throw a few chips at some long term puts?

Given how out of sync tech people are with the general population I 'd be tempted to think it's a buy opportunity. Time will tell.

Re: Inside the longest Atlassian outage

#93

Post by bombcar »

dxf wrote: post #62
politelemon wrote: post #31

> it takes between 4 and 5 elapsed days to hand a site back to a customer. Atlassian's SLA page says, Premium Cloud Products 99.9% That's 43 minutes of downtime per month. That works out to, Atlassian can't have any more downtime for the next 14 years. Are SLAs even real? I'm being slightly facetious. From the page text it's just a threshold after which I think you're entitled to some money back for that month.

>Are SLAs even real? SLI: Some metric you use to measure a thing (e.g. uptime, latency, etc.) SLO: Some objective you try to hit, as measured by the SLI (e.g. "99.99% of requests are processed within 3 seconds) SLA: A promise to a customer that they will meet some SLO, and consequences if they don't. If there aren't consequences for not meeting the SLO, then measuring and tracking the metrics is a pointless exercise.…

Most SLAs say "if we miss this, you get time for free" which means that these companies will hopefully get a refund ... for the time they can't use the service.

SLAs are mostly aspirational.

Re: Inside the longest Atlassian outage

#94

Post by qiskit »

h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point

How would they lose committed data? Even after restoring the backups can't they run the logs so that everyone is caught up?

Re: Inside the longest Atlassian outage

#95

Post by doliveira »

dabeeeenster wrote: post #33
radicaldreamer wrote: post #28

Earlier quoted context omitted.

I can't believe that they would intermix the data in that way... but if they did, godspeed to them, they're likely still overpromising what can be done in this time frame.

How else do you run a multitenancy platform?

By segregating as much as you can. Definitely not by putting everything in a single table. At the very least separate databases/schemas with proper permissions so there's not any chance of data intermiBy segregating as much as you can. Definitely not by putting everything in a single table. At the very least separate databases/schemas with proper permissions so there's no chance of data intermixing.

The best would be multiple separate database instances, which is not even hard to manage specially for qualified engineers like Atlassian surely has plenty of. The problem are business decisions of ignoring the tech debt, usually...

Re: Inside the longest Atlassian outage

#96

Post by mrits »

brianwawok wrote: post #25
R0ger wrote: post #13

I guess this is wake call for the people rushing to SaaS solutions.

Is it? We use JIRA. Not impacted. If this had hit us.. we would just switch to excel or something for a week/month? But maybe we are a very light user of JIRA. Nothing in there can't be replaced. It's "nice" to be able to go look up a 3 year old bug and which client reported it, but not really crucial for day to day ops.

I wonder why you use Jira if a spreadsheet is sufficient for your use case.

Re: Inside the longest Atlassian outage

#97

Post by bombcar »

dylan604 wrote: post #79
notdang wrote: post #70

Earlier quoted context omitted.

it seems that it worked as intended, thus they have a QA process. The problem was in the wrong IDs provided and I doubt that at their scale they have a staging environment that duplicates the customer data.

Would it be bad practice to append values to a GUID type of ID that would help a human recognize them? For instance, in this specific case they wanted app IDs as APP-XXXXX-XXXX-blahblah and CLOUD-XXXXX-blahblah. I'm not looking to help their specific problems, but this is more from a general question I've thought of doing but never have done just because I'm sure I'd get laughed at for blazing my own trail

This is recommended in my experience, but you do have some potential issues when a UUID gets reused or repurposed.

WHENEVER a human is involved in the chain, UUIDs can be suspicious because there's no easy way to verify what it is, whereas a human has a good chance of realizing that $1,342.34 is probably not a valid date.

Re: Inside the longest Atlassian outage

#98

Post by brightball »

radicaldreamer wrote: post #40
dabeeeenster wrote: post #33

Earlier quoted context omitted.

How else do you run a multitenancy platform?

Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?

There are basically two options for multi-tenancy with their own tradeoffs.

1. An account/tenant_id field for each table

2. A schema for each tenant wrapping all of the tables

Option 2 gives you cleaner separation but complicates your deployment process because now you have to run every database change across every schema every time you deploy. This gets more complicated as your code is deploying in case the code itself gets out of sync, there's a rollback or an error mid deploy due to an issue with some specific data.

The benefit of the approach is the option to do different backup policies for different customers, makes moving specific customers to specific instances easier and you avoid the extra index on tenant_id in every table.

Option 1 is significantly easier to shard out horizontally and simplifies the database change process, but you lose space on the extra indexes. Plus in many databases you can partition on the tenant_id.

Most people typically end up with option 1 after dealing with or reading horror stories about the operational complexity of option 2.

Re: Inside the longest Atlassian outage

#99

Post by scottlamb »

Gmail had a vaguely similar outage years ago. [1] tl;dr:

1. Different root cause. There was a bug in a refactoring of gmail's storage layer (iirc a missing asterisk caused a pointer to an important bool to be set to null, rather than setting the bool to false), which slipped through code review, automated testing, and early test servers dedicated to the team, so it got rolled out to some fraction of real users. Online data was lost/corrupted for 0.02% of users (a huge amount of email).

2. There were tape backups, but the tooling wasn't ready for a restore at scale. It was all hands on deck to get those accounts back to an acceptable state, and it took four days to get back to basically normal (iirc no lost mail, although some got bounced).

3. During the outage, some users could log in and see something frightening: an empty/incomplete mailbox, and no banner or anything telling them "we're fixing it".

4. Google communicated more openly, sooner, [2] which I think helped with customer trust. Wow, Atlassian really didn't say anything publicly for nine days?!?

Aside from the obvious "have backups and try hard to not need them", a big lesson is that you have to be prepared to do a mass restore, and you have to have good communication: not only traditional support and PR communication but also within the UI itself.

[1] https://static.googleusercontent.com/media/www.google.com/en...

[2] https://gmail.googleblog.com/2011/02/gmail-back-soon-for-eve...

Re: Inside the longest Atlassian outage

#100

Post by bombcar »

a-dub wrote: post #85

i hate deleting things. prefer flags that hide things instead (like a boolean deleted flag in an rdbms table). prevents data integrity issues in relational databases, makes debugging easier and prevents disasters. ideally also include a timestamp, both for bookkeeping and safe tools that only remove things that have been soft deleted for some time and are safe to delete without compromising integrity of anything that…

The GDPR and various things have made companies more skittish in doing things this way, because they get scared.

Perhaps an effective measure would be to create a key that encrypts a customer's data, and give them a copy of the key, and let them know that after a certain point your copy of the key will be deleted, and if they want a restore past that point they'll need to provide the key.

Post reply on HN