Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

471–480 of 772 posts

Re: Inside the longest Atlassian outage

#471

Post by originalvichy »

crummy wrote: post #391
originalvichy wrote: post #221

Earlier quoted context omitted.

I am biased but I can tell you what works best for mid-large companies: having a solution provider. Basically a partner that hosts and maintains the instance and has enough Atlassian certified people to help you with any question so that you will never have to hire people to just maintain the beasts or tell you about features, tricks or plugins that could solve problem X. Experienced people hosting and tuning Atlassi…

I can't see the difference between a "solution provider" that hosts your Jira and just getting Atlassian to do it. What's stopping the solution provider from accidentally running a script that deletes some customer's files and struggling to do a partial backup restore?

Because you can get the best parts of self-hosted and managed services. And on that backup question: self-hosted Atlassian is vastly easier to protect against disasters. The problem these Atlassian guys had arose from multi-tenant architecture. Usually managed service providers will host your stack on individual databases and VMs, and backing up the software is just a matter of taking pg_dumps and rsyncing certain directories (pretty old school) or just taking disk level snapshots.

Many medium-large corporations have their own cloud environments that their IT Ops control. Solution providers can host Atlassian stacks on their own cloud environment where they are not affected by data privacy concerns (it's in their already green-lit cloud providers data center) so they can host it behind a firewall with only VPN access allowed. They can also do all the magic you can usually do with web software like put a frontend proxy in front of it, or use more flexible/legacy authentication methods. Not to mention that for example you could have a Jira Cloud that you would need to integrate with a SCM program. Jira data could be "OK" to live in the cloud but code would be a big no-no. These problems can be solved by having them all live behind the firewall.

A competent managed solution provider also has consultants that can train or instruct on usage. It costs but it is simpler and faster than having to go through the forums or send a support ticket for every small issue to Atlassian itself.

Re: Inside the longest Atlassian outage

#472

Post by Animats »

beachy wrote: post #411

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…

Rather, customers must stop using Atlassian cloud services.

Re: Inside the longest Atlassian outage

#473

Post by originalvichy »

snark42 wrote: post #402
crummy wrote: post #391

Earlier quoted context omitted.

I can't see the difference between a "solution provider" that hosts your Jira and just getting Atlassian to do it. What's stopping the solution provider from accidentally running a script that deletes some customer's files and struggling to do a partial backup restore?

I would assume the MSP is running a dedicated instance and can do a full/backup restore just for the user they're supporting. If it's some multi-tenant solution it's no better.

Correct. There are probably not a lot of MSPs that have so many customers that they need to share that much data, and their customers probably use MSPs for the strict purpose that they don't want to share things with other companies.

Re: Inside the longest Atlassian outage

#474

Post by deckard1 »

bombcar wrote: post #100
a-dub wrote: post #85

i hate deleting things. prefer flags that hide things instead (like a boolean deleted flag in an rdbms table). prevents data integrity issues in relational databases, makes debugging easier and prevents disasters. ideally also include a timestamp, both for bookkeeping and safe tools that only remove things that have been soft deleted for some time and are safe to delete without compromising integrity of anything that…

The GDPR and various things have made companies more skittish in doing things this way, because they get scared. Perhaps an effective measure would be to create a key that encrypts a customer's data, and give them a copy of the key, and let them know that after a certain point your copy of the key will be deleted, and if they want a restore past that point they'll need to provide the key.

> The GDPR and various things have made companies more skittish in doing things this way, because they get scared.

They may be scared. But are they scared enough to reload every single backup they have, purge the desired records, and resave each and every single backup they have? And not also worry they will corrupt/break the backups in the process.

GDPR compliance is a mess of contradictions and unreasonable asks which all seem to amount to "depends on who you ask."

Re: Inside the longest Atlassian outage

#475

Post by oriki »

NationalPark wrote: post #312
kgeist wrote: post #142

We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is l…

What do you do if the on-premises guy gets hit by a car and isn't in his office?

The same thing that the cloud company would do. If there are other people there who share that guy's responsibilities, have them do it. If there aren't, you should have an on-call.

Cloud just outsources that problem to another business. Sure, they have better reasons to actually cover those positions and make sure they have on-calls and backup and a disaster plan, but just because you pay extra money for it doesn't actually make it work better if the company underlying it sucks.

Re: Inside the longest Atlassian outage

#476

Post by [deleted] »

spookthesunset wrote: post #453
mdoms wrote: post #234

Earlier quoted context omitted.

This is the case. I won't comment on your "hundreds of thousands" figure because the number of Cloud customers was a closely guarded secret at least when I worked there, but yes one DB per tenant, dozens to hundreds of DBs per server, and some complicated shuffling of tenant DBs when you run into noisy neighbours.

If they had multi-tenant databases for SaaS it would mean either the self-hosted jira instances also had the same multi-tenant database schema or they'd have to maintain two almost entirely different data access layers for cloud vs. on-prem. Since their cloud offering came from a historically on-prem codebase, I would expect the easiest way to offer cloud stuff is to do a DB per tenant. Otherwise there would a shit-t…

[deleted]

Re: Inside the longest Atlassian outage

#477

Post by iamtheworstdev »

kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

Can confirm. Saw them there while I was on vacation.

Re: Inside the longest Atlassian outage

#478

Post by ibejoeb »

drsim wrote: post #442
beachy wrote: post #411

This is extremely poor for a large SaaS company. A standard RFP question for SaaS should be: - Can you restore data for a single customer, and if so, what is the RTO for that operation? A smaller SaaS could be excused for only thinking about full database restores. When you're a scrappy upstart, thinking about hypotheticals is less important than survival. But for any decent size multi-tenanted SaaS, it's imperative…

Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.

I did. It was an first-principles architectural decision. A client could request any point-in-time within the contracted period, and it could be either a restoration or a fully operational, parallel instance of the account.

It was initially a cover-my-own-ass design, but it turned out to be an extremely popular feature that was never even used for disaster recovery. Instead, it was used for audit support, trial scenarios, projections, and all kinds of other stuff.

Re: Inside the longest Atlassian outage

#479

Post by encryptluks2 »

miketria wrote: post #396
bluedino wrote: post #145

Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had over almost 40 hours. At the end of the second day, it was decided to restore from backup. We had done this before, as a test. It took about 12 hours to restore the data and another 12 hours to import the data and get back up and running. One small thing was diffe…

Hi, I'm Mike and I work in Engineering at Atlassian. Here's our approach to backup and data management: https://www.atlassian.com/trust/security/data-management - we certainly have the backups and have a restore process that we keep to. However, this incident stressed our ability to do this at scale, which has led to the very long times to restore.

You mean your poor practices and bad design. The only way to prevent this type of issue in the future is to admit the failures.

Re: Inside the longest Atlassian outage

#480

Post by syshum »

kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

sales never takes the blame. If anyone is fired it will be scapegoats in engineering once they have busted their ass to restore their reward will be the door
Post reply on HN