Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

601–610 of 772 posts

Re: Inside the longest Atlassian outage

#601

Post by MuffinFlavored »

sizzle wrote: post #515
miketria wrote: post #396

Earlier quoted context omitted.

Hi, I'm Mike and I work in Engineering at Atlassian. Here's our approach to backup and data management: https://www.atlassian.com/trust/security/data-management - we certainly have the backups and have a restore process that we keep to. However, this incident stressed our ability to do this at scale, which has led to the very long times to restore.

How’s the atmosphere internally Mike? Must be crazy times there. I know this isn’t your fault, so hang in there. Cheers!

Whose fault is it? Is it any one person/team’s fault? Management? Culture?

“Corporations are people too”

Re: Inside the longest Atlassian outage

#602

Post by jimbokun »

ibejoeb wrote: post #478
drsim wrote: post #442

Earlier quoted context omitted.

Which SaaS platforms provide account-level restores? If you contact them and say "please restore our data to as it was last week" those I know do not offer this.

I did. It was an first-principles architectural decision. A client could request any point-in-time within the contracted period, and it could be either a restoration or a fully operational, parallel instance of the account. It was initially a cover-my-own-ass design, but it turned out to be an extremely popular feature that was never even used for disaster recovery. Instead, it was used for audit support, trial scena…

What kind of data volumes were you dealing with?

Re: Inside the longest Atlassian outage

#603

Post by jtc331 »

Karrot_Kream wrote: post #537
syshum wrote: post #480

Earlier quoted context omitted.

sales never takes the blame. If anyone is fired it will be scapegoats in engineering once they have busted their ass to restore their reward will be the door

Why should sales take the blame when it's engineering's problem? If Sales promised a feature to a customer that was infeasible _then_ it should be Sales problem, but engineers made a mistake so only engineers can clean up the mistake.

You’re missing something huge. Sales almost certainly sold features that meant engineering couldn’t focus on things the engineers thought were valuable, like better automation.

Re: Inside the longest Atlassian outage

#604

Post by nijave »

iancarroll wrote: post #552
theteapot wrote: post #541

Earlier quoted context omitted.

Not having per tenant backups is sensible? Seems like a bit of an oversight. Doesn't really matter if the hosting is multi-tenant or not.

You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants; i.e. any cross-tenant primary keys which will have shifted since one tenant's last backup. Separating the backups wouldn't help with the restoration.

I can imagine plenty of issues but it's definitely a limitation in their design and I'd be surprised if that haven't ran into it before. Surely they've had a tenant destroy their instance and request a restore before.

They're in control of the architecture: rollback, backup, and recovery should all be considerations

Re: Inside the longest Atlassian outage

#605

Post by nijave »

brazzledazzle wrote: post #573
iancarroll wrote: post #552

Earlier quoted context omitted.

You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants; i.e. any cross-tenant primary keys which will have shifted since one tenant's last backup. Separating the backups wouldn't help with the restoration.

It is a complex problem but it’s one worth solving. Just spit balling but I think you could reduce some of the difficulty maintaining it by shifting it away from ops to development. Keep the disaster recovery level database backups for that rainy day but make customer level backup/restores an integrated feature developed and maintained like other services.

I wouldn't be surprised if a lot of the time was spent just waiting on the ops team to perform restores. If it's a manual/labor intense process, it's likely take them a while to work through the entire list

Re: Inside the longest Atlassian outage

#606

Post by wfme »

opensrcken wrote: post #540
kingofpandora wrote: post #458

Engineering mistakes happen. The most inexcusable thing is not communicating with the paying customers who have been affected for over a week. Atlassian's Global Head of Customer Success probably should have been fired but here she is promoting Atlassian Cloud on LinkedIn three days ago: https://www.linkedin.com/mwlite/in/gertie-rizzo-5b70061 Actually reading a bit more, it seems like their customer team was partying…

Unless you know, with certainty, who is responsible for customer-facing communications , how can you publicly call on a specific person to be fired? Even if you did know, and are correct in your implication, how do you know that she didn't want to do the right thing but was overruled by people above her in the chain? Overall, this is a pretty irresponsible comment, and reflects the kind of mob mentality that I would…

Completely agree. Can we please keep the pitchforks on Reddit. It's so incredibly easily to point the finger when you really have no idea. I'm all for calling out their lack of communication, but picking someone you deem to be responsible and calling for their head is pretty medieval.

Re: Inside the longest Atlassian outage

#608

Post by astroalex »

420official wrote: post #257
yabones wrote: post #215

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

I'm not at all familiar but a tweet linked from the OP and written by the author plugs https://linear.app/

I recently started using Linear at my new job, and I am a big fan.

1. It's snappy. Moving between issues and views is a breeze.

2. The UI is extremely functional and consistent.

3. Great keyboard shortcuts!

Using Jira felt like having brain fog, and Linear is such a relief after that experience.

Re: Inside the longest Atlassian outage

#609

Post by JohnBooty »

iancarroll wrote: post #552
theteapot wrote: post #541

Earlier quoted context omitted.

Not having per tenant backups is sensible? Seems like a bit of an oversight. Doesn't really matter if the hosting is multi-tenant or not.

You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants; i.e. any cross-tenant primary keys which will have shifted since one tenant's last backup. Separating the backups wouldn't help with the restoration.

I don't have Postgres/MySQL scaling experience beyond wrangling some largeish monoliths with But,

    You can imagine problems restoring one 
    individual tenant's data to an otherwise 
    active database with many tenants

    any cross-tenant primary keys
Why would multiple tenants share a database? Sharing a database server, yes, but sharing databases and mingling primary keys and such?

That's such a recipe for disaster; giving each client their own database seems like the easiest win in the world.

But I'm not intimate with Atlassian products. Maybe they have some products where that's not practical for some reason.

Re: Inside the longest Atlassian outage

#610

Post by atlasgone »

miketria wrote: post #396
bluedino wrote: post #145

Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had over almost 40 hours. At the end of the second day, it was decided to restore from backup. We had done this before, as a test. It took about 12 hours to restore the data and another 12 hours to import the data and get back up and running. One small thing was diffe…

Hi, I'm Mike and I work in Engineering at Atlassian. Here's our approach to backup and data management: https://www.atlassian.com/trust/security/data-management - we certainly have the backups and have a restore process that we keep to. However, this incident stressed our ability to do this at scale, which has led to the very long times to restore.

A friend in Atlassian engineering said the numbers on the trust site are closer to wishful thinking than actual capabilities, and that there has been an engineering wide disaster recovery project running because things were in such bad shape. The recovery part hasn't even started. If Atlassian could actually restore full products in under six hours, they should have been able to restore a second copy of the products exclusively for the impacted customers.
Post reply on HN