Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

711–720 of 772 posts

Re: Inside the longest Atlassian outage

#711

Post by lelanthran »

buscoquadnary wrote: post #489
systemvoltage wrote: post #486

Earlier quoted context omitted.

This is an engineering problem. They should own it and improve things, make sure it doesn't happen again. Also, GP's quote > Engineering mistakes happen. I don't like this statement because it offers consolation at the expense of unintentional normalization.

And coders that say all code has bugs are just defeatists that are trying to make excuses for being lazy. Sometimes manure will always hit the fan. Being robust means being able to handle that.

> Sometimes manure will always hit the fan. Being robust means being able to handle that.

You're never going to get perfect error handling in any non-trivial system.

Being robust means that you plan for particular states (like "deleting the production data"). That doesn't mean that your plan is any good, or that your plan will fix the problem, only that you have a sequence of steps developed in advance of the problem.

Sometimes the state in question is considered too unlikely[1] to ever occur, so is ignored with the caveat "too unlikely", such as planning for the case when the company files for bankruptcy and all software needs to be sped up by a factor of two in order to halve computing costs.

Not all possible future states need to be accommodated for in the tech stack - that doesn't mean stack is not "robust".

[1] Or if likely, is such a large problem that all the other problems are irrelevant.

Re: Inside the longest Atlassian outage

#712

Post by Traster »

celim307 wrote: post #451
Traster wrote: post #313

I remember finding out one of the senior managers from my company ended up as head of software at Atlassian. It was at that point I was convinced Atlassian has no idea what the hell they're doing. I think this demonstrates the point nicely.

After this they might have to boomerang back to your company lol

It's funny you say that, because he actually did end up coming back to work for the company, but it was a while ago now. He doesn't even list Atlassian on his Linkedin, he just pretends he never left hte company

Re: Inside the longest Atlassian outage

#714

Post by originalvichy »

max599 wrote: post #701
originalvichy wrote: post #461

Earlier quoted context omitted.

Trust me, if someone could clone Jira and its functionality they would have done so already. Truth is that if you build one product for 20 years you have a giant lead in features. If all it took was having a Kanban board then Jira would have died years ago.

isn't it one of those "No one ever got fired for buying XXXXXX" type of situation? Maybe i'm wrong, but the impression I had of Jira is that just like using sharepoint for file storage, the C-level people want it because they were told that's what big enterprise are using. And if it doesn't fit the need of the company and everyone hates it, they just blame the employees or lack of training.

It’s just so flexible that tracking projects and working together is easier with it. Decades of feature requests have made it good for people who want everything made for them or people who want to customise.

Re: Inside the longest Atlassian outage

#715

Post by shp0ngle »

kgeist wrote: post #142

We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is l…

We migrated from Slack to self-hosted Mattermost so we avoid being down. (And I guess money.)

Mattermost is so much worse that the slowness and general issues are not worth it. And in the end it is more down than Slack ever was, because it has performance issues.

I am not sure if it is Mattermost fault or our fault; but my friend from other corporation has similar experience with it. But maybe in general just don't know how to host MM, I donno

Re: Inside the longest Atlassian outage

#716

Post by fishnchips »

fletchowns wrote: post #703
fishnchips wrote: post #499

Earlier quoted context omitted.

Funny enough, most of what we restored then was spam (ex gTape SRE, remember the outage).

Even though you are no longer there...I had a friend who recently had her gmail inbox mysteriously emptied, all emails seemingly permanently deleted. She paid for Google One to be able to talk to support, and they said that the data is gone. Do you know if there's a way to recover this data? She is quite heartbroken at all the attachments that she will never get to see again.

I honestly wouldn't know, I left 8 years ago.

Re: Inside the longest Atlassian outage

#717

Post by shp0ngle »

bastardoperator wrote: post #666
shp0ngle wrote: post #628

Earlier quoted context omitted.

> Atlassian - MUST DO BETTER. it’s not like people will stop using jira and confluence, lol they basically have a monopoly there

Nobody likes using this stuff though. I think the new github boards might give jira a run for their money given the cost... free.

Once a project gets to feature parity with atlassian, it will become atlassian.

Same with SAP. Once your ERP has same feature parity with SAP, it will become SAP.

JIRA is SAP of engineering... yeah why I haven't thought of that before.

Re: Inside the longest Atlassian outage

#718

Post by febeling »

Reading this piece is kinda boring. As usual, the root cause is a design defect in their backup-restore functionality. And it's at a complexity level any senior developer could have pointed out to be posing a fatal risk to the company.

My guess is many people new about the problem inside, but corporate taboos made it impossible to discuss. I'd bet a fortune on this being the case.

Re: Inside the longest Atlassian outage

#719

Post by 2muchcoffeeman »

febeling wrote: post #718

Reading this piece is kinda boring. As usual, the root cause is a design defect in their backup-restore functionality. And it's at a complexity level any senior developer could have pointed out to be posing a fatal risk to the company. My guess is many people new about the problem inside, but corporate taboos made it impossible to discuss. I'd bet a fortune on this being the case.

It’s always good to know that these guys with their huge budgets are just doing what I’m doing and aren’t any better at it though.

Re: Inside the longest Atlassian outage

#720

Post by jtc331 »

JohnBooty wrote: post #609
iancarroll wrote: post #552

Earlier quoted context omitted.

You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants; i.e. any cross-tenant primary keys which will have shifted since one tenant's last backup. Separating the backups wouldn't help with the restoration.

I don't have Postgres/MySQL scaling experience beyond wrangling some largeish monoliths with But, You can imagine problems restoring one individual tenant's data to an otherwise active database with many tenants any cross-tenant primary keys Why would multiple tenants share a database? Sharing a database server, yes, but sharing databases and mingling primary keys and such? That's such a recipe for disaster; giving e…

It’s actually a real pain to handle “database per tenant”. Now for Postgres, for example, that’d mean database connections per tenant, which is wildly unscalable with lots of (particularly small) tenants per server.
Post reply on HN