Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

381–390 of 772 posts

Re: Inside the longest Atlassian outage

#381

Post by [deleted] »

grog454 wrote: post #307
a4isms wrote: post #76

Earlier quoted context omitted.

They used to say of Microsoft Word, "Nobody uses more than 5% of its features, but every company uses a different 5%." The saying is apocryphal and unlikely to be accurate, but the shape of the thing its describing applies to almost every piece of enterprise software whether installed on-prem or SaaS. And as another comment points out, at Enterprise scale you can substitute "team" or "group" for customer. Every team…

>"Nobody uses more than 5% of its features, but every company uses a different 5%." >The saying is apocryphal and unlikely to be accurate Well its mathematically impossible to be accurate as soon as you have > 20 users.

[deleted]

Re: Inside the longest Atlassian outage

#382

Post by chousuke »

nitinagg wrote: post #2

Selectively restoring data only for certain rows is super hard. But the communications by Atlassian has been the worst I have ever seen in the industry.

If the database schema for Jira on the cloud is anything like the Datacenter version, I'm not surprised they're having a hard time restoring data. I once tried to figure out how to find duplicate / redundant project schemas by querying the database (the required APIs are cloud-only) and could not even find which tables stored half the data, never mind how they referred to each other.

Re: Inside the longest Atlassian outage

#383

Post by callamdelaney »

Phelinofist wrote: post #356
ordiel wrote: post #266

All I can say as an Attlassian Server products user is that the moment they say it was Cloud or nothing, I choose nothing. I much rather running Gittea on a raspberry pi that I CONTROL than having to have the impotence of doing nothing for more than a week. + having work at cloud companies and having been requested to "collect customer data" to hand it over to the government I would NEVER move critical pieces to anyo…

As I understood it is not "Cloud or Nothing" but "Cloud or Data Center" - is this wrong?

Where do you think the cloud lives?

Re: Inside the longest Atlassian outage

#384

Post by caymanjim »

elesbao wrote: post #376

In a side note that someone else already made: it is interesting to see that many companies that uses JIRA also uses Slack but the noise/complaint/mentions comparing when Slack is down is way different. I barely saw people complaning.

I dunno about everyone else, but I'm generally frustrated and feel blocked when Slack is down, and I celebrate Jira being down because I've never had a pleasant experience using it. Jira is bureaucracy that gets in the way of me getting things done, and Slack is a critical communication path.

Re: Inside the longest Atlassian outage

#385

Post by yibg »

kgeist wrote: post #142

We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is l…

What do you do if your on prem setup lost data? There is an implicit assumption here that on prem is more reliable than cloud. Less downtime, less chances of data loss etc. Obviously it depends on which cloud product we're talking about but I don't think a blanket "my on prem goes down less and when it does go down I can get it back up sooner" is true.

Re: Inside the longest Atlassian outage

#386

Post by upbeat_general »

elesbao wrote: post #376

In a side note that someone else already made: it is interesting to see that many companies that uses JIRA also uses Slack but the noise/complaint/mentions comparing when Slack is down is way different. I barely saw people complaning.

I don’t believe slack has been down as long?

Slack is generally much more critical than JIRA in order to keep working.

Re: Inside the longest Atlassian outage

#387

Post by liamwire »

agalunar wrote: post #309
codycraven wrote: post #185

Earlier quoted context omitted.

It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates. However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system that manages the data about the work that has been completed, is being worked, and is planned to be worked on . Part of that system might be defining…

Caveat: this is going to be a meta comment rather than a comment about the topic proper, and so maybe not appropriate for HN, but I think it's worth discussing. > It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates. When we assume someone was hurt, and imply they hold an opinion only because they were hurt, we r…

I sincerely appreciate your articulation of this, thank you for taking the time.

Re: Inside the longest Atlassian outage

#388

Post by MajorBee »

mywittyname wrote: post #180
dylan604 wrote: post #64

Earlier quoted context omitted.

Lawyers are involved, so I'd assume some text about "excluding acts of god, sabotage,etc" to weasel their way out of things. They might even be able to get away with "acts of incompetence" how ever a lawyer might phrase that to allow their client to weasel.

That's a good way to get executive approval to replace a system. Google or Apple can get away with this kind of behavior, I doubt Atlassian can. This outage alone has spurred conversations in slack about how terrible JIRA is and why we should replace it. If this kind of shit was pulled, I can guarantee we'd be on shortcut, linear, or something else in short order.

> Google or Apple can get away with this kind of behavior, I doubt Atlassian can

Atlassian absolutely can in enterprise settings. In my company (a large cloud company), if JIRA goes down, large swathes of the business will also stall, including code deployment (deployments are tracked through change management JIRA tickets). We also use the DC version of Atlassian products, so presumably we aren't be at the mercy of Atlassian cloud engineers.

Re: Inside the longest Atlassian outage

#389

Post by a2800276 »

Alex3917 wrote: post #44

A few years ago we didn't renew our subscription on time because we got the email over Christmas break, and iirc they deleted all of our data in less than two weeks. They were eventually able to manually restore it from backups, but they restored it incorrectly so there was a bunch of stuff broken. This whole thing isn't even remotely surprising to me.

You can sleep soundly: it seems like they back _everything_ up:

> Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 customers were improperly deleted.

> To recover from this incident, our global engineering team has implemented a > methodical process for restoring our impacted customers.

[https://www.atlassian.com/engineering/april-2022-outage-upda...]

Anyone else find it disturbing that they are able to restore data that they deleted permanently for "compliance" reasons? If this is true, how were they ever compliant? I guess data is only permanently deleted when the engineering team is following their typical, non-methodical process...

Re: Inside the longest Atlassian outage

#390

Post by bpicolo »

inopinatus wrote: post #242
seanwilson wrote: post #236

Earlier quoted context omitted.

> Selectively restoring data only for certain rows is super hard. What's the right way to structure your data here that would make restoring more straightforward here? Is this backup/restore scenario niche or they should have designed for it?

in theory, shard your customer databases 1:1, job done. alas, in practice, many SaaS compromise this two ways: a) overwhelmed by creeping featuritis, each customer's data has relationships to global tables, and b) they backup their entire database cluster in one snapshot and there maybe other gotchas for restoration, like relying on denormalized views and caches that have to be rebuilt. they may also have erroneously…

Heck, it's worse now - if your data deletion tooling did a good job, there are dozens or hundreds of microservice databases to restore.
Post reply on HN