Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

391–400 of 772 posts

Re: Inside the longest Atlassian outage

#391

Post by crummy »

originalvichy wrote: post #221
kgeist wrote: post #142

We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is l…

I am biased but I can tell you what works best for mid-large companies: having a solution provider. Basically a partner that hosts and maintains the instance and has enough Atlassian certified people to help you with any question so that you will never have to hire people to just maintain the beasts or tell you about features, tricks or plugins that could solve problem X. Experienced people hosting and tuning Atlassi…

I can't see the difference between a "solution provider" that hosts your Jira and just getting Atlassian to do it. What's stopping the solution provider from accidentally running a script that deletes some customer's files and struggling to do a partial backup restore?

Re: Inside the longest Atlassian outage

#392

Post by notatoad »

a2800276 wrote: post #389
Alex3917 wrote: post #44

A few years ago we didn't renew our subscription on time because we got the email over Christmas break, and iirc they deleted all of our data in less than two weeks. They were eventually able to manually restore it from backups, but they restored it incorrectly so there was a bunch of stuff broken. This whole thing isn't even remotely surprising to me.

You can sleep soundly: it seems like they back _everything_ up: > Second, the script we used provided both the "mark for deletion" capability ... (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons . The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for app…

No, I don't think that's disturbing. That's the point of backups - even when something is permanently and completely erased in the production database, it's still in the backup. Eventually it will get rotated out as the backups expire.

Going back and purging things from the backups as part of the delete process would be overdoing it to a ridiculous degree.

Re: Inside the longest Atlassian outage

#393

Post by mountainriver »

caymanjim wrote: post #384
elesbao wrote: post #376

In a side note that someone else already made: it is interesting to see that many companies that uses JIRA also uses Slack but the noise/complaint/mentions comparing when Slack is down is way different. I barely saw people complaning.

I dunno about everyone else, but I'm generally frustrated and feel blocked when Slack is down, and I celebrate Jira being down because I've never had a pleasant experience using it. Jira is bureaucracy that gets in the way of me getting things done, and Slack is a critical communication path.

Yup Jira is bureaucracy incarnate. Middle managers love it though

Re: Inside the longest Atlassian outage

#394

Post by herpderperator »

Alex3917 wrote: post #141
jacquesm wrote: post #124

Earlier quoted context omitted.

Did you continue as their customer after that?

Nope. I exported our data after they restored the backup and then we cancelled less than a month later. Like I obviously understand suspending our logins, but why would you ever delete someone's data when it's literally only 160 KB of text? The whole thing made zero sense.

I don't think people write code saying "if accountSize < 160kB { skipDelete() }" - THAT would make zero sense. So, the size is not relevant here. The process was likely to delete data after some event occurred, or lack of event occurred.

Re: Inside the longest Atlassian outage

#395

Post by LadyCailin »

HeyLaughingBoy wrote: post #168
profmonocle wrote: post #3

Earlier quoted context omitted.

I actually got an email from our Atlassian contact just the other day encouraging us to switch to their cloud service. Crazy that no one thought to pause those. (I assume it must have been scheduled.)

This article on HN is the only time I've even heard that Atlassian was having a problem. I suspect that 99% of the tech "community" has absolutely no idea this is happening. We use Jira, but it's self-hosted for my team. Maybe other teams that have transitioned to the cloud version are aware that there's a problem, but I haven't heard about it.

Apparently the self hosted version goes out of support in 2024, so there will only be cloud hosting. Dumb dumb dumb.

Re: Inside the longest Atlassian outage

#396

Post by miketria »

bluedino wrote: post #145

Regarding the backup restores: I once worked a company that had a data loss issue. There was nothing else we could do, we had exhausted every option we had over almost 40 hours. At the end of the second day, it was decided to restore from backup. We had done this before, as a test. It took about 12 hours to restore the data and another 12 hours to import the data and get back up and running. One small thing was diffe…

Hi, I'm Mike and I work in Engineering at Atlassian. Here's our approach to backup and data management: https://www.atlassian.com/trust/security/data-management - we certainly have the backups and have a restore process that we keep to. However, this incident stressed our ability to do this at scale, which has led to the very long times to restore.

Re: Inside the longest Atlassian outage

#397

Post by lfpeb8b45ez »

judge2020 wrote: post #101
chupchap wrote: post #57

Earlier quoted context omitted.

A lot of companies have integrations to atlassian suite which might not be easy to shift from. Secondly, there are a lot of individual competitors to Jira, Confluence and Bitbucket but which competitor can offer all three under a single invoice? May be Microsoft, can't think of anyone else. Also for such an extended downtime the customers are entitled to a discount or a credit note which a lot of CXOs consider in the…

> May be Microsoft, Is there a Jira replacement/offering in the Microsoft 365 suite?

Azure DevOps is really underrated: https://www.thoughtworks.com/radar/platforms/azure-devops

Re: Inside the longest Atlassian outage

#398

Post by snark42 »

kgeist wrote: post #142

We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is l…

> but I don't understand why a medium/large business would choose anything but an on-premises setup.

Atlassian is in the process of killing the on-premise small/medium business option, already announced an EOL date.

Move to the cloud, buy a 500+ user solution for a much higher price or migrate away are my choices. Of course I use the local database and have local services JIRA/Confluence talk to so it's not really an option to move to the cloud.

I assume lack of competent on-site staff 24/7, having someone else to blame as well as lower costs are why people choose the cloud over on-premise though.

Re: Inside the longest Atlassian outage

#399

Post by nemo1618 »

Alex3917 wrote: post #141
jacquesm wrote: post #124

Earlier quoted context omitted.

Did you continue as their customer after that?

Nope. I exported our data after they restored the backup and then we cancelled less than a month later. Like I obviously understand suspending our logins, but why would you ever delete someone's data when it's literally only 160 KB of text? The whole thing made zero sense.

After I met my now-fiancée on OkCupid, I deactivated my profile, turned off notifications and forgot about it for a while. A while later, I thought it be nice to revisit the first messages we sent to each other, only to find that... OkCupid had deleted both of our accounts. They didn't give me any advance warning, either, because I turned off notifications, remember? :^)

I'm still kinda salty about it. I understand why big services can't retain data indefinitely, but like... it's just a few KB of text, and that text happens to have a lot of sentimental value. Besides, OkCupid knows that I deactivated my account because I am a success story -- why not hold onto those profiles a bit longer? Or better yet, how about emailing an archive of those messages immediately when you click the "I'm leaving because I'm in a happy relationship now" button? /rant

Re: Inside the longest Atlassian outage

#400

Post by adhesive_wombat »

upbeat_general wrote: post #386
elesbao wrote: post #376

In a side note that someone else already made: it is interesting to see that many companies that uses JIRA also uses Slack but the noise/complaint/mentions comparing when Slack is down is way different. I barely saw people complaning.

I don’t believe slack has been down as long? Slack is generally much more critical than JIRA in order to keep working.

It's it though? You can hop onto any of a constellation of other IM platforms, FOSS and not fairly quickly for an instant comms channel, even if you're missing the history. Having all your issue tickets missing is something you can't really deal with unless you have a very recent dump, and even then you can't just fire up Bugzilla and get something working without a lot of migration and administrative effort.

You can do without JIRA for a week or two as long as managers understand and you all have a good concept of what work needed doing anyway. Then it starts getting dicey unless someone becomes a human JIRA to connect temporary manual bug tracking systems with everyone involved.

Post reply on HN