Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

251–260 of 772 posts

Re: Inside the longest Atlassian outage

#251
post #19

Earlier quoted context omitted.

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 cu…

I suppose that’s why you don’t combine a tazer and gun into 1 device with 2 triggers.

the problem is that sometimes that gun looks like a taser.

Re: Inside the longest Atlassian outage

#252

Earlier quoted context omitted.

Lawyers are involved, so I'd assume some text about "excluding acts of god, sabotage,etc" to weasel their way out of things. They might even be able to get away with "acts of incompetence" how ever a lawyer might phrase that to allow their client to weasel.

That's a good way to get executive approval to replace a system. Google or Apple can get away with this kind of behavior, I doubt Atlassian can. This outage alone has spurred conversations in slack about how terrible JIRA is and why we should replace it. If this kind of shit was pulled, I can guarantee we'd be on shortcut, linear, or something else in short order.

[deleted]

Re: Inside the longest Atlassian outage

#253

Earlier quoted context omitted.

Did you continue as their customer after that?

Nope. I exported our data after they restored the backup and then we cancelled less than a month later. Like I obviously understand suspending our logins, but why would you ever delete someone's data when it's literally only 160 KB of text? The whole thing made zero sense.

[deleted]

Re: Inside the longest Atlassian outage

#254
post #157

Earlier quoted context omitted.

Oh, nonsense. People buy Atlassisn because the licensing is cheap, not because it's particularly good at what it does or designed with any particular workflow in mind.

Cheaper than whatever is the open-source alternative?

Free software has zero acquisition cost, but non-zero TCO, which can measure in millions USD (recurring salary of dedicated IT team), depending on the size of organization and complexity of the setup. You will need to maintain on-premise infrastructure, automate backups and recovery, automate security, automate updates (including testing and rollbacks) etc etc, basically doing all the jobs of the people responsible for the infrastructure at the SaaS provider, but at much smaller scale and not achieving the same efficiency. You will have to do those jobs considerably better to justify the costs.

Re: Inside the longest Atlassian outage

#255

Earlier quoted context omitted.

It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates. However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system that manages the data about the work that has been completed, is being worked, and is planned to be worked on . Part of that system might be defining…

>ensure the agreed upon process is followed for quality or consistency That is what I mean here by "assembly line" and "control." Making sure that processes lead and individuals follow. Citing consistency as a terminal value in the same breath as quality is also exactly what I mean by the middle-manager aversion to local differences.

Beyond trivial scale, you need good processes so that individuals can do their jobs. If you have no processes, change and development becomes extremely difficult because people will be hunting for documentation all the time, stepping on each other's toes, and making mistakes that they should not be making because they forgot a trivial procedure that was a prerequisite to solving their actual problem.

I work with a variety of different environments, and depending on the environment I can either solve my problem in minutes and get it deployed in another few minutes or solve the problem in minutes and spend hours figuring out how to safely deploy it without breaking everything. JIRA is terrible if you do anything that it offers by default, but when used properly it can absolutely help with this.

Re: Inside the longest Atlassian outage

#256

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

I'm on the same boat. Currently best choice seems to be youtrack, which has reasonable licensing model for self hosted option.

Re: Inside the longest Atlassian outage

#257

What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

I'm not at all familiar but a tweet linked from the OP and written by the author plugs https://linear.app/

Re: Inside the longest Atlassian outage

#259
post #181

Earlier quoted context omitted.

Most likely the database tables themselves are just a mixture of everyone's data. There's no true multitenancy. So they have to load the backups into a separate database. Then just go through and individually select/insert into the old database. And then you have to worry about things like foreign key constraints complicating the bulk data loading. Are you going to disable constraint enforcement while you bulk load t…

Yup. The database schema of one of our products uses a tenant_id in most tables to separate customers logically. I've eventually gotten a tenant exporter to work. Practically, this requires some deep and nasty digging through the information_schema to build a graph of tables and foreign key constraints. Once it had that, it generates selects with a simple where clause for tables with the tenant_id, and selects with w…

[deleted]

Re: Inside the longest Atlassian outage

#260
PSA, because I’m seeing a lot of JIRA in this thread: Since the 2017 rebranding, Jira is no longer officially written in all caps: https://community.atlassian.com/t5/Feedback-Forum-articles/A...

(You can argue how successful it was when people are still using the old style in 2022).

It also makes more sense, since Jira is not an acronym, it’s a truncation of Gojira, inspired by Bugzilla/Mozilla.

Post reply on HN