Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

211–220 of 772 posts

Re: Inside the longest Atlassian outage

#211

Earlier quoted context omitted.

I feel you here, but I've been at multiple companies that used JIRA and never once had any of those requirements. I've also never seen it come up when deciding which ticketing system to use. Teams have always been free to move tickets at-will.

One very large video game studio has tons of automation for Jira. Imagine someone deciding to add new weapon. The automation creates 100s of tasks for concept artists, 3d artists, animators, sound artists, software developers with complex dependencies better those. Most importantly, automation creates multiple QA steps for each element of completed work. The same exists for levels, enemies, quests and tons of other e…

See, that is great. Automate what can logically be deduced from the information available and set up templates to provide that information. For developers, it should be automated enough you shouldn't have to write the same info twice, once in commit messages/merges/branch names, once in the ticket itself. If the workflow is so streamlined, all that information can be deduced and the ticket can be advanced automatically. Most information is available and documented for other parties.

However, that's just not what most people go through in companies using JIRA. Worse, they have to toggle between pages multiple times, each taking at least a few decent seconds to reload. I'd like to give JIRA the benefit of the doubt here, but it sounds like the tool is just very easy to misconfigure and abuse.

Re: Inside the longest Atlassian outage

#212

Earlier quoted context omitted.

JIRA is a framework for making assembly lines out of knowledge workers. When you're a middle manager at a decent sized company, a major problem you face is that the mass of knowledge workers beneath you are opaque : you have no way of knowing whether they're working or not. Another problem you face is that they're uppity : people who went to college and got used to managing their own time now have all kinds of idiosy…

It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates. However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system that manages the data about the work that has been completed, is being worked, and is planned to be worked on . Part of that system might be defining…

>ensure the agreed upon process is followed for quality or consistency

That is what I mean here by "assembly line" and "control." Making sure that processes lead and individuals follow.

Citing consistency as a terminal value in the same breath as quality is also exactly what I mean by the middle-manager aversion to local differences.

Re: Inside the longest Atlassian outage

#213

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

And yet, it actually is set up in an extremely opinionated annoying way. For example there is no way to actually assign multiple users to the same ticket, which is a big problem if your org legitimately does pair programming (mine does for juniors)

Re: Inside the longest Atlassian outage

#214
post #176
post #87

Earlier quoted context omitted.

Wait. Why? This sounds like something that feels hard, if you are used to the giant DBs of old. But you can probably get many many instances of the smaller databases without much trouble. Would still be some maintenance, don't get me wrong. But far from impossible.

Imagine the database schema migrations...

the good news is by the time you get to the 100th client, you'll likely have run into all possible bugs and the remaining 6900 will be pretty smooth.

Re: Inside the longest Atlassian outage

#215
What's a good Jira replacement? Redmine? Phabricator? OpenProject? Just leaving the jira server alone and hoping there's no new and exciting zero-days? One thing is clear, these guys are a bunch of cowboys who can't be trusted with any amount of data.

Re: Inside the longest Atlassian outage

#217

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

> Simple concepts aren't that simple at their scale

It's true that nothing is simple at scale, but it's important to note that simple concepts are the only concepts that work at scale.

Re: Inside the longest Atlassian outage

#218
post #150
post #142

We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is l…

You're assuming every team would have better uptime with in-house solutions I think many would have worse uptime even with more headcount

I find this argument to be totally bs these days.

If anything, a smaller company with smaller footprint and fewer total requirements is going to be more likely to manage a vertical slice of some SAAS product.

The reason things like github go down so often is because they are public/shared resources.

Re: Inside the longest Atlassian outage

#219

Earlier quoted context omitted.

A lot of companies have integrations to atlassian suite which might not be easy to shift from. Secondly, there are a lot of individual competitors to Jira, Confluence and Bitbucket but which competitor can offer all three under a single invoice? May be Microsoft, can't think of anyone else. Also for such an extended downtime the customers are entitled to a discount or a credit note which a lot of CXOs consider in the…

> May be Microsoft, Is there a Jira replacement/offering in the Microsoft 365 suite?

Azure DevOps has boards and tickets and whatnot, so probably that?

Re: Inside the longest Atlassian outage

#220
post #176
post #87

Earlier quoted context omitted.

Wait. Why? This sounds like something that feels hard, if you are used to the giant DBs of old. But you can probably get many many instances of the smaller databases without much trouble. Would still be some maintenance, don't get me wrong. But far from impossible.

Imagine the database schema migrations...

Having worked at shops that used this architecture it's really not that bad. Can you write the code to do one schema migration? Great, now you can do 1000. App server boots and runs the schema migrations, drops privs and launches the app. Now you've staved off your scaling issues from "how to have a db large enough to hold all our customer data" to "how to have a db large enough to hold our biggest customer's data." Much easier.
Post reply on HN