Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

201–210 of 772 posts

Re: Inside the longest Atlassian outage

#201
post #15

They have recently killed off on premise offerings, it's cloud only now. And this makes it harder to trust both the security and integrity of your data.

Afaik, the Data Center option still allows for on-premise deployment, incl. Kubernetes and cloud deployments [1, 2, 3].

[1] https://www.atlassian.com/enterprise/data-center

[2] https://confluence.atlassian.com/enterprise/jira-data-center...

[3] https://confluence.atlassian.com/enterprise/deploying-enterp...

Re: Inside the longest Atlassian outage

#202

Earlier quoted context omitted.

JIRA is a framework for making assembly lines out of knowledge workers. When you're a middle manager at a decent sized company, a major problem you face is that the mass of knowledge workers beneath you are opaque : you have no way of knowing whether they're working or not. Another problem you face is that they're uppity : people who went to college and got used to managing their own time now have all kinds of idiosy…

It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates. However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system that manages the data about the work that has been completed, is being worked, and is planned to be worked on . Part of that system might be defining…

Agreed. This is a problem of scale in my opinion. When we have 10 engineers, it is easy to check in with everyone and know what they are working on and get a status update. When we have 500 engineers, making sure all their tasks are aligning (organizations are one big race condition) is not just hard but impossible without some sort of tracking system. We all want to grow big. To do so, your processes need to change as you add more people. The exceptions (Valve, Netflix, etc.) that can handle being flat or semi-flat are very unique.

Re: Inside the longest Atlassian outage

#203
post #150
post #142

We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is l…

You're assuming every team would have better uptime with in-house solutions I think many would have worse uptime even with more headcount

In our experience, this strongly depends on the services involved, as well as the scale.

For example, for our own service: If you have a hundred or two hundred licenses, you can drop our system on a linux box and usually you have to throw a yum update and one or two service restarts at it every few months and it just works. I honestly wouldn't be surprised if many of our small on-prem solutions have better uptime than the SaaS clusters, or be capped in uptime by some externality, rendering the system downtime irrelevant. If their VMWare cluster is down, our system is down, but no one cares.

This also mirrors a lot of our internal systems. At a small scale, you can just dump chef, jenkins, sonar, nexus, whatever on a linux box and forget about it.

However, this changes with high license counts. We have singular customers in our SaaS offering that are more than 50 - 100x bigger than the small on prem systems. At that point, our SaaS offering is better than anything the customer could to on-prem. I'm confident to say this about all of our customers, except maybe 2.

Re: Inside the longest Atlassian outage

#204

Earlier quoted context omitted.

How else do you run a multitenancy platform?

This. It would be an impossible nightmare for every account to have their own DB. Hundreds of thousands of accounts and databases....

I worked at company that architected their multi-tenancy in almost exactly this style. In their particular case, only a few of the very largest customers had their database set aside on their own dedicated instance, but every customer did have their own DB with their own set of tables. Having worked in that world (every customer had their own DB) and on a product where all customers had their data intermingled in one gigantic set of tables in one giant DB on one logical instance, I'd definitely encourage the "every customer gets their own DB".

Giving every customer their own table means you're going to need database administrators. For these folks their dedicated job was maintaining, operating, and changing their fleet of databases, but they where very technical and were amazing to work with.

Re: Inside the longest Atlassian outage

#206
post #19

Earlier quoted context omitted.

> The fact it's been so long and they still haven't revealed and explained the root cause of the outage They did last night: https://www.atlassian.com/engineering/april-2022-outage-upda...

> Faulty script. Second, the script we used provided both the "mark for deletion" capability used in normal day-to-day operations (where recoverability is desirable), and the "permanently delete" capability that is required to permanently remove data when required for compliance reasons. The script was executed with the wrong execution mode and the wrong list of IDs. The result was that sites for approximately 400 cu…

If coding is theatrical then ops is operatic. You have to telegraph stuff so over the top that the people in the cheap seats know what’s going on.

I think what we’ve lost in the post-XP world is that just because you build something incrementally doesn’t mean it’s designed incrementally (read: myopically).

My idiot coworkers are “fixing” redundancy issues by adding caching, which recreates the same problem they’re (un?)knowingly trying to avoid, which is having to iterate over things twice to accomplish anything. They’ve just moved the conditional branches to the cache and added more.

Most of the time, and especially on a concurrent system, you are better off building a plan of action first and then executing it second. You can dedupe while assembling the plan (dynamic programming) and you don’t have to worry about weird eviction issues dropping you into a logic problem like an infinite loop.

More importantly, you can build the plan and then explain the plan. You can explain the plan without running it. You can abort the plan in the middle when you realize you’ve clicked the wrong button. And you can clean up on abort because the plan is not twelve levels deep in a recursive call, where trying to clean up will have bugs you don’t see in a Dev sandbox.

    Deleting 500 users…
Versus

    Permanently deleting 500 users…
Maybe with a nice 10 second pause (what’s an extra ten seconds for a task that takes five minutes?)

Re: Inside the longest Atlassian outage

#207

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

In a lot of ways, JIRA disrupted Remedy Action Request System, which had a painful transition from X to Windows client. Remedy was even more admin dependent and unwieldy.

Re: Inside the longest Atlassian outage

#209

Earlier quoted context omitted.

A lot of companies have integrations to atlassian suite which might not be easy to shift from. Secondly, there are a lot of individual competitors to Jira, Confluence and Bitbucket but which competitor can offer all three under a single invoice? May be Microsoft, can't think of anyone else. Also for such an extended downtime the customers are entitled to a discount or a credit note which a lot of CXOs consider in the…

> May be Microsoft, Is there a Jira replacement/offering in the Microsoft 365 suite?

GitHub / GitHub Enterprise?

Re: Inside the longest Atlassian outage

#210

Earlier quoted context omitted.

A lot of companies have integrations to atlassian suite which might not be easy to shift from. Secondly, there are a lot of individual competitors to Jira, Confluence and Bitbucket but which competitor can offer all three under a single invoice? May be Microsoft, can't think of anyone else. Also for such an extended downtime the customers are entitled to a discount or a credit note which a lot of CXOs consider in the…

> May be Microsoft, Is there a Jira replacement/offering in the Microsoft 365 suite?

visual studio online is what it was called internally, the marketing may have changed. It's okay, and is what was/probably still is used at MS internally to develop windows.
Post reply on HN