Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

221–230 of 772 posts

Re: Inside the longest Atlassian outage

#221

Post by originalvichy »

kgeist wrote: post #142

We use on-premises setups for almost everything (we generally avoid cloud solutions to have full control of our data), sometimes (approximately once a month) it goes down for a few minutes which already feels like a torture because all our processes depend on it, I can't imagine having no access to it for several weeks, all our work would stop to a halt... The office of the guy who administers on-premise servers is l…

I am biased but I can tell you what works best for mid-large companies: having a solution provider. Basically a partner that hosts and maintains the instance and has enough Atlassian certified people to help you with any question so that you will never have to hire people to just maintain the beasts or tell you about features, tricks or plugins that could solve problem X.

Experienced people hosting and tuning Atlassian products has a greater success rate than someone doing it alone for a large company. Almost every time I’ve migrated an old Atlassian installation under our wing it’s given me shock how users have been made to suffer the loading times and perfs that come from underprovisioning (db or actual machine) and messy configuration. I’m not blaming the former admins but it just happens. Usually end users are happy after we clean the mess up and everything feels snappy.

Disclosure: I’ve worked in this kind of expert role.

Re: Inside the longest Atlassian outage

#222

Post by truffdog »

radicaldreamer wrote: post #40
dabeeeenster wrote: post #33

Earlier quoted context omitted.

How else do you run a multitenancy platform?

Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?

Separate DB instances doesn't scale as well cost wise, and generally means onboarding takes a few minutes instead of being instant. It is very common though.

Re: Inside the longest Atlassian outage

#223

Post by more_corn »

h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

Yeah, I’m thinking the exact same thing.

Perhaps they don’t have the right people on hand to do hard things like this.

They also apparently lack an incident response plan since a critical component of that is coms to affected customers.

They also lack good practices around preventing human error. It should not have even been possible to make the initial mistake. It certainly should have involved multiple steps of “are you sure” and potentially even review.

Sounds like an operations shit show. Glad it’s not my circus.

Re: Inside the longest Atlassian outage

#224

Post by Spivak »

outworlder wrote: post #197
brightball wrote: post #98

Earlier quoted context omitted.

There are basically two options for multi-tenancy with their own tradeoffs. 1. An account/tenant_id field for each table 2. A schema for each tenant wrapping all of the tables Option 2 gives you cleaner separation but complicates your deployment process because now you have to run every database change across every schema every time you deploy. This gets more complicated as your code is deploying in case the code its…

Option 2 has many unforeseen consequences. Business wants to run a query across customers? In most DBs you need either custom code or to create a stored procedure to iterate across schemas. Every table that you create is multiplied by the number of customers. This has implications for some database systems (like PG's vacuum). Your migrations will take _forever_ to run. Etc.

The second problem is mitigated by the fact that schemas are trivially migratable between database servers. Once you grow too big for one cluster just make another.

Re: Inside the longest Atlassian outage

#225

Post by mistrial9 »

codycraven wrote: post #185
closeparen wrote: post #136

Earlier quoted context omitted.

JIRA is a framework for making assembly lines out of knowledge workers. When you're a middle manager at a decent sized company, a major problem you face is that the mass of knowledge workers beneath you are opaque : you have no way of knowing whether they're working or not. Another problem you face is that they're uppity : people who went to college and got used to managing their own time now have all kinds of idiosy…

It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates. However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system that manages the data about the work that has been completed, is being worked, and is planned to be worked on . Part of that system might be defining…

I was told by a lifetime manager turned successful consultant, that roughly fifty percent of engineering firms govern their engineers basically using fear.

Re: Inside the longest Atlassian outage

#226

Post by [deleted] »

throwaway894345 wrote: post #36
h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

[deleted]

Re: Inside the longest Atlassian outage

#227

Post by Spivak »

andyjohnson0 wrote: post #60
radicaldreamer wrote: post #40

Earlier quoted context omitted.

Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?

> Is it not a good idea to spin up separate db instances for each client/company? It depends, really. There is a trade-off in terms of software and operational complexity vs scalability/perf and isolation. And probably a bunch of other factors. If you have separate databases for each customer, schema migrations can be staged over time. But that means your software backend needs to be able to work with different schem…

> But that means your software backend needs to be able to work with different schemas concurrently.

Not if you're truly multi-tenant and each customer has their own app servers. Then your code and schema version are always in lock-step.

Re: Inside the longest Atlassian outage

#229

Post by malermeister »

codycraven wrote: post #185
closeparen wrote: post #136

Earlier quoted context omitted.

JIRA is a framework for making assembly lines out of knowledge workers. When you're a middle manager at a decent sized company, a major problem you face is that the mass of knowledge workers beneath you are opaque : you have no way of knowing whether they're working or not. Another problem you face is that they're uppity : people who went to college and got used to managing their own time now have all kinds of idiosy…

It sounds like you've been hurt by the some terrible management practices, I'm truly sorry that some managers think their job is to control their subordinates. However, regarding ticketing systems, in team environments, it is very effective and helpful to have a system that manages the data about the work that has been completed, is being worked, and is planned to be worked on . Part of that system might be defining…

> ensure the agreed upon process is followed for quality or consistency.

Isn't that just a more corporate way of phrasing "control"?

Re: Inside the longest Atlassian outage

#230

Post by Spivak »

truffdog wrote: post #222
radicaldreamer wrote: post #40

Earlier quoted context omitted.

Sorry, I'm not actually sure... maybe someone who's experienced in backend db can elucidate here. Is it not a good idea to spin up separate db instances for each client/company?

Separate DB instances doesn't scale as well cost wise, and generally means onboarding takes a few minutes instead of being instant. It is very common though.

The solution that satisfies everyone is having a separate schema per customer and a number of database clusters. Then each customer is assigned to a particular cluster. Always make sure you have excess capacity on your pool of clusters and onboarding is still instant.
Post reply on HN