Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Inside the longest Atlassian outage

newsletter.pragmaticengineer.com

Post reply on HN
Print view View original

351–360 of 772 posts

Re: Inside the longest Atlassian outage

#351

Post by lalaithion »

kubami wrote: post #314
oauea wrote: post #270

Earlier quoted context omitted.

You'll quickly run into limitations of how many tcp connections you can hold open. Unless you also want to run separate app servers for each customer, which will cost a lot of $$$ Oh, and just forget about allowing your customers to share their data with each other, which most enterprises want in one way or another.

Wait. What? None of the enterprise customers want to share data with each other. And definitely not on a DB level. That should happen in the business logic.

Lots of companies have consultants, and want to be able to share their consulting-related tickets with their consultants. And the consultants want one system they can log into and see the tickets from all of the companies that are hiring them.

Re: Inside the longest Atlassian outage

#352

Post by malermeister »

robertlagrant wrote: post #296
malermeister wrote: post #229

Earlier quoted context omitted.

> ensure the agreed upon process is followed for quality or consistency. Isn't that just a more corporate way of phrasing "control"?

Not in a negative way. You want to trust engineers to always have changes built and tested before they go to production, but when something egregious happens you need to go back and see what went wrong. You can choose to interpret that as control, but really the only alternative (often cited) is "Well that shouldn't ever happen, so you don't need tooling to support that situation". And that is not a useful way of thi…

I think the problem is that the processes are often not mutually agreed, but instead dictated by middle managers.

JIRA then becomes a tool for enforcing arbitrary rules, e.g. control

Re: Inside the longest Atlassian outage

#353

Post by seanwilson »

inopinatus wrote: post #290
seanwilson wrote: post #277

Earlier quoted context omitted.

> shard your customer databases 1:1 What are the downsides to this?

* makes it much harder to distribute your tables by any other factor, for whatever reason (usually performance, sometimes archival) * disaggregates data that the SaaS might be interested in querying/updating as an aggregate * not all ORM frameworks handle this case well, if at all * dumps are more than a single trivial command basically all your data operations gain an additional dimension of complexity, and you may…

Would it be fair to estimate that the majority of SaaS companies aren't sharding like this then? Seems like a lot of downsides that impact everything often except for backups, which you'd restore rarely.

Re: Inside the longest Atlassian outage

#354

Post by TheRealDunkirk »

closeparen wrote: post #136
throwaway894345 wrote: post #36

Earlier quoted context omitted.

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

JIRA is a framework for making assembly lines out of knowledge workers. When you're a middle manager at a decent sized company, a major problem you face is that the mass of knowledge workers beneath you are opaque : you have no way of knowing whether they're working or not. Another problem you face is that they're uppity : people who went to college and got used to managing their own time now have all kinds of idiosy…

Even worse, companies with the resources to buy JIRA will probably hire consultants to set it up, and you wind up with a system 1) bought by people who don't understand how programmers work, 2) configured by people who don't know how your company works. So end users usually wind up with a terrible system that continually generates complaints (along MANY axes), and the people responsible for foisting it on them think they're just being difficult.

Re: Inside the longest Atlassian outage

#355

Post by shukantpal »

grog454 wrote: post #307
a4isms wrote: post #76

Earlier quoted context omitted.

They used to say of Microsoft Word, "Nobody uses more than 5% of its features, but every company uses a different 5%." The saying is apocryphal and unlikely to be accurate, but the shape of the thing its describing applies to almost every piece of enterprise software whether installed on-prem or SaaS. And as another comment points out, at Enterprise scale you can substitute "team" or "group" for customer. Every team…

>"Nobody uses more than 5% of its features, but every company uses a different 5%." >The saying is apocryphal and unlikely to be accurate Well its mathematically impossible to be accurate as soon as you have > 20 users.

False. If you have 100 features, there are nCr(100, 5) combinations of 5% features = 75287520.

Re: Inside the longest Atlassian outage

#356

Post by Phelinofist »

ordiel wrote: post #266

All I can say as an Attlassian Server products user is that the moment they say it was Cloud or nothing, I choose nothing. I much rather running Gittea on a raspberry pi that I CONTROL than having to have the impotence of doing nothing for more than a week. + having work at cloud companies and having been requested to "collect customer data" to hand it over to the government I would NEVER move critical pieces to anyo…

As I understood it is not "Cloud or Nothing" but "Cloud or Data Center" - is this wrong?

Re: Inside the longest Atlassian outage

#357

Post by danielovichdk »

richardw wrote: post #105
throwaway894345 wrote: post #36

Earlier quoted context omitted.

I would really like to understand who makes the decision to purchase JIRA. It's like the C++ of ticketing software--it does everything because no one wanted to sit down and think critically about the use cases and instead decided it would be easier to say "yes" to every single feature request. It definitely feels like whoever is buying JIRA is not on the team who is using it (maybe IT or finance) because it ticks the…

I’m just a user but totally happy with all our Atlassian apps. Confluence is a huge win across our multi-thousand person company and the best teams use it very well. I like the integration between Jira and Bitbucket. We don’t over complicate things and it works fine. It’s like my taste in wine. I don’t want an overdeveloped sense of taste where only a $400 bottle will do. I’m fine with what we have because the work i…

Good point.

Wine that costs 400$ is for fun.

You don't drink that professionally.

Re: Inside the longest Atlassian outage

#358

Post by hallway_monitor »

hinkley wrote: post #189
Alex3917 wrote: post #141

Earlier quoted context omitted.

Nope. I exported our data after they restored the backup and then we cancelled less than a month later. Like I obviously understand suspending our logins, but why would you ever delete someone's data when it's literally only 160 KB of text? The whole thing made zero sense.

Someone somewhere got a promotion sooner because they lowered the slope of a line a little bit.

Or some overzealous engineer said hey guys let's delete all data 7 days after an account is canceled. This is called over optimizing.

Re: Inside the longest Atlassian outage

#359

Post by kikki »

jjice wrote: post #336
kikki wrote: post #285

Earlier quoted context omitted.

Not quite the same but at Fandom (Wikia), every wiki has its own DB (over 300,000 wikis), and they are clustered across a bunch of servers (usually balanced by traffic). It works well - but we don't ever really need to query across databases. There's a bunch of logic around instance/db selection but that's about as complex as it gets.

Interesting architecture. From a design point of view, I like the idea of full isolation. From an infrastructure point of view I'm a little scared. I'd assume it's actually not that bad and there's a good way to manage the individual DBs and scale them individually. Really interested if you can share any details. Edit: I know each wiki is on a subdomain. Does each wiki also have it's own server?

There are _many_ databases on each server, last I checked there was around 8 servers (or: "clusters") - and we have it so the traffic is somewhat evenly distributed across each server. There are reasonable capacity limits, and when servers get full we spin up a new one and start accepting new wikis there. I am not in OPS, and they do a lot of work behind the scenes to make this all run smoothly - but from an eng perspective we rarely have issues with this at scale.

Some of this was open source before we unified all of our wiki products, which has a lot of the selection / db logic, at https://github.com/Wikia/app.

Re: Inside the longest Atlassian outage

#360

Post by tus666 »

h2odragon wrote: post #4

> However, if they [restore backups], while the impacted ~400 companies would get back all their data, everyone else would lose all data committed since that point OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production. Simple concepts aren't that simple at their scale, sure, but I suspect this is skimping details on some truly horrendous monolithic architec…

> OK, so you restore backups to a separate system, and selectively copy the stomped accounts data back to production

You don't think that's exactly what they are doing?

Post reply on HN