Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use one big server

specbranch.com

Post reply on HN
Print view View original

581–590 of 601 posts

Re: Use one big server

#581

Post by arinlen »

jd_mongodb wrote: post #562
arinlen wrote: post #560

Earlier quoted context omitted.

> Yes, Jepsen did find some issues with the initial release of ACID transactions (...) By "some issues" you mean lost data and violated causal by default. https://www.infoq.com/news/2020/05/Jepsen-MongoDB-4-2-6/ And no, the issues were not fixed. Jepsen pointed out that "the newer MongoDB 4.2.6 has more problems" including “retrocausal transactions”. https://jepsen.io/analyses/mongodb-4.2.6 From Jepsen's report: > Je…

Also read the addendum from the same report: 1 Updates 2020-05-26: MongoDB identified a bug in the transaction retry mechanism which they believe was responsible for the anomalies observed in this report; a patch is scheduled for 4.2.8.

> Also read the addendum from the same report:

Your initial claim was that these issues were addressed in 4.0.

Jepsen's report refutes your claim,and demonstrates MongoDB had serious reliability problems even in 4.2.6.

Frankly, your insistence in pulling the wool over everyone's eyes, specially on a topic that's easily verified, does not help built up trust on MongoDB

Re: Use one big server

#582

Post by pythonaut_16 »

banku_brougham wrote: post #461
throwfh80h82 wrote: post #406

Earlier quoted context omitted.

I can share some. Had a similar experience as the parent comment. I do support "one big database" but it requires a dedicated db admin team to solve the tragedy of the commons problem. Say you have one big database. You have 300 engineers and 30-50 product managers shipping new features every day accountable to the C-Suite. They are all writing queries to retrieve the data they want. One more join, one more N+1 query…

I've seen this too. I guess 50% of query load were jobs that got deprecated in the next quarterly baseline. It felt a system was needed to allocate query resource to teams, some kind of tradeable tokens that were scarce maybe, to incentivise more care and consciousness of the resource from the many users. What we did was have a few levels of priority managed by a central org. It resulted in a lot of churn and hectare…

Let me tell you about my new Blockchain CorpoCoin... (/s)

Re: Use one big server

#583

Post by porker »

rlpb wrote: post #55
porker wrote: post #23

I like One Big (virtual) Server until you come to software updates. At a current project we have one server running the website in production. It runs an old version of Centos, the web server, MySQL and Elasticsearch all on the one machine. No network RTTs when doing too many MySQL queries on each page - great! But when you want to upgrade one part of that stack... we end up cloning the server, upgrading it, testing…

You could just run system containers (eg. lxd) for each component, but still on one server. That gets you multiple "servers" for the purposes of upgrades, but without the rest of the paradigm shift that Docker requires.

Doesn't the container help with versioning the software inside it, but you're still tied to the host computer's operating system, and so when you upgrade that you have to test every single container to see if anything broke?

Whereas if running a VM you have a lot more OS upgrades to do, but you can do them individually and they have no other impact?

This is the bit I've never understood with containers...

Re: Use one big server

#584

Post by redwood »

arinlen wrote: post #581
jd_mongodb wrote: post #562

Earlier quoted context omitted.

Also read the addendum from the same report: 1 Updates 2020-05-26: MongoDB identified a bug in the transaction retry mechanism which they believe was responsible for the anomalies observed in this report; a patch is scheduled for 4.2.8.

> Also read the addendum from the same report: Your initial claim was that these issues were addressed in 4.0. Jepsen's report refutes your claim,and demonstrates MongoDB had serious reliability problems even in 4.2.6. Frankly, your insistence in pulling the wool over everyone's eyes, specially on a topic that's easily verified, does not help built up trust on MongoDB

Another take https://arxiv.org/pdf/2111.14946.pdf

Re: Use one big server

#585

Post by spopejoy »

hnlmorg wrote: post #496
lelanthran wrote: post #483

Earlier quoted context omitted.

> The long term solutions end up being difficult to implement and can be high risk because now you have real customers (maybe not so happy because now slow db) and probably not much in house experience for dealing with such large scale data; and an absolute lack of ability to hire existing talent as the few people that really can solve for it are up to their ears in job offers. This is a problem of having succeeded b…

> This is a problem of having succeeded beyond your expectations, which is a problem only unicorns have. Nope. I've worked on a few projects that are not "unicorns" yet have legitimately hit that wall. Particularly around on line gaming and gambling.

Maybe unicorn is not the right word? If your app has millions of DAUs choking your DB, you should at least be tacking your next big investment round or some other success milestone.

Otherwise, your product is on it's way to failure, so good thing you did One Big DB...

Re: Use one big server

#586

Post by gfody »

cogman10 wrote: post #563
gfody wrote: post #561

Earlier quoted context omitted.

fwiw hacking hundreds of apps literally making them worse by fragmenting their source of record doesn't sound like a good plan. it's no surprise you have saboteurs, your company probably wants to survive and your plan is to shatter its brain. outside view: you should be trying to debottleneck your sql server if that's the plan the whole org can get behind. when they all want you to succeed you'll find a way.

> fwiw hacking hundreds of apps literally making them worse by fragmenting their source of record doesn't sound like a good plan. it's no surprise you have saboteurs, your company probably wants to survive and your plan is to shatter its brain. The brain is already shattered. This wouldn't "literally make them worse", instead it would say that "now instead of everyone in the world hitting the users table directly and…

assuming some beastly server with terabytes of ram, hundreds of fast cores, and an exotic io subsystem capable of ridiculous amounts of low latency iops, I'd guess the perf issue with that example is not sql server struggling with load but rather lock contention from the users table being heavily updated. unless that beast of a server is sitting pegged with a hardware bottleneck it can probably be debottlenecked by vertically partitioning the users table. ie: split the table into two (or more) to isolate the columns that change frequently from the ones that don't, replace the table with a view that joins it back together w/instead-of triggers conditionally updating the appropriate tables, etc. etc. then when this happens:

    SELECT b.*, u.username FROM Bar b JOIN users u ON b.userId = u.id
sql server sees that you're only selecting username from the users view and eliminates the joins for the more contentious tables and breathes easy peasy

> And why is this query doing that? To get a human readable username that isn't needed but at one point years ago made it nicer to debug the application.

imo users should be able to do this and whatever else they want and it's not even unreasonable to want usernames for debugging purposes forever. I'd expect the db team to support the requirements of the apps teams and wouldn't want to have to get data from different sources

Re: Use one big server

#587

Post by cogman10 »

gfody wrote: post #586
cogman10 wrote: post #563

Earlier quoted context omitted.

> fwiw hacking hundreds of apps literally making them worse by fragmenting their source of record doesn't sound like a good plan. it's no surprise you have saboteurs, your company probably wants to survive and your plan is to shatter its brain. The brain is already shattered. This wouldn't "literally make them worse", instead it would say that "now instead of everyone in the world hitting the users table directly and…

assuming some beastly server with terabytes of ram, hundreds of fast cores, and an exotic io subsystem capable of ridiculous amounts of low latency iops, I'd guess the perf issue with that example is not sql server struggling with load but rather lock contention from the users table being heavily updated. unless that beast of a server is sitting pegged with a hardware bottleneck it can probably be debottlenecked by v…

> assuming some beastly server with terabytes of ram, hundreds of fast cores, and an exotic io subsystem capable of ridiculous amounts of low latency iops, I'd guess the perf issue with that example is not sql server struggling with load but rather lock contention from the users table being heavily updated.

You'd guess wrong. The example above is not the only query our server runs. It's an example of some of the queries that can be run. We have a VERY complex relationship graph, far more than what you'll typically find. This is finance, after all.

I used the user example for something relatable without getting into the weeds of the domain.

We are particularly read heavy and write light. The issue is quiet literally that we have too many applications doing too many reads. We are literally running into problems where our tempDb can't keep up with the requests because there are too many of them doing too complex of work.

You are assuming we can just partition a table here or there and everything will just work swimmingly, that's simply not the case. Our tables do not so easily partition. (perhaps our users table would, but again, that was for illustrative purposes and by no means the most complex example).

Do you think that such a simple solution hasn't been explored by a team of 50 DBAs? Or that this sort of obvious problem wouldn't have been immediately fixed?

Re: Use one big server

#588

Post by gfody »

cogman10 wrote: post #587
gfody wrote: post #586

Earlier quoted context omitted.

assuming some beastly server with terabytes of ram, hundreds of fast cores, and an exotic io subsystem capable of ridiculous amounts of low latency iops, I'd guess the perf issue with that example is not sql server struggling with load but rather lock contention from the users table being heavily updated. unless that beast of a server is sitting pegged with a hardware bottleneck it can probably be debottlenecked by v…

> assuming some beastly server with terabytes of ram, hundreds of fast cores, and an exotic io subsystem capable of ridiculous amounts of low latency iops, I'd guess the perf issue with that example is not sql server struggling with load but rather lock contention from the users table being heavily updated. You'd guess wrong. The example above is not the only query our server runs. It's an example of some of the quer…

> Do you think that such a simple solution hasn't been explored by a team of 50 DBAs? Or that this sort of obvious problem wouldn't have been immediately fixed?

based on what you've shared, yeah. I also wouldn't expect a million DBAs to replace a single DBE

Re: Use one big server

#589

Post by reillyse »

gizzlon wrote: post #508
reillyse wrote: post #111

Earlier quoted context omitted.

Ok, so to your points. "It depends" is the correct answer to the question, but the least informative. One Big Server or multiple small servers? It depends. It always depends. There are many workloads where one big server is the perfect size. There are many workloads where many small servers are the perfect solution. What my point is, is that the ideas put forward in the article are flawed for the vast majority of use…

> All in I'm well under $200 per month including database. You forgot all the crucial numbers.. Like QPS.. My blog runs on 0 to 1 Cloud Run instances and costs < 3$ per month, including database

You should be using One Big Server mate :)

Re: Use one big server

#590

Post by jd_mongodb »

arinlen wrote: post #581
jd_mongodb wrote: post #562

Earlier quoted context omitted.

Also read the addendum from the same report: 1 Updates 2020-05-26: MongoDB identified a bug in the transaction retry mechanism which they believe was responsible for the anomalies observed in this report; a patch is scheduled for 4.2.8.

> Also read the addendum from the same report: Your initial claim was that these issues were addressed in 4.0. Jepsen's report refutes your claim,and demonstrates MongoDB had serious reliability problems even in 4.2.6. Frankly, your insistence in pulling the wool over everyone's eyes, specially on a topic that's easily verified, does not help built up trust on MongoDB

I can see the source of confusion. Apologies. I mentioned ACID transactions were released in 4.0 but did not explicitly mention when the problems arose which of course was in 4.2 which was actually released a year later. The version numbers are clearly referenced in the Jepsen article.
Post reply on HN