Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use one big server

specbranch.com

Post reply on HN
Print view View original

341–350 of 601 posts

Re: Use one big server

#341

Post by riku_iki »

lmm wrote: post #336
riku_iki wrote: post #329

Earlier quoted context omitted.

> Shard your datastore from day 1 what about using something like cocroach from day 1?

I don't know the characteristics of bikesheddb's upstream in detail (if there's ever a production-quality release of bikesheddb I'll take another look), but in general using something that can scale horizontally (like Cassandra or Riak, or even - for all its downsides - MongoDB) is a great approach - I guess it's a question of terminology whether you call that "sharding" or not. Personally I prefer that kind of datas…

> over an SQL database

it is actually distributed SQL Db with auto sharding, their goal is to be SQL compatible with Postgres.

Re: Use one big server

#342

Post by cogman10 »

cfors wrote: post #56

Yep, there's a premium on making your architecture more cloudy. However, the best point for Use One Big Server is not necessarily running your big monolithic API server, but your database. Use One Big Database. Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want t…

> Use One Big Database.

> Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care).

This works until it doesn't and then you land in the position my company finds itself in where our databases can't handle the load we generate. We can't get bigger or faster hardware because we are using the biggest and fastest hardware you can buy.

Distributed systems suck, sure, and they make querying cross systems a nightmare. However, by giving those aspects up, what you gain is the ability to add new services, features, etc without running into scotty yelling "She can't take much more of it!"

Once you get to that point, it becomes SUPER hard to start splitting things out. All the sudden you have 10000 "just a one off" queries against several domains that are broken by trying carve out a domain into a single owner.

Re: Use one big server

#343

Post by zmmmmm »

Joeri wrote: post #254
rbanffy wrote: post #150

Earlier quoted context omitted.

> Use One Big Database. I emphatically disagree. I've seen this evolve into tightly coupled microservices that could be deployed independently in theory, but required exquisite coordination to work. If you want them to be on a single server, that's fine, but having multiple databases or schemas will help enforce separation. And, if you need one single place for analytics, push changes to that space asynchronously. Ha…

I have done both models. My previous job we had a monolith on top of a 1200 table database. Now I work in an ecosystem of 400 microservices, most with their own database. What it fundamentally boils down to is that your org chart determines your architecture. We had a single team in charge of the monolith, and it was ok, and then we wanted to add teams and it broke down. On the microservices architecture, we have man…

> splitting off services when it becomes apparent they need to change at a different pace from the main body

yes - this seems to get lost, but the microservice argument is no different to the bigger picture software design in general. When things change independently, separate and decouple them. It works in code and so there is no reason it shouldn't apply at the infrastructure layer.

If I am responsible for the FooBar and need to update it once a week and know I am not going to break the FroggleBot or the Bazlibee which are run by separate teams who don't care about my needs and update their code once a year, hell yeah I want to develop and deploy it as a separate service.

Re: Use one big server

#344

Post by Foomf »

fleddr wrote: post #331

Our industry summarized: Hardware engineers are pushing the absolute physical limits of getting state (memory/storage) as close as possible to compute. A monumental accomplishment as impactful as the invention of agriculture and the industrial revolution. Software engineers: let's completely undo all that engineering by moving everything apart as far as possible. Hmmm, still too fast. Let's next add virtualization an…

You've more or less described Wirth's Law: https://en.wikipedia.org/wiki/Wirth%27s_law

Re: Use one big server

#345

Post by fleddr »

Foomf wrote: post #344
fleddr wrote: post #331

Our industry summarized: Hardware engineers are pushing the absolute physical limits of getting state (memory/storage) as close as possible to compute. A monumental accomplishment as impactful as the invention of agriculture and the industrial revolution. Software engineers: let's completely undo all that engineering by moving everything apart as far as possible. Hmmm, still too fast. Let's next add virtualization an…

You've more or less described Wirth's Law: https://en.wikipedia.org/wiki/Wirth%27s_law

I had no idea, thanks. Consider this a broken clock being sometimes right.

Re: Use one big server

#346

Post by zmmmmm »

marcosdumay wrote: post #192
rbanffy wrote: post #150

Earlier quoted context omitted.

> Use One Big Database. I emphatically disagree. I've seen this evolve into tightly coupled microservices that could be deployed independently in theory, but required exquisite coordination to work. If you want them to be on a single server, that's fine, but having multiple databases or schemas will help enforce separation. And, if you need one single place for analytics, push changes to that space asynchronously. Ha…

Yeah... Dividing your work into microservices while your data is in an interdependent database doesn't lead to great results. If you are creating microservices, you must segment them all the way through.

I have to say I disagree with this ... you can only separate them if they are really, truly independent. Trying to separate things that are actually coupled will quickly take you on a path to hell.

The problem here is that most of the microservice architecture divisions are going to be driven by Conway's law, not what makes any technical sense. So if you insist on separate databases per microservice, you're at high risk of ending up with massive amounts of duplicated and incoherent state models and half the work of the team devoted to synchronizing between them.

I quite like an architecture where services are split except the database, which is considered a service of its own.

Re: Use one big server

#347

Post by Flow »

cogman10 wrote: post #342
cfors wrote: post #56

Yep, there's a premium on making your architecture more cloudy. However, the best point for Use One Big Server is not necessarily running your big monolithic API server, but your database. Use One Big Database. Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want t…

> Use One Big Database. > Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care). This works until it doesn't and then you land in the position…

Do you have Spectre countermeasures active in the kernel of that machine?

Re: Use one big server

#348

Post by kedean »

cogman10 wrote: post #342
cfors wrote: post #56

Yep, there's a premium on making your architecture more cloudy. However, the best point for Use One Big Server is not necessarily running your big monolithic API server, but your database. Use One Big Database. Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want t…

> Use One Big Database. > Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care). This works until it doesn't and then you land in the position…

Many databases can be distributed horizontally if you put in the extra work, would that not solve the problems you're describing? MariaDB supports at least two forms of replication (one master/replica and one multi-master), for example, and if you're willing to shell out for a MaxScale license it's a breeze to load balance it and have automatic failover.

Re: Use one big server

#349

Post by runjake »

Flow wrote: post #347
cogman10 wrote: post #342

Earlier quoted context omitted.

> Use One Big Database. > Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care). This works until it doesn't and then you land in the position…

Do you have Spectre countermeasures active in the kernel of that machine?

What does it matter, in this context?

If it's about bare metal vs. virtual machines, know that Spectre affects virtual machines, too.

Re: Use one big server

#350

Post by GuB-42 »

fleddr wrote: post #331

Our industry summarized: Hardware engineers are pushing the absolute physical limits of getting state (memory/storage) as close as possible to compute. A monumental accomplishment as impactful as the invention of agriculture and the industrial revolution. Software engineers: let's completely undo all that engineering by moving everything apart as far as possible. Hmmm, still too fast. Let's next add virtualization an…

Actually, for those who push for these cloudy solutions, they do that in part to make data close to you. I am talking mostly about CDNs, I don't thing YouTube and Netflix would have been possible without them.

Google is a US company, but you don't want people in Australia to connect to the other side of the globe every time they need to access Google services, it would be an awful waste of intercontinental bandwidth. Instead, Google has data centers in Australia to serve people in Australia, and they only hit US servers when absolutely needed. And that's when you need to abstract things out. If something becomes relevant in Australia, move it in there, and move it out when it no longer matters. When something big happens, copy it everywhere, and replace the copies by something else as interest wanes.

Big companies need to split everything, they can't centralize because the world isn't centralized. The problem is when small businesses try to do the same because "if Google is so successful doing that, it must be right". Scale matters.

Post reply on HN