Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use one big server

specbranch.com

Post reply on HN
Print view View original

441–450 of 601 posts

Re: Use one big server

#441

Post by dalyons »

throwfh80h82 wrote: post #406
endisneigh wrote: post #402

Earlier quoted context omitted.

Without details it's hard to really get anything from this. Could you share some?

I can share some. Had a similar experience as the parent comment. I do support "one big database" but it requires a dedicated db admin team to solve the tragedy of the commons problem. Say you have one big database. You have 300 engineers and 30-50 product managers shipping new features every day accountable to the C-Suite. They are all writing queries to retrieve the data they want. One more join, one more N+1 query…

This is so painfully painfully true. I’ve seen in born out personally at three different companies so far. Premature splitting up is bad too, but I think the “just use one Postgres for everything” crowd really underestimate how bad it gets in practice at scale

Re: Use one big server

#442

Post by cmrdporcupine »

Swizec wrote: post #273
BenoitEssiambre wrote: post #226

Earlier quoted context omitted.

I'm glad this is becoming conventional wisdom. I used to argue this in these pages a few years ago and would get downvoted below the posts telling people to split everything into microservices separated by queues (although I suppose it's making me lose my competitive advantage when everyone else is building lean and mean infrastructure too). In my mind, reasons involve keeping transactional integrity, ACID compliance…

> I'm glad this is becoming conventional wisdom My hunch is that computers caught up. Back in the early 2000's horizontal scaling was the only way. You simply couldn't handle even reasonably mediocre loads on a single machine. As computing becomes cheaper, horizontal scaling is starting to look more and more like unnecessary complexity for even surprisingly large/popular apps. I mean you can buy a consumer off-the-sh…

Honestly from my perspective it feels like microservices arose strongly in popularity precisely when it was becoming less necessary. In particular the mass adoption of SSD storage massively changed the nature of the game, but awareness of that among regular developers seemed not as pervasive as it should have been.

Re: Use one big server

#443

Post by lmm »

endisneigh wrote: post #412
throwfh80h82 wrote: post #406

Earlier quoted context omitted.

I can share some. Had a similar experience as the parent comment. I do support "one big database" but it requires a dedicated db admin team to solve the tragedy of the commons problem. Say you have one big database. You have 300 engineers and 30-50 product managers shipping new features every day accountable to the C-Suite. They are all writing queries to retrieve the data they want. One more join, one more N+1 query…

I do not disagree at all that what you are describing can happen. What I'm not understanding is why they're failing at multi year attempts to fix this. Even in your scenario you could identify schemas and tables that can be separated and moved into a different database or at maturity into a more scalable NoSQL variety. Generally once you get to the point that is being described that means you have a very strong sense…

> Even in your scenario you could identify schemas and tables that can be separated and moved into a different database or at maturity into a more scalable NoSQL variety.

How? There's nothing tracking or reporting that (unless database management instrumentation has improved a lot recently), SQL queries aren't versioned or typechecked. Usually what happens is you move a table out and it seems fine, and then at the end of the month it turns out the billing job script was joining on that table and now your invoices aren't getting sent out.

> Generally once you get to the point that is being described that means you have a very strong sense on the of queries you are making.

No, just the opposite; you have zillions of queries being run from all over the case and no idea what they all are, because you've taught everyone that everything's in this one big database and they can just query for whatever it is they need.

Re: Use one big server

#444

Post by ithrow »

bigtunacan wrote: post #431
Retric wrote: post #420

Earlier quoted context omitted.

The other side of this is once you actually can’t scale a single DB the project has proved it’s value and you have a solid idea what you actually want. Designing let alone building something scaleable on the other hand is a great way to waste extreme effort up front when it’s completely superfluous. That’s vastly more likely to actually kill a project than some growing pains especially when most projects never scale…

You're not wrong. Probably more than 95% of applications will never outgrow one large relational database. I just think that this leads to an unfortunate, but mostly inevitable issue of complexity for the few that do hit such a level of success and scale. Alex DeVrie (author of 'The DynamoDB Book') discusses that his approach is to essentially start all new projects with DynamoDB. Now I don't really agree with him, y…

Devrie works AWS ;)

Re: Use one big server

#445

Post by lazide »

lmm wrote: post #443
endisneigh wrote: post #412

Earlier quoted context omitted.

I do not disagree at all that what you are describing can happen. What I'm not understanding is why they're failing at multi year attempts to fix this. Even in your scenario you could identify schemas and tables that can be separated and moved into a different database or at maturity into a more scalable NoSQL variety. Generally once you get to the point that is being described that means you have a very strong sense…

> Even in your scenario you could identify schemas and tables that can be separated and moved into a different database or at maturity into a more scalable NoSQL variety. How? There's nothing tracking or reporting that (unless database management instrumentation has improved a lot recently), SQL queries aren't versioned or typechecked. Usually what happens is you move a table out and it seems fine, and then at the en…

Every database I know of can generate query logs. Why not just log every query and do some statistical analysis on it?

Re: Use one big server

#446

Post by klysm »

sagarm wrote: post #414
alexpotato wrote: post #262

My favorite summary of why not to use microservices is from Grug: "grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too seem very confusing to grug" https://grugbrain.dev/#grug-on-microservices

IMO microservices primarily solve organizational problems, not technical problems. They allow a team to release independently of other teams that have or want to make different risk/velocity tradeoffs. Also smaller units being released means fewer changes and likely fewer failed releases.

> Also smaller units being released means fewer changes and likely fewer failed releases.

The interfaces are the hard part, so you may have fewer internal failures but problems between services seem more likely.

Re: Use one big server

#447

Post by EddySchauHai »

pclmulqdq wrote: post #167
porker wrote: post #23

I like One Big (virtual) Server until you come to software updates. At a current project we have one server running the website in production. It runs an old version of Centos, the web server, MySQL and Elasticsearch all on the one machine. No network RTTs when doing too many MySQL queries on each page - great! But when you want to upgrade one part of that stack... we end up cloning the server, upgrading it, testing…

Containers are your friend here. The sysadmin tools that have grown out of the cloud era are actually really helpful if you don't cloud too much.

Yup! Docker is probably the greatest language-agnostic tool a developer can add to their toolbox.

Re: Use one big server

#448

Post by EddySchauHai »

hot_gril wrote: post #361

The way I code now after 10 years: Use one big file. No executable I'm capable of writing on my own is complex enough to need 50 files spread across a 3-layers-deep directory tree. Doesn't matter if it's a backend, a UI, or what. There's no way your React or whatever tutorial example code needs that either. And you don't gain any meaningful organization splitting into files when there are already namespaces, classes,…

I like simplicity but this sounds pretty awful if you work in a team - file structure can help with the domain design too.

Re: Use one big server

#449

Post by hazz99 »

icelancer wrote: post #369
maxbond wrote: post #195

Earlier quoted context omitted.

The thing that confuses me is, isn't every publicly accessible service bursty on a long timescale? Everything looks seasonal and predictable until you hit the front page of Reddit, and you don't know what day that will be. You don't decide how much traffic you get, the world does.

If you can't handle traffic from reddit or a larger site, you configured static pages and caching incorrectly, or you run your site on a Raspberry Pi, I guess.

Having your caching set up I correctly is verrryyy easy to do. There’s lots of things you can miss, and don’t realise until a whole lot of traffic hits you

Re: Use one big server

#450

Post by bigtunacan »

bigtunacan wrote: post #431
Retric wrote: post #420

Earlier quoted context omitted.

The other side of this is once you actually can’t scale a single DB the project has proved it’s value and you have a solid idea what you actually want. Designing let alone building something scaleable on the other hand is a great way to waste extreme effort up front when it’s completely superfluous. That’s vastly more likely to actually kill a project than some growing pains especially when most projects never scale…

You're not wrong. Probably more than 95% of applications will never outgrow one large relational database. I just think that this leads to an unfortunate, but mostly inevitable issue of complexity for the few that do hit such a level of success and scale. Alex DeVrie (author of 'The DynamoDB Book') discusses that his approach is to essentially start all new projects with DynamoDB. Now I don't really agree with him, y…

@ithrow, yeah I know he is clearly biased which is why I don't really agree with him. I do however think it would have helped me to start using/learning before I needed it since the paradigm is so foreign to the relational model that is now second nature.
Post reply on HN