Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use one big server

specbranch.com

Post reply on HN
Print view View original

541–550 of 601 posts

Re: Use one big server

#541

Post by dalyons »

Aeolun wrote: post #517
dalyons wrote: post #441

Earlier quoted context omitted.

This is so painfully painfully true. I’ve seen in born out personally at three different companies so far. Premature splitting up is bad too, but I think the “just use one Postgres for everything” crowd really underestimate how bad it gets in practice at scale

Maybe it’s all a matter of perspective? I’ve seen the ‘split things everywhere’ thing go wrong a lot more times than the ‘one big database’ thing. So I prefer the latter, but I imagine that may be different for other people. Ultimately I think it’s mostly up to the quality of the team, not the technical choice.

I’ve seen splitting things go bad too. But less often and to a lesser degree of pain than mono dbs - a bad split is much easier to undo than monodb spaghetti.

However I think it’s “thou shall” rules like this blog post that force useless arguments. The reality is it depends, and you should be using your judgement, use the simplest thing (monodb) until it doesn’t work for you, then pursue splitting (or whatever). Just be aware of your problem domain, your likely max scale, and design for splitting the db sooner than you think before you’re stuck in mud.

And if you’re building something new in an already-at-scale company you should perhaps be starting with something like dynamo if it fits your usecase.

Re: Use one big server

#542

Post by jd_mongodb »

Aeolun wrote: post #509
bigtunacan wrote: post #450

Earlier quoted context omitted.

@ithrow, yeah I know he is clearly biased which is why I don't really agree with him. I do however think it would have helped me to start using/learning before I needed it since the paradigm is so foreign to the relational model that is now second nature.

DynamoDB (and Mongo) is nice, right up until you need those relations. I haven’t found a document oriented database that gives me the consistency guarantees of a RDBMS yet.

You must not have looked at MongoDB. We have been delivering fully consistent ACID transactions since 4.0 which shipped several years. Yes, Jepsen did find some issues with the initial release of ACID transactions and yes, we fixed those problems pretty rapidly.

Re: Use one big server

#543

Post by runjake »

rkagerer wrote: post #355
runjake wrote: post #349

Earlier quoted context omitted.

What does it matter, in this context? If it's about bare metal vs. virtual machines, know that Spectre affects virtual machines, too.

I think they are implying disabling them (if on) could squeeze you out a bit more performance.

Ah, I did not think of that angle, which is substantial. Thanks.

Re: Use one big server

#544

Post by Aeolun »

threeseed wrote: post #292
BenoitEssiambre wrote: post #226

Earlier quoted context omitted.

I'm glad this is becoming conventional wisdom. I used to argue this in these pages a few years ago and would get downvoted below the posts telling people to split everything into microservices separated by queues (although I suppose it's making me lose my competitive advantage when everyone else is building lean and mean infrastructure too). In my mind, reasons involve keeping transactional integrity, ACID compliance…

> I'm glad this is becoming conventional wisdom It's not though. You're just seeing the most popular opinion on HN. In reality it is nuanced like most real-world tech decisions are. Some use cases necessitate a distributed or sharded database, some work better with a single server and some are simply going to outsource the problem to some vendor.

> outsource the problem to some vendor

At least that way you can be certain of failure.

Re: Use one big server

#545

Post by branko_d »

throwfh80h82 wrote: post #406
endisneigh wrote: post #402

Earlier quoted context omitted.

Without details it's hard to really get anything from this. Could you share some?

I can share some. Had a similar experience as the parent comment. I do support "one big database" but it requires a dedicated db admin team to solve the tragedy of the commons problem. Say you have one big database. You have 300 engineers and 30-50 product managers shipping new features every day accountable to the C-Suite. They are all writing queries to retrieve the data they want. One more join, one more N+1 query…

Do you think it would make it better to have the tables hidden behind an API of views and stored procedures? Perhaps a small team of engineers maintaining that API would be be able to communicate effectively enough to avoid this "tragedy of commons" and balance the performance (and security!) needs of various clients?

Re: Use one big server

#546

Post by lazide »

GeorgeDewar wrote: post #466
lmm wrote: post #458

Earlier quoted context omitted.

> It's resource intensive - but so is being in a giant tarpit/morass. Agreed, but it means it's not really a viable option for digging yourself out of that hole if you're already in it. Most of the time if you're desperately trying to split up your database it's because you're already hitting performance issues. > Adding client query logging is cheaper and can be distributed. Right, but that only works if you've got…

If you are in the hole where you really cannot add load to your database server but want to log the queries, there is a technique called zero impact monitoring where you literally mirror the network traffic going to your database server, and use a separate server to reconstruct it into query logs. These logs identify the queries that are being run, and critically, who/what is running them. A past workplace of mine ha…

Excellent pointer! Do you happen to know of anything which can do this out of the box?

Re: Use one big server

#547

Post by sanderjd »

gwbas1c wrote: post #531
sanderjd wrote: post #425

I didn't want to write a top-level comment and I'm sure few people will see this, but I scrolled down very far in this thread and didn't see this point made anywhere: The article focuses almost entirely on technical questions, but the technical considerations are secondary; the reason so many organizations prefer cloud services, VMs, and containers is to manage the challenges of scaling organizationally, not technica…

I agree with almost everything you said; except that the article offers extremely valuable advice for small startups going the cloud / rented VM route: Yearly payments, or approaching a salesperson, can lead to much lower costs. (I should point out that yesterday, in Azure, I added a VM in a matter of seconds and it took all of 15 minutes to boot up and start running our code. My employer is far too small to have ded…

Yep. To be clear, I thought it was a great article with lots of great advice, just too focused on the technical aspects of cloud benefits, whereas I think the real value is organizational.

Re: Use one big server

#548

Post by sanderjd »

tablespoon wrote: post #465
sanderjd wrote: post #421

Earlier quoted context omitted.

I think this hits the nail right on the head, and it's the same criticism I have of and article itself: the framing is that you split up a database or use small vms or containers for performance reasons, but that's not the primary reason these things are useful; they are useful for people scaling first and foremost, and for technical scaling only secondarily. The tragedy of the commons with one big shared database is…

> Defining service boundaries and APIs with clarity around backwards compatibility is a good solution. Can't you do that with one big database? Every application gets an account that only gives it access to what it needs. Treat database tables as APIs: if you want access to someone else's, you have to negotiate to get it, so it's known who uses what. You don't have to have one account with access to everything that e…

It would be easier to create different databases to achieve the same thing. Those could be in the same database server, but clear boundaries is the key.

Re: Use one big server

#549

Post by sanderjd »

Wicher wrote: post #501
cerved wrote: post #471

Earlier quoted context omitted.

Schemas can be useful in this regard

Indeed! And functions with security definers can be useful here too. With those one can define a very strict and narrow API that way, with functions that write or query tables that users don't have any direct access to. Look at it as an API written in DB functions, rather than in HTTP request handlers. One can even have neat API versioning through, indeed, the schema, and give different users (or application accounts…

I think we're getting hung up on database server vs. database as conceptual entity. I think separation between the entities is more important (organizationally) and don't think it matters as much whether or not the server is shared.

Re: Use one big server

#550

Post by sanderjd »

Aeolun wrote: post #510
sanderjd wrote: post #421

Earlier quoted context omitted.

I think this hits the nail right on the head, and it's the same criticism I have of and article itself: the framing is that you split up a database or use small vms or containers for performance reasons, but that's not the primary reason these things are useful; they are useful for people scaling first and foremost, and for technical scaling only secondarily. The tragedy of the commons with one big shared database is…

> Teams not having the flexibility to evolve their own schemas because they have no idea who depends on them This sounds like a problem of testing and organization to me, not a problem with single big databases.

Sure. My point is that the organization problems are more difficult and interesting than the technical problems being discussed in the article and in most of the threads.
Post reply on HN