Live data from Hacker News

Use one big server

specbranch.com

511–520 of 601 posts

Re: Use one big server

#511

Earlier quoted context omitted.

> Use One Big Database. > Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care). This works until it doesn't and then you land in the position…

I don't know what's the complexity of your project, but more often than not the feeling of doom coming from hitting that wall is bigger than the actual effort it takes to solve it. People often feel they should have anticipated and avoid the scaling issues altogether, but moving from a single DB to master/replica model, and/or shards or other solutions is fairly doable, and it doesn't come with worse tradeoffs than i…

One nice compromise is to migrate to using read-only database connections for read tasks from the moment you upgrade from medium sized DB hardware to big hardware. Keep talking to the one big DB with both connections.

Then when you are looking at the cost of upgrading from big DB hardware to huge DB hardware, you've got another option available to compare cost-wise: a RW main instance and one more read-only replicas, where your monolith talks to both: read/write to the master and read-only to the replicas via a load balancer.

Re: Use one big server

#512

Earlier quoted context omitted.

> Use One Big Database. > Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care). This works until it doesn't and then you land in the position…

I don't know what's the complexity of your project, but more often than not the feeling of doom coming from hitting that wall is bigger than the actual effort it takes to solve it. People often feel they should have anticipated and avoid the scaling issues altogether, but moving from a single DB to master/replica model, and/or shards or other solutions is fairly doable, and it doesn't come with worse tradeoffs than i…

I don’t see anyone mentioning memcached which can reduce the load on the db tremendously.

Re: Use one big server

#513

Let's be clear here, everything you can do in a "cloudy" environment, you could do on big servers yourself - but at what engineering and human resource cost? Because that's something many - if not most - hardware and 'on-prem' infra focussed people seem to miss. While cloud might seem expensive, most of the times, humans will be even more expensive (unless you're in very niche markets like HPC)

You could also have those big servers in the cloud (I think this is what many are doing; I certainly have). That gives you a lot of the cloud services e.g. for monitoring, but you get to not have to scale horizontally or rebuild for serverless just yet. Works great for Kubernetes workloads, too – have a single super beefy node (i.e. single-node node pool) and target just your resource-heavy workload onto that node.

As far as costs are concerned, however, I've found that for medium+ sized orgs, cloud doesn't actually save money in the HR department, the HR spend just shifts to devops people, who tend to be expensive and you can't really leave those roles empty since then you'll likely get an ungovernable mess of unsecured resources that waste a huge ton of money and may expose you to GDPR fines and all sorts of nasty breaches.

If done right, you get a ton of execution speed. Engineers have a lot of flexibility in terms of the services they use (which they'd otherwise have to buy through processes that tend to be long and tedious), scale as needed when needed, shift work to the cloud provider, while the devops/governance/security people have some pretty neat tools to make sure all that's done in a safe and compliant manner. That tends to be worth it many times over for a lot of orgs, if done effectively with that aim, though it may not do much for companies with relatively stagnant or very simple products. If you want to reduce HR costs, cloud is probably not going to help much.

Re: Use one big server

#514
Design systems such that eventual completion/consistency is a core tenant.

When it gets too slow, improve only the parts that are currently the slowest.

Re: Use one big server

#515
post #452

Earlier quoted context omitted.

I've never understood this logic for webapps. If you're building a web application, congratulations, you're building a distributed system, you don't get a choice. You can't actually use transactional integrity or ACID compliance because you've got to send everything to and from your users via HTTP request/response. So you end up paying all the performance, scalability, flexibility, and especially reliability costs of…

I really don't understand how anything of what you wrote follows from the fact that you're building a web-app. Why do you lose user data when two users do anything at the same time? That has never happened to me with any RDBMS. And why would HTTP requests prevent me from using transactional logic? If a user issues a command such as "copy this data (a forum thread, or a Confluence page, or whatever) to a different pla…

> I really don't understand how anything of what you wrote follows from the fact that you're building a web-app. Why do you lose user data when two users do anything at the same time? That has never happened to me with any RDBMS.

> And why would HTTP requests prevent me from using transactional logic? If a user issues a command such as "copy this data (a forum thread, or a Confluence page, or whatever) to a different place" and that copy operation might actually involve a number of different tables, I can use a transaction and make sure that the action either succeeds fully or is rolled back in case of an error; no extra logic required.

Sure, if you can represent what the user wants to do as a "command" like that, that doesn't rely on a particular state of the world, then you're fine. Note that this is also exactly the case that an eventually consistent event-sourcing style system will handle fine.

The case where transactions would actually be useful is the case where a user wants to read something and modify something based on what they read. But you can't possibly do that over the web, because they read the data in one request and write it in another request that may never come. If two people try to edit the same wiki page at the same time, either one of them loses their data, or you implement some kind of "userspace" reconciliation logic - but database transactions can't help you with that. If one user tries to make a new post in a forum thread at the same time as another user deletes that thread, probably they get an error that throws away all their data, because storing it would break referential integrity.

Re: Use one big server

#516
post #212

Last year I did some consulting for a client using Google cloud services such as Spanner and cloud storage. Storing and indexing mostly timeseries data with a custom index for specific types of queries. It was difficult for them to define a schema to handle the write bandwidth needed for their ingestion. In particular it required a careful hashing scheme to balance load across shards of the various tables. (It seems…

I find disk io to be a primary reason to go with bare metal. The vm abstractions just kill io performance. In a single server you can fill up the PCI lanes with flash and hit some ridiculous throughput numbers.

Re: Use one big server

#517

Earlier quoted context omitted.

I can share some. Had a similar experience as the parent comment. I do support "one big database" but it requires a dedicated db admin team to solve the tragedy of the commons problem. Say you have one big database. You have 300 engineers and 30-50 product managers shipping new features every day accountable to the C-Suite. They are all writing queries to retrieve the data they want. One more join, one more N+1 query…

This is so painfully painfully true. I’ve seen in born out personally at three different companies so far. Premature splitting up is bad too, but I think the “just use one Postgres for everything” crowd really underestimate how bad it gets in practice at scale

Maybe it’s all a matter of perspective? I’ve seen the ‘split things everywhere’ thing go wrong a lot more times than the ‘one big database’ thing. So I prefer the latter, but I imagine that may be different for other people.

Ultimately I think it’s mostly up to the quality of the team, not the technical choice.

Re: Use one big server

#518
post #331

Our industry summarized: Hardware engineers are pushing the absolute physical limits of getting state (memory/storage) as close as possible to compute. A monumental accomplishment as impactful as the invention of agriculture and the industrial revolution. Software engineers: let's completely undo all that engineering by moving everything apart as far as possible. Hmmm, still too fast. Let's next add virtualization an…

Agreed and I think it's easier to compare tech to the movie industry. Just look at all the crappy movies they produce with IMDB ratings below 5 out of 10, that is movies that nobody's going to even watch; then there are the shitty blockbusters with expensive marketing and greatly simplified stories optimized for mindless blockbuster movie goers; then there are rare gems, true works of art that get recognized at festivals at best but usually not by the masses. The state of the movie industry is overall pathetic, and I see parallels with the tech here.

Re: Use one big server

#519
post #515

Earlier quoted context omitted.

I really don't understand how anything of what you wrote follows from the fact that you're building a web-app. Why do you lose user data when two users do anything at the same time? That has never happened to me with any RDBMS. And why would HTTP requests prevent me from using transactional logic? If a user issues a command such as "copy this data (a forum thread, or a Confluence page, or whatever) to a different pla…

> I really don't understand how anything of what you wrote follows from the fact that you're building a web-app. Why do you lose user data when two users do anything at the same time? That has never happened to me with any RDBMS. > And why would HTTP requests prevent me from using transactional logic? If a user issues a command such as "copy this data (a forum thread, or a Confluence page, or whatever) to a different…

> Sure, if you can represent what the user wants to do as a "command" like that, that doesn't rely on a particular state of the world, then you're fine. Note that this is also exactly the case that an eventually consistent event-sourcing style system will handle fine.

Yes, but the event-sourcing system (or similar variants, such as CRDTs) is much more complex. It's true that it buys you some things (like the ability to roll back to specific versions), but you have to ask yourself whether you really need that for a specific piece of data.

(And even if you use event sourcing, if you have many events, you probably won't want to replay all of them, so you'll maybe want to store the result in a database, in which case you can choose a relational one.)

> If two people try to edit the same wiki page at the same time, either one of them loses their data, or you implement some kind of "userspace" reconciliation logic - but database transactions can't help you with that.

Yes, but

a) that's simply not a problem in all situations. People will generally not update their user profile concurrently with other users, for example. So it only applies to situations where data is truly shared across multiple users, and it doesn't make sense to build a complex system only for these use cases,

b) the problem of users overwriting other users' data is inherent to the problem domain; you will, in the end, have to decide which version is the most recent regardless of which technology you use. The one thing that evens etc. buy you is a version history (which btw can also be implemented with a RDBMS), but if you want to expose that in the UI so the user can go back, you have to do additional work anyway - it doesn't come for free.

c) Meanwhile, the RDBMS will at least guarantee that the data is always in a consistent state. Users overwriting other users' data is unfortunate, but corrupted data is worse.

d) You can solve the "concurrent modification" issue in a variety of ways, depending on the frequency of the problem, without having to implement a complex event-sourced system. For example, a lock mechanism is fairly easy to implement and useful in many cases. You could also, for example, hash the contents of what the user is seeing and reject the change if there is a mismatch with the current state (I've never tried it, but it should work in theory).

I don't wish to claim that a relational database solves all transactionality (and consistency) problems, but they certainly solve some of them - so throwing them out because of that is a bit like "tests don't find all bugs, so we don't write them anymore".

Re: Use one big server

#520
post #131
post #84

I didn’t see a point of cloudy services being easier to manage. If some team gets a capital budget to buy that one big server, they will put every thing on it, no matter your architectural standards. Cron jobs editing state on disk, tmux sessions shared between teams, random web servers doing who knows what, non-DBA team Postgres installs, etc. at least in cloud you can limit certain features and do charge back calcu…

One of our projects uses 1 big server and indeed, everyone started putting everything on it (because it's powerful): the project itself, a bunch of corporate sites, a code review tool, and god knows what else. Last week we started having issues with the projects going down because something is overloading the system and they still can't find out what exactly without stopping services/moving them to a different machin…

I believe that the "one big server" is intended for an application rather than trying to run 500 applications.

Does your application run on a single server? If yes. Don't use a distributed system for it's architecture or design. Simply buy bigger hardware when necessary. Because the top end of servers are insanely big and fast.

It does not mean, IMHO, throw everything on a single system without suitable organization, oversight, isolation, and recovery plans.

Post reply on HN