Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use one big server

specbranch.com

Post reply on HN
Print view View original

451–460 of 601 posts

Re: Use one big server

#451

Post by ycombinete »

fleddr wrote: post #331

Our industry summarized: Hardware engineers are pushing the absolute physical limits of getting state (memory/storage) as close as possible to compute. A monumental accomplishment as impactful as the invention of agriculture and the industrial revolution. Software engineers: let's completely undo all that engineering by moving everything apart as far as possible. Hmmm, still too fast. Let's next add virtualization an…

Jonathan Blow has a talk about exactly this, called Preventing the Collapse of Civilisation []

[] https://www.youtube.com/watch?v=ZSRHeXYDLko

Re: Use one big server

#452

Post by lmm »

BenoitEssiambre wrote: post #226
cfors wrote: post #56

Yep, there's a premium on making your architecture more cloudy. However, the best point for Use One Big Server is not necessarily running your big monolithic API server, but your database. Use One Big Database. Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want t…

I'm glad this is becoming conventional wisdom. I used to argue this in these pages a few years ago and would get downvoted below the posts telling people to split everything into microservices separated by queues (although I suppose it's making me lose my competitive advantage when everyone else is building lean and mean infrastructure too). In my mind, reasons involve keeping transactional integrity, ACID compliance…

I've never understood this logic for webapps. If you're building a web application, congratulations, you're building a distributed system, you don't get a choice. You can't actually use transactional integrity or ACID compliance because you've got to send everything to and from your users via HTTP request/response. So you end up paying all the performance, scalability, flexibility, and especially reliability costs of an RDBMS, being careful about how much data you're storing, and getting zilch for it, because you end up building a system that's still last-write-wins and still loses user data whenever two users do anything at the same time (or you build your own transactional logic to solve that - exactly the same way as you would if you were using a distributed datastore).

Distributed systems can also make efficient use of cache, in fact they can do more of it because they have more of it by having more nodes. If you get your dataflow right then you'll have performance that's as good as a monolith on a tiny dataset but keep that performance as you scale up. Not only that, but you can perform a lot better than an ACID system ever could, because you can do things like asynchronously updating secondary indices after the data is committed. But most importantly you have easy failover from day 1, you have easy scaling from day 1, and you can just not worry about that and focus on your actual business problem.

Relational databases are largely a solution in search of a problem, at least for web systems. (They make sense as a reporting datastore to support ad-hoc exploratory queries, but there's never a good reason to use them for your live/"OLTP" data).

Re: Use one big server

#453

Post by lmm »

lazide wrote: post #445
lmm wrote: post #443

Earlier quoted context omitted.

> Even in your scenario you could identify schemas and tables that can be separated and moved into a different database or at maturity into a more scalable NoSQL variety. How? There's nothing tracking or reporting that (unless database management instrumentation has improved a lot recently), SQL queries aren't versioned or typechecked. Usually what happens is you move a table out and it seems fine, and then at the en…

Every database I know of can generate query logs. Why not just log every query and do some statistical analysis on it?

> Every database I know of can generate query logs.

Every one I know of warns that it comes with significant performance implications and isn't intended to be used in production.

> Why not just log every query and do some statistical analysis on it?

And then what? If you know this table only gets queried a few times a month, what does that actually tell you that you can use?

Re: Use one big server

#454

Post by chii »

makeitdouble wrote: post #422
throwfh80h82 wrote: post #406

Earlier quoted context omitted.

I can share some. Had a similar experience as the parent comment. I do support "one big database" but it requires a dedicated db admin team to solve the tragedy of the commons problem. Say you have one big database. You have 300 engineers and 30-50 product managers shipping new features every day accountable to the C-Suite. They are all writing queries to retrieve the data they want. One more join, one more N+1 query…

For whatever reason I've been thrown into a lot of companies at that exact moment when "hardware is cheap" and "not my problem" approaches couldn't cut it anymore... So yes, it's super painful, and requires a lot of change in processes, mindsets, and it's hard to get everyone to understand things will get slower from there. On the other end, micro-services and/or multi-DB is also super hard to get right. One of the s…

> teams that are good at split architectures are also usually good at monolith, and vice-versa.

aka, low-competency engineers will not outperform using better processes or project management.

The way, imho, is to up-skill the team (which is only possible if it was small unfortunately).

Re: Use one big server

#455

Post by lazide »

lmm wrote: post #453
lazide wrote: post #445

Earlier quoted context omitted.

Every database I know of can generate query logs. Why not just log every query and do some statistical analysis on it?

> Every database I know of can generate query logs. Every one I know of warns that it comes with significant performance implications and isn't intended to be used in production. > Why not just log every query and do some statistical analysis on it? And then what? If you know this table only gets queried a few times a month, what does that actually tell you that you can use?

It's resource intensive - but so is being in a giant tarpit/morass. Adding client query logging is cheaper and can be distributed. I just double checked, and neither Oracle nor Postgres warn 'never use it in production'

And if you have logs, you can see what actually gets queried, and by whom, and what doesn't get queried, and by whom.

That will also potentially let you start constructing views and moving actual underlying tables out of the way to where you can control them.

Which can let you untangle the giant spaghetti mess you're in.

But then, that's just me having actually done that a few times. You're welcome to complain about how it's actually unsolvable and will never get better, of course.

Re: Use one big server

#456

Post by zasdffaa »

altdataseller wrote: post #284
IggleSniggle wrote: post #276

Earlier quoted context omitted.

In regards to Elasticsearch, you basically opt-in to which behavior you want/need. You end up in the same place: potentially losing some data points or introducing some "fuzziness" to the results in exchange for speed. When you ask Elasticsearch to behave in a guaranteed atomic manner across all records, performing locks on data, you end up with similar constraints as in a RDBMS. Elasticsearch is for search. If you'r…

Having just an Elasticsearch index without also having the data in a primary store like a RDMS is an anti-pattern and not recommended by almost all experts. Whether you want to call it a “system of record”, i wont argue semantics. But the point is, its recommended hacing your data in a primary store where you can index into elasticsearch.

Have you a link for this? Never heard of this requirement (but not an elastic user so no surprise).

Re: Use one big server

#457

Post by conradev »

cogman10 wrote: post #342
cfors wrote: post #56

Yep, there's a premium on making your architecture more cloudy. However, the best point for Use One Big Server is not necessarily running your big monolithic API server, but your database. Use One Big Database. Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want t…

> Use One Big Database. > Seriously. If you are a backend engineer, nothing is worse than breaking up your data into self contained service databases, where everything is passed over Rest/RPC. Your product asks will consistently want to combine these data sources (they don't know how your distributed databases look, and oftentimes they really do not care). This works until it doesn't and then you land in the position…

Why can’t the databases handle the load? That is to say, did you see this coming from a while away or was it a surprise?

Re: Use one big server

#458

Post by lmm »

lazide wrote: post #455
lmm wrote: post #453

Earlier quoted context omitted.

> Every database I know of can generate query logs. Every one I know of warns that it comes with significant performance implications and isn't intended to be used in production. > Why not just log every query and do some statistical analysis on it? And then what? If you know this table only gets queried a few times a month, what does that actually tell you that you can use?

It's resource intensive - but so is being in a giant tarpit/morass. Adding client query logging is cheaper and can be distributed. I just double checked, and neither Oracle nor Postgres warn 'never use it in production' And if you have logs, you can see what actually gets queried, and by whom, and what doesn't get queried, and by whom. That will also potentially let you start constructing views and moving actual unde…

> It's resource intensive - but so is being in a giant tarpit/morass.

Agreed, but it means it's not really a viable option for digging yourself out of that hole if you're already in it. Most of the time if you're desperately trying to split up your database it's because you're already hitting performance issues.

> Adding client query logging is cheaper and can be distributed.

Right, but that only works if you've got a good handle on what all your clients are. If you've got a random critical script that you don't know about, client logging isn't going to catch that one's queries.

> But then, that's just me having actually done that a few times. You're welcome to complain about how it's actually unsolvable and will never get better, of course.

I've done it a few times too, it's always been a shitshow. Query logging is a useful tool to have in some cases but it's often not an option, and even when it is not a quick or easy fix. You're far better off not getting into that situation in the first place, by enforcing proper datastore ownership and scalable data models from the start, or at least from well before you start hitting the performance limits of your datastores.

Re: Use one big server

#459

Post by manigandham »

threeseed wrote: post #312
manigandham wrote: post #200

Earlier quoted context omitted.

Why would you break apart a microservice? Any why do you need to use/split into microservices anyway? 99% of apps are best fit as monolithic apps and databases and should focus on business value rather than scale they'll never see.

> 99% of apps are best fit as monolithic apps and databases and should focus on business value rather than scale they'll never see You incorrectly assume that 99% of apps are building these architectures for scalability reasons. When in reality it's far more for development productivity, security, use of third party services, different languages etc.

The vast majority is for (unreached) scalability reasons. Also monolithic app/db tend to have far higher productivity, and better security as well.
Post reply on HN