Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres scaling advice

cybertec-postgresql.com

Post reply on HN
Print view View original

21–30 of 207 posts

Re: Postgres scaling advice

#21

Post by Turbots »

jjice wrote: post #13

In the opinion of a last semester CS student who has never written an application from scratch that needed more than a SQLite DB (so take me with a half grain of salt), it seems like premature optimization, while always talked about, is very common. I see people talking about using Kubernetes for internal applications and I just can't figure out why. If it's a hobby project and you want to learn Kubernetes, that's a…

For one application, kubernetes is obviously giant overkill. Companies with hundreds and thousands and applications need a platform that can offer standardised deployment pattern, runtime pattern and scaling pattern for those apps. Kubernetes is a great start for managing all those containers in a consistent, secure, multi tenant environment. I know customers with literally 10000s of VMs that are moving to more cloud like environments, either on premise in their own datacenter or in the public cloud. They need these kind of platforms to automate away the repetitive stuff and have a secure posture, throughout the company.

Re: Postgres scaling advice

#22

Post by rc_hackernews »

jjice wrote: post #13

In the opinion of a last semester CS student who has never written an application from scratch that needed more than a SQLite DB (so take me with a half grain of salt), it seems like premature optimization, while always talked about, is very common. I see people talking about using Kubernetes for internal applications and I just can't figure out why. If it's a hobby project and you want to learn Kubernetes, that's a…

I haven’t worked at a FAANG or any other company even close to that level of scale, so you can take me with half a grain of salt too.

But what you said is absolutely true. It’s also something you will very much experience once you start working professionally.

I’m in no position to give you advice, and I think I might be giving advice to myself...just don’t let it get to you.

Re: Postgres scaling advice

#23

Post by ComodoHacker »

It's not surprising to hear such advice from PostgreSQL consultancy shop: don't bother and relax until the day you desperately need PostgreSQL consultancy! /s

Seriously though, the main point stands. PostgreSQL isn't designed with "distributed" in mind, so try vertical scaling tricks first.

Re: Postgres scaling advice

#24

Post by kevincox »

cuu508 wrote: post #16
kevincox wrote: post #8

I wonder when using a distributed database (like CockroachDB) will be the default for new applications. Right now it seems that they are less feature and harder to set up than traditional RDBMSes but I can only assume that this gap will narrow and at some point in the future things will be "scalable by default". (Of course no DB is going to prevent all ways to shoot yourself in the foot)

To me it looks like there are not many affordable options right now. CockroachDB understandably wants you to use their Cloud or Enterprise products: the OSS version is quite limited. For example it doesn't support row-level partitioning ( https://www.cockroachlabs.com/docs/stable/configure-replicat... ). Which means, if I understand correctly, it is not of much help for scaling writes to a single big table.

Oh, I didn't realize that the free version was that limited. I guess I need another name to use as the default open source distributed database.

Re: Postgres scaling advice

#25

Post by h0l0cube »

gowld wrote: post #17
jjice wrote: post #13

In the opinion of a last semester CS student who has never written an application from scratch that needed more than a SQLite DB (so take me with a half grain of salt), it seems like premature optimization, while always talked about, is very common. I see people talking about using Kubernetes for internal applications and I just can't figure out why. If it's a hobby project and you want to learn Kubernetes, that's a…

Kubernetes is a container system, mostly orthogonal to 3- or 4-tier application design.

And typically a single DO droplet would suffice for a toy project or POC, for which Ansible is probably the more expedient option. But maybe they're not in a rush, and learning K8s is just another feather in their cap .

Re: Postgres scaling advice

#26

Post by pg_bot »

jjice wrote: post #13

In the opinion of a last semester CS student who has never written an application from scratch that needed more than a SQLite DB (so take me with a half grain of salt), it seems like premature optimization, while always talked about, is very common. I see people talking about using Kubernetes for internal applications and I just can't figure out why. If it's a hobby project and you want to learn Kubernetes, that's a…

What you're describing is called resumé driven development. It happens every few years when people want to cash in on trends/buzzwords that people believe will be disruptive to all industries but are just tools to have in the toolbox for most. New tools pop up all the time that fit this mould. Over the past ten years I can think of Hadoop (Big data), MongoDB (NoSQL), Kubernetes, "Serverless" computing, and TensorFlow. While all these tools have legitimate use cases, they are often overused due to marketing or industry hype.

Adding artificial intelligence to your recipe application is unlikely to make any sense, but people do it because they want to have AI software engineer on their resumé.

Re: Postgres scaling advice

#27

Post by ahachete »

bmcahren wrote: post #11
wilsonrocks wrote: post #4

This was an interesting read for a database novice. It seems like a lot of the quoted stats are about in memory datasets - is that realistic?

Yes. Memory can reach 768GB on a single instance today and I imagine that to expand. From there you can scale by sharding. In memory provides real-time transactions you can't guarantee when using disk-based storage.

EC2's u-24tb1.metal is 224/448 cores/hyperthreads and 24TB RAM ;)

https://aws.amazon.com/sap/instance-types/

Re: Postgres scaling advice

#28

Post by dap »

wilsonrocks wrote: post #4

This was an interesting read for a database novice. It seems like a lot of the quoted stats are about in memory datasets - is that realistic?

It depends a lot on what you’re doing. In my last role, keeping the Postgres database in memory was absolutely not an option, and we ran into major issues related to physical I/O.

Re: Postgres scaling advice

#29

Post by ahachete »

fabian2k wrote: post #3

One point I found very interesting was the following paragraph: > For example, on my (pretty average) workstation, I can do ca. 25k simple read transactions per 1 CPU core on an “in memory” pgbench dataset…with the default config for Postgres v13! In my own very unscientific experiments I never got values as high as that, but in the area of around 4k transactions per second total on multiple cores. Of course I'm comp…

As this is a public reference: GitLab's Postgres cluster handles peaks of 300K tps, where the master node alone supports around 60K-80K. And this is not in-memory (datasize in the order of 8TB, RAM 600GB).

https://about.gitlab.com/blog/2020/09/11/gitlab-pg-upgrade/

And there's still room for vertical scaling.

Disclaimer: we provide Postgres support for GitLab.

Re: Postgres scaling advice

#30

Post by yuribro »

kevincox wrote: post #19
WJW wrote: post #10

Earlier quoted context omitted.

This question is similar to asking on a car forum when using a 40 foot lorry will be the default starter car for everyone. The answer is "probably never" because while it does offer superior cargo transport scalability, the tradeoffs are not worth it for the vast majority of users. The question is posed like distributed databases have no disadvantages over non-distributed databases, but that is simply not the case. C…

I don't think it is quite the same. - Switching from a car to a van to a lorry is fairly low cost. You don't need to recreate your product (probably). - You don't need to run distributed databases in a cluster to start. But I think most importantly the decrease in dev speed and performance is an investment in future scalability. And I only imagine that this different will shrink over time to where for example a 1 nod…

> But I think most importantly the decrease in dev speed and performance is an investment in future scalability.

It makes sense only if you'll ever need this scalability. And you take a hit on other fronts too: Infra cost, Deployment complexity. And both deployment complexity and code complexity also increase QA cost, instability, product and company reputation.

>> much like people don't often start with SQLite today

Maybe they should start with SQLite by default

Post reply on HN