Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres Postmaster does not scale

recall.ai

Post reply on HN
Print view View original

21–30 of 94 posts

Re: Postgres Postmaster does not scale

#21

Post by axiolite »

thayne wrote: post #20
kayson wrote: post #19

> sudo echo $NUM_PAGES > /proc/sys/vm/nr_hugepages This won't work :) echo will run as root but the redirection is still running as the unprivileged user. Needs to be run from a privileged shell or by doing something like sudo sh -c "echo $NUM_PAGES > /proc/sys/vm/nr_hugepages" The point gets across, though, technicality notwithstanding.

Or echo $NUM_PAGES | sudo tee /proc/sys/vm/nr_hugepages I've always found it odd that there isn't a standard command to write stdin to a file that doesn't also write it to stdout. Or that tee doesn't have an option to supress writing to stdout.

You forgot the "sudo" before "tee"

> write stdin to a file that doesn't also write it to stdout

You mean like "dd of=/path/file" ?

Re: Postgres Postmaster does not scale

#24

Post by timetoogo »

kayson wrote: post #19

> sudo echo $NUM_PAGES > /proc/sys/vm/nr_hugepages This won't work :) echo will run as root but the redirection is still running as the unprivileged user. Needs to be run from a privileged shell or by doing something like sudo sh -c "echo $NUM_PAGES > /proc/sys/vm/nr_hugepages" The point gets across, though, technicality notwithstanding.

Great point, I was running as root so I didn't pick this up. Corrected, thank you!

Re: Postgres Postmaster does not scale

#25

Post by sgt »

moomoo11 wrote: post #14

maybe this is silly but these days cloud resources are so cheap. just loading up instances and putting this stuff into memory and processing it is so fast and scalable. even if you have billions of things to process daily you can just split if needed. you can keep things synced across databases easily and keep it super duper simple.

It's not really my experience that cloud resources are very cheap.

Re: Postgres Postmaster does not scale

#26

Post by BatteryMountain »

hinkley wrote: post #23
atherton94027 wrote: post #7

I'm a bit confused here, do they have a single database they're writing to? Wouldn't it be easier and more reliable to shard the data per customer?

When one customer is 50 times bigger than your average customer then sharding doesn't do much.

Combination of partitioning + sharding perhaps? Often times its is only a handful of tables that grows large, so even less so for a single large customer, thus sharding that customer out and then partitioning the data by a common/natural boundary should get you 90% there. Majority of data can be partitioned, and it doesn't have to be by date - it pays dividends to go sit with the data and reflect what is being stored, its read/write pattern and its overall shape, to determine where to slice the partitions best. Sometimes splitting a wide table into two or three smaller tables can work if your joins aren't too frequent or complex. Can also help if you can determine which of the rows can be considered hot or cold, so you move the colder/hotter data to a separate tables to speed up read/writes. There are always opportunities for storage optimization large datasets but it does take time & careful attention to get it right.

Re: Postgres Postmaster does not scale

#29

Post by thayne »

atherton94027 wrote: post #7

I'm a bit confused here, do they have a single database they're writing to? Wouldn't it be easier and more reliable to shard the data per customer?

Sharding is often not easy. Depending on the application, it may add significant complexity to the application. For example, what do you do if you have data related to multiple customers? How do you handle customers of significantly different sizes?

And that is assuming you have a solution for things like balancing, and routing to the correct shard.

Re: Postgres Postmaster does not scale

#30

Post by foota »

paulkre wrote: post #28

Can’t believe they needed this investigation to realize they need a connection pooler. It’s a fundamental component of every large-scale Postgres deployment, especially for serverless environments.

No? It sounds like they rejected the need for a connection pooler and took an alternative approach. I imagine they were aware of connection poolers and just didn't add one until they had to.
Post reply on HN