Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We scaled PgBouncer to 4x throughput

clickhouse.com

Post reply on HN
Print view View original

51–60 of 67 posts

Re: We scaled PgBouncer to 4x throughput

#51

Post by kmeaw »

bradfitz wrote: post #34
saisrirampur wrote: post #19

Earlier quoted context omitted.

It improved quite a lot! It scales pretty well to thousands of connections: https://techcommunity.microsoft.com/blog/adforpostgresql/imp... . However, if pooling isn’t used, there’s always an overhead (tens of milliseconds or more) when creating a new connection because Postgres needs to fork a process. And yes, applications can be written without pooling, which isn’t ideal, but happens quite a lot. Application frame…

In other words, it's still super heavy if it's forking a process per connection. I find it ridiculous that PgBouncer even needs to exist. Postgres should be doing this.

Why Postgres should be doing this? Not every client creates a lot of connections and spinning up PgBouncer is easy. On the other hand, debugging async multithreaded complex code is hard.

Re: We scaled PgBouncer to 4x throughput

#52

Post by Yajirobe »

zarzavat wrote: post #33
Yajirobe wrote: post #30

Earlier quoted context omitted.

nginx was built by one person. I'm talking about a russian corporation that has (inescapable) ties to kremlin

If open source code can have political beliefs then the only reasonable solution is to live as a hermit. I can guarantee that there is software on your computer written by communists, nazis, mormons, every single political ideology is represented by lines of code that you run every day, because open source doesn't require any political vetting to contribute.

Again, there's a difference between software written by a person that represents a 'bad' (to me) ideology, and software written by corporations in terrorist states.

Re: We scaled PgBouncer to 4x throughput

#53

Post by Yajirobe »

UnfitFootprint wrote: post #31
Yajirobe wrote: post #29

Earlier quoted context omitted.

[flagged]

except for perhaps… a scalable pg bouncer? And good nuclear reactors [1]? Not to say I support many Russian political moves, but I think discrediting an entire people and their outputs on Russian politics is brusque - particularly open source ones. [1]: https://www.abc.net.au/news/2026-05-08/russia-nuclear-power-...

Neither pg bouncer nor nuclear reactors are unique inventions in and of themselves - these are just iterations on what has existed previously. Of course russia can take an existing thing and iterate on it (computers, rockets, AK-47, etc.) but that wasn't my point.

Re: We scaled PgBouncer to 4x throughput

#54

Post by marksomnian »

Yajirobe wrote: post #50
femiagbabiaka wrote: post #38

Earlier quoted context omitted.

Objectively speaking, Russia is one of the most creative and innovative societies of all time. Literature, science, music, sporting, mathematics, computing, you name it, a Russian has innovated in it.

What would be some of these innovations and contributions in the last 20 years?

nginx (2004, ok, 22 years), ClickHouse (2016)

Re: We scaled PgBouncer to 4x throughput

#55

Post by ramon156 »

Yajirobe wrote: post #50
femiagbabiaka wrote: post #38

Earlier quoted context omitted.

Objectively speaking, Russia is one of the most creative and innovative societies of all time. Literature, science, music, sporting, mathematics, computing, you name it, a Russian has innovated in it.

What would be some of these innovations and contributions in the last 20 years?

I don't get the "its from [country] so I don't use it". Unless this is somehow funding e.g. a war, what did the maintainers do to deserve such backlash?

Edit: just read what ties ClickHouse has, nevermind.

Re: We scaled PgBouncer to 4x throughput

#56

Post by z3ratul163071 »

is this for microservice scenarios where you gate access to the psql server through a connection pool thing? because if there is monolithic backend this is not needed. most decent backend frameworks have built-in connection pooling. that covers 98% of use cases for which microservices are not needed, nor recommended.

Re: We scaled PgBouncer to 4x throughput

#57

Post by [deleted] »

Yajirobe wrote: post #50
femiagbabiaka wrote: post #38

Earlier quoted context omitted.

Objectively speaking, Russia is one of the most creative and innovative societies of all time. Literature, science, music, sporting, mathematics, computing, you name it, a Russian has innovated in it.

What would be some of these innovations and contributions in the last 20 years?

[deleted]

Re: We scaled PgBouncer to 4x throughput

#58

Post by ezekiel68 »

__s wrote: post #32
ezekiel68 wrote: post #23

> The cancel lands on a process that has never heard of the query, and nothing happens. > Peering fixes this. The processes are aware of one another, so a cancel that lands on the wrong process is forwarded to the one that actually owns the session. I understand "peering" as a concept here but have never tried this with PostgreSQL before. May I ask: A) Does PostgreSQL have a mode/setting for peering that makes this e…

you encode information in the token, this was a motivation for postgres to remove 32 byte cap on tokens: https://www.postgresql.org/message-id/508d0505-8b7a-4864-a68... See from slide 26 https://www.pgevents.ca/events/pgconfdev2024/sessions/sessio... Jelte's a pgbouncer maintainer, video of a talk on this by him: https://www.youtube.com/watch?v=X-nCHcZ6vQU

Thanks for this!

Re: We scaled PgBouncer to 4x throughput

#59

Post by didip »

z3ratul163071 wrote: post #56

is this for microservice scenarios where you gate access to the psql server through a connection pool thing? because if there is monolithic backend this is not needed. most decent backend frameworks have built-in connection pooling. that covers 98% of use cases for which microservices are not needed, nor recommended.

if your backend app is one big binary per host? Sure.

But if your backend app is a shared nothing fork of many processes then this is surely needed.

Re: We scaled PgBouncer to 4x throughput

#60

Post by paulryanrogers »

kmeaw wrote: post #51
bradfitz wrote: post #34

Earlier quoted context omitted.

In other words, it's still super heavy if it's forking a process per connection. I find it ridiculous that PgBouncer even needs to exist. Postgres should be doing this.

Why Postgres should be doing this? Not every client creates a lot of connections and spinning up PgBouncer is easy. On the other hand, debugging async multithreaded complex code is hard.

Postres project also once lacked replication, calling it unnecessary to the core effort. Now it has two means of replication in core and I'd argue is better for it, especially after suffering through both Pgpool and Slony.

I hope they do develop a native, threaded pooling, even if it were incompatible with some libraries or extensions.

Post reply on HN