Live data from Hacker News

Does anyone run Postgres without PgBouncer?

brandur.org

41–50 of 121 posts

Re: Does anyone run Postgres without PgBouncer?

#41
Article title is pure click-bait. PgBouncer adds complexity. If you need it, you need it. If you don't need it, then you added complexity for nothing.

> since neither IBM nor Oracle is a service that any self-respecting person not part of an enterprise sales cycle would actually use

Author needs a serious ego check. There are legitimate engineering reasons to pick IBM Cloud (like if you need to support Z mainframes, which you will sometimes need if you sell to those enterprise folk) as well as Oracle Cloud (they built datacenters in cities that are not served by other cloud providers and can thus offer the lowest latency). These reasons may not be common, but they're certainly legitimate.

Re: Does anyone run Postgres without PgBouncer?

#43

(I work on the postgres proxy layer at Neon) PgBouncer is entirely optional and it's not always the right choice. If you have a classical app (non serverless) and you can maintain a connection pool from your app, then I recommend avoiding pgbouncer. The benefits of pgbouncer mostly come from irregular client connections (too many, too much churn). If you don't have that problem, go direct to postgres. I'm exploring r…

And the only comment in this whole thread, who argues for a similar architectural alternative...is the ONLY one in the whole thread down voted. HN continue to excel in technical chops...

https://news.ycombinator.com/item?id=49319988

https://news.ycombinator.com/item?id=49320460

Re: Does anyone run Postgres without PgBouncer?

#44
Transaction mode has one real footgun: SET statements are session-scoped, but in transaction mode your session gets reassigned between transactions. Easy to miss until you're chasing a mysterious search_path bug at 2am.

psycopg3 prepares statements by default now, which breaks in transaction mode unless you explicitly opt out.

For apps with a persistent server process and an in-app pool the external bouncer is mostly ceremony. The math changes with serverless: no persistent process means no persistent pool, so a dedicated pooler starts pulling its weight.

Re: Does anyone run Postgres without PgBouncer?

#45
I see many comments comparing PgBouncer with application connection poolers without addressing the conceptual difference between them. Here it is:

1. Most application connection poolers follow a first-in-first-out (FIFO) algorithm, which is simple enough to implement and is enough to make sure the application always has a connection available to connect to the database. It optimizes low latency, and works great from the point of view from the application. The problem is that it has few mechanisms to remove redundant connections, since the application is constantly keeping them all "warm".

2. PgBouncer and very few external poolers follow the inverse idea – last-in-first-out (LIFO), and they optimize for reducing the number of connections that reach Postgres, thus improving its throughput. The idea might seem crazy at first – the last connection used is the first one to be picked up again – but this algorithm automatically removes excess connections, which will get cold and get closed.

When starting a new application, option (1) is enough, but as it scales up enough, at some time it is recommended to use (2), since having hundreds of open connections to Postgres is bad for performance if you can use PgBouncer or similar to cut it by 90%. Postgres' process-per-connection design works much better when there are fewer connections reaching it.

Re: Does anyone run Postgres without PgBouncer?

#46
post #33

Earlier quoted context omitted.

Your high level buckets are languages but the constraints you list are usage patterns. You can build applications in any language that have many short-lived transactions in single connections or a few huge, blocking one, or anything in between. The former case is a good one for PGBouncer (it can interleave transactions) and the latter isn’t (it can’t do much about your three minute long BEGIN…COMMIT). Neither has to…

They addressed this though? I suspect you are unfamiliar with the GIL in python. The reason for a language distinction is because, as OP says: "As it's much easier to share a connection pool locally". This may change vaguely soon, but right now you typically scale python apps by starting multiple python processes, while for Java you can just add threads. Python processes can't share a thread pool among all of them, w…

> I suspect you are unfamiliar with the GIL in python.

Sadly, I’m deeply familiar. What made Evan Phoenix’s Rubinius work so exciting in ~2012 was getting rid of the GIL and seeing what was “fixed” by parallelism and what was not.

You make a good point. I didn’t read the parent comment that way but you’re right about how you scale python with the GIL vs JVM.

Re: Does anyone run Postgres without PgBouncer?

#47

(I work on the postgres proxy layer at Neon) PgBouncer is entirely optional and it's not always the right choice. If you have a classical app (non serverless) and you can maintain a connection pool from your app, then I recommend avoiding pgbouncer. The benefits of pgbouncer mostly come from irregular client connections (too many, too much churn). If you don't have that problem, go direct to postgres. I'm exploring r…

Also some people with use cases that shouldn't need PgBouncer end up thinking they do because something else is misconfigured. This was brought up in a parent of https://news.ycombinator.com/item?id=49019695; FastAPI recommends dep-injecting transactions into your HTTP handlers. Ties up all your connections and creates idle xact spam. There are valid reasons to use PgBouncer, this isn't one.

Even on serverless platforms like Heroku, I've been fine giving each worker a pool such that max_workers * pool_size < max_connections.

Re: Does anyone run Postgres without PgBouncer?

#50
post #23

This question will get more interesting responses if it was qualified as: "Does anyone run Postgres without PgBouncer for non-trivial workloads?" Because, as we can see from the comments so far, lots of people are going to say you don't need it for your blog that gets 10 hits a month. I've personally never heard of anyone not using PGbouncer, or some connection pooling proxy, for reasonably concurrent workloads. PG's…

Yes, I've done a variety of typical, nontrivial workloads on Postgres for about a decade and have never used PgBouncer. Though I can imagine use cases where it'd make sense.
Post reply on HN