Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We scaled PgBouncer to 4x throughput

clickhouse.com

Post reply on HN
Print view View original

31–40 of 67 posts

Re: We scaled PgBouncer to 4x throughput

#31

Post by UnfitFootprint »

Yajirobe wrote: post #29
inigyou wrote: post #26

Earlier quoted context omitted.

Do you boycott American companies too? I'm assuming you boycott Russian companies because Russia causes death and destruction. But the USA caused much more death and destruction than Russia ever did - do you boycott them too?

[flagged]

except for perhaps… a scalable pg bouncer? And good nuclear reactors [1]?

Not to say I support many Russian political moves, but I think discrediting an entire people and their outputs on Russian politics is brusque - particularly open source ones.

[1]: https://www.abc.net.au/news/2026-05-08/russia-nuclear-power-...

Re: We scaled PgBouncer to 4x throughput

#32

Post by __s »

ezekiel68 wrote: post #23

> The cancel lands on a process that has never heard of the query, and nothing happens. > Peering fixes this. The processes are aware of one another, so a cancel that lands on the wrong process is forwarded to the one that actually owns the session. I understand "peering" as a concept here but have never tried this with PostgreSQL before. May I ask: A) Does PostgreSQL have a mode/setting for peering that makes this e…

you encode information in the token, this was a motivation for postgres to remove 32 byte cap on tokens: https://www.postgresql.org/message-id/508d0505-8b7a-4864-a68...

See from slide 26 https://www.pgevents.ca/events/pgconfdev2024/sessions/sessio... Jelte's a pgbouncer maintainer, video of a talk on this by him: https://www.youtube.com/watch?v=X-nCHcZ6vQU

Re: We scaled PgBouncer to 4x throughput

#33

Post by zarzavat »

Yajirobe wrote: post #30
arcanemachiner wrote: post #27

Earlier quoted context omitted.

nginx is also Russian, so don't use... a third of the web.

nginx was built by one person. I'm talking about a russian corporation that has (inescapable) ties to kremlin

If open source code can have political beliefs then the only reasonable solution is to live as a hermit.

I can guarantee that there is software on your computer written by communists, nazis, mormons, every single political ideology is represented by lines of code that you run every day, because open source doesn't require any political vetting to contribute.

Re: We scaled PgBouncer to 4x throughput

#34

Post by bradfitz »

saisrirampur wrote: post #19
bradfitz wrote: post #18

I'm 46 now. I remember being shocked at Postgres's heavy connection model when I was 23. I gather things haven't improved since?

It improved quite a lot! It scales pretty well to thousands of connections: https://techcommunity.microsoft.com/blog/adforpostgresql/imp... . However, if pooling isn’t used, there’s always an overhead (tens of milliseconds or more) when creating a new connection because Postgres needs to fork a process. And yes, applications can be written without pooling, which isn’t ideal, but happens quite a lot. Application frame…

In other words, it's still super heavy if it's forking a process per connection.

I find it ridiculous that PgBouncer even needs to exist. Postgres should be doing this.

Re: We scaled PgBouncer to 4x throughput

#35

Post by hinkley »

saisrirampur wrote: post #11
x4m wrote: post #6

Just use https://github.com/yandex/odyssey :) It's a scalable PgBouncer.

We started with the most battle-tested and native option to Postgres, which is PgBouncer and tried tuning it the right way. Also now that long due kinks like support for prepared statements are solved, it’s been working really well. There are many customers scaling well with 10K+ Postgres connections. We will consider other options like odyssey, pgdog in the future! Side note: I’m not a big fan of having 10K+ connect…

> 100s are more than enough to scale Postgres well

I'd want to know what the workload is. That's true of lots of projects, especially internal tools (even for multinationals). But for my last project, that would have been tough. And by FAANG standards my last project was 'medium' sized, even though it was large by the standards of many places I've worked.

(The galling thing is I shrunk the hardware by 40% but if I'd been there during the architecture phase I'm pretty sure I could have shrunk it by 8x by solving a completely different problem that had higher margins than what we actually did)

Re: We scaled PgBouncer to 4x throughput

#37

Post by hinkley »

bradfitz wrote: post #18

I'm 46 now. I remember being shocked at Postgres's heavy connection model when I was 23. I gather things haven't improved since?

You were coming into the field just as companies were fielding their first reasonable answers to the Threading Model that Java put forward, which was sort of Windows' but with extra features. Even Solaris choked on Java. HP UX did worse and I can't recall if SGI was worse or better than HP. But getting compatible with Java shook a lot of companies up, in how they handled concurrency.

Postgres and SQLite were being designed at that same time but by industry veterans. People who had been deploying high load systems before any of this threading nonsense was around. And they were supporting people running on old hardware.

Re: We scaled PgBouncer to 4x throughput

#38

Post by femiagbabiaka »

Yajirobe wrote: post #29
inigyou wrote: post #26

Earlier quoted context omitted.

Do you boycott American companies too? I'm assuming you boycott Russian companies because Russia causes death and destruction. But the USA caused much more death and destruction than Russia ever did - do you boycott them too?

[flagged]

Objectively speaking, Russia is one of the most creative and innovative societies of all time. Literature, science, music, sporting, mathematics, computing, you name it, a Russian has innovated in it.

Re: We scaled PgBouncer to 4x throughput

#39

Post by __mharrison__ »

zarzavat wrote: post #33
Yajirobe wrote: post #30

Earlier quoted context omitted.

nginx was built by one person. I'm talking about a russian corporation that has (inescapable) ties to kremlin

If open source code can have political beliefs then the only reasonable solution is to live as a hermit. I can guarantee that there is software on your computer written by communists, nazis, mormons, every single political ideology is represented by lines of code that you run every day, because open source doesn't require any political vetting to contribute.

There are probably even Mormons who post on HN...

Re: We scaled PgBouncer to 4x throughput

#40

Post by bradfitz »

hinkley wrote: post #37
bradfitz wrote: post #18

I'm 46 now. I remember being shocked at Postgres's heavy connection model when I was 23. I gather things haven't improved since?

You were coming into the field just as companies were fielding their first reasonable answers to the Threading Model that Java put forward, which was sort of Windows' but with extra features. Even Solaris choked on Java. HP UX did worse and I can't recall if SGI was worse or better than HP. But getting compatible with Java shook a lot of companies up, in how they handled concurrency. Postgres and SQLite were being de…

Yeah, I remember writing epoll libraries for Perl (https://metacpan.org/pod/Sys::Syscall, first out 2005-08-01) doing raw system calls because libc on the distros of the time (at least Debian) didn't have epoll support yet.

So in 2005 I didn't expect Postgres to do super well here, but it's 21 years later and we're still pgbouncin'. It's just kinda sad.

Post reply on HN