Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Odyssey: Scalable PostgreSQL Connection Pooler

github.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Odyssey: Scalable PostgreSQL Connection Pooler

#41

Post by duelingjello »

For scaling (LB) reads + HA, HAproxy + built-in replication to hot standbys is much simpler. You can use keepalived to promote a new master automatically if it dies. HAproxy can also give you two different ports on the vIP, one for read-only and one for read-write that automatically find which server is the master and which servers are replicas, using periodic banner testing. It's free, doesn't depend on someone coding a Postgres statement proxy and getting every corner-case just right, and lower latency. I suggest using LVS too so the vIP can move around if a box dies.

For scaling (LB) read&writes + HA, consider postgres-xl instead. It's FOSS w/ optional commercial support.

If you have crazy OLAP or metrics data, consider Greenplum. Freemium FOSS.

Re: Odyssey: Scalable PostgreSQL Connection Pooler

#43

Post by x4m »

tuldia wrote: post #25

Scalable in which sense? Assuming that this is something like pgbouncer or pgpool that sit between the client and the database, and that you have a limit of connections with the database as well as the number of client connections you can keep, what value does this adds compared with those other (more mature, battle tested, included in major distros) projects?

Pgbouncer is single-threaded. If you have 80-core box it is a huge waste to terminate your SSL in single-threaded proxy pooler.

Re: Odyssey: Scalable PostgreSQL Connection Pooler

#44

Post by x4m »

hinkley wrote: post #34

> Odyssey has sophisticated asynchronous multi-threaded architecture which is driven by custom made coroutine engine: machinarium Let me find my space helmet. The voice in my head is crying 'architectural astronaut'. We are forgetting again to ask the important question of "what else did you try, and why didn't it work so you had to write your own?" I think I get the 'why' for pooling. I know a guy at work who strugg…

We were using libevent and had to remove an extra abstraction layer. Machinarium is there for two reasons: 1. Express network protocol state machine as normal control flow. Just to make code simpler. 2. Reduce contention of multithreaded operations. In Machinarium you have coroutines which can yield context not to OS scheduler but to other coroutines. You can organize local data around worker (thread - owner of coroutines) and avoid false cache line sharing.

Re: Odyssey: Scalable PostgreSQL Connection Pooler

#46

Post by takeda »

x4m wrote: post #43
tuldia wrote: post #25

Scalable in which sense? Assuming that this is something like pgbouncer or pgpool that sit between the client and the database, and that you have a limit of connections with the database as well as the number of client connections you can keep, what value does this adds compared with those other (more mature, battle tested, included in major distros) projects?

Pgbouncer is single-threaded. If you have 80-core box it is a huge waste to terminate your SSL in single-threaded proxy pooler.

Starting from version 1.12.0 you can actually run multiple pgbouncer processes listening on the same port, that eliminates some of the issues.

Re: Odyssey: Scalable PostgreSQL Connection Pooler

#47

Post by takeda »

x4m wrote: post #45
arcticfox wrote: post #42

Earlier quoted context omitted.

This is definitely my top question coming in here. I know what pgbouncer can do for me, and it's pretty good, is there a good reason to look at this instead?

It's multithreaded PgBouncer.

pgbouncer is event based for a reason, they recently added an option to run multiple processes listening on the same socket so that allows it to utilize all available cores.

Re: Odyssey: Scalable PostgreSQL Connection Pooler

#48

Post by SergeAx »

pepemon wrote: post #40
PeterZaitsev wrote: post #37

Earlier quoted context omitted.

I respect Yandex a lot. They are fantastic at doing 80% of what Google, Uber, Amazon (combined) do with like 10% of resources - people and otherwise. So their software tends to be super efficient.

They are fantastic at loosing their best talent to the western counterparts. Nothing fantastic at all for a big company within other reaches.

It's not the Yandex' fault, it is a political situation in Russia. IT skills is the best ticket out for honest people today, can't blame them.

Re: Odyssey: Scalable PostgreSQL Connection Pooler

#49

Post by x4m »

takeda wrote: post #46
x4m wrote: post #43

Earlier quoted context omitted.

Pgbouncer is single-threaded. If you have 80-core box it is a huge waste to terminate your SSL in single-threaded proxy pooler.

Starting from version 1.12.0 you can actually run multiple pgbouncer processes listening on the same port, that eliminates some of the issues.

Yes, we use a cascade of PgBouncers. But it's hard to maintain, actually. Also, PgBouncer was not actively developed for years. I consider new PgBouncer features (port reuse, SCRAM-SHA-256) accomplishment of Odyssey to some (small) extent :)

Re: Odyssey: Scalable PostgreSQL Connection Pooler

#50

Post by x4m »

pepemon wrote: post #40
PeterZaitsev wrote: post #37

Earlier quoted context omitted.

I respect Yandex a lot. They are fantastic at doing 80% of what Google, Uber, Amazon (combined) do with like 10% of resources - people and otherwise. So their software tends to be super efficient.

They are fantastic at loosing their best talent to the western counterparts. Nothing fantastic at all for a big company within other reaches.

Your claim is wrong in that "best talent" part. Though, Yandex, from time to time, loose new grads, sad but true.
Post reply on HN