let's say i have a primary with 100M rows of addresses and indexes on things like city, state, zip code (all in memory). I also have 3 read replicas that struggle to do 1000 lookups per minute each. Does PgDog help?
PgDog is funded and coming to a database near you
221–230 of 275 posts
Re: PgDog is funded and coming to a database near you
#222Re: PgDog is funded and coming to a database near you
#223Earlier quoted context omitted.
I've extensively used Dynamo (internally at Amazon and externally) and even founded a DB startup with it at it's core. Boiling down scalability of Postgres vs Dynamo as it's written in blog is a bit terse. Dynamo scales writes horizontally with the keyspace, forever. Postgres simply can't, and no number of layers between the machines and the developer changes that. Sharding, pooling, Citus are all layered on top of a…
Dynamo DB isn’t even good at being a KV store. Almost every time we have to also back it with S3 because of size limitations.
did you use single table design?
and yeah you have to spend a lot of upfront time designing your data models
Re: PgDog is funded and coming to a database near you
#224It’s surprising they don’t mention advantages over other sharding systems like Citus. Maybe it’s just the fact that it’s only a proxy and not core extensions? But that could limit capabilities.
We do, just buried deep in our blog: https://pgdog.dev/blog/pgdog-vs-citus The same old processes vs. threads debate, plus having the ability to scale the coordinator past a single machine. So, if you're OLTP, definitely consider PgDog. OLAP - Citus still wins because of its advanced query engine. We'll get there.
> having the ability to scale the coordinator past a single machine
Since Citus v11 (released 4 years ago), any worker node can also work as a "query router" (a node that you can query against [1], and works from this perspective as a pure coordinator:
> for very demanding applications, you now have the option to load balance distributed queries across the workers
You can also setup such query routers as dedicated nodes by setting the `shouldhaveshards` to `false`, becoming an effective coordinator (for querying; not for metadata operations).
So with Citus you can absolutely have as many query routers (coordinators if you wish) as you want.
[1]: https://www.citusdata.com/updates/v11-0/#metadata-sync
Edit: formatting, typo
Re: PgDog is funded and coming to a database near you
#225Earlier quoted context omitted.
Have you looked into things like CloudnativePG? https://cloudnative-pg.io/
CNPG is quite nice and robust but I'd still be a bit reluctant to stack PG on k8s for really big clusters just because k8s ecosystem moves quite quickly and there's lots of patching/maintenance/churn which means more PG failovers so depends on how well your workload handles that (they're normally only a few seconds)
Most K8s upgrades can happen independently of node reboots etc., you only need to update for OS updates really, but that would be true of anywhere you run PG, even RDS.
Re: PgDog is funded and coming to a database near you
#226Is this like on prem RDS?
Re: PgDog is funded and coming to a database near you
#227Is this comparable to Supabase's just announced multigres?
Re: PgDog is funded and coming to a database near you
#228Congrats on the funding Lev!
Just to say we're happy pgdog users here! One feature we quite like (of the proxy) is the handling of different connection settings per connection (i.e. statement_timeout). When we investigated RDS proxy (ages ago) it wasn't supported, I think the same was true for pgbouncer so it required a bunch of application changes. With pgdog, it just works transparently.
Re: PgDog is funded and coming to a database near you
#229We sharded over 20 TB that we know about. This is probably a typo, right? 20TB isn't that big. I would imagine they've sharded a lot more than that
If you think 20TB "isn't that big" I want to know what size of DBs you're working with 0_0
It's really not that big for a postgres db in a lot of places, honestly.
Re: PgDog is funded and coming to a database near you
#230Earlier quoted context omitted.
That's the idea of sharding. If you read the pgdog docs, you'll notice you need to tell it which shard server to route your request to - it doesn't just magically work. It's still providing value by reusing connections, which are particularly expensive in postgres. Because it's not magic, you do still have to know what's going on under the hood, e.g. no cross-shard transactions. I'd see if my application can benefit…
fwiw, we support cross-shard transactions. They are not magic though, just good old 2pc and a bit of coordination.
2pc is only safe if every part of the system has guaranteed uptime, which it never does. Assume that cross-shard transactions only work in the happy case and may result in inconsistent data otherwise.
They also reduce the benefit of sharding, possibly down to worse performance than a non-sharded DB.