"Why Us" => "I ran Postgres at Instacart, where we scaled the company 5x in April of 2020. The biggest problem we had was making Postgres serve 100,000s of grocery delivery orders per minute"
Couldn't be a better why us :)
PgDog is funded and coming to a database near you
71–80 of 275 posts
Re: PgDog is funded and coming to a database near you
#72Earlier quoted context omitted.
It's weird that PostgreSQL still doesn't have a proper, open source, general multi-master implementation. At this point i wonder if i'll ever see that.
Do other RDBMSs have this? I genuinely have no clue. I've been fortunate enough to be able to get away with one primary and multiple secondaries at my largest usage of Postgres. Multi-master is the kind of thing I am fully out of my depth on, so I'm curious if there's a well defined path for implementation here or what.
MySQL has Galera cluster for that.
Re: PgDog is funded and coming to a database near you
#73Three real-world issues I've run into recently with PgBouncer + Postgres are: 1. pool exhaustion from idle connections inside open long-running transactions 2. SQLAlchemy's client-side pool using dead connections that PgBouncer had already killed, causing periodic request errors 3. Some tasks have to bypass PgBouncer when they use SET or prepared statements I've already sharded large datasets at the application layer…
SQLA async is a bit of a struggle with pgbouncer.
I had to disable application pooling as it was causing read only transactions I could couldnt pin down the cause.
Re: PgDog is funded and coming to a database near you
#74How are 3 developers going to QA this properly ?
They are not just some random 3 have decades of real db experience behind them. They also just got funded which gives them the ability to expand and stay longer in the game.
Re: PgDog is funded and coming to a database near you
#75I'm curious how this might help with our biggest downtime-causer with postgres, which is major version upgrades. Poolers do a great job for failover and load balancing, but we consistently need ~10-20 minutes of downtime once or twice a year to do upgrades. Logical replication between old->new versions could probably help, but it would still require flipping everything over to the new cluster without partial writes o…
It's weird that PostgreSQL still doesn't have a proper, open source, general multi-master implementation. At this point i wonder if i'll ever see that.
It has been tried many times. Good luck to pgdog, but there’s a reason these projects don’t stick.
Multi master, from even a conceptual perspective, is incredibly complicated. Databases, transactions, consistency, parallelism are all very complicated.
It’s something that always seems promising at the start but as soon as maintenance and long term improvements enter the picture(ie integrating new Postgres versions), the complexity becomes too much.
Re: PgDog is funded and coming to a database near you
#76I've seen a couple of these "distributed" postgres extensions.
My question is, has any of them been talked about being upstreamed to postgres itself? Or, adding a custom built in feature to postgres itself?
Re: PgDog is funded and coming to a database near you
#77I'm curious how this might help with our biggest downtime-causer with postgres, which is major version upgrades. Poolers do a great job for failover and load balancing, but we consistently need ~10-20 minutes of downtime once or twice a year to do upgrades. Logical replication between old->new versions could probably help, but it would still require flipping everything over to the new cluster without partial writes o…
Logical replication is how this is typically done. If you have some infra-as-code setup, you create a new cluster with identical settings except for the major version, import the schema, start copying data from a read-replica running the old version, stop accepting writes from the old version (downtime starts), sync the sequence numbers, and point your services to the new cluster (downtime ends). If you use something…
RDS has blue green deployments that can help. It was rough at first, though seems they worked out the kinks.
Re: PgDog is funded and coming to a database near you
#78I've seen a couple of these "distributed" postgres extensions. My question is, has any of them been talked about being upstreamed to postgres itself? Or, adding a custom built in feature to postgres itself?
This is not an extension, it's a proxy! Very different. You can deploy it anywhere already without having to wait for upstreaming or your cloud provider adding support for it. It's one of the two reasons why we built it this way, the other being performance (it's much faster to do this in the proxy than inside Postgres).
Re: PgDog is funded and coming to a database near you
#79PgDog, Neki, multigres, awesome to see. And yes this is the main issue with postgres. Well this and not having index hints, looking forward to 19
The pg_hint_plan extension isn't in core, yet is pretty competent when you need to override planner.
Re: PgDog is funded and coming to a database near you
#80I am trying to gain a basic understanding of this: Right now I have a 4TB DB on one large box. Is the idea that using a proxy tool like PGDog I could spin up 8 smaller boxes handling ~500GB each and then one medium box for the proxy? Right now I have a project that has very heavy write traffic from multiple services and a web app that reads from this. We are starting to hit the point where no amount of indexing, quer…
> 8 smaller boxes handling ~500GB each and then one medium box for the proxy?
That's exactly right. Get in touch (lev@pgdog.dev), happy to help or at the very least tell you what current works (or doesn't) so you know what your options are.