Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PgDog is funded and coming to a database near you

pgdog.dev

Post reply on HN
Print view View original

61–70 of 275 posts

Re: PgDog is funded and coming to a database near you

#61

Post by hylaride »

levkk wrote: post #3
htrp wrote: post #2

>PgDog is a sharder, connection pooler and load balancer for PostgreSQL. Written in Rust, PgDog is fast, reliable and scales databases horizontally without requiring changes to application code. Still trying to figure out how this works technically, is the performance gain really just re-write in rust?

Not quite. The performance gain is to bring those features to Postgres! Edit: Performance gains are from having the ability to load balance reads (horizontal scaling for read queries) and scale out writes (with sharding). Once instance bottleneck in Postgres has many faces: 1. Behind schedule vacuums because of too many dead tuples (too many writes) 2. The WALWriter is single-threaded and IO-bound - Postgres can only…

So to oversimplify, is the idea to bring an AWS Aurora-style storage mechanism natively to Postgres?

Re: PgDog is funded and coming to a database near you

#62

Post by welder »

tschellenbach wrote: post #49

PgDog, Neki, multigres, awesome to see. And yes this is the main issue with postgres. Well this and not having index hints, looking forward to 19

Don't forget the original PgBouncer. Hard to setup, but with the help of AI these days it's easier to configure.

Re: PgDog is funded and coming to a database near you

#63

Post by znpy »

yabones wrote: post #30

I'm curious how this might help with our biggest downtime-causer with postgres, which is major version upgrades. Poolers do a great job for failover and load balancing, but we consistently need ~10-20 minutes of downtime once or twice a year to do upgrades. Logical replication between old->new versions could probably help, but it would still require flipping everything over to the new cluster without partial writes o…

It's weird that PostgreSQL still doesn't have a proper, open source, general multi-master implementation.

At this point i wonder if i'll ever see that.

Re: PgDog is funded and coming to a database near you

#64

Post by jjice »

znpy wrote: post #63
yabones wrote: post #30

I'm curious how this might help with our biggest downtime-causer with postgres, which is major version upgrades. Poolers do a great job for failover and load balancing, but we consistently need ~10-20 minutes of downtime once or twice a year to do upgrades. Logical replication between old->new versions could probably help, but it would still require flipping everything over to the new cluster without partial writes o…

It's weird that PostgreSQL still doesn't have a proper, open source, general multi-master implementation. At this point i wonder if i'll ever see that.

Do other RDBMSs have this? I genuinely have no clue. I've been fortunate enough to be able to get away with one primary and multiple secondaries at my largest usage of Postgres. Multi-master is the kind of thing I am fully out of my depth on, so I'm curious if there's a well defined path for implementation here or what.

Re: PgDog is funded and coming to a database near you

#65

Post by levkk »

hylaride wrote: post #61
levkk wrote: post #3

Earlier quoted context omitted.

Not quite. The performance gain is to bring those features to Postgres! Edit: Performance gains are from having the ability to load balance reads (horizontal scaling for read queries) and scale out writes (with sharding). Once instance bottleneck in Postgres has many faces: 1. Behind schedule vacuums because of too many dead tuples (too many writes) 2. The WALWriter is single-threaded and IO-bound - Postgres can only…

So to oversimplify, is the idea to bring an AWS Aurora-style storage mechanism natively to Postgres?

Yes, except it doesn't have any cross-dependencies on the same volume, so the uptime here should be higher.

Re: PgDog is funded and coming to a database near you

#66

Post by jeltz »

Blackthorn wrote: post #38
boxed wrote: post #33

Earlier quoted context omitted.

Seconded. Coming from MySQL this is a huge regression that makes Postgres look like something from the 80s. I still wonder why this isn't seen as the absolutely highest priority.

Probably because it's an open source project and apparently none of its users cared about this feature enough to develop it or fund it.

It is also a bit tricky tradeoff. You do not want to be stuck with the same data format forever. So databases like MySQL and PostgreSQL need a downtime when doing a major version upgrade. They both try to keep it short, usually seconds, but minutes can happen in either database.

Re: PgDog is funded and coming to a database near you

#67

Post by welder »

Three real-world issues I've run into recently with PgBouncer + Postgres are:

1. pool exhaustion from idle connections inside open long-running transactions

2. SQLAlchemy's client-side pool using dead connections that PgBouncer had already killed, causing periodic request errors

3. Some tasks have to bypass PgBouncer when they use SET or prepared statements

I've already sharded large datasets at the application layer, but looks like PgDog solves the above problems for any future work?

Re: PgDog is funded and coming to a database near you

#68

Post by sandeepkd »

Nit-Pick: It might be anti-marketing, still it would be helpful if the use cases can be articulated in a way where it would make sense to use this Vs any other type of database. Honesty goes a long way with the more technical folks for anything related to infrastructure.

Surfacing where and how PG is better than Dynamo or any other database is probably a good starting point instead of calling out PG a silver bullet for everything. At the end of the day its all a trade-off.

Re: PgDog is funded and coming to a database near you

#69

Post by briffle »

tschellenbach wrote: post #51
yabones wrote: post #30

I'm curious how this might help with our biggest downtime-causer with postgres, which is major version upgrades. Poolers do a great job for failover and load balancing, but we consistently need ~10-20 minutes of downtime once or twice a year to do upgrades. Logical replication between old->new versions could probably help, but it would still require flipping everything over to the new cluster without partial writes o…

Logical replication solves this. You roll the cluster, downtime is minimal. like 60s maybe.

Logical replication needs a special 'upgrade' use case that will automate most of its pain points away. I understand why DDL does not replicate, and that you may want to replicate to a data warehouse that only needs some columns, etc, but there should be a case just for upgrading that handles all DDL, sequences all existing everything, and just works...

Re: PgDog is funded and coming to a database near you

#70

Post by boxed »

jeltz wrote: post #39
boxed wrote: post #33

Earlier quoted context omitted.

Seconded. Coming from MySQL this is a huge regression that makes Postgres look like something from the 80s. I still wonder why this isn't seen as the absolutely highest priority.

I have not ran MySQL for some years but it at least used to have exactly the same issue. Upgrading a database with MySQL can take a long time if you have many tables. The main difference is only really that PostgreSQL does it with a separate tool, pg_upgrade, while MySQL does it as part of the main binary. For both MySQL and PostgreSQL you will need to use some kind of logical upgrades if you want no downtime.

They don't change the on-disk structure all the time though...
Post reply on HN