Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PgDog is funded and coming to a database near you

pgdog.dev

Post reply on HN
Print view View original

51–60 of 275 posts

Re: PgDog is funded and coming to a database near you

#51

Post by tschellenbach »

yabones wrote: post #30

I'm curious how this might help with our biggest downtime-causer with postgres, which is major version upgrades. Poolers do a great job for failover and load balancing, but we consistently need ~10-20 minutes of downtime once or twice a year to do upgrades. Logical replication between old->new versions could probably help, but it would still require flipping everything over to the new cluster without partial writes o…

Logical replication solves this. You roll the cluster, downtime is minimal. like 60s maybe.

Re: PgDog is funded and coming to a database near you

#52

Post by parthdesai »

levkk wrote: post #25
melon_tsui wrote: post #7

2M qps in production is legit. Curious how much RAM and CPU that takes on average per deployment though

Depends. Only pooling, very little. Load balancing/sharding needs to parse queries, so a bit more. Could go up to a GB per pod, sometimes more if you have a lot of unique SQL queries (unique by text, not by parameters). We cache query ASTs to avoid parsing them on each request - that's the bulk of memory usage.

Semi related question - I have always wondered, how do you tackle OOM issues at the proxy layer, i.e. let's say a particular SQL query requires proxy to fan out the query to multiple shards, which return a pretty large dataset. I'm assuming you would need to load this dataset in the ram to perform certain operations. What happens if the resulting dataset causes the proxy pod to go OOM?

Re: PgDog is funded and coming to a database near you

#53

Post by levkk »

e12e wrote: post #32
simonw wrote: post #18

Suggestion: have more than just helm and Docker in your quickstart documentation. I'd like to try this out just to see what it can do, but not quite enough to fire up one of those systems for it. Is there a binary I can run directly?

In addition - the docker compose example doesn't set up any data volumes for the postgres instances - that might be considered a bug? Then again, sharding on a single host probably isn't very useful anyway - but it might work with docker in swarm mode?

The docker compose example is just a demo. I don't know anyone who runs Postgres with docker compose / swarm in prod :) But yes, happy to add volumes so it seems more real.

Re: PgDog is funded and coming to a database near you

#54

Post by levkk »

parthdesai wrote: post #52
levkk wrote: post #25

Earlier quoted context omitted.

Depends. Only pooling, very little. Load balancing/sharding needs to parse queries, so a bit more. Could go up to a GB per pod, sometimes more if you have a lot of unique SQL queries (unique by text, not by parameters). We cache query ASTs to avoid parsing them on each request - that's the bulk of memory usage.

Semi related question - I have always wondered, how do you tackle OOM issues at the proxy layer, i.e. let's say a particular SQL query requires proxy to fan out the query to multiple shards, which return a pretty large dataset. I'm assuming you would need to load this dataset in the ram to perform certain operations. What happens if the resulting dataset causes the proxy pod to go OOM?

Two schools of thought:

1. Let it crash. Increase the RAM, try again.

2. Page to disk (swap), make it slow but ultimately work.

Both have their trade-offs. There is no free lunch here.

Re: PgDog is funded and coming to a database near you

#55

Post by rbranson »

Ozzie_osman wrote: post #31

We sharded over 20 TB that we know about. This is probably a typo, right? 20TB isn't that big. I would imagine they've sharded a lot more than that

You are correct. As a point of comparison: almost ten years ago at Segment we had a single Aurora PostgreSQL instance with ~50T of data, it was used to index potential identity data in a much larger corpus of files stored in S3.

Re: PgDog is funded and coming to a database near you

#56

Post by 999900000999 »

rswail wrote: post #50
pantulis wrote: post #37

Earlier quoted context omitted.

How are 3 developers going to sell that to any company? Procurement will have a field day.

They have funding. That's what it will be for. I wish them well and appreciate that people are still doing FOSS. As long as they don't get undercut by the equivalent of AWS https://aws.amazon.com/rds/proxy/ which is a managed pgbouncer.

The issue is if the DB layer fails your product is going to completely stop working.

You’d need a ton of faith in these 3 people.

Feels more like it would work better inside of a bigger organization.

The QA tester in me is kinda risk adverse.

Re: PgDog is funded and coming to a database near you

#57

Post by jeltz »

tomnipotent wrote: post #48
jeltz wrote: post #39

Earlier quoted context omitted.

I have not ran MySQL for some years but it at least used to have exactly the same issue. Upgrading a database with MySQL can take a long time if you have many tables. The main difference is only really that PostgreSQL does it with a separate tool, pg_upgrade, while MySQL does it as part of the main binary. For both MySQL and PostgreSQL you will need to use some kind of logical upgrades if you want no downtime.

MySQL has advocated for decades spinning up a replica with the upgraded version, waiting for it to catch up to master before promoting it to the new master. You can do the same thing with Postgres.

Exactly, MySQL and PostgreSQL are the same here. Maybe one is a bit faster than the other at doing major version upgrades but the behaviours are quite similar.

Re: PgDog is funded and coming to a database near you

#58

Post by levkk »

drchaim wrote: post #35

Good stuff, although I’m not quite sure about the fast OLAP use case. If you’re already sharding by tenant for other reasons, OK… But I see CDC to a true OLAP system as more scalable. PostgreSQL still needs real columnar tables in the core, hopefully one day

OLAP means different things to different people. For us, it's just making sure your admin dashboard keeps working basically:

  SELECT tenant_id, COUNT(clicks)
  FROM users
  GROUP BY tenant_id
  ORDER BY 2 DESC
  LIMIT 25;
Performance is a side effect - definitely needed and we'll do everything we can, but we are not competing with ClickHouse or Snowflake - just trying to make sharded Postgres work with your app.

Re: PgDog is funded and coming to a database near you

#59

Post by jeltz »

GiorgioG wrote: post #42
Ozzie_osman wrote: post #31

We sharded over 20 TB that we know about. This is probably a typo, right? 20TB isn't that big. I would imagine they've sharded a lot more than that

For a vast majority of use cases 20TB is positively enormous.

Yes. But for most workloads it is not much for PostgreSQL. You often will not have to shard at all.

Re: PgDog is funded and coming to a database near you

#60

Post by mplanchard »

GiorgioG wrote: post #42
Ozzie_osman wrote: post #31

We sharded over 20 TB that we know about. This is probably a typo, right? 20TB isn't that big. I would imagine they've sharded a lot more than that

For a vast majority of use cases 20TB is positively enormous.

RDS caps out at 64 TB unless you use Aurora, so 20 TB is totally manageable without sharding.
Post reply on HN