Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Peerdb Streams – Simple, native Postgres change data capture

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#11

Post by saisrirampur »

take-five wrote: post #10

How do you handle Postgres cluster failover? Does PeerDB automatically restore logical replication slot on a new primary?

Great question! We have retry logic in place to handle Postgres restarts. If the failure occurs in-place, you should be fine as the slot will persist. If Postgres fails over to the standby, PeerDB will wait until the slot is created. We did consider automatically creating the slot if it doesn't exist on retries, but ensuring data reliability (creating the slot right after failover without data being ingested) is tricky. So, as of now, we leave it to the user.

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#13

Post by CuriouslyC »

arsalanb wrote: post #12

Noob question: What is the advantage of replicating data into a warehouse vs. just querying it in place on a postgres database?

If the postgres database is recording business transactions, you don't want to cause your business to stop being able to take credit cards because you generated a report.

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#14

Post by arsalanb »

CuriouslyC wrote: post #13
arsalanb wrote: post #12

Noob question: What is the advantage of replicating data into a warehouse vs. just querying it in place on a postgres database?

If the postgres database is recording business transactions, you don't want to cause your business to stop being able to take credit cards because you generated a report.

Assuming you use a connection pool, why would it stop? Either the query returns a result or it doesnt? Am I missing something?

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#15

Post by CuriouslyC »

arsalanb wrote: post #14
CuriouslyC wrote: post #13

Earlier quoted context omitted.

If the postgres database is recording business transactions, you don't want to cause your business to stop being able to take credit cards because you generated a report.

Assuming you use a connection pool, why would it stop? Either the query returns a result or it doesnt? Am I missing something?

Reporting queries can put a significant load on the db, to the point that it interrupts service.

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#16

Post by arsalanb »

CuriouslyC wrote: post #15
arsalanb wrote: post #14

Earlier quoted context omitted.

Assuming you use a connection pool, why would it stop? Either the query returns a result or it doesnt? Am I missing something?

Reporting queries can put a significant load on the db, to the point that it interrupts service.

I see, thanks!

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#17

Post by necubi »

CuriouslyC wrote: post #15
arsalanb wrote: post #14

Earlier quoted context omitted.

Assuming you use a connection pool, why would it stop? Either the query returns a result or it doesnt? Am I missing something?

Reporting queries can put a significant load on the db, to the point that it interrupts service.

Futhermore, Postgres is an OLTP (transactional) database, designed to efficiently perform updates and deletes on individual rows. OLAP (analytical) databases/query engines like Clickhouse, Presto, Druid, etc. are designed for efficient processing across many rows of mostly unchanging data.

Analytical queries (like "find the average sales price across all orders over the past year, grouped by store and region") can be 100-1000x faster in an OLAP database compared to Postgres.

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#18

Post by gniting »

Nice to see more product development and offerings in this area. Well done.

[Full disclosure, I work for Prisma and we have a similar product called Pulse (https://prisma.io/pulse)]

Another use case for CDC is compliance. I reckon that in the near future, to ensure with data compliance regulations, CDC will become the better option for devs vs traditional seek/update/delete functions.

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#19

Post by matthieucan »

arsalanb wrote: post #12

Noob question: What is the advantage of replicating data into a warehouse vs. just querying it in place on a postgres database?

Additionally, unless your data model is designed as append-only (which is unusual and requires logic downstream), you won't be able to track updates and deletions, which are valuable for reporting

Re: Show HN: Peerdb Streams – Simple, native Postgres change data capture

#20

Post by ralfhn »

arsalanb wrote: post #12

Noob question: What is the advantage of replicating data into a warehouse vs. just querying it in place on a postgres database?

Data warehouses are structured to handle large volumes of data and complex queries more efficiently than a typical transactional database like PostgreSQL.
Post reply on HN