Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks

github.com

Post reply on HN
Print view View original

11–20 of 39 posts

Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks

#11

Post by mesid »

odammit wrote: post #5

Looks cool. Seems like a good alternative if you can’t use listen/notify for some reason. I’ve used listen/notify with triggers for a while with good success. The flexibility of my apps controlling what they want to listen for instead of having that config elsewhere. Downside to l/n is that if you aren’t listening you aren’t getting changes. Seems like the same is true if the webhook fails here since there isn’t a re…

Hi

I did use Debezium in a Kubernetes cluster to stream Postgres changes into Kafka and then through a custom application that can send them to clients via HTTP or WebSockets. But the entire system had multiple points of failure. So we then decided to write our own thing that connects to the LR slot and sends over WebSockets. Working on open sourcing that too!

Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks

#12

Post by btown »

odammit wrote: post #5

Looks cool. Seems like a good alternative if you can’t use listen/notify for some reason. I’ve used listen/notify with triggers for a while with good success. The flexibility of my apps controlling what they want to listen for instead of having that config elsewhere. Downside to l/n is that if you aren’t listening you aren’t getting changes. Seems like the same is true if the webhook fails here since there isn’t a re…

I remember this old article about Debezium's now-unmaintained predecessor, Bottled Water. Its diagrams may still be useful for those wanting to understand how this might be used.

https://www.confluent.io/blog/bottled-water-real-time-integr...

Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks

#13

Post by chatmasta »

hderms wrote: post #8
ryanworl wrote: post #6

Earlier quoted context omitted.

The most reliable implementation of this pattern is to use the replication binary logs of the database. Your application cannot atomically publish a write to both a database and a messaging system (without introducing a massive headache that you will implement incorrectly).

Yeah but I think it's important to consider whether any microservice should be able to listen to any other microservice inserts or updates. I think using the replication log as the underlying mechanism is a good idea, but you should really be choosing which messages to expose because it forms a public API.

The receivers choose which messages to expose by selecting which events to listen for. The broker then only forwards those events. The question is, how do the receivers specify the event type? Do they use the schema of the publisher, or some intermediate interface at the broker?

Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks

#15

Post by bearjaws »

Very cool, at my current job we have a realtime ETL process using Maxwell & Apache Kafka to stream replication events to a set of Node servers for a variety of transformations.

Seeing that this isn't hooked into replication is a bit worrying, but it seems they are going to implement it as a replication client which should make it much more robust.

Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks

#16

Post by matte_black »

Lord_Zero wrote: post #3
anc84 wrote: post #2

Very cool and I see myself using this in the future. Is an approach like this best practise or rather meant as intermediate hack?

In general I have not seen the DBMS emitting changes as a best practice. The applications modifying the data should be emitting events or messages appropriately. Someone can correct me if I am wrong.

I’d venture to say that is wrong at least in the case of Postgres.

In order to be able to reason with certainty about your data, information should be coming from the source closest to the truth, and that would be your database. You can’t really trust the application, it may have been written wrong.

Postgres’s LISTEN/NOTIFY commands, which are not part of the SQL standard, exist for these cases.

Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks

#19

Post by octernion »

odammit wrote: post #5

Looks cool. Seems like a good alternative if you can’t use listen/notify for some reason. I’ve used listen/notify with triggers for a while with good success. The flexibility of my apps controlling what they want to listen for instead of having that config elsewhere. Downside to l/n is that if you aren’t listening you aren’t getting changes. Seems like the same is true if the webhook fails here since there isn’t a re…

Unfortunately, most of these LR programs don't work in AWS (RDS) land, since they require PG plugins, and RDS LR only supports the built-in test decoder.

I wrote a similar program that uses LR to stream changes to AWS's Kinesis[1], using a mini library that parses the output of the default test decoder[2] as a result, in case anyone else is in RDS and has the same limitation.

[1] https://github.com/nickelser/pg_kinesis [2] https://github.com/nickelser/parselogical

Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks

#20

Post by nrmitchi »

octernion wrote: post #19
odammit wrote: post #5

Looks cool. Seems like a good alternative if you can’t use listen/notify for some reason. I’ve used listen/notify with triggers for a while with good success. The flexibility of my apps controlling what they want to listen for instead of having that config elsewhere. Downside to l/n is that if you aren’t listening you aren’t getting changes. Seems like the same is true if the webhook fails here since there isn’t a re…

Unfortunately, most of these LR programs don't work in AWS (RDS) land, since they require PG plugins, and RDS LR only supports the built-in test decoder. I wrote a similar program that uses LR to stream changes to AWS's Kinesis[1], using a mini library that parses the output of the default test decoder[2] as a result, in case anyone else is in RDS and has the same limitation. [1] https://github.com/nickelser/pg_kines…

IIRC Debezium is RDS compatible since version 0.7.0
Post reply on HN