Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks
1–10 of 39 posts
Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks
#2Very cool and I see myself using this in the future.
Is an approach like this best practise or rather meant as intermediate hack?
Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks
#3Very cool and I see myself using this in the future. Is an approach like this best practise or rather meant as intermediate hack?
In general I have not seen the DBMS emitting changes as a best practice. The applications modifying the data should be emitting events or messages appropriately. Someone can correct me if I am wrong.
Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks
#4Interesting. A quick look at the code indicates you're just doing a notify on the whole row. Have you tested with large rows? The postgres notify function fails with a payload larger than 8000 bytes, and that limit cannot be changed at runtime, last I looked.
I wrote a similar thing to notify on changes, but had to just fire notifies on what changed (id, column names) and there was a second step in the listening code to fetch the full details.
Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks
#5Looks cool. Seems like a good alternative if you can’t use listen/notify for some reason.
I’ve used listen/notify with triggers for a while with good success. The flexibility of my apps controlling what they want to listen for instead of having that config elsewhere.
Downside to l/n is that if you aren’t listening you aren’t getting changes. Seems like the same is true if the webhook fails here since there isn’t a retry.
If you want to check out another cool alternative see Debezium[1]. It’s uses PGs logical decoding feature to stream changes. Supports protobufs and JSON output.
Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks
#6Very cool and I see myself using this in the future. Is an approach like this best practise or rather meant as intermediate hack?
In general I have not seen the DBMS emitting changes as a best practice. The applications modifying the data should be emitting events or messages appropriately. Someone can correct me if I am wrong.
The most reliable implementation of this pattern is to use the replication binary logs of the database. Your application cannot atomically publish a write to both a database and a messaging system (without introducing a massive headache that you will implement incorrectly).
Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks
#7Very cool and I see myself using this in the future. Is an approach like this best practise or rather meant as intermediate hack?
In general I have not seen the DBMS emitting changes as a best practice. The applications modifying the data should be emitting events or messages appropriately. Someone can correct me if I am wrong.
That was the conclusion I came to as well. If you just want to push data to a data warehouse by listening to the WAL or NOTIFY and then batch a bunch of updates together that's one thing. If you intend it to be application level events then coupling yourself to the schema of some other service (among other issues) seems problematic.
Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks
#8Earlier quoted context omitted.
In general I have not seen the DBMS emitting changes as a best practice. The applications modifying the data should be emitting events or messages appropriately. Someone can correct me if I am wrong.
The most reliable implementation of this pattern is to use the replication binary logs of the database. Your application cannot atomically publish a write to both a database and a messaging system (without introducing a massive headache that you will implement incorrectly).
Yeah but I think it's important to consider whether any microservice should be able to listen to any other microservice inserts or updates. I think using the replication log as the underlying mechanism is a good idea, but you should really be choosing which messages to expose because it forms a public API.
Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks
#9Very cool and I see myself using this in the future. Is an approach like this best practise or rather meant as intermediate hack?
It’s more of an intermediate hack I’d say unless you don’t mind missing the odd event.
We’re working on another implementation that uses the replication stream (like dbezium) that will be a more best-practice solution.
(I work at Hasura)
Re: Show HN: Skor – Drop-in microservice to get Postgres changes as JSON webhooks
#10Very cool and I see myself using this in the future. Is an approach like this best practise or rather meant as intermediate hack?
In general I have not seen the DBMS emitting changes as a best practice. The applications modifying the data should be emitting events or messages appropriately. Someone can correct me if I am wrong.
Part of the thought process around that is that most DBs don’t have a good option to do it well.
In larger applications, if you’ve got a need to have a stored procedure handle a complex modification your application will be blind to it.
Plus, at the application level you don’t have any way to enforce that the particular place in your code that modified the data and then emits the change is the ONLY place in your code that can modify that data. This becomes more pronounced over time too.
The problem is amplified if you find something that needs to be done to inbound data that is better handled in another language for some reason. Then you have to remember to duplicate the logic or hook your new code into the application, forcing a new layer in front of your database.
There are some things that the database is better left handling. :)