Nice write-up of the webhook data consistency problem - I've been bitten by some of those Stripe issues myself. I agree that webhooks should come hand-in-hand with a "records updated since" endpoint that allows high-level discrepancy checks and API endpoints to pull full logs for any record. There needs to be some sort of reconciliation process that is both fully verifiable and not a firehose of data.
The Valley of Webhooks
11–20 of 108 posts
Re: The Valley of Webhooks
#12I had the exact same thing with the Quickbooks api recently. You cannot trust the responses or webhooks at all.
On create a user or invoice for example sometimes it will return an error, yet it actually created the entity. This means you have to check manually after creating everything to know if its created properly.
Then you have the issue that sometimes quickbooks takes a while to update, and locks the company file while it does some background magic. This means you cannot immediately do the existence check, and also sometimes the check errors or times out which essentially means you need to keep checking forever until you can properly reconcile your db against theirs. But with hundreds/thousands of transactions per minute this state is never reached. You perpetually live in a state of trying to catch up but never managing it.
When I brought it up with Quickbooks dev support their response was literally "Its your job to make sure things are created properly in our system".
How did we get to this place where we started putting up with systems that cannot ever be trusted?
Re: The Valley of Webhooks
#13This topic always surprises me. I do not understand the sequence of logic that leads people to build synchronisation mechanisms based only on webhooks. Webhooks aren't at-least-once, nor at-most-once, nor are they guaranteed in-order. Some people build systems to make them more reliable, but if you really care about the data you need to think of a webhook delivery as best-effort, a bit like UDP. That's before you get…
Maybe the LLM has written maximum 50 words of the article by just being directed to switch things around and improve grammar and or internal consistency
Re: The Valley of Webhooks
#14> Ask with a cursor and you resume where you left off.
How does the provider know what event the cursor you provided refers to?
Sounds like external state that needs to be managed ("cursor" -> timestamp)
Re: The Valley of Webhooks
#15> Ask with a cursor and you resume where you left off. How does the provider know what event the cursor you provided refers to? Sounds like external state that needs to be managed ("cursor" -> timestamp)
That is completely fine. In the spec cursors just need to be lexicographically comparable. They could easily be timestamps if the provider chooses to. In many of my example they are just alphanumeric dictionaries.
Re: The Valley of Webhooks
#16Webhooks are a painful problem. To clarify, Stripe's events API definitely ships a cursor and polling it has been the method preferred by large consumers for a long time.
Stripe events API is one of the examples of how to do things properly. And SCROLL is just trying to create a common spec so that everyone offers a stripe-like event polling api.
Re: The Valley of Webhooks
#17This is really a great article from implementation perspective and the challenges associated with it. The OP already captured the reasons. Even though the reason 1 has been mentioned it did not go deep into it and somehow focussed a lot more on 2
> 1. Trigger a side effect: send the receipt, start the build, ping the channel. > 2. Keep a copy of the provider’s data correct:
On high level
1. System can either be PUSH or PULL, webhooks are essentially push and towards the end the OP is exploring the possibility with PULL. The caveat is that OP already iterated the PUSH mechanisms thrice and is aware of all the hardships and is somehow hoping that PULL would solve them. Unfortunately the grass is same on other side too.
a) The availability of the server can always be questionable in PULL mechanisms and its a lot of load on servers to support this kind of data at scale in bulk to multiple customers. You are essentially getting into database table scans. Its becomes a lot more costly with NOSQL databases.
b) The customer would end up making way too many calls to server even if data is not available or there would be additional latency when data was updated and when it was queried. This is one of the reasons why servers prefer to push instead of pull if they can find a listener available on other side.
c) CRLs (Certificate revocation lists) are good example which are available for PULL, same for all clients and yet rarely anyone does it correctly or does it at all even though its in security domain. In fact they are simple files on webservers in most of the implementations.
3. Logging data can and always have gaps, it should never be treated as source of truth. I might question the need for deduplication, usually there is a unique identifier and almost all databases support insert ignore kind of clause. Logging the event data just provides you with better availability and latency, the source of truth is still with the provider if a next step needs to happen.
4. If user cancelled the subscription in stripe and the event never arrived then its a system design issue or system availability issue on the client side. The complete data checksum or bulk imports at night are attempt to fix the problem in a hammerhead way . I understand it exists in lot of places, however it defeats the whole purpose.
Re: The Valley of Webhooks
#18This topic always surprises me. I do not understand the sequence of logic that leads people to build synchronisation mechanisms based only on webhooks. Webhooks aren't at-least-once, nor at-most-once, nor are they guaranteed in-order. Some people build systems to make them more reliable, but if you really care about the data you need to think of a webhook delivery as best-effort, a bit like UDP. That's before you get…
Maybe the LLM has written maximum 50 words of the article by just being directed to switch things around and improve grammar and or internal consistency
"And here’s the absurd part: that history exists."
"It’s a jigsaw puzzle where the manufacturer had the original picture, cut it up, mailed me the pieces one at a time, lost a few in the post, mailed some twice, and printed nothing on the box."
"and that’s the entire problem: nothing announces a gap."
"None of this is any provider’s bug. Their webhooks work exactly as documented. The problem is what a webhook is: a notification, “something happened, here’s a POST about it.”"
Almost this entire section is clearly written by an LLM
Re: The Valley of Webhooks
#19This is really a great article from implementation perspective and the challenges associated with it. The OP already captured the reasons. Even though the reason 1 has been mentioned it did not go deep into it and somehow focussed a lot more on 2 > 1. Trigger a side effect: send the receipt, start the build, ping the channel. > 2. Keep a copy of the provider’s data correct: On high level 1. System can either be PUSH…
> its a system design issue or system availability issue on the client side
If all events need to arrive, then the problem is not "notification" (which would be solved by webhooks) but "database replication": subscribe to new events, fetch the full snapshot, fetch the updates in range, have the monotonic value to establish the "range" in the first place. Reach the eventual consistency.
The proposed SCROLL handles half of these, which limits its use-cases.
Re: The Valley of Webhooks
#20With webhooks, consumers get to asynchronously respond to updates from a provider. If no data has changed, a provider will not send any updates. With SCROLL, consumers are responsible for choosing when to ask a provider for updates. Without a mechanism for knowing when data has changed, consumers will be forced to be pessimistic and poll providers for new data on some cadence. I see two issues with the proposal: (1)…
Assuming you're not using the proposed streaming option, I suppose you could always send a webhook for that fact alone? In other words, an empty notification, with semantics of "something has probably changed, better poll the SCROLL if you aren't already".
Doesn't doing that just reinvent eTags on hypertext resources from the RESTful wars and XML Web Services days 20 years ago?
1. Long Poll the cursor to pull down the latest events
2. Trigger a long-poll even if in exponential backoff because they shot you a webhook saying 'eTag changed!'