You Can't Guarantee Webhook Ordering
1–10 of 27 posts
Re: You Can't Guarantee Webhook Ordering
#2You can also poll an /events endpoint to get a consistent ordering.
I used /events to apply writes from Stripe to a local database for this reason in the tdog CLI:
Re: You Can't Guarantee Webhook Ordering
#3This is rubbish, we've run with guaranteed webhook ordering for years, so the idea that you can't do is laughable.
Timestamps don't solve the issue, and neither do "thin payloads" since the receiver has no idea how long to wait before assuming that the order is certain, and if you have a problem on the sender side it could cause logic errors for all of your clients.
Most of these problems are solved if the receiver doesn't process the webhook immediately, but instead queues it internally. You don't have issues with the queue being stalled due to one bad webhook, because there is no event-specific processing happening on the receiver (other than perhaps ignoring some events). The queue can still be stalled if there is a wider problem, but as soon as the problem is resolved, the system can catch up on those queued webhooks, and synchronization integrity is maintained.
Having said all that, if I were to design a new system I would go with a pull-based system instead. In this system, the client would request a range (start time, max count) of events via an HTTP request, and the response would include the "end time" that can be used in the next query. A "webhook" would contain an empty payload, and would simply indicate that the queue had become non-empty - this could be omitted entirely if realtime updates are not required, instead having the client poll.
The advantages of this approach are that it's easy for consumers to "replay" a set of events if they accidentally lose them, and it's also a lot more efficient, since many events can be sent per request (we gain some of this benefit at the moment by supporting "batch" webhooks containing multiple events, but it requires opt-in from the client.) Additionally, it allows webhooks to be versioned more easily, since you can have versioned endpoints for fetching events, and it also allows you to have an arbitrary number of consumers of the same set of events with no additional complexity.
Re: You Can't Guarantee Webhook Ordering
#4Those who fear or do not know Kafka are doomed to work around its absence. Like I just cannot understand this mindset of "just educate people" when the system doesn't meet the requirements of its users. If your users want event ordering just give them event ordering.
You can even keep your existing webhook code by providing a synchronous bridge to Kafka so "just send them in order but wait for the 200 before sending the next one." Boom, now you are guaranteed the events are recorded and processed in order.
Re: You Can't Guarantee Webhook Ordering
#5This is rubbish, we've run with guaranteed webhook ordering for years, so the idea that you can't do is laughable. Timestamps don't solve the issue, and neither do "thin payloads" since the receiver has no idea how long to wait before assuming that the order is certain, and if you have a problem on the sender side it could cause logic errors for all of your clients. Most of these problems are solved if the receiver d…
[deleted]
Re: You Can't Guarantee Webhook Ordering
#6This is rubbish, we've run with guaranteed webhook ordering for years, so the idea that you can't do is laughable. Timestamps don't solve the issue, and neither do "thin payloads" since the receiver has no idea how long to wait before assuming that the order is certain, and if you have a problem on the sender side it could cause logic errors for all of your clients. Most of these problems are solved if the receiver d…
Author here.
You obviously CAN guarantee ordering, it's just that you can't guarantee it as the sender, you need cooperation from the receiver. Additionally, putting them in a receive queue on the receiver doesn't solve the issue unless the receiver takes extra care to also read from the queue in strict (non-overlapping) order which is rarely the case, and even then has significant throughput implications. So it really is all on the receiver. This piece was written from the context of the sender.
Timestamps definitely don't solve the issue, I explicitly said to use a centralized sequence number if you must (not a great idea in most cases). Thin payloads: the idea behind that is essentially to use the webhooks as a "please update" kind of notification and then you get the most recent data from the server. Essentially what you called a "pull system", it's a combination of both a push (webhook) to know when to pull, and the pull to get the data. This also doesn't work as nicely in many scenarios (because oftentimes, receivers want the data immediately without having to fetch), but it's good in others.
Please take a look at the content of the article (rather than just the title), I've addressed most of it there too.
Re: You Can't Guarantee Webhook Ordering
#7Those who fear or do not know Kafka are doomed to work around its absence. Like I just cannot understand this mindset of "just educate people" when the system doesn't meet the requirements of its users. If your users want event ordering just give them event ordering. You can even keep your existing webhook code by providing a synchronous bridge to Kafka so "just send them in order but wait for the 200 before sending…
[deleted]
Re: You Can't Guarantee Webhook Ordering
#8This is rubbish, we've run with guaranteed webhook ordering for years, so the idea that you can't do is laughable. Timestamps don't solve the issue, and neither do "thin payloads" since the receiver has no idea how long to wait before assuming that the order is certain, and if you have a problem on the sender side it could cause logic errors for all of your clients. Most of these problems are solved if the receiver d…
Author here. You obviously CAN guarantee ordering, it's just that you can't guarantee it as the sender, you need cooperation from the receiver. Additionally, putting them in a receive queue on the receiver doesn't solve the issue unless the receiver takes extra care to also read from the queue in strict (non-overlapping) order which is rarely the case, and even then has significant throughput implications. So it real…
[deleted]
Re: You Can't Guarantee Webhook Ordering
#9Would have thought this is self evident. Intermediates exist and they can do essentially anything. Without absolute control over every aspect of the systems involved you have no guarantees about ordering.
Re: You Can't Guarantee Webhook Ordering
#10This is rubbish, we've run with guaranteed webhook ordering for years, so the idea that you can't do is laughable. Timestamps don't solve the issue, and neither do "thin payloads" since the receiver has no idea how long to wait before assuming that the order is certain, and if you have a problem on the sender side it could cause logic errors for all of your clients. Most of these problems are solved if the receiver d…
> In this system, the client would request a range (start time, max count) of events via an HTTP request, and the response would include the "end time" that can be used in the next query.
What happens if two transactions commit out of order? tx1 with a lower timestamp commits after tx2 with a higher timestamp has committed - and your client just saw tx2's timestamp.
Or if you have ≥$maxCount number of events changed the same exact timestamp?