Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You Can't Guarantee Webhook Ordering

svix.com

Post reply on HN
Print view View original

11–20 of 27 posts

Re: You Can't Guarantee Webhook Ordering

#11

Post by makeitdouble »

tasn wrote: post #6
Diggsey wrote: post #3

This is rubbish, we've run with guaranteed webhook ordering for years, so the idea that you can't do is laughable. Timestamps don't solve the issue, and neither do "thin payloads" since the receiver has no idea how long to wait before assuming that the order is certain, and if you have a problem on the sender side it could cause logic errors for all of your clients. Most of these problems are solved if the receiver d…

Author here. You obviously CAN guarantee ordering, it's just that you can't guarantee it as the sender, you need cooperation from the receiver. Additionally, putting them in a receive queue on the receiver doesn't solve the issue unless the receiver takes extra care to also read from the queue in strict (non-overlapping) order which is rarely the case, and even then has significant throughput implications. So it real…

I agree with the parent on the weirdness of how the problem is stated in the first place.

On your customer and card example, the issue is not message delivery order but processing order, or more precisely prerequisite satisfaction.

My first thought looking at it was to just store the data of any of the hooks coming in, check the prerequisites each time, and only process the whole when everything needed has arrived.

Trying to dictate order from the sender without any cooperation from the receiver seems like a fool's errand, as in any real world scenario where it really matters, the receiver will also want a way to check it actually received everything in order.

Re: You Can't Guarantee Webhook Ordering

#12

Post by rexreed »

topspin wrote: post #9

Would have thought this is self evident. Intermediates exist and they can do essentially anything. Without absolute control over every aspect of the systems involved you have no guarantees about ordering.

There's no way to guarantee that the receiver will process the webhooks in order since you have no control over the receiver, but you CAN guarantee you can send them in a given order or at least identify the order of those webhooks and provide ways to identify or discover that order if the receiver so chooses to pay attention to ordering or use order discovery.

Re: You Can't Guarantee Webhook Ordering

#13

Post by rexreed »

makeitdouble wrote: post #11
tasn wrote: post #6

Earlier quoted context omitted.

Author here. You obviously CAN guarantee ordering, it's just that you can't guarantee it as the sender, you need cooperation from the receiver. Additionally, putting them in a receive queue on the receiver doesn't solve the issue unless the receiver takes extra care to also read from the queue in strict (non-overlapping) order which is rarely the case, and even then has significant throughput implications. So it real…

I agree with the parent on the weirdness of how the problem is stated in the first place. On your customer and card example, the issue is not message delivery order but processing order, or more precisely prerequisite satisfaction. My first thought looking at it was to just store the data of any of the hooks coming in, check the prerequisites each time, and only process the whole when everything needed has arrived. T…

I responded in much the same way below - your job as a sender is not to guarantee that the receiver will do its job in processing but to provide a reliable set of webhook messages so that if the receiver does fail, at least they can discover they've missed or skipped messages or are processing them out of order. As a sender, you certainly can provide guaranteed ordering or a way to identify the order of those messages. What you can't guarantee is that the receiver will process them in any given order if they choose to ignore the ordering you provide.

Re: You Can't Guarantee Webhook Ordering

#14

Post by tasn »

makeitdouble wrote: post #11
tasn wrote: post #6

Earlier quoted context omitted.

Author here. You obviously CAN guarantee ordering, it's just that you can't guarantee it as the sender, you need cooperation from the receiver. Additionally, putting them in a receive queue on the receiver doesn't solve the issue unless the receiver takes extra care to also read from the queue in strict (non-overlapping) order which is rarely the case, and even then has significant throughput implications. So it real…

I agree with the parent on the weirdness of how the problem is stated in the first place. On your customer and card example, the issue is not message delivery order but processing order, or more precisely prerequisite satisfaction. My first thought looking at it was to just store the data of any of the hooks coming in, check the prerequisites each time, and only process the whole when everything needed has arrived. T…

I understand it may not have been very clear. Though the point is that no one cares about delivery order, what they really care about is processing order. So it doesn't matter if you ensure delivery order if they process it out of order.

As for relying on the customers to get ordering correctly: it's actually more involved and easier to get wrong than people realize, and it's better to avoid it altogether in how you design your API if possible.

Re: You Can't Guarantee Webhook Ordering

#15

Post by spiffytech »

This is a big pain point with Stripe's webhooks, and I think there's ample room for improvement.

Senders could guarantee ordering by only sending webhook n+1 after the HTTP request for webhook n completes, rather than sending them concurrently or in arbitrary order. For efficiency, perhaps only guarantee ordering for hooks related to each resource rather than all of a customer's hooks.

Or, include a monotonic counter in the webhook so the recipient can tell when it would apply an old state on top of a new one.

What the recipient does when they receive the webhook is up to them (delays, parallelism, etc.), but at least they'd know the correct event order.

The author raises a good point about what to do in the face of errors, but I'd vastly prefer to handle special behavior upon recipient error (stall, dead letter queue) to the current Stripe reality of "things come in out of order, and we don't give you the info needed to reassemble the order on your end".

Re: You Can't Guarantee Webhook Ordering

#16

Post by topspin »

rexreed wrote: post #12
topspin wrote: post #9

Would have thought this is self evident. Intermediates exist and they can do essentially anything. Without absolute control over every aspect of the systems involved you have no guarantees about ordering.

There's no way to guarantee that the receiver will process the webhooks in order since you have no control over the receiver, but you CAN guarantee you can send them in a given order or at least identify the order of those webhooks and provide ways to identify or discover that order if the receiver so chooses to pay attention to ordering or use order discovery.

> but you CAN guarantee you can send them in a given order

That's not sufficient. Intermediate proxies can reorder your requests however they wish for whatever reason they want, and then change behavior with no notice at any time. In the real world of HTTP you'll get duplicates, false positives and every other conceivable failure mode.

> or at least identify the order of those webhooks and provide ways to identify or discover that order

Sure, you might invent some protocol that incorporates a sequence number or uses some chaining mechanism.

Thing is this; if you find yourself engaging in such gymnastics and you're any good as an engineer it needs to occur to you that you're using the wrong medium, hopefully long before you obligate yourself to the task. "Webhooks" are a pretty fragile thing to use when your requirements involve stuff like "order." If it did fail to occur to you then you're unlikely to get whatever sequencing mechanism you invent working properly either, because that's actually a hard problem that doesn't submit to the sort of muddlers unaware that "You Can't Guarantee Webhook Ordering."

Re: You Can't Guarantee Webhook Ordering

#17

Post by spiffytech »

The solution to receiving webhooks in unknown order is to ignore the payload and refetch the resource. Yet naively implemented, this still leaves race conditions on the recipient end: if two webhooks can come in at once, you have to make sure you process them serially, since your refetch or database write could complete in arbitrary order.

That's non-trivial engineering to foist upon every recipient of your webhooks.

I like the idea of the /events pull-based endpoint, which keeps engineering much simpler on for recipient: https://blog.sequin.io/events-not-webhooks/

Re: You Can't Guarantee Webhook Ordering

#18

Post by lliamander »

emadda wrote: post #2

You can also poll an /events endpoint to get a consistent ordering. I used /events to apply writes from Stripe to a local database for this reason in the tdog CLI: https://table.dog/blog/principles/events-are-better/

I think those are some excellent points, and after plenty of experience with webhook-based integrations, I agree that they are definitely a pain.

While I largely agree with you, I'm hesitant to say that it is always preferable to use an /events endpoint. There are two reasons:

1. This requires the client to essentially implement an event-sourced architecture. There are many advantages to such architectures, but they are more complicated and can be tricky to implement.

2. It's important to consider the direction of coupling in systems, and how that affects you're ability to evolve the architecture of the whole system.

3. Polling is generally going to involve a higher amount of network traffic, and will have to be weighed against the latency requirements for processing an event.

Re: You Can't Guarantee Webhook Ordering

#19

Post by tasn »

spiffytech wrote: post #15

This is a big pain point with Stripe's webhooks, and I think there's ample room for improvement. Senders could guarantee ordering by only sending webhook n+1 after the HTTP request for webhook n completes, rather than sending them concurrently or in arbitrary order. For efficiency, perhaps only guarantee ordering for hooks related to each resource rather than all of a customer's hooks. Or, include a monotonic counter…

The problem with "n+1 after HTTP request for webhooks n completes" is that your throughput is very adversely impacted by this. Let's assume that a webhook takes 1s to process (usually much slower when you include network latency, and endpoint processing time), you're effectively limited to 1 request per second.

Counter makes it slightly better because then you can reconstruct the order without the above artificial limit, though it's also not great (though indeed much better!).

Re: You Can't Guarantee Webhook Ordering

#20

Post by tasn »

spiffytech wrote: post #17

The solution to receiving webhooks in unknown order is to ignore the payload and refetch the resource. Yet naively implemented, this still leaves race conditions on the recipient end: if two webhooks can come in at once, you have to make sure you process them serially, since your refetch or database write could complete in arbitrary order. That's non-trivial engineering to foist upon every recipient of your webhooks.…

I mentioned it in the post, but yeah, also not trivial.

Polling /events solves some problems but introduces others. A mix of push (webhooks) and pull (/events) can also work, which is what I was referring to with the "thin clients", though it's not a great experience for many use-cases and it requires state (many webhooks recipients are stateless - e.g Zapier or Slack).

Post reply on HN