Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Webhooks do’s and dont’s: what we learned after integrating APIs

restful.io

Post reply on HN
Print view View original

31–40 of 83 posts

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#31

Post by brandur »

madamelic wrote: post #21
bazizbaziz wrote: post #16

How do people in production handle the possibility that your service might miss a webhook notification? If you miss a notification you'll end up with stale data and you won't know it. Slack has a retry policy for a while but will then just give up. Another webhook provider I've looked at says nothing at all about this sort of thing. How do folks deal with this in production systems? Seems to me like the best way to a…

Stripe has a retry policy as well. You can set up something where it will alert you if there are too many failures in a certain time period. That isn't offered by Stripe but you can build it. If you mean in the case of "catastrophic failure", there is none. If there is a "catastrophic failure" (machine gets shut off for a week, data center blown up, whatever), there are probably bigger issues or we probably would alr…

Stripe has an "events" API that can be polled to receive the same content that you would have received via Webhook [1].

(Disclaimer: I work there.)

If you missed some Webhooks due to an application failure, it's possible to page through it and look for omissions. I've spoken to at least one person integrating who had this sort of setup running as a regular process to protect against the possibility of dropped Webhooks. This usually works pretty well, but does start to break down at very large scale where events are being created faster than you can page back.

The possibility of dropped events is a major disavantage of Webhooks in my mind -- if you consider other alternatives for streaming APIs like a Kafka/Kinesis-like stream (over HTTP) that's simply iterated through periodically with a cursor, you avoid this sort of degenerate case completely, and also get nice things like a vastly reduced number of total HTTP requests, and guaranteed event ordering.

(But to be clear, Webhooks are overall pretty good.)

[1] https://stripe.com/docs/api#events

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#32

Post by nimblegorilla »

jarcoal wrote: post #30

Sort of disagree with the send-everything-in-the-payload approach. It opens your system up to all sorts of weird edge case bugs like receiving hooks out of order which could mean stale data is considered fresh. It also means you have to care a lot more about verifying the authenticity of the request.

"trust but verify"

Sometimes you want to ignore webhooks based on the payload (or put them into a different queue). It's faster to do that if you get the payload up front.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#33

Post by beachy »

sly010 wrote: post #22

Aside: Webhooks are always a pain. Implementing polling is easier for both sides. I routinely have to integrate with random 3rd party systems, some with no or broken webhooks, some with no API at all. It turns out for my customers (this may not be always the case) eventual consistency is more important than timelyness. What I do now every time I need to sync data from a third party is I always implement some sort of…

Long polling works brilliantly (where your API call blocks until there are some results or until timeout occurs - then you loop and call again).

Long polling gives you the best of both worlds - easy programming model with instant alerting rather than the delay of normal polling.

The only downside really is the need for a more or less permanently open connection per client. As long as the server does not use a naive "thread per connection" model this can scale up to many hundreds of thousands of clients or more.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#34

Post by brandur »

madamelic wrote: post #14

Also, in your documentation, please show what the webhook events will look like since developers actually want to write code and not guess at what we will get. cough Stripe. ( https://stripe.com/docs/api#events )

The implication was meant to be that the information under `data/object` is simply a full representation of another API resource of the type on which the event occurred, and that you can look elsewhere in the documentation to see exactly what each type will look like (you can see a subscription embedded in the sample response for example).

Fair enough that we could rewrite this to be more explicit about that though! We'll see what we can do to make that section more clear.

(I work for Stripe.)

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#35

Post by beachy »

jarcoal wrote: post #30

Sort of disagree with the send-everything-in-the-payload approach. It opens your system up to all sorts of weird edge case bugs like receiving hooks out of order which could mean stale data is considered fresh. It also means you have to care a lot more about verifying the authenticity of the request.

Agree. Its better to use webhooks as a pure signal that something has changed, and then in the case of update or insert, have the client pull whatever they want using normal API.

Otherwise, you end up in a descending vortex of madness trying to specify some protocol whereby the client can specify in advance which properties they care about.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#36

Post by draaglom »

jarcoal wrote: post #30

Sort of disagree with the send-everything-in-the-payload approach. It opens your system up to all sorts of weird edge case bugs like receiving hooks out of order which could mean stale data is considered fresh. It also means you have to care a lot more about verifying the authenticity of the request.

This is correct, and important.

Webhook payloads need to be logically monotonic[0]; this probably means either:

- having a lamport-clock timestamp for each payload so you can entirely discard older payloads in favour of new ones

- a well defined / consistent "merge" function over the subset of the payload you care about (e.g. maybe you know a customer's state can never go back from "registered" to "guest")

[0]: http://bloom-lang.net/calm/

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#37

Post by qyv »

giuliano84 wrote: post #10
detaro wrote: post #8

Earlier quoted context omitted.

The entire guide seems mostly consumer-centric: What do they as the one being called want, sender-side concerns are missing entirely.

Yes but that's why you build webhook for, to let people consume your content. I think DX is critical today and big companies can afford to do the heavy-lifting. As I mentioned in the Subscription Expiration paragraph I totally get Microsoft's reason to put a 72hrs expire date on subscriptions but it adds some friction on the consumer side.

Consumer convenience cannot be made at the expense of security measures and abuse mitigation, see: IoT.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#38

Post by draaglom »

bazizbaziz wrote: post #16

How do people in production handle the possibility that your service might miss a webhook notification? If you miss a notification you'll end up with stale data and you won't know it. Slack has a retry policy for a while but will then just give up. Another webhook provider I've looked at says nothing at all about this sort of thing. How do folks deal with this in production systems? Seems to me like the best way to a…

I like what Shopify does here - because your app is tied to a partner account, they can email you saying "this payload has failed 20 times in succession". If it fails too many times then the webhook is uninstalled.

Not to be snarky - but it's a distributed system. There's no way to guarantee you've got all updates! At a certain combination of latency and volume polling becomes impossible so webhooks (or something analogous) are all you've got :)

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#39

Post by Animats »

beachy wrote: post #33
sly010 wrote: post #22

Aside: Webhooks are always a pain. Implementing polling is easier for both sides. I routinely have to integrate with random 3rd party systems, some with no or broken webhooks, some with no API at all. It turns out for my customers (this may not be always the case) eventual consistency is more important than timelyness. What I do now every time I need to sync data from a third party is I always implement some sort of…

Long polling works brilliantly (where your API call blocks until there are some results or until timeout occurs - then you loop and call again). Long polling gives you the best of both worlds - easy programming model with instant alerting rather than the delay of normal polling. The only downside really is the need for a more or less permanently open connection per client. As long as the server does not use a naive "…

The good thing about long polling is that if the connection breaks, the keep-alive will time out and you'll know you're not getting updates. Assuming there's some keep-alive feature.
Post reply on HN