Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Webhooks do’s and dont’s: what we learned after integrating APIs

restful.io

Post reply on HN
Print view View original

21–30 of 83 posts

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#21

Post by madamelic »

bazizbaziz wrote: post #16

How do people in production handle the possibility that your service might miss a webhook notification? If you miss a notification you'll end up with stale data and you won't know it. Slack has a retry policy for a while but will then just give up. Another webhook provider I've looked at says nothing at all about this sort of thing. How do folks deal with this in production systems? Seems to me like the best way to a…

Stripe has a retry policy as well.

You can set up something where it will alert you if there are too many failures in a certain time period. That isn't offered by Stripe but you can build it.

If you mean in the case of "catastrophic failure", there is none.

If there is a "catastrophic failure" (machine gets shut off for a week, data center blown up, whatever), there are probably bigger issues or we probably would already know.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#22

Post by sly010 »

Aside: Webhooks are always a pain.

Implementing polling is easier for both sides.

I routinely have to integrate with random 3rd party systems, some with no or broken webhooks, some with no API at all. It turns out for my customers (this may not be always the case) eventual consistency is more important than timelyness.

What I do now every time I need to sync data from a third party is I always implement some sort of pull first with idempotent logic on my side. It's easier, and it allows me to just re-run things if something fails (e.g. network error, unexpected data in production, etc).

Only when that works reliably and only if required by the customer I implement a webhook, but I usually throw away most of the message and just wake up my polling worker that is otherwise polling relatively slowly.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#23

Post by madamelic »

enraged_camel wrote: post #20
madamelic wrote: post #14

Also, in your documentation, please show what the webhook events will look like since developers actually want to write code and not guess at what we will get. cough Stripe. ( https://stripe.com/docs/api#events )

You don't have to guess anything. Stripe lets you send test webhooks to the endpoint you specify[1]. You can set up something like ngrok[2] on your localhost to examine the headers and bodies, then write your code to parse them accordingly. [1] http://i.imgur.com/oCpxYwE.png [2] https://ngrok.com/

Yeah, that is one method.

I also learned about services that will set up test webhooks without having to go about setting up a server, etc.

I think I might use that, but I still think that docs should at least explain what will get sent. Maybe that is a bit too verbose in Stripe's case though.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#24

Post by niftich »

misterbowfinger wrote: post #3

This is going to sound bizarre, but why do webhooks and not just an AMQP queue? I get that receiving HTTP POSTs is easier, but it just seems better to setup a publisher/subscriber relationship. That way, if a subscriber goes down, they can always catch up. And publishers can allow messages to sit in the queue with a TTL and max_size. It seems like a win-win for everyone.

Because anyone can throw another route onto port 443 if you already host a website. A non-HTTP protocol running on a dedicated port, despite often being a superior solution, requires extra effort to set up, if the hosting environment even provides that option.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#25

Post by johns »

bazizbaziz wrote: post #16

How do people in production handle the possibility that your service might miss a webhook notification? If you miss a notification you'll end up with stale data and you won't know it. Slack has a retry policy for a while but will then just give up. Another webhook provider I've looked at says nothing at all about this sort of thing. How do folks deal with this in production systems? Seems to me like the best way to a…

When I was at IFTTT (a few years ago, so it's definitely changed since then) we tried not to rely on the content of the webhooks and just used them as a hint as you describe to fetch new data. Not every API made this easy though.

If receiving a webhook is critical, you should make your receiver do as little as possible to place the event into a resilient queueing system and then process them separately. That won't save you from bad DNS, TLS, etc. configs but it should help reduce the possibility that you DoS yourself with a flood of webhook events.

Also (shameless plug), you could monitor and log them (we offer retries if your server fails): https://www.runscope.com/product/alerts

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#26

Post by johns »

I did a talk a little while back on providing a good developer experience around webhooks that covers a lot of the same topics. I wish I had a recording of it, but the slides are here: https://speakerdeck.com/johnsheehan/crafting-a-great-webhook...

Edit: found the video https://www.youtube.com/watch?v=xc5ezyJjz1k&feature=youtu.be...

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#27

Post by iddqd »

bazizbaziz wrote: post #16

How do people in production handle the possibility that your service might miss a webhook notification? If you miss a notification you'll end up with stale data and you won't know it. Slack has a retry policy for a while but will then just give up. Another webhook provider I've looked at says nothing at all about this sort of thing. How do folks deal with this in production systems? Seems to me like the best way to a…

Maybe webhook providers could provide an endpoint where one could poll for events that failed to deliver.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#28

Post by boubiyeah »

Webhook only makes sense if you don't care a single bit about missing updates. If not, it's deeply flawed.

A pull model (polling, long-polling, SSE, etc) is strictly superior for synchronisation. You just can't "miss" updates, can restart from the beginning again and reinterpret past events in a different light, the client goes at its own pace, etc.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#29

Post by pdkl95 »

shizcakes wrote: post #5

I think the "securing webhooks" section is missing some critical tips that we've learned in production. 1) Resolve the DNS of the webhook URL, and compare all returned addresses from that resolution against an IP blacklist, which includes all RFC1918 addresses, EC2 instance metadata, and any other concerning addresses. 2) Even though it seems like you'd want to, do NOT blindly return an unexpected response to the per…

> IP blacklist

Please stop trying to enumerate badness. This is and always will be incomplete.

http://www.ranum.com/security/computer_security/editorials/d...

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#30

Post by jarcoal »

Sort of disagree with the send-everything-in-the-payload approach. It opens your system up to all sorts of weird edge case bugs like receiving hooks out of order which could mean stale data is considered fresh. It also means you have to care a lot more about verifying the authenticity of the request.
Post reply on HN