Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Webhooks do’s and dont’s: what we learned after integrating APIs

restful.io

Post reply on HN
Print view View original

11–20 of 83 posts

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#11

Post by jbert »

misterbowfinger wrote: post #3

This is going to sound bizarre, but why do webhooks and not just an AMQP queue? I get that receiving HTTP POSTs is easier, but it just seems better to setup a publisher/subscriber relationship. That way, if a subscriber goes down, they can always catch up. And publishers can allow messages to sit in the queue with a TTL and max_size. It seems like a win-win for everyone.

One difference is in "dead connection detection". How do you know that your AMQP connection is down? At some level you're polling, whether that be TCP keepalive, application keepalive or something else.

If you're doing polling, you're actually back at the same pre-webhook place - polling their server on some timescale which is a compromise between latency and load.

Yes, a TCP keepalive is generally cheaper than an HTTP long poll request, but only by a constant factor.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#12

Post by giuliano84 »

shizcakes wrote: post #5

I think the "securing webhooks" section is missing some critical tips that we've learned in production. 1) Resolve the DNS of the webhook URL, and compare all returned addresses from that resolution against an IP blacklist, which includes all RFC1918 addresses, EC2 instance metadata, and any other concerning addresses. 2) Even though it seems like you'd want to, do NOT blindly return an unexpected response to the per…

Point 3 is spot on. I think it would be a good strategy to avoid expire dates on subscriptions. Producers could take decisions on whether or not keep sending data by monitoring responses on the consumer's target URL.

Another thing that it should be worth mentioning is that some services batch notifications (e.g Facebook Messenger) so that they can send more data in a single POST request.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#13

Post by [deleted] »

giuliano84 wrote: post #10
detaro wrote: post #8

Earlier quoted context omitted.

The entire guide seems mostly consumer-centric: What do they as the one being called want, sender-side concerns are missing entirely.

Yes but that's why you build webhook for, to let people consume your content. I think DX is critical today and big companies can afford to do the heavy-lifting. As I mentioned in the Subscription Expiration paragraph I totally get Microsoft's reason to put a 72hrs expire date on subscriptions but it adds some friction on the consumer side.

[deleted]

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#16

Post by bazizbaziz »

How do people in production handle the possibility that your service might miss a webhook notification? If you miss a notification you'll end up with stale data and you won't know it.

Slack has a retry policy for a while but will then just give up. Another webhook provider I've looked at says nothing at all about this sort of thing. How do folks deal with this in production systems?

Seems to me like the best way to address this issue is to use the webhook as a hint that you need to run some other process that guarantees you've got all updates.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#17

Post by jon-wood »

misterbowfinger wrote: post #3

This is going to sound bizarre, but why do webhooks and not just an AMQP queue? I get that receiving HTTP POSTs is easier, but it just seems better to setup a publisher/subscriber relationship. That way, if a subscriber goes down, they can always catch up. And publishers can allow messages to sit in the queue with a TTL and max_size. It seems like a win-win for everyone.

It's not AMQP (sadly) but something I've done previously is to have the actual webhook endpoint be as dumb as possible, doing nothing but accepting the payload (maybe with some very high level validation that the request was expected) and pushing it into a real queueing system.

This means you can handle all sorts of failure modes, not just the backend going down, but also bugs in the consumer that would otherwise result in losing the request. I've not tried it, but I imagine this is a pretty good usecase for AWS Lambda as it's a small bit of glue code.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#18

Post by detaro »

arxpoetica wrote: post #15

Honest question. Why webhooks over something like push/pull handshaked socket?

For the receiver: Everything that can run a dynamic website can run a webhook receiver, opening an arbitrary socket connection isn't possible in all environments (e.g. software running on shared hosting or PaaS). You'd also need to define and implement a protocol on top of said socket, whereas more or less every web developer knows what to do with HTTP POST with a JSON payload.

And for the sender, keeping many concurrent connections open can be quite a challenge. Sending Webhooks also takes resources, but at least you can easily distribute it over many machines/processes if necessary.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#19

Post by ec109685 »

arxpoetica wrote: post #15

Honest question. Why webhooks over something like push/pull handshaked socket?

Slack offers both. Their realtime API is especially nice because you connect to it rather than needing to deploy public facing web services to receive events from them.

On the other hand, if you are not connected, messages could be lost unless you build in syncing capability, whereas with web hooks, Slack will handle the retries for you.

Re: Webhooks do’s and dont’s: what we learned after integrating APIs

#20

Post by enraged_camel »

madamelic wrote: post #14

Also, in your documentation, please show what the webhook events will look like since developers actually want to write code and not guess at what we will get. cough Stripe. ( https://stripe.com/docs/api#events )

You don't have to guess anything. Stripe lets you send test webhooks to the endpoint you specify[1]. You can set up something like ngrok[2] on your localhost to examine the headers and bodies, then write your code to parse them accordingly.

[1]http://i.imgur.com/oCpxYwE.png

[2]https://ngrok.com/

Post reply on HN