It's also really handy when API providers give a nice webhook UI that lets you view and resend webhooks during development.
[deleted]
Webhooks do’s and dont’s: what we learned after integrating APIs
51–60 of 83 posts
Re: Webhooks do’s and dont’s: what we learned after integrating APIs
#52How do people in production handle the possibility that your service might miss a webhook notification? If you miss a notification you'll end up with stale data and you won't know it. Slack has a retry policy for a while but will then just give up. Another webhook provider I've looked at says nothing at all about this sort of thing. How do folks deal with this in production systems? Seems to me like the best way to a…
I like what Shopify does here - because your app is tied to a partner account, they can email you saying "this payload has failed 20 times in succession". If it fails too many times then the webhook is uninstalled. Not to be snarky - but it's a distributed system. There's no way to guarantee you've got all updates! At a certain combination of latency and volume polling becomes impossible so webhooks (or something ana…
At a certain combination of latency and volume polling becomes impossible so webhooks (or something analogous) are all you've got :)
Isn't it the opposite? At a certain volume, when each polling request aways returns results, polling becomes more efficient than "interrupts". It's only at low volumes that webhooks are more efficient, since polling would have to issue a lot of requests with no response if a low latency is required.
Re: Webhooks do’s and dont’s: what we learned after integrating APIs
#53Webhook only makes sense if you don't care a single bit about missing updates. If not, it's deeply flawed. A pull model (polling, long-polling, SSE, etc) is strictly superior for synchronisation. You just can't "miss" updates, can restart from the beginning again and reinterpret past events in a different light, the client goes at its own pace, etc.
Luckily they aren't incompatible, you can take advantage of both.
Re: Webhooks do’s and dont’s: what we learned after integrating APIs
#54I think the "securing webhooks" section is missing some critical tips that we've learned in production. 1) Resolve the DNS of the webhook URL, and compare all returned addresses from that resolution against an IP blacklist, which includes all RFC1918 addresses, EC2 instance metadata, and any other concerning addresses. 2) Even though it seems like you'd want to, do NOT blindly return an unexpected response to the per…
> IP blacklist Please stop trying to enumerate badness. This is and always will be incomplete. http://www.ranum.com/security/computer_security/editorials/d...
I understand what you're saying here. But the baseline sanity set is pretty fixed. Localhost, RFC1918, IPv6 link local, etc. I'm not advocating folks blacklist every bad actor on the internet - that obviously cannot work - but there's some simple things you can do to prevent a malicious user from configuring webhooks that attack your internal services.
Re: Webhooks do’s and dont’s: what we learned after integrating APIs
#55I think the "securing webhooks" section is missing some critical tips that we've learned in production. 1) Resolve the DNS of the webhook URL, and compare all returned addresses from that resolution against an IP blacklist, which includes all RFC1918 addresses, EC2 instance metadata, and any other concerning addresses. 2) Even though it seems like you'd want to, do NOT blindly return an unexpected response to the per…
Point 1, can be a little more tricky than it seems. At first you'll think, I'll just use a regex to match known local addresses to protect again evil callback urls like http://127.0.0.1/status.
You'll realize though you have to actually resolve hostnames, because users can just create an A record of foo.bar.com that points to 127.0.0.1.
Re: Webhooks do’s and dont’s: what we learned after integrating APIs
#56This is going to sound bizarre, but why do webhooks and not just an AMQP queue? I get that receiving HTTP POSTs is easier, but it just seems better to setup a publisher/subscriber relationship. That way, if a subscriber goes down, they can always catch up. And publishers can allow messages to sit in the queue with a TTL and max_size. It seems like a win-win for everyone.
It's not AMQP (sadly) but something I've done previously is to have the actual webhook endpoint be as dumb as possible, doing nothing but accepting the payload (maybe with some very high level validation that the request was expected) and pushing it into a real queueing system. This means you can handle all sorts of failure modes, not just the backend going down, but also bugs in the consumer that would otherwise res…
Shameless plug: https://requesthub.xyz is ideal for these cases.
Re: Webhooks do’s and dont’s: what we learned after integrating APIs
#57Earlier quoted context omitted.
I like what Shopify does here - because your app is tied to a partner account, they can email you saying "this payload has failed 20 times in succession". If it fails too many times then the webhook is uninstalled. Not to be snarky - but it's a distributed system. There's no way to guarantee you've got all updates! At a certain combination of latency and volume polling becomes impossible so webhooks (or something ana…
At a certain combination of latency and volume polling becomes impossible so webhooks (or something analogous) are all you've got :) Isn't it the opposite? At a certain volume, when each polling request aways returns results, polling becomes more efficient than "interrupts". It's only at low volumes that webhooks are more efficient, since polling would have to issue a lot of requests with no response if a low latency…
Assuming here you mean something like a classic REST-alike "/events" endpoint which returns a bunch of stuff that's changed since the last time you requested it.
In that case, as the number of events grows, the HTTP transaction overhead goes to zero with polling, yeah.
But now you have a bunch of extra things which will impact your latency:
- The third-party service will do more work preparing the payload, meaning that the earliest event on the list no longer hits the wire right away
- related: someone might be holding a lock on event 63 of 100. Now other events have to wait for it before they can hit the wire
- In your application code, you may have to read the entire request before you can validate it or do anything with it (at least, this goes for APIs which speak JSON)
- You probably have to commit your transaction for the previous page of events before you can start your next request. Otherwise, whichever side of the network is keeping tabs on your current pointer in the list, that pointer may end up in the wrong place. Oops!
- If more events happen during the time it takes you to request a page than will fit on a page, then you're really stuck.
- An error anywhere in the super-http-transaction (network, user code...) now means that an entire page of updates has been delayed rather than just one.
It's possible to remove the sequential-ness constraint from our hypothetical "/events" but not without introducing other fun new problems.
Re: Webhooks do’s and dont’s: what we learned after integrating APIs
#58How do people in production handle the possibility that your service might miss a webhook notification? If you miss a notification you'll end up with stale data and you won't know it. Slack has a retry policy for a while but will then just give up. Another webhook provider I've looked at says nothing at all about this sort of thing. How do folks deal with this in production systems? Seems to me like the best way to a…
By periodic reconciliation of the full dataset.
[deleted]
Re: Webhooks do’s and dont’s: what we learned after integrating APIs
#59How do people in production handle the possibility that your service might miss a webhook notification? If you miss a notification you'll end up with stale data and you won't know it. Slack has a retry policy for a while but will then just give up. Another webhook provider I've looked at says nothing at all about this sort of thing. How do folks deal with this in production systems? Seems to me like the best way to a…
I would prefer to implement the sending of webhooks in bulk - if the consumer falls behind, they receive up to 100-1000 webhooks per request (depending on the size and complexity of each individual webhook - ids only is 1000, complex documents 100). This drastically cuts down on the number of concurrent requests to a single client when load is high, or the consumer broke down for a period of time.
Unfortunately, developers writing code to receive batch requests are often... inadequate, to say the least. They'll write basic looping code without any error/exception handling; so if the 3rd item in a bulk request of 100 items causes a server-side error for them, they throw a 500 Internal Server Error or similar and fail to continue processing items 4 through 100. You simply cannot batch webhooks as a producer, unless you detect a single failure from the client to process a batch as a cue to drop to performing "batches" of size 1 until you receive an error for a single request, at which point you return to bulk. Rinse and repeat.
Honestly, being the producer sending webhooks to consumers which are written by random developers is a nightmare. You have to understand that your customers will not write proper code to accept your webhook requests, even if each request is for a single webhook. You also must understand that your customers will not look to blame themselves for shitty code. You can retry 1,000 times over a 48 hour period, and if their code still fails to process the webhook, it will be YOUR fault, not theirs. Truthfully, it is horrible to be on the sending end of webhooks to random developers/customers.
Re: Webhooks do’s and dont’s: what we learned after integrating APIs
#60Webhook only makes sense if you don't care a single bit about missing updates. If not, it's deeply flawed. A pull model (polling, long-polling, SSE, etc) is strictly superior for synchronisation. You just can't "miss" updates, can restart from the beginning again and reinterpret past events in a different light, the client goes at its own pace, etc.
To expand on icebraining's comment, you can use weebhooks as notifications to poll.