Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Collection of best practices for providing and consuming webhooks

webhooks.fyi

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Collection of best practices for providing and consuming webhooks

#21

Post by tasn »

danpalmer wrote: post #18

Great resource! I’ve encountered most of these problems at some point. It would be great to see advice about implementations. Things like when to process in a queue and when not to, tying idempotency tokens to database transactions, etc. These are subtle issues, but when done well can help make very robust systems.

I recorded a video with advice about designing a webhooks system a couple of months ago: https://www.svix.com/blog/webhook-architecture-design/

It's high level, so it doesn't touch everything you mentioned, but it's a good primer.

Re: Collection of best practices for providing and consuming webhooks

#22

Post by rajamaka »

m3h wrote: post #7
cuu508 wrote: post #6

There are some interesting attack vectors to be aware of if you run a service where users can define webhooks, and your service will will call the user-defined webhooks to notify about certain system events. In my case, a monitoring service which can send notifications by calling user-defined webhook. * Timeouts: the user can set up a webhook receiver that takes very long to generate a response. Your service must be…

It seems like webhooks have enough corner cases for the sender to require a specialized tool to protect itself from malicious users and to stay performant. Does anyone have suggestion for such tools/services that they might have used in production?

Outgoing webhooks dispatched from a Lambda seemed to solve most of the above problems for us

Re: Collection of best practices for providing and consuming webhooks

#23

Post by caseysoftware »

Hello all! I'm one of the creators of webhooks.fyi over at https://ngrok.com/

Happy to answer any questions!

First, we started this project when we launched in-product webhook verification. We realized that we were collecting great information, had uncovered some clear and obvious patterns (both good & bad), and knew we could publish the results to help the ecosystem as a whole.

Next, yes it's ridiculously hard to build webhooks correctly. There are so many shortcuts that feel "okay" and you don't really think about until you realize that it didn't quite work as you had planned and now you have a gap. We're hoping the site can help people move more of that thinking earlier and make better decisions.

Finally, we're missing a bunch of webhooks! While we looked at 100+ in our own research, we only had time to add ~50 to this initial pass. If you'd like to add your favorites, pull requests welcome: https://github.com/ngrok/webhooks.fyi

And yes, seriously, you can use ngrok to verify webhooks pretty easily:

ngrok http 80 --verify-webhook=slack --verify-webhook-secret=[secret]

Re: Collection of best practices for providing and consuming webhooks

#24

Post by sudobinbash »

cuu508 wrote: post #6

There are some interesting attack vectors to be aware of if you run a service where users can define webhooks, and your service will will call the user-defined webhooks to notify about certain system events. In my case, a monitoring service which can send notifications by calling user-defined webhook. * Timeouts: the user can set up a webhook receiver that takes very long to generate a response. Your service must be…

Some webhook providers implement one-time verifications, CRCs, and heartbeat checks to address some of these problems:

https://webhooks.fyi/security/one-time-verification-challeng... https://webhooks.fyi/ops-experience/resiliency

Some also don't follow 301s or 302s and will process only specific http responses.

Re: Collection of best practices for providing and consuming webhooks

#25

Post by caseysoftware »

tasn wrote: post #15
sparsely wrote: post #9

As this page makes very clear, it's actually pretty hard to make a robust webhooks implementation! What alternatives are there? I've looked at: * Publishing AWS EventBridge events to other accounts. * /events instead of webhooks https://blog.sequin.io/events-not-webhooks/ * ???

We built Svix[1] exactly for this reason. We make it super easy for companies to send webhooks reliably. There are only three companies with a "perfect score" on webhooks.fyi's list, we power one of them. ;) [1] https://www.svix.com

That's awesome and glad to see others making it easier and better to do this right.

Obviously you should add some of your other customers to make sure they get the attention they deserve.

Also, there are other patterns or practices that should be in the list, let us know. PRs are welcome but feel free to email me directly: danger@ngrok.com

Re: Collection of best practices for providing and consuming webhooks

#26

Post by tasn »

caseysoftware wrote: post #23

Hello all! I'm one of the creators of webhooks.fyi over at https://ngrok.com/ Happy to answer any questions! First, we started this project when we launched in-product webhook verification. We realized that we were collecting great information, had uncovered some clear and obvious patterns (both good & bad), and knew we could publish the results to help the ecosystem as a whole. Next, yes it's ridiculously hard to bu…

Oh, I didn't know about the new --verify-webhook addition to ngrok, that's cool!

Who can I chat with about adding Svix style webhooks[1] to ngrok? We do webhook sending as a service and power the webhooks for many companies, including one of the only three that got a "perfect score" on your list.

Adding Svix will make ngrok automatically support many webhooks services. I'd be happy to chat further my email is my name at the domain of the Svix service.

[1] https://docs.svix.com/receiving/verifying-payloads/how-manua...

Re: Collection of best practices for providing and consuming webhooks

#27

Post by [deleted] »

tasn wrote: post #26
caseysoftware wrote: post #23

Hello all! I'm one of the creators of webhooks.fyi over at https://ngrok.com/ Happy to answer any questions! First, we started this project when we launched in-product webhook verification. We realized that we were collecting great information, had uncovered some clear and obvious patterns (both good & bad), and knew we could publish the results to help the ecosystem as a whole. Next, yes it's ridiculously hard to bu…

Oh, I didn't know about the new --verify-webhook addition to ngrok, that's cool! Who can I chat with about adding Svix style webhooks[1] to ngrok? We do webhook sending as a service and power the webhooks for many companies, including one of the only three that got a "perfect score" on your list. Adding Svix will make ngrok automatically support many webhooks services. I'd be happy to chat further my email is my name…

[deleted]

Re: Collection of best practices for providing and consuming webhooks

#28

Post by tasn »

caseysoftware wrote: post #25
tasn wrote: post #15

Earlier quoted context omitted.

We built Svix[1] exactly for this reason. We make it super easy for companies to send webhooks reliably. There are only three companies with a "perfect score" on webhooks.fyi's list, we power one of them. ;) [1] https://www.svix.com

That's awesome and glad to see others making it easier and better to do this right. Obviously you should add some of your other customers to make sure they get the attention they deserve. Also, there are other patterns or practices that should be in the list, let us know. PRs are welcome but feel free to email me directly: danger@ngrok.com

Will email you now in a bit! I also just replied to your other comment about another topic, so there's much to discuss. :)

Re: Collection of best practices for providing and consuming webhooks

#29

Post by dgudkov »

If only there was a single standard for webhook subscription verification. Cloud services invent their own authentication protocols. Webhook verification in Dropbox is different from that in Trello. The lack of a single standard makes it hard to design a universal incoming webhook service (we faced this problem).

Re: Collection of best practices for providing and consuming webhooks

#30

Post by tasn »

dgudkov wrote: post #29

If only there was a single standard for webhook subscription verification. Cloud services invent their own authentication protocols. Webhook verification in Dropbox is different from that in Trello. The lack of a single standard makes it hard to design a universal incoming webhook service (we faced this problem).

Amen. FWIW, we are open source, and we publish libraries that let anyone sign and verify webhooks, hoping to save people the work and have a unified standard.

Libs: https://github.com/svix/svix-webhooks

Docs: https://docs.svix.com/receiving/verifying-payloads/how-manua...

The docs mention `Svix-Signature`, but we (and the libs) also support `Webhooks-Signature` which are useful for the generic standard.

The fact that we don't have REST for webhooks is both silly and annoying. Especially since (as you can see on the list in the original post) many companies who implement webhooks end up doing it poorly.

Post reply on HN