Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Collection of best practices for providing and consuming webhooks

webhooks.fyi

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Collection of best practices for providing and consuming webhooks

#11

Post by charcircuit »

m3h wrote: post #7
cuu508 wrote: post #6

There are some interesting attack vectors to be aware of if you run a service where users can define webhooks, and your service will will call the user-defined webhooks to notify about certain system events. In my case, a monitoring service which can send notifications by calling user-defined webhook. * Timeouts: the user can set up a webhook receiver that takes very long to generate a response. Your service must be…

It seems like webhooks have enough corner cases for the sender to require a specialized tool to protect itself from malicious users and to stay performant. Does anyone have suggestion for such tools/services that they might have used in production?

For half of those it's called a firewall and for the other half you should specify timeouts / max redirects.

Re: Collection of best practices for providing and consuming webhooks

#12

Post by freeqaz »

calyhre wrote: post #8
m3h wrote: post #7

Earlier quoted context omitted.

It seems like webhooks have enough corner cases for the sender to require a specialized tool to protect itself from malicious users and to stay performant. Does anyone have suggestion for such tools/services that they might have used in production?

At Slite, for all outgoing calls we use a sandboxed proxy. It has saved us a few times already. We detailed the trick in a blog post -> https://slite-tech-blog.ghost.io/anti-ssrf-solution/

This is the path I've seen be fairly robust at a few tech companies I've helped sort out this defense for. I've helped write libraries too but the proxy is the easiest approach when targeting many languages.

Re: Collection of best practices for providing and consuming webhooks

#13

Post by spiffytech »

cuu508 wrote: post #6

There are some interesting attack vectors to be aware of if you run a service where users can define webhooks, and your service will will call the user-defined webhooks to notify about certain system events. In my case, a monitoring service which can send notifications by calling user-defined webhook. * Timeouts: the user can set up a webhook receiver that takes very long to generate a response. Your service must be…

Stripe built an OSS egress network proxy that prevents webhooks from reaching internal resources:

https://github.com/stripe/smokescreen

Re: Collection of best practices for providing and consuming webhooks

#15

Post by tasn »

sparsely wrote: post #9

As this page makes very clear, it's actually pretty hard to make a robust webhooks implementation! What alternatives are there? I've looked at: * Publishing AWS EventBridge events to other accounts. * /events instead of webhooks https://blog.sequin.io/events-not-webhooks/ * ???

We built Svix[1] exactly for this reason. We make it super easy for companies to send webhooks reliably. There are only three companies with a "perfect score" on webhooks.fyi's list, we power one of them. ;)

[1] https://www.svix.com

Re: Collection of best practices for providing and consuming webhooks

#16

Post by technion »

cuu508 wrote: post #6

There are some interesting attack vectors to be aware of if you run a service where users can define webhooks, and your service will will call the user-defined webhooks to notify about certain system events. In my case, a monitoring service which can send notifications by calling user-defined webhook. * Timeouts: the user can set up a webhook receiver that takes very long to generate a response. Your service must be…

Just adding to that - don't forget about users defining AWS metadata addresses for a webhook. Returning IAM data to them can be .. bad.

Re: Collection of best practices for providing and consuming webhooks

#17

Post by tasn »

egorfine wrote: post #14

Excellent. I'm in process of building a service delivering notifications via webhooks right now. Thank you!

Shameless plug: we make it super easy for companies to send webhooks. https://www.svix.com

There are only 3 companies on webhooks.fyi's list with 3 green checks, and we're powering webhooks for one of those. :)

Re: Collection of best practices for providing and consuming webhooks

#18

Post by danpalmer »

Great resource! I’ve encountered most of these problems at some point.

It would be great to see advice about implementations. Things like when to process in a queue and when not to, tying idempotency tokens to database transactions, etc. These are subtle issues, but when done well can help make very robust systems.

Re: Collection of best practices for providing and consuming webhooks

#19

Post by tasn »

cuu508 wrote: post #6

There are some interesting attack vectors to be aware of if you run a service where users can define webhooks, and your service will will call the user-defined webhooks to notify about certain system events. In my case, a monitoring service which can send notifications by calling user-defined webhook. * Timeouts: the user can set up a webhook receiver that takes very long to generate a response. Your service must be…

Those are all great points! At Svix (we do webhooks as a service), we disallow redirects because of what you mentioned above, and because it's also just bad for performance (for both sides) and is most likely a configuration error anyway.

Resolution and timeouts: the aiohttp library for Python is slightly better in terms of letting you configure these things, though it's better to just use a sending proxy that does it all for you and is also located in an isolated VPC to make sure that you're protected.

Re: Collection of best practices for providing and consuming webhooks

#20

Post by egorfine »

tasn wrote: post #17
egorfine wrote: post #14

Excellent. I'm in process of building a service delivering notifications via webhooks right now. Thank you!

Shameless plug: we make it super easy for companies to send webhooks. https://www.svix.com There are only 3 companies on webhooks.fyi's list with 3 green checks, and we're powering webhooks for one of those. :)

Thank you!

Looks promising!

Post reply on HN