Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Valley of Webhooks

weli.dev

Post reply on HN
Print view View original

51–60 of 108 posts

Re: The Valley of Webhooks

#51

Post by zbentley »

__MatrixMan__ wrote: post #23

If you have two or more services that need to agree about state, and you have some set of rules that govern what state changes are valid, and you don't want to mess around with any of this "what do we do when we miss and update vs when we get two of the same update" nonsense, and the services aren't in a position to query the same database, then you should really consider a permissioned blockchain. Consensus hard, bu…

Why is cryptographically verifying record sequentiality an important property here?

If I offer my customers a source of ordered records, the "trust" in that system is the fact that they pay me to make sure records are ordered. If I sell a fast or slow log database, approximately zero customers in the world care to verify ordering cryptographically.

Or by "blockchain" do you just mean .... records with sequential IDs? Because sequential, guaranteed IDs surface gappiness/idempotency a lot easier than Markov chains over cryptographic primitives.

That also doesn't address the other core problems in the article: the replication (or data retrieval/polling) protocol is a lot more complex than a blockchain's "I can verify and replicate the entire chain state from the beginning of time to you" single behavior. People want more specificity than that.

Re: The Valley of Webhooks

#52

Post by cyberax »

Just treat webhooks as a hint.

Write your code to work as a "reconciler" that checks the state of the remote system and reconciles it with the local view of that system. Run reconciliation for the full state periodically using a scheduler, and then treat webhooks as a hint to run the reconciler immediately.

This way, you will have a robust system that can survive logical bugs and outages because you don't store the synchronization state per se.

In the case of Stripe, for example, have a process that polls every open checkout session every couple of minutes. A webhook then just triggers the run earlier. If you're worried about DDoS, have an exponential backoff for the poll period.

Theoretically polling doesn't scale, but in practice it works just fine.

Re: The Valley of Webhooks

#53

Post by tlonny »

I much prefer cursor paginated API requests vs. webhooks. The obvious downside being that in order to not get 429'd you need a respectable poll frequency - meaning you lose reactivity to new events.

Thus I think webhooks still have a place - but as a simple "poke" that can be sent to the client to tell them something has changed - supplementing a default low frequency polling interval.

This gives us the best of both worlds:

1. No need to bother de-duping/retrying pokes - if you miss a webhook you will shortly recover anyway when you next poll. 2. No need for any local-specific tunnelling/tooling - the local app will work just fine with the default poll interval. 3. No need to keep a connection live for each client. 4. All the good stuff OP mentioned in his blog post.

Re: The Valley of Webhooks

#54

Post by zbentley »

tasn wrote: post #45
zbentley wrote: post #44

This article resonates with similar struggles I've had at multiple companies. Its content is valuable. I wish the writing and the article itself had less of a slop odor. That aside, what I don't understand (especially having worked on the side of the webhook sender , which is itself really tricky to get correct/performant/cheap) is why more companies which broadcast webhooks don't, say, provide direct access to Kafka…

We (Svix, webhooks infra) actually help our customers directly write to Kafka topics, S3 buckets, SQS, etc. and have for a few years now. There are definitely people that adopt that, but receivers as well prefer the simplicity of webhooks.

> receivers as well prefer the simplicity of webhooks.

Do you see receive-side customers as prepared to outright reject paying for vendors that only offer non-webhook event streams if there's a webhook-ful competitor available? Or is that preference more of the "well, it's easier to add a webhook route to our existing webapp than it is to run a stream consumer/cron/whatever, but neither of those two is cost- or effort-preventative" situation, where customers don't consider event delivery systems to be the main differentiating factor?

Re: The Valley of Webhooks

#55

Post by oasisbob »

bytesandbots wrote: post #43

With the proposed solution every consumer will have a persistent connection to the server irrespective of the frequency of events. This setup seems inefficient unless you have a very high volume of events coming in. Many CDN networks have a limit on how long a connection you can open. And data providers will not prefer serving persistent requests. Problems listed are signatures, dedup, buffering, bootstrap, cron. Eve…

Open TCP connections can also be wildly cheap and efficient - Apple Push Notifications (APNS) and Android's push systems maintain open TCP connections to just about every mobile device on this planet.

An open connection is just a bit of state on either end. The C10K problem has been solved for ages.

Anyone remember consuming Twitter hoses back in the day? Those were also long-lived persistent connections for efficiency reasons.

Re: The Valley of Webhooks

#56

Post by throwaway7783 »

tlonny wrote: post #53

I much prefer cursor paginated API requests vs. webhooks. The obvious downside being that in order to not get 429'd you need a respectable poll frequency - meaning you lose reactivity to new events. Thus I think webhooks still have a place - but as a simple "poke" that can be sent to the client to tell them something has changed - supplementing a default low frequency polling interval. This gives us the best of both…

Absolutely. I have been so livid at so many applications for not providing a decent CDC API (and dont forget deletes). Salesforce perhaps is the best out there. Imagine if every application exposed a standard CDC API, the world of integrations would be so much better.

Webhooks are fine, but a pollable API is a must have. The amount of hacks I had to do at work to workaround shitty APIs gives me nightmares.

Re: The Valley of Webhooks

#57

Post by lelanthran »

alt227 wrote: post #12

I had the exact same thing with the Quickbooks api recently. You cannot trust the responses or webhooks at all. On create a user or invoice for example sometimes it will return an error, yet it actually created the entity. This means you have to check manually after creating everything to know if its created properly. Then you have the issue that sometimes quickbooks takes a while to update, and locks the company fil…

> You cannot trust the responses or webhooks at all.

Well... yeah. I mean it's pretty obvious, no?

Here's some things that could go wrong regardless of what care the software tries to provide:

- The transaction completed on the backend cluster but the app instance died before if could create the response and after it committed the transaction.

- The transaction completed, the app instance transmitted a response, but the load-balancer/reverse-proxy in-between died before it could relay that response.

- Everything went well, but the ISP dropped some packets before it could get to you.

- Everything completed and the ISP stayed up, but on your end the response was flagged as malicious, or never made it through your load-balancer.

So, yeah. in general when you make an API request and get an error you have to check if the state was changed anyway, and if you aren't doing that you're doing it wrong anyway and cannot blame the system on the other side for returning errors.

Re: The Valley of Webhooks

#58

Post by weli »

oasisbob wrote: post #55
bytesandbots wrote: post #43

With the proposed solution every consumer will have a persistent connection to the server irrespective of the frequency of events. This setup seems inefficient unless you have a very high volume of events coming in. Many CDN networks have a limit on how long a connection you can open. And data providers will not prefer serving persistent requests. Problems listed are signatures, dedup, buffering, bootstrap, cron. Eve…

Open TCP connections can also be wildly cheap and efficient - Apple Push Notifications (APNS) and Android's push systems maintain open TCP connections to just about every mobile device on this planet. An open connection is just a bit of state on either end. The C10K problem has been solved for ages. Anyone remember consuming Twitter hoses back in the day? Those were also long-lived persistent connections for efficien…

Not only this but the spec explictly allows provider-terminated connections followed by a 429 with a respected "Retry-After" if you want to definitely kick "slow" feeds. This combined with HTTP2/3 multiplexing makes holding feeds open pretty cheap.

Re: The Valley of Webhooks

#59

Post by pphysch »

lobofta wrote: post #40
pphysch wrote: post #39

The core tradeoff here is "Push vs. Pull". Webhooks are a model for pushing data to subscribers. A traditional API allows clients to request and pull data. The data flow is in the same direction, but control flow is opposite. Pull-oriented models are much easier to reason about and should be preferred where possible (cybernetically they are a closed loop, vs. push models which could literally just be a barrage of UDP…

You mean pull-oriented models are easier to reason about, right?

Yes, thanks. Corrected.

Re: The Valley of Webhooks

#60

Post by weli »

zbentley wrote: post #44

This article resonates with similar struggles I've had at multiple companies. Its content is valuable. I wish the writing and the article itself had less of a slop odor. That aside, what I don't understand (especially having worked on the side of the webhook sender , which is itself really tricky to get correct/performant/cheap) is why more companies which broadcast webhooks don't, say, provide direct access to Kafka…

> That seems unlikely to me, but I might be surprised.

I'm pretty sure you've hit the nail on the head. Especially now with LLM's you'd be surprised of how many people are running full on production apps that are coded solely by an agent, deployed with a mix of random providers (supabase + fly.io + whatever) and have no clue about what Kafka is. They are never going to ask for it because... they don't know they can, and why they should. They ask an agent "connect to stripe" and the agent codes your typical webhook ingestion mechanism, never proposes advocating for an event polling loop. Whenever there is a bug they will tell the agent to fix it and the agent will just write a webhook deduplication mechanism and that's it. You'd be surprised how little supposedly technical people care about the current state of technologies.

PS: Thanks for the feedback on the writing.

Post reply on HN