Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Valley of Webhooks

weli.dev

Post reply on HN
Print view View original

71–80 of 110 posts

Re: The Valley of Webhooks

#71

Post by zbentley »

tasn wrote: post #64
zbentley wrote: post #54

Earlier quoted context omitted.

> receivers as well prefer the simplicity of webhooks. Do you see receive-side customers as prepared to outright reject paying for vendors that only offer non-webhook event streams if there's a webhook-ful competitor available? Or is that preference more of the "well, it's easier to add a webhook route to our existing webapp than it is to run a stream consumer/cron/whatever, but neither of those two is cost- or effor…

I know (from our customers) that some of their customers make buying decisions based on the availability of webhooks. I'm unaware of anyone doing it based on the availability of our other functionality (but it could be I just don't have the data). Though the data I do have: how many people adopt these advanced endpoints in practice (as we offer these), and it's less than webhooks.

Interesting. There’s probably a market for the consumer side of that, then: really easy-to-integrate-with-common-app-frameworks-and-hosting-platforms runtimes that provide a polling/socketful event consumer and sidestep all the concerns that come with scaling webhooks on the consumer side. Your company looks like it helps with those concerns a lot! But if HTTP webhooks are the lingua franca, there’s only so much I can do as a consumer of them per TFA, you know?

Re: The Valley of Webhooks

#73

Post by tux3 »

lelanthran wrote: post #57
alt227 wrote: post #12

I had the exact same thing with the Quickbooks api recently. You cannot trust the responses or webhooks at all. On create a user or invoice for example sometimes it will return an error, yet it actually created the entity. This means you have to check manually after creating everything to know if its created properly. Then you have the issue that sometimes quickbooks takes a while to update, and locks the company fil…

> You cannot trust the responses or webhooks at all. Well... yeah. I mean it's pretty obvious, no? Here's some things that could go wrong regardless of what care the software tries to provide: - The transaction completed on the backend cluster but the app instance died before if could create the response and after it committed the transaction. - The transaction completed, the app instance transmitted a response, but…

Those are all network errors. You wouldn't just catch all errors and treat them the same. An application error should mean the transaction did not complete.

Of course none of that applies in this case, since it's about Quickbooks. If the quickbooks api says the sky is blue, you should double check just to make sure.

Re: The Valley of Webhooks

#76

Post by Terr_ »

zbentley wrote: post #47
Terr_ wrote: post #3

The end here reminds me of "The Log: Real-time data's unifying abstraction" [0], which has unfortunately had a bit of link-rot since 2013. One complication in this approach involves access-windows: What if my system is only supposed to be seeing stuff that happened during two separate weeks in the year, because those are the spans when it was subscribed or authorized? So the data-host would need to maintain a concept…

That requirement seems ... uncommon. How many systems/industries have a frequent notion of transient/sliced views of history for their customers? If that is a real requirement, it seems like it'd be easier to meet by giving customers a realtime-stream/log API whose history starts when they were most recently granted access, and providing them older historical events via a separate API of the classic "ask for a report…

Perhaps less a requirement and more a kind of consistency with prior limitations? For example:

1. I have events in a Calendar service.

2. I want to authorize Reminder service to see upcoming events, so that it can send reminders to attendees in a way the Calendar service does not directly support, e.g. SMS/WhatsApp.

3. With the necessary credentials/SSO, the Reminder service subscribes to Calendar and Calendar periodically POSTSs webhook updates. Reminder needs to recognize when an event is cancelled or rescheduled, so that can alter its reminders.

Do I want to give Reminder potential access to all events ever, or just ones active across the usage period? Meanwhile, the Reminder guys probably don't want to step through the whole Calendar-wide event stream to reconstruct which events haven't finally happened yet.

Re: The Valley of Webhooks

#77

Post by zrail »

rawgabbit wrote: post #49
thingification wrote: post #28

I wonder if this is a CS problem somebody solved in 1954. Does somebody have the link to that paper? (I'm not serious about 1954 in particular, I am about hoping somebody here knows the CS literature better than me)

The old timey systems modeled this problem using accounting 101. When things change, you don’t immediately update the balance. Instead it is written to a transaction journal aka a log. The thing is this log is the source of truth. State or the balance is derived from the log. You don’t send a continuous stream of logs. Instead it is batched and sent asynchronously. It is also applied asynchronously. It also records i…

I dunno about "old timey", I wrote a system that does exactly this like six months ago. Granted, it's a money tracking system.

It's always surprising to me how much of the real world runs on CSV and EDI files sent back and forth over SFTP.

Re: The Valley of Webhooks

#78

Post by angrysponge »

toomim wrote: post #32

This is a nice writeup of the problems in using Webhooks for State Synchronization. I also noticed that the proposed solution is a pseudo IETF-style draft protocol called SCROLL... that happens to be remarkably similar to an actual IETF draft I am bringing to IETF 127 this November called "Braid-HTTP Subscriptions." Both drafts request a subscription with a GET plus a header: Scroll Request: GET /scroll/feed/customer…

I'm sorry, but I have to ask. bog-standard?

Re: The Valley of Webhooks

#79

Post by [deleted] »

zbentley wrote: post #51
__MatrixMan__ wrote: post #23

If you have two or more services that need to agree about state, and you have some set of rules that govern what state changes are valid, and you don't want to mess around with any of this "what do we do when we miss and update vs when we get two of the same update" nonsense, and the services aren't in a position to query the same database, then you should really consider a permissioned blockchain. Consensus hard, bu…

Why is cryptographically verifying record sequentiality an important property here? If I offer my customers a source of ordered records, the "trust" in that system is the fact that they pay me to make sure records are ordered. If I sell a fast or slow log database, approximately zero customers in the world care to verify ordering cryptographically. Or by "blockchain" do you just mean .... records with sequential IDs?…

[deleted]

Re: The Valley of Webhooks

#80

Post by __MatrixMan__ »

zbentley wrote: post #51
__MatrixMan__ wrote: post #23

If you have two or more services that need to agree about state, and you have some set of rules that govern what state changes are valid, and you don't want to mess around with any of this "what do we do when we miss and update vs when we get two of the same update" nonsense, and the services aren't in a position to query the same database, then you should really consider a permissioned blockchain. Consensus hard, bu…

Why is cryptographically verifying record sequentiality an important property here? If I offer my customers a source of ordered records, the "trust" in that system is the fact that they pay me to make sure records are ordered. If I sell a fast or slow log database, approximately zero customers in the world care to verify ordering cryptographically. Or by "blockchain" do you just mean .... records with sequential IDs?…

If your consumers are going to end up in this position:

> I do not trust the copy I built, and I have no way to know when it’s wrong, so I will re-derive it from scratch every night, forever.

Then guaranteed sequentiality means that they only have to verify each new block rather than fetch the whole thing every night.

Without it, you have this ever growing probability, which resets to 0 each night, that you unknowingly hold an invalid state. You might've acted on that state and so now when the nightly cleanup runs you have add code to go back out the consequences and instead apply the prosequences.

The complexity you think you're avoiding by not having a consensus protocol you're instead embracing as a data cleanup job, except instead of the same code everywhere, each reader has their own separate implementation.

That all goes away if you just don't process inbound data until you're sure nothing else is going to come along and invalidate it.

Sequential ID's work when there is only one writer and their implementation can be trusted but sometimes we get different events which both say they're number 12 and then we have to go call upstream and learn that so-and-so was on vacation and it won't happen again we promise. It takes days to resolve during which the potential of propagating bad state based on the lack of resolution continues to rise.

Ideally you can just avoid coordination delays entirely by keeping things monotonic and leaning on the CALM theorem but when that's not in the cards it's way better to put that delay on the writer's side, which is what blockchains do. Waiting for the consensus protocol to spit out a block before assuming that a write landed saves so many headaches on the reader side.

Post reply on HN