Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Valley of Webhooks

weli.dev

Post reply on HN
Print view View original

101–108 of 108 posts

Re: The Valley of Webhooks

#101

Post by NortySpock »

Yeah, been fighting variants of this problem as a data engineer where the upstream database would only let you see the current state of the table and did not provide you a changelog / cdc

So you have to

(a) repeatedly query incremental date ranges, with some overlap (b) deal with late arriving rows

(c) deal with occasional bugs where a bug in a sproc caused the timestamp not to update

(d) query each day for all primary keys and remove rows that had been deleted (but only if you have newer data, otherwise you delete old values but don't have the updates)

(e) Sweat about if this goes wrong on a large table, how can you rapidly determine what state is out of sync? (Idea: how do I "hash" this in a way that hints at where the problem is?)

(f) Deduplicate, as mentioned

Martin Kleppmann's "Designing Data-Intensive Applications" book has some detailed discussions of some of the challenges...

Re: The Valley of Webhooks

#102

Post by bowersbros »

angrysponge wrote: post #78
toomim wrote: post #32

This is a nice writeup of the problems in using Webhooks for State Synchronization. I also noticed that the proposed solution is a pseudo IETF-style draft protocol called SCROLL... that happens to be remarkably similar to an actual IETF draft I am bringing to IETF 127 this November called "Braid-HTTP Subscriptions." Both drafts request a subscription with a GET plus a header: Scroll Request: GET /scroll/feed/customer…

I'm sorry, but I have to ask. bog-standard?

It's a British colloquial term meaning basic / expected / ordinary

Re: The Valley of Webhooks

#103

Post by zbentley »

Terr_ wrote: post #76
zbentley wrote: post #47

Earlier quoted context omitted.

That requirement seems ... uncommon. How many systems/industries have a frequent notion of transient/sliced views of history for their customers? If that is a real requirement, it seems like it'd be easier to meet by giving customers a realtime-stream/log API whose history starts when they were most recently granted access, and providing them older historical events via a separate API of the classic "ask for a report…

Perhaps less a requirement and more a kind of consistency with prior limitations? For example: 1. I have events in a Calendar service. 2. I want to authorize Reminder service to see upcoming events, so that it can send reminders to attendees in a way the Calendar service does not directly support, e.g. SMS/WhatsApp. 3. With the necessary credentials/SSO, the Reminder service subscribes to Calendar and Calendar period…

Fair, and thanks for explaining. Other than my original proposal of a simple webhook stream and a complicated offline backfill, all the good ways to do that with current tools probably involve building support for fine-grained read permissions (Reminder service's windowing) onto the producer (Calendar) side and emitting a tailored reminder stream.

If that challenge is indeed common, then there might be a market/demand/opportunity to implement ... I don't know what to call it, "Postgres RLS for Kafka" or something--a record-level security model for event streams, basically. Event compaction/deduplication would make this hard, though.

Re: The Valley of Webhooks

#104

Post by __MatrixMan__ »

Xirdus wrote: post #94
__MatrixMan__ wrote: post #87

Earlier quoted context omitted.

That might be true for 99% of data, but I find that it's the remaining 1% of that data occupies the majority of the time. If you don't have a problem that is solved by a blockchain don't use one, but if you're saying things like this: > I do not trust the copy I built, and I have no way to know when it’s wrong, so I will re-derive it from scratch every night, forever. Then your life would probably be better if you ju…

Not 99% of data - 99% of applications. In 99% of applications, 100% of data is such that any piece of this 100% has only one authoritative source. Only in 1% of applications, the percent of data that doesn't have one authoritative source is less than 100%. So the absolute upper bound of when blockchain is even appropriate at all is 1% of applications. > Then your life would probably be better if you just had to consu…

That is true, but I think it's a pattern we need to get away from. It's the source of everything that is wrong with the modern web.

- These single sources of truth become high value targets for corruption.

- Our apps become totally useless if they can't connect to whichever source we've hard-coded as authoritative.

- Even when connectivity is good and upstream sources are trustworthy, we end up creating unnecessary burdens on infrastructure for a connections that spans the globe from a single point to millions of them even when the data we're after is elsewhere on the LAN, ignored because it doesn't have the authoritative hostname.

It's much more fault tolerant to trust the data on the basis of its verifiable properties, not because of where you got it. That way, if the source becomes unavailable or untrustworthy, somebody more relevant can start being the leader (supposing you need a leader at all).

> you should always pick SCROLL and never a blockchain

The SCROLL RFC is very honest about how mature it is:

> draft-scroll-protocol-00 · request for comments · no implementations in the wild, one document

People have built some truly stupid things on a blockchain, and SCROLL does seem like a good idea for some cases, but I think you're going a bit far by suggesting that a protocol which has never been implemented is always preferable to one with dosens of implementations to chose from.

Re: The Valley of Webhooks

#105

Post by Xirdus »

__MatrixMan__ wrote: post #104
Xirdus wrote: post #94

Earlier quoted context omitted.

Not 99% of data - 99% of applications. In 99% of applications, 100% of data is such that any piece of this 100% has only one authoritative source. Only in 1% of applications, the percent of data that doesn't have one authoritative source is less than 100%. So the absolute upper bound of when blockchain is even appropriate at all is 1% of applications. > Then your life would probably be better if you just had to consu…

That is true, but I think it's a pattern we need to get away from. It's the source of everything that is wrong with the modern web. - These single sources of truth become high value targets for corruption. - Our apps become totally useless if they can't connect to whichever source we've hard-coded as authoritative. - Even when connectivity is good and upstream sources are trustworthy, we end up creating unnecessary b…

The real world operates on single sources of truth. The software merely models the real world - as it should. Saying we should move away from this pattern is synonymous with saying we should fundamentally change how are society works. Hint: if your preferred software solution only makes sense if we fundamentally change how society works, maybe it's not the best solution.

If Stripe says you didn't get the payment, then you didn't get the payment. End of story. The payment isn't magically going to appear in your wallet just because you verify some properties of some data.

If your only complaint against SCROLL is that an implementation doesn't exist yet, then how about implementing SCROLL instead of creating yet another blockchain? It's not much work anyway, SCROLL is like 1000 times simpler than even the simplest of blockchains.

Re: The Valley of Webhooks

#106

Post by gregw2 »

thingification wrote: post #28

I wonder if this is a CS problem somebody solved in 1954. Does somebody have the link to that paper? (I'm not serious about 1954 in particular, I am about hoping somebody here knows the CS literature better than me)

I'm not an academic but I've worked through practical problems in this space for longer than I'd like to admit, often in ignorance of the literature. The most CS-y fundamental paper would probably be Lamport(1978) below, but the database papers are pretty fundamental on this topic in their own right.

Here are some pointers:

* Astrahan et al., "System R: Relational Approach to Database Management" (1976), oldest paper talking about logs in a database which is kinda what at its root this is recapitulating

* Jim Gray, "Notes on Data Base Operating Systems" (1978), covers logs with the goal of transaction management, undo, redo, the famous "two phase commmit" process for aligning state across a network boundary in two systems

* Leslie Lamport, "Time, Clocks, and the Ordering of Events in a Distributed System" (1978), not a database paper, more about state synchronization in general

* Bruce Lindsey, "Notes on Distributed Databases" (1979), standards for replicating data across multiple identical database nodes

* Jim Grey et al, "The Recovery Manager of the System R Data Manager" (1981), describes "write to the log first, then the database (WAL)" pattern

* Silberschatz, Stonebraker, Ullman: "Database Systems: Achievements and Opportunities"/"A Architecture for Heterogeneous Database Replication" (1980), discusses tricky bits about replaying logs in client database systems that work differently from the producer

* Leslie Lamport, "The Byzantine Generals Problem" (1982), on the math and needed consensus error handling when distributing state

For other later topics on state synchronization, read up on Paxos (Lamport, 1989/98), RAFT (2013), CRDTs (2011), etc.

The above approaches of "write-to-the-log-then-update-state" were applied to UNIX filesystem first in AIX 3.1 in 1990 and then adopted by other UNIX vendors and then in Linux ReiserFS/ext3/XFS(SGI) in 2001ish.

Lotus Notes took a different path in the 1990s to synchronizing state within documents which was not ACID-oriented like the above, but was more like the modern append-only-log with optimistic eventual consistency. Not sure about the best paper on this.

Then Martin Fowler popularized event sourcing in 2005 with Enterprise Application Architecture, and later described CQRS in 2011 and microservices. So "we" decoupled everything with Webhooks (2007) and Kafka (2011) and reinvented this problem. Oh, and did I mention Blockchain (2008), etc. Oh, did I mention Bitkeeper dvcs (1998) and git (2005) handling of distributed state?

Along the way databases (Snowflake, BigQuery, Iceberg, etc) started exploiting logs to show "older" state via features like "Time Travel" queries. Which is actually what Stonebreaker tried to do in the earliest versions of PostgreSQL but the first implementation in the 80s didn't perform well without good compaction support and on much more constrained resources.

Ask your local LLM for a good survey paper and it might be a bit easier to digest, but the above gives you some color and keywords.

Re: The Valley of Webhooks

#107

Post by __MatrixMan__ »

Xirdus wrote: post #105
__MatrixMan__ wrote: post #104

Earlier quoted context omitted.

That is true, but I think it's a pattern we need to get away from. It's the source of everything that is wrong with the modern web. - These single sources of truth become high value targets for corruption. - Our apps become totally useless if they can't connect to whichever source we've hard-coded as authoritative. - Even when connectivity is good and upstream sources are trustworthy, we end up creating unnecessary b…

The real world operates on single sources of truth. The software merely models the real world - as it should. Saying we should move away from this pattern is synonymous with saying we should fundamentally change how are society works. Hint: if your preferred software solution only makes sense if we fundamentally change how society works, maybe it's not the best solution. If Stripe says you didn't get the payment, the…

I don't think society worked that way until the web made it possible for society to work that way. And it's not working particularly well. It's brittle, coercive, and it asks operators to have a hand in supporting it in any way to support it in every way, raising the challenge of participating to a level where only the largest companies can.

If we think we can fundamentally change it for the better, we should.

I'm not especially enamored of blockchains. I think that partition tolerance is more important than global consistency, so mostly it's CRDT's that I'm trying to apply. I just brought up blockchains because they seemed especially applicable to the problems that SCROLL is trying to solve.

If you have a single source and a reliable connection to it, SCROLL seems great, I just think we should be preparing fallbacks for when those assumptions need to change. Because the real world doesn't operate on single sources of truth. Pick anything that people talk about, and you'll find that they're not on the same page about which sources are trustworthy for that topic. The software world operates on single sources of truth not because the real world does, but because software has so far failed to be useful outside of that domain.

Re: The Valley of Webhooks

#108

Post by __MatrixMan__ »

Xirdus wrote: post #95
__MatrixMan__ wrote: post #85

Earlier quoted context omitted.

I'm not sure what a private blockchain is, but a permissioned blockchain is one where only certain parties have keys that allow them to write blocks. You end up with a message queue optimized to eliminate anything that would lead to the inconsistency nonsense that this article is talking about as soon as it is detected. It then becomes the writer's problem to retransmit in a way that doesn't cause a problem next time…

What if the keys leak and need to be rotated?

If you're exchanging public keys to configure your node with in the first place, then presumably you have a trusted side channel for that kind of thing. It's equivalent to the webhook situation re: what if the domain name changes?

If you don't have such a channel then there's always social key recovery schemes, but I don't think there's a particular pattern that you can just adopt. You're now in participate-in-the-research mode.

Post reply on HN