This article resonates with similar struggles I've had at multiple companies. Its content is valuable. I wish the writing and the article itself had less of a slop odor.
That aside, what I don't understand (especially having worked on the side of the webhook sender, which is itself really tricky to get correct/performant/cheap) is why more companies which broadcast webhooks don't, say, provide direct access to Kafka topics, S3 buckets with ordered data objects landing, SQS queues, or any of the alternatives to those things.
"But it's irresponsible to expose an internal-use-only datastore directly to clients" goes one objection. But plenty of log-store systems have the notion of sharing a subpart of the log with a less-than-trusted external peer, so while exposing Kafka directly might be asking for the same kind of trouble as exposing your customer's SQL database for authenticated connection over the open internet (e.g. "we said you could issue reads, not that you could open/close TCP connections a million times a second! You just took out our message broker!"), exposing, say, an S3 bucket or Kinesis stream is much less risky because those systems have put some thought towards semi-trusted sharing.
"But everyone is used to getting HTTP webhooks and doesn't have the expertise to connect to something else"--that'd be true if, say, reading from a websocket or Postgres NOTIFY stream or Kafka topic or S3-change-notification stream were advanced techniques, but libraries around those things are so good nowadays that even the most web-tech-only low-skill developer can probably integrate with them with minimal hassle. Maybe it's just that a lot of shops literally only know how to run their code in a webserver, and have never deployed any other kind of application service/cronjob/queue worker? That seems unlikely to me, but I might be surprised.
"If we do something weird our competition will beat us on ease-of-use" goes another objection. But is it that hard given the libraries available? And can't you hedge back on the ease-of-use sell with "our data is fresher and more provably ordered and correct"?
I'm glad that SCROLL exists as a possible solution here. I'm just puzzled why more people haven't been using existing technologies to achieve this property.
Do most webhook senders literally not have a log store? Are they just firing webhooks in the middle of business event handlers and giving up synchronously if they can't be delievered?
Because if that's not the case (and I don't think it's the case), then it seems like the SCROLL API is ... basically just the Kafka consumer API. Or Kinesis. Or SQS. And so on.