81–90 of 108 posts
This is a nice writeup of the problems in using Webhooks for State Synchronization. I also noticed that the proposed solution is a pseudo IETF-style draft protocol called SCROLL... that happens to be remarkably similar to an actual IETF draft I am bringing to IETF 127 this November called "Braid-HTTP Subscriptions." Both drafts request a subscription with a GET plus a header: Scroll Request: GET /scroll/feed/customer…
I'm sorry, but I have to ask. bog-standard?
This is a nice writeup of the problems in using Webhooks for State Synchronization. I also noticed that the proposed solution is a pseudo IETF-style draft protocol called SCROLL... that happens to be remarkably similar to an actual IETF draft I am bringing to IETF 127 this November called "Braid-HTTP Subscriptions." Both drafts request a subscription with a GET plus a header: Scroll Request: GET /scroll/feed/customer…
I'm sorry, but I have to ask. bog-standard?
Earlier quoted context omitted.
99% of the time (and all 3 times in the blog post), there is only one source of truth for any piece of data, and state transitions are completely arbitrary. Blockchain is almost always the wrong solution.
> Blockchain is almost always the wrong solution. Especially since in most of the cases where it's not-totally-insane to use, the right solution is still the classic distributed database which already existed. In those, the ledger is kept among a predefined/controlled node-membership... as opposed to a bloated mass of workarounds and limitations to make it barely survive being ungovernable. I've seen some boosters pi…
The craziness comes in when you'll accept blocks from anybody willing to burn enough electricity to do so, or gamble enough tokens to do so, or whatever other artificial scarcity game people like to play. But if you're only planning to consume data from the eight other companies you do business with then there's no reason to bother with any of that, you can just hard code their public keys into your consensus protocol and you've sidestepped the nonsense.
They go on about how we've settled in the trough, but the whole local optimum problem is about settling in an early peak, and being unwilling to cross a trough to get to a higher peak. The workarounds would be propping up the first peak, not evidence that we've settled in the trough.
Either way, the metaphor feels kinda forced IMO.
If you have two or more services that need to agree about state, and you have some set of rules that govern what state changes are valid, and you don't want to mess around with any of this "what do we do when we miss and update vs when we get two of the same update" nonsense, and the services aren't in a position to query the same database, then you should really consider a permissioned blockchain. Consensus hard, bu…
99% of the time (and all 3 times in the blog post), there is only one source of truth for any piece of data, and state transitions are completely arbitrary. Blockchain is almost always the wrong solution.
> I do not trust the copy I built, and I have no way to know when it’s wrong, so I will re-derive it from scratch every night, forever.
Then your life would probably be better if you just had to consume block-at-a-time and not the whole dataset every night. Better to be persistently five minutes behind, then to go all day not knowing whether you're wrong, with a brief moment of certainty each night.
Your nightly reconciliation job just ends up being an inside out version of the consensus protocol that you failed to enforce up front (which may be a necessary evil if you have no influence over the people who publish your data, but let's not let those people off the hook for failing to support incremental verification of sync).
The end here reminds me of "The Log: Real-time data's unifying abstraction" [0], which has unfortunately had a bit of link-rot since 2013. One complication in this approach involves access-windows: What if my system is only supposed to be seeing stuff that happened during two separate weeks in the year, because those are the spans when it was subscribed or authorized? So the data-host would need to maintain a concept…
I had the exact same thing with the Quickbooks api recently. You cannot trust the responses or webhooks at all. On create a user or invoice for example sometimes it will return an error, yet it actually created the entity. This means you have to check manually after creating everything to know if its created properly. Then you have the issue that sometimes quickbooks takes a while to update, and locks the company fil…
> You cannot trust the responses or webhooks at all. Well... yeah. I mean it's pretty obvious, no? Here's some things that could go wrong regardless of what care the software tries to provide: - The transaction completed on the backend cluster but the app instance died before if could create the response and after it committed the transaction. - The transaction completed, the app instance transmitted a response, but…
Some Apis are designed to always fail safe and send the correct response, or at least accept multiple messages without creating duplicates, so you can just keep firing the same message until you get the correct response. Not doing that is just lazy design.
I had the exact same thing with the Quickbooks api recently. You cannot trust the responses or webhooks at all. On create a user or invoice for example sometimes it will return an error, yet it actually created the entity. This means you have to check manually after creating everything to know if its created properly. Then you have the issue that sometimes quickbooks takes a while to update, and locks the company fil…
Instead they deep dive into distributed systems, without ever learning about them and all the issues that can arise.
There is a reason many CS degrees have two semesters full of distributed systems content, between networking approaches, architecture design and algorithms to make everything robust.