21–30 of 157 posts
This is basically CSP no?
Interesting analogy - in a way it is doing something CSP-like in a distributed app/service architecture with the all the different processes and components that are there. The shared log (or a partition of that) being a way to establish a sequential order.
I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does not matter. - Users of the log are peers. No client / server distinction. - When appending a log entry, you can send a copy of the append to all your peers. - You can ask your peers to refresh the latest log entries. - When creating a new entry, it is a very go…
I.e. "we have 10 customers as of this date" can become immediately invalid if a new entry is appended afterwards with a date before that summary entry (i.e. because it was on a peer which hadn't yet sent it)
I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does not matter. - Users of the log are peers. No client / server distinction. - When appending a log entry, you can send a copy of the append to all your peers. - You can ask your peers to refresh the latest log entries. - When creating a new entry, it is a very go…
This is surprising, Kafka-like logs are all strictly ordered.
I’ve been doing a similar thing, although I called it “append only transaction ledgers”. Same idea as a log. A few principles: - The order of log entries does not matter. - Users of the log are peers. No client / server distinction. - When appending a log entry, you can send a copy of the append to all your peers. - You can ask your peers to refresh the latest log entries. - When creating a new entry, it is a very go…
Well, yes, but then you've backed into CAP again because you only have one log.
The end result is https://pipe.pico.sh which is an authenticated, networked *nix pipes over SSH. Since it relies on stdin/stdout via SSH it's one of the easiest pubsub systems we've used and we keep finding its ergonomics powerful. We have a centralized log-drain, metric-drain, and cache-clearing-drain all using `pipe`.
On the contrary. Everything becomes coordinated.
The entire "log" becomes a giant ass mutex lock. Good luck scaling it.
A notable example of a large-scale app built with a very similar architecture is ATproto/Bluesky[1]. "ATProto for Distributed Systems Engineers" describes how updates from the users end up in their own small databases (called PDS) and then a replicated log. What we traditionally think of as an API server (called a view server in ATProto) is simply one among the many materializations of this log. I personally find thi…
In a way this article here suggests to extend that
(1) from a log that represents data (upserts, cdc, etc.) to a log of coordination commands (update this, acquire that log, journal that steo)
(2) have a way to link the events related to a broader operation (handler execution) together
(3) make the log aware of handler execution (better yet, put it in charge), so you can automatically fence outdated executions
[1] https://engineering.linkedin.com/distributed-systems/log-wha...
A notable example of a large-scale app built with a very similar architecture is ATproto/Bluesky[1]. "ATProto for Distributed Systems Engineers" describes how updates from the users end up in their own small databases (called PDS) and then a replicated log. What we traditionally think of as an API server (called a view server in ATProto) is simply one among the many materializations of this log. I personally find thi…
That blog post is a great read as well. Truely, the log abstraction [1] and "Turning the DB inside out" [2] have been hugely influential. In a way this article here suggests to extend that (1) from a log that represents data (upserts, cdc, etc.) to a log of coordination commands (update this, acquire that log, journal that steo) (2) have a way to link the events related to a broader operation (handler execution) toge…