Ask HN: Options for handling state at the edge?

If records are allowed to change, then you end up in situations where changes don't converge. But if you instead collect a history of unchanging events, then you can untangle these scenarios.

Event Sourcing is the most popular implementation of a history of immutable events. But I have found that a different model works better for data at the edge. An event store tends to be centrally localized within your architecture. That is necessary because the event store determines the one true order of events. But if you relax that constraint and allow events to be partially ordered, then you can have a history at the edge. If you follow a few simple rules, then those histories are guaranteed to converge.

Rule number 1: A record is immutable. It cannot be modified or deleted.

Rule number 2: A record refers to its predecessors. If the order between events matters, then it is made explicit with this predecessor relationship. If there is no predecessor relationship, then the order doesn't matter. No timestamps.

Rule number 3: A record is identified only by its type, contents, and set of predecessors. If two records have the same stuff in them, then they are the same record. No surrogate keys.

Following these rules, analyze your problem domain and build up a model. The immutable records in that model form a directed acyclic graph, with arrows pointing toward the predecessors. Send those records to the edge nodes and let them make those millisecond decisions based only on the records that they have on hand. Record their decisions as new records in this graph, and send those records back.

Jeff Doolittle and I talk about this system on a recent episode of Software Engineering Radio: https://www.se-radio.net/2021/02/episode-447-michael-perry-o...

No matter how you store it, treat data at the edge as if you could not update or delete records. Instead, accrue new records over time. Make decisions at the edge with autonomy, knowing that they will be honored within the growing partially-ordered history.

Most of the approaches mentioned here will give you fast reads everywhere, but writes only fast if you're close to some arbitrarily chosen primary region.

A few technologies I've experimented with for doing fast, eventually consistently replicated writes: DynamoDB Global Tables, CosmosDB, Macrometa, KeyDB.

None of them are perfect, but in terms of write latency, active-active replicated KeyDB in my fly.io cluster has everything else beat. It's the only solution that offered _reliable_ sub-5ms latency writes (most are close to 1-2ms). Dynamo and Cosmos advertise sub-10ms, but in practice, while _most_ writes fall in that range, I've seen them fluctuate wildly to over 200ms (Cosmos was much worse than Dynamo IME), which is to be expected on the public internet with noisy neighbors.

Unfortunately, I got too wary of the operational complexity of running my own global persistent KeyDB cluster with potentially unbounded memory/storage requirements, and eventually migrated most app state over to use Dynamo as the source of truth, with the KeyDB cluster as a auto-replicating caching layer so I don't have to deal with perf/memory/storage scaling and backup. So far that has been working well, but I'm still pre-launch so it's not anywhere close to battle tested.

Would love to hear stories from other folks building systems with similar requirements/ambitions!

cloudflare kv store is great if the supported write pattern fits

if you need something with more consistency between pops durable objects should be on your radar

i also found that cloudant/couchdb is a perfect fit for a lot of usecases with heavy caching in the cf worker. its also possible to have multiple master replication with each couchdb cluster close to the local users, so you dont have to wait for writes to reach a single master on the other side of the world