Databases = Frameworks for Distributed Systems

“There are only three modes for a distributed system - it implements paxos/raft itself, it relies on a data store that implements paxos/raft or it’s wrong.”

Obviously a bit of an over simplification since there have been some advancements in that area but not far off.

This article introduces an approach that falls under the second mode and I think it’s a good way to build a distributed system without having to re invent the wheel especially if you can just rely on the data store for the consensus and make your brokers and nodes relatively dumb.

Look at k8s and etcd for a very widely used example of this approach as well.

I’m also currently building a distributed search engine using etcd for the service registry, broker peer announcements, and worker queue and it’s been a good experience so far.

As an elementary observation, isolating the designs of storage, compute, and networking as separate modules is a reliable way to achieve poor performance and resource efficiency. Some of the most important scalability and performance optimizations in modern database architecture are schedule-driven, which requires tight coupling across the design of storage, compute, and networking. Ignoring this class of optimizations produces very large reductions in real-world workload performance, and feeds a narrative that the software is wasteful and not environmentally friendly.

Heterogeneous and specialized resource collections with complex topologies are becoming more common for a diverse set of reasons. This breaks most horizontal resource abstractions. The practice of treating every thread as a complete monolithic database kernel connected together with protocols that are good at equilibrium finding lends itself to this environment very well, and the edge (for which database technology is broadly broken), in addition to being ideal for the boring data center case where the topology is highly regular and every node has uniform resources.

Most of our distributed systems are modeled as giant distributed file systems, and that fits okay with the idea of horizontally partitioning compute, storage, and network. This is easy for developers to reason about because they are familiar with file systems. However, it is difficult to express important database-y operations in this model. For example, if you want to do ad hoc joins in a scale-out environment, you need mechanics like decentralized parallel orchestration which isn't really a concept in a "distributed file system" model of "distributed".

Databases have been slowly converging on latency-hiding HPC software architectures (which has no implication of HPC hardware), which look very different than a distributed file system, can express more powerful capabilities, and are generally more scalable. To the extent there is a general principle and framework behind databases, it is more this than traditional distributed systems.