Live data from Hacker News

Spanner vs. Calvin: distributed consistency at scale

fauna.com

51–52 of 52 posts

Re: Spanner vs. Calvin: distributed consistency at scale

#51
post #48

I think it's often a better strategy to do sharding or micro-services and if possible keep the service so small that it's state can fit in a single machine. If you can have data boundaries, like for example one customer do not need to access the data of another customer, then you can separate customers's data and place them in different databases.

Sharding your product so that no cross-shard links are possible is robust and scalable, but it requires you to understand your final product very well at design time. If you start down that path and then bolt on some cross-shard data, the end result is almost always worse then if you had planned for that from the start.

Re: Spanner vs. Calvin: distributed consistency at scale

#52

The all-to-all dependency step between Calvin's sequencer layer and scheduler layer seems like it will be a problem as things scale, because it means that a single stalled sequencer [edit, orig: scheduler] blocks all writes in the system whether they conflict or not. This is the kind of dependence structure that magnifies outlier latencies and unavailability at scale. In Spanner's design, on the other hand, a transac…

You should think of the sequencer layer as a shared log abstraction along the lines of the Corfu project from Microsoft. It is distributed and replicated, with the scheduler layer reading from their local copy. Stalled scheduler nodes do not block writes in the system.

Corfu looks interesting. I need a distributed, consistent, persistent log for a project. Corfu is quite complex (three different RAM-hungry Java components that each wants to be distributed), however. Do you know of any about other, more lightweight, but still decently scalable alternatives? I was thinking about Etcd, but apparently it's not designed for large amounts of data.
Post reply on HN