Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Spanner vs. Calvin: distributed consistency at scale

fauna.com

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Spanner vs. Calvin: distributed consistency at scale

#21

Post by elvinyung »

evanweaver wrote: post #10
wwilson wrote: post #8

Earlier quoted context omitted.

Yeah, I've read that post, and I have no idea where it gives the impression that we don't support interactive transactions. What paragraph are you looking at? Could you be referring to the fact that we make you do writes via a mutation API rather than DML? Obviously that has no impact on interactivity...

Yes, our misunderstanding and we misinformed Daniel. Fixed, and thank you. It would be cool to know why Spanner is like that.

+1, also curious about this. I speculated that Cloud Spanner is supposed to be F1[1], but the fact that F1 seems to fully support SQL DML makes this difference even more perplexing.

> Updates are also supported using SQL data manipulation statements, with extensions to support updating fields inside protocol buffers and to deal with repeated structure inside protocol buffers.

[1] https://research.google.com/pubs/pub41344.html

Re: Spanner vs. Calvin: distributed consistency at scale

#23

Post by zzzcpan »

"It’s not relevant for the type of modern architectural deployments discussed in this post where network partitions are rare"

I think the post needs to be more precise about this. Modern networks are asynchronous and so they are essentially always partitioned. And this is important, because later, when snapshot and transactional read latencies are discussed, they are not exactly "very low", not on "AP" systems level. Because "CP" systems need consensus they can't just send requests to all of the replicas and wait for the first reply or even read one locally, "very low" there is more about talking to a remote node "low", with all of the problems associated with it, like poor tail latency, etc.

Re: Spanner vs. Calvin: distributed consistency at scale

#24

Post by freels »

imoldfella wrote: post #16

The Spanner design seems more resilient in the face of server failures. The initial Calvin papers call for taking the entire replica offline if a single server in the replica fails. Are there more advanced versions of Calvin that get around this?

Yes. For log data, it's simply a matter of reading from a replica peer of the down node.

For transaction resolution it's a bit easier if you are able to assume more about the storage layer's semantics.

For example, if you store versioned values for some bounded period of time (ala MVCC), you can go to other replicas for the version required to resolve a transaction, removing the restriction that transaction resolution must proceed in lock-step across all nodes, and allows transaction reads to route to live peers assuming they have the required version of each read dependency.

Re: Spanner vs. Calvin: distributed consistency at scale

#25

Post by CalvinDB »

imoldfella wrote: post #16

The Spanner design seems more resilient in the face of server failures. The initial Calvin papers call for taking the entire replica offline if a single server in the replica fails. Are there more advanced versions of Calvin that get around this?

Yes --- the current version of Calvin (in the Yale research group) does not have this limitation. We're actually not sure which paper you're talking about, but either way, it's not fundamental to the Calvin approach. In general, if a single server in a replica fails, the other servers within the replica that need data from the failed server can access that data from one of the replicas of the failed server. (We can't speak for FaunaDB, but like the current version of Calvin, it is unlikely they have this limitation.)

Re: Spanner vs. Calvin: distributed consistency at scale

#26

Post by fdsfsafasfdas »

cobookman wrote: post #19
atombender wrote: post #13

Tangentially, while FaunaDB and Spanner both seem great, and I love consuming the technical details in articles like these, neither I or my companies are ever going to commit to using something which isn't open source. I have a real problem understanding why Google (and now FaunaDB) are so clueless and/or inconsistent about software licensing. Google seemed to finally get it with Kubernetes, which has grown into a re…

> The fact that I can't run these products locally on my dev machine is another major downside unrelated to the philosophical aspects of free software. You can't run a full spanner cluster unless you have the Google Hardware. Spanner needs to have an accurate clock with a known error bound. This makes it difficult to open source without open sourcing a bunch of hardware.

> This makes it difficult to open source without open sourcing a bunch of hardware.

That's simply not true; there are many consumer-grade GPS clocks out there well within the bounds that spanner operates (iirc skew bounded at +- 5ms).

Not only that, but you can bet that if spanner were open-sourced, you'd see additional pressure to lower the cost of that hardware.

Not that the answer is any more nefarious: They don't open source it because it extensively uses google-only code, and open sourcing spanner would entail open sourcing many, many other parts of code, which is expensive. It'd be much easier to have something like the dynamo db local front to sqlite3 like amazon offers.

Re: Spanner vs. Calvin: distributed consistency at scale

#28

Post by vgt »

fdsfsafasfdas wrote: post #26
cobookman wrote: post #19

Earlier quoted context omitted.

> The fact that I can't run these products locally on my dev machine is another major downside unrelated to the philosophical aspects of free software. You can't run a full spanner cluster unless you have the Google Hardware. Spanner needs to have an accurate clock with a known error bound. This makes it difficult to open source without open sourcing a bunch of hardware.

> This makes it difficult to open source without open sourcing a bunch of hardware. That's simply not true; there are many consumer-grade GPS clocks out there well within the bounds that spanner operates (iirc skew bounded at +- 5ms). Not only that, but you can bet that if spanner were open-sourced, you'd see additional pressure to lower the cost of that hardware. Not that the answer is any more nefarious: They don't…

A critical design feature of Spanner is owning the packets end-to-end (as in, between data centers and across oceans). This lets Google minimize the "A" in "CAP", and would be quite difficult to replicate outside of a network like Google's SDN.

Spanner is the software, plus the network, plus the hardware.

(work on Google Cloud)

Re: Spanner vs. Calvin: distributed consistency at scale

#29

Post by rdtsc »

> Before we get started, allow me to suggest the following: Ignore the CAP theorem in the context of this discussion. Just forget about it. It’s not relevant for the type of modern architectural deployments discussed in this post where network partitions are rare.

Anyone worried about that statement. Ok so they are rare, but what happens when they do occur? File corruption is rare as well but in our large deployment over thousands of servers I see it happen pretty often.

Re: Spanner vs. Calvin: distributed consistency at scale

#30

Post by fdsfsafasfdas »

vgt wrote: post #28
fdsfsafasfdas wrote: post #26

Earlier quoted context omitted.

> This makes it difficult to open source without open sourcing a bunch of hardware. That's simply not true; there are many consumer-grade GPS clocks out there well within the bounds that spanner operates (iirc skew bounded at +- 5ms). Not only that, but you can bet that if spanner were open-sourced, you'd see additional pressure to lower the cost of that hardware. Not that the answer is any more nefarious: They don't…

A critical design feature of Spanner is owning the packets end-to-end (as in, between data centers and across oceans). This lets Google minimize the "A" in "CAP", and would be quite difficult to replicate outside of a network like Google's SDN. Spanner is the software, plus the network, plus the hardware. (work on Google Cloud)

A very good point.
Post reply on HN