Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NewSQL databases fail to guarantee consistency and I blame Spanner

dbmsmusings.blogspot.com

Post reply on HN
Print view View original

41–50 of 319 posts

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#41

Post by abadid »

felixgallo wrote: post #28
abadid wrote: post #2

I'm the author of that post. I'm happy to respond to comments on the post on this thread for the next several hours. You can also leave comments on the post itself, and I will respond there at any time.

why didn't you disclose your connections in your post? It appears you've written much the same article for FaunaDB's official blog in the past ( https://fauna.com/blog/distributed-consistency-at-scale-span... ).

That post was actually quite different than this one. This one focuses on distributed vs. global consensus.

But I added a note at the end that clearly documents my connection to FaunaDB. As far as Calvin, the post itself clearly says that it came out of my research group.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#42

Post by CydeWeys »

It's definitely true that putting the burden of consistency on developers (instead of on the DB) results in a lot more tricky work for developers. On my project, which started six years ago, we use Cloud Datastore, because Cloud Spanner hadn't come out yet. It results in complicated, painful code that would be completely unnecessary with stronger transactional guarantees. Some examples: https://github.com/google/nomulus/blob/master/java/google/re... https://github.com/google/nomulus/blob/master/java/google/re... https://github.com/google/nomulus/blob/master/java/google/re...

It's no surprise that we're currently running screaming to something with stronger transactional guarantees.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#43

Post by cloakandswagger »

romed wrote: post #24
abadid wrote: post #2

I'm the author of that post. I'm happy to respond to comments on the post on this thread for the next several hours. You can also leave comments on the post itself, and I will respond there at any time.

I question your statement that building apps on weakly-consistent systems is so difficult. I’ve worked on very large scale systems that you’ve definitely heard of and probably used that are built atop storage systems with very weak semantics and asynchronous replication. Aren’t such systems existence proofs, or do you think there’s just a huge difference between the abilities of engineers in various organizations?

He said it was difficult, not impossible.

I think it's a fairly noncontroversial statement. Dealing with eventual consistency is always going to be more difficult and require more careful thought and preparation than immediately consistent systems.

How many programmers do you think are out there that have only ever worked on systems that use a single RDBMS instance, and what would happen if they tried to apply their techniques to a distributed, eventually consistent environment?

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#44

Post by hosay123 »

romed wrote: post #29

I’ve seen comments on HN over the years in which someone Dunning-Kruegers their way into saying that TrueTime is easily replicated. I always wonder if they have sixteen senior SREs in their pocket, because that’s the level of production engineering Google applies to the problem. Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up. If yo…

> Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up

It's another cute anecdote, but Google culture is full of these, always scant on details and always intended to show how big/smart/important/complex/indispensable their engineering is.

"Had to" is a strong term here, it's made to sound like USAF could not possibly have noticed some deviation they were likely to correct of their own accord as a matter of routine as they had been doing for the 20 years of the GPS project prior to Google being founded.

The reality is drift and bad clocks are and always have been a feature of GPS, one explicitly designed for, one an entire staff exists to cope with, and designs depending on the absolute accuracy of a single clock have never been correct

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#45

Post by reilly3000 »

sabujp wrote: post #35
romed wrote: post #29

I’ve seen comments on HN over the years in which someone Dunning-Kruegers their way into saying that TrueTime is easily replicated. I always wonder if they have sixteen senior SREs in their pocket, because that’s the level of production engineering Google applies to the problem. Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up. If yo…

the only solution to this is to launch your own satellites :)

If we seal the surface of, then boil the ocean we can launch our own time satellites with completely sustainable propulsion.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#46

Post by peterwwillis »

Distributed databases are like systems of government. Some are worse than others, but all of them suck. I'm not aware of any study that shows that X type of database reduces bugs, increases availability, and makes customers happier. Pick one that fits your application and deal with the suckiness.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#47

Post by otterley »

Cshelton wrote: post #37
romed wrote: post #29

I’ve seen comments on HN over the years in which someone Dunning-Kruegers their way into saying that TrueTime is easily replicated. I always wonder if they have sixteen senior SREs in their pocket, because that’s the level of production engineering Google applies to the problem. Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up. If yo…

Also, I believe all the major cloud providers provide a "TrueTime" API service. I forgot the name that AWS uses, but you can call it on your EC2 instances and make sure your hosts are all in sync. It's pretty cool.

AFAIK AWS is only offering NTP service with a GPS source (i.e. stratum 1). TrueTime appears to be a service offering that is a step above that in terms of the guarantees it provides.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#48

Post by nemothekid »

>Systems that guarantee consistency only experience a necessary reduction in availability in the event of a network partition. As networks become more redundant, partitions become an increasingly rare event. And even if there is a partition, it is still possible for the majority partition to be available

In my experience, yes network partitions are incredibly rare. However 99% of my distributed ststem partitions have little do with the network. When running databases on a cloud environment network partitions can occur for a variety of reasons that don’t actually include the network link between databases:

1. The host database is written in a GC’d language and experiences a pathological GC pause.

2. The Virtual machine is migrated and experiences a pathological pause

3. Google migrates your machine with local SSDs, fucks up that process and you lose all your data on that machine (you do have backups right?)

4. AWS retires your instance and you need to reboot your VM.

You may never see these issue if you are running a 3 or 5 cluster database. I began seeing issues like this semi regularly once the cluster grew to 30-40 machines (Cassandra). Now I will agree that none of the issues took down majority, but if your R=3, it really only takes an unlucky partition to fuck up an entire shard

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#49

Post by zzzcpan »

haberman wrote: post #13
zzzcpan wrote: post #9

Earlier quoted context omitted.

Too many false claims for my taste. Especially wrt to eventual consistency and consistency guarantees. Also some misinterpretation of CAP theorem to fit the narrative. Very typical FaunaDB promotional post.

Without some description of what "false claims" and "misinterpretation" you mean, this is a content-free post.

Too much is false to explain it, like half of the article. So I just called it out instead. I know it's not very convincing, but I'm not getting paid to write giant FUD/PR posts like they do. I work on distributed stuff completely independently.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#50

Post by niftich »

dgreensp wrote: post #23
zzzcpan wrote: post #9

Earlier quoted context omitted.

Too many false claims for my taste. Especially wrt to eventual consistency and consistency guarantees. Also some misinterpretation of CAP theorem to fit the narrative. Very typical FaunaDB promotional post.

The author of the CAP theorem says the same thing about CAP: https://www.infoq.com/articles/cap-twelve-years-later-how-th...

The middle part of 'CAP Twelve Years Later' [1], especially the part about 'Managing Partitions', and the associated diagram, really should be mandatory reading for anyone who invokes CAP, as it drives home the fact that a partition creates alternate universes (invariants), which have to be reconciled after the fact, and points out that allowing reads only is a valid strategy to provide Consistency (erroring out on writes), while systems that choose to provide Availability (trying to fulfill every transaction) will find themselves in a situation where they have to reconcile mutations that happened in separate universes, exactly like a version control system would. It also provides some strategies on how some write-like operations could be done to minimize disruption to existing state, such as recording the intent to mutate some data in a way that enables duplicates to be discarded later, and for the mutation to be performed at a time that it is safe.

The author of the HN-featured article is the formulator of the PACELC theorem, which makes clear that replication is mandatory and thus always propagates state that needs to be reconciled. The ability of nodes to freely communicate with one another means that it's possible to delay answers to operations until all nodes have recorded its effects. In the event of a partition, nodes cannot share information with other nodes, so a choice must be made between denying operations that would endanger states getting out of sync, or allowing them through, and dealing with the fallout later.

[1] https://www.infoq.com/articles/cap-twelve-years-later-how-th...

Post reply on HN