Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NewSQL databases fail to guarantee consistency and I blame Spanner

dbmsmusings.blogspot.com

Post reply on HN
Print view View original

231–240 of 319 posts

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#231

Post by joemag »

ultraluminous wrote: post #218
joemag wrote: post #208

Earlier quoted context omitted.

You are totally correct that most applications do not require more than a single host to handle all the load, especially given the hosts that are easily available today. However, the far more common reason to go distributed is redundancy. This is especially true for storage systems, which are the topic of this post, and are by far some of the most complex distributed systems. Loosing a node can mean loosing data. Loo…

There's a substantial difference between High Availability (HA) clustering and Load Sharing (LS) clustering. In an HA cluster a production database can run on a single active node while replicating to a separate standby node. Network partitions will cause a lag in replication but will not present an external consistency issue as there's still only one active node at any time. When the partition is resolved the replic…

Agreed that such setups can be simpler than multi-master replication. Yet, there's still a hard distributed systems challenge even in the active/passive setup like you describe. When a failure occurs, the clients need to agree which host is the master and which host is the slave. And in the presence of network partitions, it can be impossible to say if the old master is dead, or partitioned away, and hence still happily taking writes.

The most common way in which such systems failover is by fencing off the old master prior to failover. Which in itself can be a hard distributed problem.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#232

Post by glic3rinu »

he0001 wrote: post #202
zzzcpan wrote: post #177

Earlier quoted context omitted.

> slapping a fancy name on this concept is a justification for giving up consistency (even when the network and all servers are fully functional) to retain availability This is a misconception. AP databases are not supposed to give up consistency, just not wait for all nodes to see updates. That's it. Consistency is still there, nodes resync, users always see their own updates and all that.

So then its CAP not AP? CAP was about you can’t have them all?

The C in there is for "Strong Consistency"; CAP allows for lessers forms of consistency, like "Eventual Consistency", without giving up on Availability.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#233

Post by he0001 »

zzzcpan wrote: post #205
he0001 wrote: post #202

Earlier quoted context omitted.

So then its CAP not AP? CAP was about you can’t have them all?

No, it's still AP, it's just CAP consistency is very specific thing and you can't generalize it into "giving up consistency". AP systems don't give up consistency, they just don't explicitly wait for it.

Then you are saying that all the trouble Spanner goes through to actually be consistent is not needed? And by saying a AP system can be “consistent” there’s no need for a product as Spanner. Consistency is obviously a timing problem. Something that is not consistent at some time will not be consistent since it will be inconsistent at that time.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#234

Post by lllr_finger »

cdcfa78156ae5 wrote: post #213
guelo wrote: post #166

Earlier quoted context omitted.

> Common languages like node.js, Python, R, OCaml, etc. do NOT let you use all your cores "by default" If the code is written in a distributed fashion from the start then it can be designed so it's one python/node/R process per core. And that makes the transition to distributed servers simpler anyway.

> If the code is written in a distributed fashion from the start then it can be designed so it's one python/node/R process per core. That's a really glib dismissal of how hard the problem is. Python and node have pretty terrible support for building distributed systems. With Python, in practice most systems end up based on Celery, with huge long-running tasks. This configuration basically boils down to using Celery,…

I think the intention was that if you're gently coerced into working with a single thread, like with node, then you're also coerced into writing your code in a way that's independent from other threads. In theory, it's easier reasoning about doing parallel work when you start from this point - I've certainly noticed this effect before.

I don't think any reasonable developer would dismiss concurrency/parallelism as easy problems.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#235

Post by he0001 »

glic3rinu wrote: post #232
he0001 wrote: post #202

Earlier quoted context omitted.

So then its CAP not AP? CAP was about you can’t have them all?

The C in there is for "Strong Consistency"; CAP allows for lessers forms of consistency, like "Eventual Consistency", without giving up on Availability.

Well yes but that’s not what we are talking about here. And “eventual consistency” is not “consistency”. And I would argue that “eventual consistency” is not consistent since it can result in fake states since it’s not consistent.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#236

Post by almostdeadguy »

zzzcpan wrote: post #205
he0001 wrote: post #202

Earlier quoted context omitted.

So then its CAP not AP? CAP was about you can’t have them all?

No, it's still AP, it's just CAP consistency is very specific thing and you can't generalize it into "giving up consistency". AP systems don't give up consistency, they just don't explicitly wait for it.

CAP consistency is linearizability, so if what you mean is there are other models of consistency available to "AP systems" (kind of hate the framing that these are binary features of a distributed system rather than a huge space of design choices, but anyways), then yes that's true. But AP systems don't always guarantee things like read-your-own-write consistency models, and not all conflicts that occur during a partition can be resolved in many of these databases.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#237

Post by lllr_finger »

toast0 wrote: post #101
Quekid5 wrote: post #78

Earlier quoted context omitted.

I may be a bit of an old fart, but this is the exact reasoning behind my decision to never go with "distributed X" if there's a "single-machine X" where you can just vertically scale. If you can afford 3-5 machines/VMs for a cluster you can almost certainly afford a single machine/VM with 2-4x the resources/CPU and chances are that it'll perform just as well (or better) because it doesn't have network latency to cont…

I think part of this is that most of the common knowledge about scaling is hard fought from the 90s/2000 era. eBay got bigger and bigger Sun boxes to run Oracle, until they couldn't get anything bigger -- then they had a problem and had to shard their listings into categories, etc. In the last few Intel cpu generations, computation performance has had small gains, but addressable memory has doubled about every other…

My experience is that scaling up can lead to owning pets instead of cattle. It's never fun having a 1TB ATS instance that's acting funky but you're terrified of bouncing it. That's more a devil's advocate anecdote than an argument against scaling up.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#238

Post by mercer »

cdcfa78156ae5 wrote: post #213
guelo wrote: post #166

Earlier quoted context omitted.

> Common languages like node.js, Python, R, OCaml, etc. do NOT let you use all your cores "by default" If the code is written in a distributed fashion from the start then it can be designed so it's one python/node/R process per core. And that makes the transition to distributed servers simpler anyway.

> If the code is written in a distributed fashion from the start then it can be designed so it's one python/node/R process per core. That's a really glib dismissal of how hard the problem is. Python and node have pretty terrible support for building distributed systems. With Python, in practice most systems end up based on Celery, with huge long-running tasks. This configuration basically boils down to using Celery,…

Am I correct in assuming Elixir/Erlang does a much better job at this compared to Node/Python/etc., putting aside (what I understand to be) the rather big problem of their relative weakness for computation?

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#239

Post by zzzcpan »

almostdeadguy wrote: post #236
zzzcpan wrote: post #205

Earlier quoted context omitted.

No, it's still AP, it's just CAP consistency is very specific thing and you can't generalize it into "giving up consistency". AP systems don't give up consistency, they just don't explicitly wait for it.

CAP consistency is linearizability, so if what you mean is there are other models of consistency available to "AP systems" (kind of hate the framing that these are binary features of a distributed system rather than a huge space of design choices, but anyways), then yes that's true. But AP systems don't always guarantee things like read-your-own-write consistency models, and not all conflicts that occur during a part…

Sure. But it's kind of is a huge space of design choices (https://jepsen.io/consistency), so it's not ok to claim you give up consistency with any of them.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#240

Post by deepsun »

ryandrake wrote: post #65
tytso wrote: post #25

The post talks about Spanner using "a specialized hardware solution that uses both GPS and atomic clocks to ensure a minimal clock skew across servers." But what it fails to mention is that is that this time is distributed using a software solution --- NTP, in fact. Google makes its "leap-smeared NTP network" available via Google's Public NTP service. And it's not expensive for someone to buy their own GPS clock and…

> And it's not expensive for someone to buy their own GPS clock and run their own leap-smared NTP service This is a fun weekend hobby project that anyone can do at home if they are curious. Get a GPS chip with a PPS output signal, a plain old “hockey puck” antenna, and a computer to hook it up to (i have a rpi in my attic) and you can deliver fairly accurate time to your home network. Not a setup I’d rely on for a co…

The problem with production usage is not clock accuracy, but accuracy guarantees. If a clock is within whopping 1ms, it is ok as long as I can prove it is definitely within the 1ms.

Which, as far as I see, boils down to hardware+software reliability.

Post reply on HN