Live data from Hacker News

NewSQL databases fail to guarantee consistency and I blame Spanner

dbmsmusings.blogspot.com

271–280 of 319 posts

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#271
post #206

Earlier quoted context omitted.

My first computer had 38911 BASIC BYTES FREE.

Luxury! My first computer (Sinclair ZX Spectrum) had 48KB RAM total, and IIRC 8KB of BASIC RAM available.

Guess I win this round, unless there are older farts than me - Commodore VIC-20 with 3.5KB RAM which was plenty for me to do really cool things with in BASIC and 6502 machine language)

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#272
post #250

Earlier quoted context omitted.

There is a "Datacenter-aware mode": https://apple.github.io/foundationdb/configuration.html#data... Here is some discussion about linearizabilty in fdb: https://news.ycombinator.com/item?id=16884882

Without knowing exactly how reads and writes are replicated, I am skeptical. I have gone through the technical documentation and haven't found many details. For reference, I've done work in storage and consensus algorithms and I can tell you for a fact that without using a consensus algorithm for either reconfiguration or request propagation, you will have consistency violations. I would love to be proven wrong, as m…

FoundationDB uses a consensus algorithm for reconfiguration, but not in the (happy path) transaction pipeline. It provides (by default) strict serializability (i.e. serializability and external consistency/linearizability) for arbitrary, ad hoc, interactive transactions, and it's expected to provide excellent performance when every single transaction is cross node (so e.g. indexes can be efficiently updated this way). It provides better fault tolerance than consensus replicated databases typically can because it needs only N+1 replicas instead of 2N+1 to survive N faults (it keeps 2N+1 replicas of tiny configuration for consensus, if course). It has the best testing story in the industry and is used at scale by, among others, the largest company in the world. Because it doesn't have consensus in the datapath it can have lower latencies than consensus replicated databases in multi region deployments, it also supports asynchronous replication, and an upcoming feature will provide a unique option for multi region failover with sub geographic write latencies, maintaining full transaction durability in failover, as if in synchronous replication, except for the exceptionally rare case where the failure of multiple datacenters in a region are exactly simultaneous.

Besides its extensive documentation, you can read its source code and run its deterministic simulation tests yourself if you are interested (it's Apache licensed). Skepticism on these points was reasonable when we originally launched it in 2012 but is getting a little silly in 2018.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#273
post #224

CTO of YugaByte here. We firmly stand by our claims, and I wanted to explain more. From the post by Daniel: YugaByte, however, continues to claim a guarantee of consistency. I would advise people not to trust this claim. YugaByte, by virtue of its Spanner roots, will run into consistency violations when the local clock on a server suddenly jumps beyond the skew uncertainty window. >> The statement about YugaByte DB i…

Looks like you cross-posted this comment here and on my blog, so I'll also cross-post in my response. I am quite confused by your statement that YugaByte does not claim linearizability. The C of CAP is linearizability (see the original CAP theorem https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.p... ). By claiming to be CP from CAP, you are claiming linearizability. CockroachDB also makes the same CP claim…

HN ate my comment, hope to figure hn out someday! Had written:

From the bailis.org link you posted:

> Linearizability is a guarantee about single operations on single objects.

The place we references "linearizability" (in the Jepsen blog and docs) are in the context of single row-key operations; I can update our docs to clarify that further.

For multi-key transactions, our docs clearly point out that we support Snapshot Isolation now and Serializable Isolation is in the roadmap.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#274
post #224

CTO of YugaByte here. We firmly stand by our claims, and I wanted to explain more. From the post by Daniel: YugaByte, however, continues to claim a guarantee of consistency. I would advise people not to trust this claim. YugaByte, by virtue of its Spanner roots, will run into consistency violations when the local clock on a server suddenly jumps beyond the skew uncertainty window. >> The statement about YugaByte DB i…

Looks like you cross-posted this comment here and on my blog, so I'll also cross-post in my response. I am quite confused by your statement that YugaByte does not claim linearizability. The C of CAP is linearizability (see the original CAP theorem https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.p... ). By claiming to be CP from CAP, you are claiming linearizability. CockroachDB also makes the same CP claim…

Karthik's response to this comment is on the original blog here:

http://dbmsmusings.blogspot.com/2018/09/newsql-database-syst...

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#277
post #238

Earlier quoted context omitted.

> If the code is written in a distributed fashion from the start then it can be designed so it's one python/node/R process per core. That's a really glib dismissal of how hard the problem is. Python and node have pretty terrible support for building distributed systems. With Python, in practice most systems end up based on Celery, with huge long-running tasks. This configuration basically boils down to using Celery,…

Am I correct in assuming Elixir/Erlang does a much better job at this compared to Node/Python/etc., putting aside (what I understand to be) the rather big problem of their relative weakness for computation?

Erlang can be a good fit -- the concurrency primatives allow for execution on multiple cores, and the in memory database (ets) scales pretty well. Large (approaching 1TB per node) mnesia databases require a good bit of operational know how, and willingness to patch things, so try to work up to it. Mnesia is erlang's optionally persistent, optionally distribution database layer that's included in the OTP distribution. It's got features for consensus if you want that, but I've almost always run it in 'dirty' mode, and used other layers to arrange so all the application level writes for a key are sent to a single process which then writes to mnesia -- this establishes an ordering on the updates and (mostly) eliminates the need for consensus.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#278
post #262
post #253

Earlier quoted context omitted.

http://www.supermicro.com/products/system/4U/8048/SYS-8048B-... Quad socket R1 (LGA 2011) supports Intel® Xeon® processor E7-8800 v4/v3, E7-4800 v4/v3 family (up to 24-Core) Up to 12TB DDR4 (128GB 3DS LRDIMM); 96x DIMM slots (8x memory module boards: X10QBi-MEM2) AFAIK the latest Skylake Xeons ("Scalable" - Platinum/Gold/Silver/Bronze) have regressed to 1.5TB support, see https://ark.intel.com/products/93794/Intel-Xe…

That's a previous-gen CPU system. Current is: https://www.supermicro.com/products/system/7U/7089/SYS-7089P... Server itself is might only be $25k but those 224 cores could add another $90k, so the total would be close to $400k. The previous-gen version, SYS-7088B-TR4FT (link in my comment upthread), has 192 DIMM slots, so if you don't need CPU horsepower, you cn get the lower-density modules and still have 12TB (or t…

I edited my answer to indicate that current gen 3.06TB RAM supporting CPUs are not out (yet?)

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#279
post #264

Earlier quoted context omitted.

I'm not sure anyone would ever actually have only one machine, without a failover option, and failing over had better not bring down the whole system.

But now we're back to a distributed system

I don't think that's true, in the usual sense of the term.

When a replica is used as a read slave, that introduces problems with (strong) consistency, that definitely starts to resemble any other distributed system.

Adding automated failover adds the need for some kind of consensus algorithm, another feature of distributed systems.

However, if failover is manual, for example, then 2 nodes can be nearly indistinguishable from 1 node, if no failover occurs.

It also strains the definition of "distributed" if both nodes are adjacent, possibly even with direct network and/or storage connections to each other.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#280
I like the realistic view of noSql vs CAP and the different tradeoffs. We do need to talk more about the disadvantages of noSql systems and all the new database alternatives.

Which brings me to my point - what the author fails to talk about is why spanner has taken the design decision they have made. He does claim scalability, but that is a very general word.

I believe the spanner decision is to avoid making a global voting, as in global across the entire world, from North America, to Asia, Europe and even as far as Australia. Such a global voting will take a lot of time - impose seconds of later on any write.

I think the author, when he talks about having a single global vote with a small window, thinks in terms of a single database, maybe two regions on the Continental USA, as apposed to spanner trying to be Geo distributed.

Post reply on HN