Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NewSQL databases fail to guarantee consistency and I blame Spanner

dbmsmusings.blogspot.com

Post reply on HN
Print view View original

241–250 of 319 posts

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#241

Post by ianopolous »

infogulch wrote: post #164
ianopolous wrote: post #114

Earlier quoted context omitted.

Of course every inertial frame in SR has a well defined time coordinate, but that is not a universal time - other frames will disagree on which of two not-causally-connected events happened first. This is normally explained through the lack of a well defined "simultaneity" across different inertial frames.

You are correct that there is no truly capital-U-Universal time, but it doesn't matter. You control the whole system, so just choose one and call it "true time" and make everything participating in the system match it. Simultaneity in all inertial frames can be translated between one another, so if you go to a new place that has, for example, more time dilation due to different gravity, just note the parameters and t…

My point is not that it isn't possible, but that it is arbitrary and has no physical meaning. Writes thrown away in one frame because another concurrent write was "later" would in fact be kept in another frame. This is why it feels like a 'bug' to me conceptually - it's not how the universe works so why should a database need it.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#242

Post by rwtxn »

jules wrote: post #110

The CAP theorem has been truly disastrous for databases. The CAP theorem simply says that if you have a database on 2 servers and the connection between those serves goes down, then queries against one server don't see new updates from the other server, so you either have to give up consistency (serve stale data) or give up availability (one of the servers refuses to process further requests). That's all that CAP is,…

> The A for availability in CAP means that ALL database servers are fully available, which is unnecessary in practice, because clients can switch to the other servers.

This assumes that only the servers are partitioned from each other and clients are not partitioned from the majority quorum. This might be rare but it is not impossible at scale.

There is also a latency cost of strict serializability or linearizability which is hard to mitigate at geo-replicated scale.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#243

Post by mjevans »

Quekid5 wrote: post #78
nemothekid wrote: post #48

> Systems that guarantee consistency only experience a necessary reduction in availability in the event of a network partition. As networks become more redundant, partitions become an increasingly rare event. And even if there is a partition, it is still possible for the majority partition to be available In my experience, yes network partitions are incredibly rare. However 99% of my distributed ststem partitions hav…

I may be a bit of an old fart, but this is the exact reasoning behind my decision to never go with "distributed X" if there's a "single-machine X" where you can just vertically scale. If you can afford 3-5 machines/VMs for a cluster you can almost certainly afford a single machine/VM with 2-4x the resources/CPU and chances are that it'll perform just as well (or better) because it doesn't have network latency to cont…

Past a given point it's time to re-design and optimize for today rather than what was quick and easy with a small user-base. Unless you know when you're starting that such a huge user-base is the target the Quekid5's approach is proper.

Past that easy single / very small cluster case it's time to start asking more important design questions. As an example, which part(s) NEED to be ACID complaint and which can have eventual consistency (or do they even need that as long as the data was valid 'at some point'? So just 'atomic'.)

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#244

Post by rwtxn »

CBLT wrote: post #150
abadid wrote: post #2

I'm the author of that post. I'm happy to respond to comments on the post on this thread for the next several hours. You can also leave comments on the post itself, and I will respond there at any time.

Great read, thank you for sharing. Do you have any opinion on the design of Eris[0]? Consistency is achieved with extra hardware, but that hardware is a network-level sequencer. [0]: https://syslab.cs.washington.edu/papers/eris-sosp17.pdf

AFAIK Eris (and previous similar work from the same group) assume network ordering guarantees that can be provided in a single datacenter but probably not in a WAN setting. This discussion is about distributed and geo-replicated databases.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#245

Post by grogers »

Dave_Rosenthal wrote: post #119
idubrov wrote: post #89

Any comments on how FoundationDB might be the same / different?

FoundationDB adheres to the highest level of consistency (linearizability) and, unlike Spanner or its derivatives, does not rely on clocks to achieve this. I'm surprised FoundationDB is not mentioned in the article. Maybe it didn't help the author make his point, but it is both the most mature of the new breed of ACID+noSQL databases and completely free/open source.

According to their docs, FoundationDB only provides serializable isolation, so it isn't the same as the others in the post which offer strict serializable isolation (multi-key version of linearizability). Without strong clocks you can't have strict serializablity and scale beyond a single log (but one log can get you pretty damn far). TBH most people probably only need serializable transactions anyway.

https://apple.github.io/foundationdb/developer-guide.html#tr...

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#246

Post by wting »

bostik wrote: post #217
how_gauche wrote: post #182

Earlier quoted context omitted.

A rational way of explaining it is -- what is my service level objective? If my SLO is for two nines of uptime, then I can be down for 3.65 days a year, and I can get away with single-homing something, or going with a simple hot-failover replicated setup. I just need to be pretty confident that I can fix it if it breaks within a few hours. If I need more nines of uptime than a single system could provide (and think a…

> what is my service level objective? There are environments where flat time distribution for SLO calculation is not acceptable. ( cough betting exchange) If your traffic patterns are extremely spiky, such as weekly peaks hitting 15-20x of your base load, and where a big chunk of your business can come from those peaks, then most normal calculations don't apply. Let's say your main system that accepts writes is 10 mi…

Time based SLOs definitely have their limitations, but in this instance isn't it fairly easy to redefine the SLO in terms of requests rather than time?

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#247

Post by jscissr »

uluyol wrote: post #215
voidmain wrote: post #210

This article lacks any reference to FoundationDB, which offers external consistency and serializable distributed transactions without trusting clocks in any way. We designed it starting in 2009 and so its lineage is independent of either Calvin or Spanner. FDB doesn't have an actively developed SQL layer at the moment, so I guess you could say it isn't a "NewSQL" database, but none of the properties under discussion…

As best as I could find, FoundationDB doesn't provide strict serializability (i.e. linearizability + serializability) and doesn't tackle multi-DC deployments. So I wouldn't put it in the same class as FaunaDB or Spanner.

There is a "Datacenter-aware mode": https://apple.github.io/foundationdb/configuration.html#data...

Here is some discussion about linearizabilty in fdb: https://news.ycombinator.com/item?id=16884882

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#248

Post by rkarthik007 »

abadid wrote: post #224
rkarthik007 wrote: post #199

CTO of YugaByte here. We firmly stand by our claims, and I wanted to explain more. From the post by Daniel: YugaByte, however, continues to claim a guarantee of consistency. I would advise people not to trust this claim. YugaByte, by virtue of its Spanner roots, will run into consistency violations when the local clock on a server suddenly jumps beyond the skew uncertainty window. >> The statement about YugaByte DB i…

Looks like you cross-posted this comment here and on my blog, so I'll also cross-post in my response. I am quite confused by your statement that YugaByte does not claim linearizability. The C of CAP is linearizability (see the original CAP theorem https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.p... ). By claiming to be CP from CAP, you are claiming linearizability. CockroachDB also makes the same CP claim…

Will cross-post as well - unclear about protocols here :)

From the Peter Bailis link you had:

> Linearizability is a guarantee about single operations on single objects.

Our references to "linearizability" in the Jepsen blog and the docs are in the context of single key operations; happy to update to update our docs to clarify that further.

For multi-key transactions in YugaByte DB, our docs clearly point out that we offer Snapshot Isolation today and Serializable Isolation is in our roadmap (https://docs.yugabyte.com/latest/architecture/transactions/i...).

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#249

Post by amaccuish »

jgalentine007 wrote: post #57
cube2222 wrote: post #30

Earlier quoted context omitted.

CockroachDB's performance is dependent on time synchronization. If a node detects it's too far behind, it will commit suicide. However, before it detects it, there is a possibility of stale reads. https://www.cockroachlabs.com/docs/stable/recommended-produc...

Thanks for pointing that out - I have yet to rtfm and dive deep. I wonder how frequently time sync problems occur in virtual environments after ntp syncing - I've seen pretty erratic behavior on virtual active directory domain controllers even after syncing with hyper-v and vmware.

Time shouldn't be a massive issue for AD no? It's a vector clock, not a UTC clock. The UTC clock is only used to solve conflicts no?

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#250

Post by uluyol »

jscissr wrote: post #247
uluyol wrote: post #215

Earlier quoted context omitted.

As best as I could find, FoundationDB doesn't provide strict serializability (i.e. linearizability + serializability) and doesn't tackle multi-DC deployments. So I wouldn't put it in the same class as FaunaDB or Spanner.

There is a "Datacenter-aware mode": https://apple.github.io/foundationdb/configuration.html#data... Here is some discussion about linearizabilty in fdb: https://news.ycombinator.com/item?id=16884882

Without knowing exactly how reads and writes are replicated, I am skeptical. I have gone through the technical documentation and haven't found many details. For reference, I've done work in storage and consensus algorithms and I can tell you for a fact that without using a consensus algorithm for either reconfiguration or request propagation, you will have consistency violations.

I would love to be proven wrong, as more systems with strong consistency guarantees is better, but for now, I don't believe that foundation db provides stronger guarantees than serializable reads and writes.

Post reply on HN