Live data from Hacker News

NewSQL databases fail to guarantee consistency and I blame Spanner

dbmsmusings.blogspot.com

51–60 of 319 posts

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#51
post #2

I'm the author of that post. I'm happy to respond to comments on the post on this thread for the next several hours. You can also leave comments on the post itself, and I will respond there at any time.

The problem isn't a lack of very accurate clocks per se, right? What you need is an accurate bound on clock error, whatever it might be. It sounds to me that non-Googles are specifying bound parameters that, unlike Google, they cannot guarantee. Doesn't that mean the blame goes to whomever put inaccurate bound parameters into the software? Why blame the algorithm for garbage input?

> It sounds to me that non-Googles are specifying bound parameters that, unlike Google, they cannot guarantee.

I've been complaining about this for years and it's so nice to see others echo the sentiment. Everyone chases timestamps but in reality they're harder to get right than most people are willing to acknowledge.

Very few places get NTP right at large-scale, or at least within accuracies required for this class of consistently. I've never seen anyone seriously measure their SLO for clock drift, often because their observability stack is incapable of achieving the necessary resolutions. Most places hand-wave the issue entirely and just assume their clocks will be fine.

The paper linked within TFA suggests a hybrid clock which is better but still carries some complications. I'll continue to recommend vector clocks despite their shortcomings.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#52
post #29

I’ve seen comments on HN over the years in which someone Dunning-Kruegers their way into saying that TrueTime is easily replicated. I always wonder if they have sixteen senior SREs in their pocket, because that’s the level of production engineering Google applies to the problem. Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up. If yo…

> Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up It's another cute anecdote, but Google culture is full of these, always scant on details and always intended to show how big/smart/important/complex/indispensable their engineering is. "Had to" is a strong term here, it's made to sound like USAF could not possibly have noticed some d…

So yes, Google can be very impressed with Google. But I'm not sure that's the issue here.

Is it really surprising that people who have extremely precise time needs and a whole team devoted to solving them would notice issues that other people wouldn't? I think it's a very common pattern that a product has some set of trailblazer users who find issues before the people who make the product.

Also, I think you're over-interpreting. "Had to" here only means that they noticed and reported the issue first because their system depended on GPS time being right. It doesn't preclude the possibility that the USAF would notice and fix the issue eventually, just with a higher latency that Google wanted.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#53
post #29

I’ve seen comments on HN over the years in which someone Dunning-Kruegers their way into saying that TrueTime is easily replicated. I always wonder if they have sixteen senior SREs in their pocket, because that’s the level of production engineering Google applies to the problem. Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up. If yo…

Or just use global consensus with batching and avoid the need for clock synchronization altogether!

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#54
post #3

In case you're wondering : this is a very worthwhile read.

The sibling comment by zzzcpan about consistency guarantees is dead and flagged, and can't be replied to, but I would be interested in reading other takes on this tradeoff.

The take on CAP from CockroachDB, for example, is https://www.cockroachlabs.com/blog/living-without-atomic-clo..., and is linked to in the GP article, but is somewhat downplayed and definitely deserves to be read.

It seems obvious to me that we're within but beyond CAP when it comes to choosing what to sacrifice.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#56

Earlier quoted context omitted.

Also, I believe all the major cloud providers provide a "TrueTime" API service. I forgot the name that AWS uses, but you can call it on your EC2 instances and make sure your hosts are all in sync. It's pretty cool.

AFAIK AWS is only offering NTP service with a GPS source (i.e. stratum 1). TrueTime appears to be a service offering that is a step above that in terms of the guarantees it provides.

Ah, I've never had a use case for anything more accurate than NTP

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#57

Earlier quoted context omitted.

I recently came across CockroachDB and thought it's capabilities interesting, almost too good to be true. I also have been looking at Citus Data which shards and distributes transactions, are you aware of any consistency shortcomings with it?

CockroachDB's performance is dependent on time synchronization. If a node detects it's too far behind, it will commit suicide. However, before it detects it, there is a possibility of stale reads. https://www.cockroachlabs.com/docs/stable/recommended-produc...

Thanks for pointing that out - I have yet to rtfm and dive deep. I wonder how frequently time sync problems occur in virtual environments after ntp syncing - I've seen pretty erratic behavior on virtual active directory domain controllers even after syncing with hyper-v and vmware.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#58
post #29

I’ve seen comments on HN over the years in which someone Dunning-Kruegers their way into saying that TrueTime is easily replicated. I always wonder if they have sixteen senior SREs in their pocket, because that’s the level of production engineering Google applies to the problem. Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up. If yo…

> Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up It's another cute anecdote, but Google culture is full of these, always scant on details and always intended to show how big/smart/important/complex/indispensable their engineering is. "Had to" is a strong term here, it's made to sound like USAF could not possibly have noticed some d…

I specifically worded this to be about money not brains. Most readers here can probably imagine how to implement a bounded time service. Most readers here also cannot afford to operate one. That is the point. Operating software reliably at large scale happens to be very expensive. 24x7 coverage with a short time-to-repair costs at a minimum several million dollars per year.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#59
post #25

The post talks about Spanner using "a specialized hardware solution that uses both GPS and atomic clocks to ensure a minimal clock skew across servers." But what it fails to mention is that is that this time is distributed using a software solution --- NTP, in fact. Google makes its "leap-smeared NTP network" available via Google's Public NTP service. And it's not expensive for someone to buy their own GPS clock and…

I feel like the author is making a huge omission by not talking about the properties of multi-raft sharded DBs coupled with the TrueTime-like APIs from cloud providers in 2018. He links to a post from the CEO of CockroachDB written in early 2016, and Amazon launched their TimeSync service in 2017.

It's likely that Spencer would have something different to say about the matter today.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#60
post #52

Earlier quoted context omitted.

> Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up It's another cute anecdote, but Google culture is full of these, always scant on details and always intended to show how big/smart/important/complex/indispensable their engineering is. "Had to" is a strong term here, it's made to sound like USAF could not possibly have noticed some d…

So yes, Google can be very impressed with Google. But I'm not sure that's the issue here. Is it really surprising that people who have extremely precise time needs and a whole team devoted to solving them would notice issues that other people wouldn't? I think it's a very common pattern that a product has some set of trailblazer users who find issues before the people who make the product. Also, I think you're over-i…

If some condition existed that exceeded GPS intended design, you most certainly wouldn't learn of it first from some random anecdote on HN.. more likely the front page of the BBC as the transportation system instantly collapses

So the anecdote itself is noise, it's intended to show how seriously intractable a problem accurate time is, but it doesn't do that, instead it only demonstrates OP's lack of familiarity with GPS and willingness to regurgitate corporate old wives' tales

Post reply on HN