Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NewSQL databases fail to guarantee consistency and I blame Spanner

dbmsmusings.blogspot.com

Post reply on HN
Print view View original

161–170 of 319 posts

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#161

Post by ihattendorf »

skrebbel wrote: post #131
hosay123 wrote: post #60

Earlier quoted context omitted.

If some condition existed that exceeded GPS intended design, you most certainly wouldn't learn of it first from some random anecdote on HN.. more likely the front page of the BBC as the transportation system instantly collapses So the anecdote itself is noise, it's intended to show how seriously intractable a problem accurate time is, but it doesn't do that, instead it only demonstrates OP's lack of familiarity with…

Sub-millisecond flaws in GPS would make the transportation system collapse? Why?

Here's an interesting article[1] about how relativity affects GPS satellites. The clock ticks in a GPS satellite need to be accurate to within 20-30 nanoseconds for accuracy, and they tick 38 microseconds/day faster to account for relativity.

[1] http://www.astronomy.ohio-state.edu/~pogge/Ast162/Unit5/gps....

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#162

Post by jhayward »

romed wrote: post #29

I’ve seen comments on HN over the years in which someone Dunning-Kruegers their way into saying that TrueTime is easily replicated. I always wonder if they have sixteen senior SREs in their pocket, because that’s the level of production engineering Google applies to the problem. Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up. If yo…

> including calling the USAF and telling them their satellites are fucked up

Citation needed. There is a worldwide organization, led by the US Naval observatory, that keeps constant watch on GPS satellite time performance. If google noticed anything that USNO and the other participants didn't that would need a paper or three published.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#163

Post by toast0 »

qaq wrote: post #154
toast0 wrote: post #101

Earlier quoted context omitted.

I think part of this is that most of the common knowledge about scaling is hard fought from the 90s/2000 era. eBay got bigger and bigger Sun boxes to run Oracle, until they couldn't get anything bigger -- then they had a problem and had to shard their listings into categories, etc. In the last few Intel cpu generations, computation performance has had small gains, but addressable memory has doubled about every other…

You can get commodity x86 server with 12TB RAM and 224 cores.

It looks like 12TB ram is available in eight socket Intel servers? Those are sort of commodity, but availability is limited, and NUMA becomes a much larger issue than in the more easily available dual socket configurations. Looks like Epyc can do 4TB in a dual socket configuration.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#164

Post by infogulch »

ianopolous wrote: post #114
amluto wrote: post #106

Earlier quoted context omitted.

Special relativity has no such problem. In SR you can easily define a “time” coordinate everywhere such that all events can be timestamped with that coordinate and will respect causality. GR at least usually has this property as well. (It doesn’t in the presence of closed timelike curves. It does in weak gravity and in the FLRW metric in cosmology. I’m not sure about the general strong gravity case.

Of course every inertial frame in SR has a well defined time coordinate, but that is not a universal time - other frames will disagree on which of two not-causally-connected events happened first. This is normally explained through the lack of a well defined "simultaneity" across different inertial frames.

You are correct that there is no truly capital-U-Universal time, but it doesn't matter. You control the whole system, so just choose one and call it "true time" and make everything participating in the system match it. Simultaneity in all inertial frames can be translated between one another, so if you go to a new place that has, for example, more time dilation due to different gravity, just note the parameters and translate it into your chosen "true time" and adjust the spread.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#165

Post by paulddraper »

qaq wrote: post #154
toast0 wrote: post #101

Earlier quoted context omitted.

I think part of this is that most of the common knowledge about scaling is hard fought from the 90s/2000 era. eBay got bigger and bigger Sun boxes to run Oracle, until they couldn't get anything bigger -- then they had a problem and had to shard their listings into categories, etc. In the last few Intel cpu generations, computation performance has had small gains, but addressable memory has doubled about every other…

You can get commodity x86 server with 12TB RAM and 224 cores.

Link?

It depends what is meant by "commodity" I guess.

The largest server AWS offers is only 64 physical cores (128 logical) and less than 4 TB RAM.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#166

Post by guelo »

chubot wrote: post #148
toast0 wrote: post #101

Earlier quoted context omitted.

I think part of this is that most of the common knowledge about scaling is hard fought from the 90s/2000 era. eBay got bigger and bigger Sun boxes to run Oracle, until they couldn't get anything bigger -- then they had a problem and had to shard their listings into categories, etc. In the last few Intel cpu generations, computation performance has had small gains, but addressable memory has doubled about every other…

Yup, Google clusters also originally had machines with 1 or 2 CPUs! SMP on Linux was a new thing! Nowadays you easily have 32 cores on a machine, and each core is significantly faster than it was back then (probably at least 10x). That is a compute cluster by the definition of 1999. So for certain "big data" tasks (really "medium data" but not everyone knows the difference), I just use a single machine and shell scri…

> Common languages like node.js, Python, R, OCaml, etc. do NOT let you use all your cores "by default"

If the code is written in a distributed fashion from the start then it can be designed so it's one python/node/R process per core. And that makes the transition to distributed servers simpler anyway.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#167

Post by zwegner »

21 wrote: post #71
wpietri wrote: post #52

Earlier quoted context omitted.

So yes, Google can be very impressed with Google. But I'm not sure that's the issue here. Is it really surprising that people who have extremely precise time needs and a whole team devoted to solving them would notice issues that other people wouldn't? I think it's a very common pattern that a product has some set of trailblazer users who find issues before the people who make the product. Also, I think you're over-i…

> Is it really surprising that people who have extremely precise time needs and a whole team devoted to solving them would notice issues that other people wouldn't If GPS timing is bad, a lot of people will notice that their position on the map is incorrect, because that's the whole purpose of the GPS network. A 1 microsecond error is 300 meters.

> A 1 microsecond error is 300 meters.

While the speed-of-light propagation is about 300 meters in a microsecond, isn't the final position error possibly much greater? For calculating position on Earth, you can think about a sphere expanding at the speed of light from each satellite. The 1 microsecond error here corresponds to a radius 300m bigger or smaller, which only corresponds to 300m horizontal distance on the ground if the satellite is on the horizon (assuming that Earth is locally a flat plane for simplicity here). For a satellite directly overhead, the 300m error is a vertical distance. Calculating the difference in horizontal position from this error is then finding the length of a leg of a right triangle with other leg length D and hypotenuse length D+300m, where D is the orbital distance from the satellite (according to Wikipedia, 20180km). The final horizontal distance error is then sqrt((D+300)^2 - D^2), or about 110km.

Of course, this is just the effect of a 1us error in a single satellite, I'm sure there's ways to detect and compensate for these errors.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#168

Post by 21 »

makomk wrote: post #159
21 wrote: post #71

Earlier quoted context omitted.

> Is it really surprising that people who have extremely precise time needs and a whole team devoted to solving them would notice issues that other people wouldn't If GPS timing is bad, a lot of people will notice that their position on the map is incorrect, because that's the whole purpose of the GPS network. A 1 microsecond error is 300 meters.

If I remember correctly, there was a bug a couple of years back which caused an incorrect time offset between GPS and UTC time to be uploaded to some of the satellites - off by a handful of microseconds. Didn't affect navigation but it did trip a bunch of alerts on systems that relied on precise time. I don't think Google was the one that alerted the USAF to that though, in fact they may not have had sufficiently acc…

> Despite the flawed data set, there were no impacts to GPS positioning and navigation. Furthermore, GPS time (tGPS) was unaffected. Only a subset of the functions that make use of the GPS-UTC offset were affected.

https://www.gps.gov/systems/gps/performance/2016-UTC-offset-...

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#169

Post by StreamBright »

peterwwillis wrote: post #62
abadid wrote: post #2

I'm the author of that post. I'm happy to respond to comments on the post on this thread for the next several hours. You can also leave comments on the post itself, and I will respond there at any time.

"Systems that guarantee consistency only experience a necessary reduction in availability in the event of a network partition." Many of the distributed clusters I've maintained had crap infrastructure and no change control, and parts of the clusters were constantly going down from lack of storage, CPU and RAM, or bad changes. The only reason the applications kept working were either (1) the not-broken vnodes continue…

Even if you maintain your infrastructure properly you run on x86 servers with disks and CPUs that need cooling, using network devices that have fascinating failure scenarios. I guess assuming that your infra is not reliable is a must for any database nowadays.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#170

Post by qaq »

paulddraper wrote: post #165
qaq wrote: post #154

Earlier quoted context omitted.

You can get commodity x86 server with 12TB RAM and 224 cores.

Link? It depends what is meant by "commodity" I guess. The largest server AWS offers is only 64 physical cores (128 logical) and less than 4 TB RAM.

https://lenovopress.com/lp0647-thinksystem-sr950-server
Post reply on HN