Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NewSQL databases fail to guarantee consistency and I blame Spanner

dbmsmusings.blogspot.com

Post reply on HN
Print view View original

71–80 of 319 posts

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#71

Post by 21 »

wpietri wrote: post #52
hosay123 wrote: post #44

Earlier quoted context omitted.

> Time SRE has at various points had take measures up to and including calling the USAF and telling them their satellites are fucked up It's another cute anecdote, but Google culture is full of these, always scant on details and always intended to show how big/smart/important/complex/indispensable their engineering is. "Had to" is a strong term here, it's made to sound like USAF could not possibly have noticed some d…

So yes, Google can be very impressed with Google. But I'm not sure that's the issue here. Is it really surprising that people who have extremely precise time needs and a whole team devoted to solving them would notice issues that other people wouldn't? I think it's a very common pattern that a product has some set of trailblazer users who find issues before the people who make the product. Also, I think you're over-i…

> Is it really surprising that people who have extremely precise time needs and a whole team devoted to solving them would notice issues that other people wouldn't

If GPS timing is bad, a lot of people will notice that their position on the map is incorrect, because that's the whole purpose of the GPS network.

A 1 microsecond error is 300 meters.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#72

Post by [deleted] »

Cshelton wrote: post #56
otterley wrote: post #47

Earlier quoted context omitted.

AFAIK AWS is only offering NTP service with a GPS source (i.e. stratum 1). TrueTime appears to be a service offering that is a step above that in terms of the guarantees it provides.

Ah, I've never had a use case for anything more accurate than NTP

[deleted]

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#73

Post by akiselev »

michaelt wrote: post #69
romed wrote: post #58

Earlier quoted context omitted.

I specifically worded this to be about money not brains. Most readers here can probably imagine how to implement a bounded time service. Most readers here also cannot afford to operate one. That is the point. Operating software reliably at large scale happens to be very expensive. 24x7 coverage with a short time-to-repair costs at a minimum several million dollars per year.

24x7 coverage with a short time-to-repair costs at a minimum several million dollars per year. Interesting - what are the constituents of that cost? What sort of challenges do you face? Do you use PTP grandmaster clocks, or something else? How many sites, and how many clocks per site? Are the support issues mostly hardware failures, configuration problems, or something else? Is 24/7 support needed because the equipme…

You generally need at least 4-5 SREs for a high availability large (big 5) scale subsystem in a multinational corp just to cover all of the timezones and make sure you're not frantically calling everyone when someone goes on vacation or has to pick up their kid from the nurse. The salary plus benefits and overhead on that is easily in the millions.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#75

Post by dekhn »

abadid wrote: post #16
haberman wrote: post #12

> We will trace the failure to guarantee consistency to a controversial design decision made by Spanner that has been tragically and imperfectly emulated in other systems. This post doesn't establish any "controversy" about Spanner's design decision. It only says that it requires special hardware, which other systems attempt to emulate despite not having this specialized hardware. To call this decision "controversial…

By controversial, I mean that in the database research community (where I spend most of my time), there is significant disagreement about global consensus vs. Spanner's choice of partitioned consensus.

There may be disagreement about Spanner's choice, but my observation of it in production environment is that the choices were reasonable tradeoffs that permitted a large number of products to be moved off BigTable with confidence about the results of the system running in a demanding environment. It has worked well enough to remain established and I don't see any successor with a "better" design coming along and replacing it within a decade.

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#76

Post by ahofmann »

michaelt wrote: post #69
romed wrote: post #58

Earlier quoted context omitted.

I specifically worded this to be about money not brains. Most readers here can probably imagine how to implement a bounded time service. Most readers here also cannot afford to operate one. That is the point. Operating software reliably at large scale happens to be very expensive. 24x7 coverage with a short time-to-repair costs at a minimum several million dollars per year.

24x7 coverage with a short time-to-repair costs at a minimum several million dollars per year. Interesting - what are the constituents of that cost? What sort of challenges do you face? Do you use PTP grandmaster clocks, or something else? How many sites, and how many clocks per site? Are the support issues mostly hardware failures, configuration problems, or something else? Is 24/7 support needed because the equipme…

I think it was meant that Google has such high costs. I read somewhere that Google operates two atomic clocks in each of its data centers, but I can't find a source for it right now, just this: https://www.wired.com/2012/11/google-spanner-time/

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#78

Post by Quekid5 »

nemothekid wrote: post #48

> Systems that guarantee consistency only experience a necessary reduction in availability in the event of a network partition. As networks become more redundant, partitions become an increasingly rare event. And even if there is a partition, it is still possible for the majority partition to be available In my experience, yes network partitions are incredibly rare. However 99% of my distributed ststem partitions hav…

I may be a bit of an old fart, but this is the exact reasoning behind my decision to never go with "distributed X" if there's a "single-machine X" where you can just vertically scale.

If you can afford 3-5 machines/VMs for a cluster you can almost certainly afford a single machine/VM with 2-4x the resources/CPU and chances are that it'll perform just as well (or better) because it doesn't have network latency to contend with.

Of course if you're around N >= 7 or N >= 9, then you should perhaps start considering "distributed X".

As long as your application is mostly built on very few assumptions about consistency it's usually pretty painless to actually go distributed.

Of course, there are legitimate cases where you want to go distributed even with N=3 or N=5, but IME they're very few and far between... but if your 3/5 machines are co-located then it's really likely that the "partitioning" problem is one of the scenarios where they actually all go down simultaneously or the availability goes to 0 because you can't actually reach any of the machines (which are likely to be on the 'same' network).

Re: NewSQL databases fail to guarantee consistency and I blame Spanner

#79

Post by [deleted] »

abadid wrote: post #8
plopz wrote: post #5

Earlier quoted context omitted.

I've only ever worked on small projects so I'm not at all familiar with these very high-scale distributed databases but from the post it seems to indicate that Spanner is in a league of its own because it integrated hardware into the mix where the others are software only. What are the differences in scale between the two categories mentioned?

Yes, Spanner is quite unusual in the distributed database world in how hardware is a pretty important part of their solution. Other systems may claim important integrations with hardware, but for Spanner, the architecture really relies on particular hardware assumptions. To answer your question about scale: there is no real practical difference in scalability between the two categories discussed in the post. Partitio…

[deleted]
Post reply on HN