Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The end of a myth: Distributed transactions can scale

muratbuffalo.blogspot.com

Post reply on HN
Print view View original

41–50 of 95 posts

Re: The end of a myth: Distributed transactions can scale

#41

Post by Cthulhu_ »

cbsmith wrote: post #6
jzelinskie wrote: post #2

The last discussion point from this article very much so rings true for me, too. It's been how many years since the Spanner paper and I still can't get a GCP VM with atomic clocks? At least now I can provision Cloud Spanner as a managed service, but is this the future of clouds keeping their services at an advantage?

I think the reality is making the SLA guarantees that Spanner in a third party cloud is just... not very practical.

That's it; there's plenty of competing products for Spanner, but Spanner's USP is the host and their hardware.

Re: The end of a myth: Distributed transactions can scale

#42

Post by rezonant »

atoav wrote: post #23
bjornsing wrote: post #13

Earlier quoted context omitted.

Time synchronization over a network with unknown latency characteristics is fundamentally impossible.

Really? It is now Tuesday in central europe and I guess most machines¹ managed to figure out that fact. Most machines will also know it is 8:11. The question is how much resolution do you need and are your clocks accuratly synced enough for the thing you plan to do. You are aware that many of the physics experiments that operate at the edge of what is possible use extremely accurate clocks synced over national and so…

Perfect time synchronization over a network with unknown latency is provably impossible.

Fairly certain distributed physics experiments will be using atomic clocks that are not synchronized over the network. And/or they'll use direct satellite or GPS time sources which have predictable latencies.

Re: The end of a myth: Distributed transactions can scale

#43

Post by rezonant »

withinboredom wrote: post #15
bjornsing wrote: post #13

Earlier quoted context omitted.

Time synchronization over a network with unknown latency characteristics is fundamentally impossible.

Wow. It’s a wonder anyone has a correct time then. /s Sarcasm aside, their NTP servers are less than 2 hops from my router. I know that because of excellent tools like ping and trace route.

Based on your other post, it sounds like you are doing NTP with your ISP and then using PTP to sync your servers on the local network. Nothing about this implies that you are accurate to the global clock within a microsecond, let alone accurate within any number of nanoseconds.

With a correctly equipped and configured PTP installation locally, you can expect your clocks to be synchronized on the order of a very small number of microseconds with respect to each other, but the relationship between those and the atomic global clock is something else.

If you are using satellite/GPS time sources, then that's another matter. Why then is it relevant to have an NTP hookup at your ISP?

Re: The end of a myth: Distributed transactions can scale

#44

Post by withinboredom »

rezonant wrote: post #43
withinboredom wrote: post #15

Earlier quoted context omitted.

Wow. It’s a wonder anyone has a correct time then. /s Sarcasm aside, their NTP servers are less than 2 hops from my router. I know that because of excellent tools like ping and trace route.

Based on your other post, it sounds like you are doing NTP with your ISP and then using PTP to sync your servers on the local network. Nothing about this implies that you are accurate to the global clock within a microsecond, let alone accurate within any number of nanoseconds. With a correctly equipped and configured PTP installation locally, you can expect your clocks to be synchronized on the order of a very small…

With A ISP, not my ISP. Also, I’m syncing with a Stratum 0 servers which are connected to an atomic clock and several GPS clocks.

Also, consistency and accuracy are orthogonal to each other. My servers clocks are consistent and fairly accurate in regards to global time.

Re: The end of a myth: Distributed transactions can scale

#45

Post by rezonant »

withinboredom wrote: post #44
rezonant wrote: post #43

Earlier quoted context omitted.

Based on your other post, it sounds like you are doing NTP with your ISP and then using PTP to sync your servers on the local network. Nothing about this implies that you are accurate to the global clock within a microsecond, let alone accurate within any number of nanoseconds. With a correctly equipped and configured PTP installation locally, you can expect your clocks to be synchronized on the order of a very small…

With A ISP, not my ISP. Also, I’m syncing with a Stratum 0 servers which are connected to an atomic clock and several GPS clocks. Also, consistency and accuracy are orthogonal to each other. My servers clocks are consistent and fairly accurate in regards to global time.

Ah yes, I figured as much, this is why I skipped over the GPS and satellite parts initially. Having a high quality highly local NTP source is better than most sources, but it is nowhere near as close as having your own local time sources, which is what a lot of time sensitive installations do.

Re: The end of a myth: Distributed transactions can scale

#46

Post by samsquire »

Thank you for this.

I implemented multiversion concurrency control in Java with threads but I want to raise it to multimachine.

The problem I have is that the read and write timestamps need to be available to detect if a transaction between machines conflicts.

How do I synchronize read and write timestamps with minimal latency?

When a database replica receives a transaction for key X, it needs (a) a timestamp that is globally accurate (b) needs to tell other replicas about it so they can detect dangerous read-write dependencies.

I feel it inherently requires serialisation of timestamps to accurately determine what other nodes are doing with data.

Otherwise you get dangerous read-write skew.

(See the whitepaper "Serializable Snapshot Isolation")

Load balancers don't do much work, except shift traffic. I wonder if a load balancer that collects timestamps and enriches requests with timestamp information would be a scalable solution?

Re: The end of a myth: Distributed transactions can scale

#47

Post by protocolture »

withinboredom wrote: post #15
bjornsing wrote: post #13

Earlier quoted context omitted.

Time synchronization over a network with unknown latency characteristics is fundamentally impossible.

Wow. It’s a wonder anyone has a correct time then. /s Sarcasm aside, their NTP servers are less than 2 hops from my router. I know that because of excellent tools like ping and trace route.

no-propagate-ttl really does clean up an ISP network.

Re: The end of a myth: Distributed transactions can scale

#48

Post by foota »

samsquire wrote: post #46

Thank you for this. I implemented multiversion concurrency control in Java with threads but I want to raise it to multimachine. The problem I have is that the read and write timestamps need to be available to detect if a transaction between machines conflicts. How do I synchronize read and write timestamps with minimal latency? When a database replica receives a transaction for key X, it needs (a) a timestamp that is…

This is an example of a system built like you're describing at the end: https://www.pingcap.com/blog/how-an-open-source-distributed-...

Also see https://tikv.org/deep-dive/distributed-transaction/timestamp...

Cockroachdb describes their approach here: https://www.cockroachlabs.com/blog/living-without-atomic-clo...

Re: The end of a myth: Distributed transactions can scale

#50

Post by mlichvar »

rezonant wrote: post #39
withinboredom wrote: post #24

Earlier quoted context omitted.

I don’t think Chrony is NTP, if we are being pedantic, just NTP compatible. My servers have hardware timestamping on the network interface and chrony uses this.

That would be PTP then

No, NTP and PTP are two different protocols. They can both use hardware timestamps and reach single-digit nanosecond accuracy in ideal conditions. The main difference is in existing support in switches and routers, which is needed to avoid the impact of asymmetric delay between ports (typically tens of nanoseconds per switch).

PTP has good support in higher-end switches and routers, but it's difficult to secure and make resilient to failures. It was designed for automation and control networks in factories etc. NTP is a better fit for computer networks, but there doesn't seem to be any switches or routers with HW NTP support. If you really need the best accuracy with NTP, you can find old 100Mb/s hubs on ebay and create a separate network.

Post reply on HN