Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The end of a myth: Distributed transactions can scale

muratbuffalo.blogspot.com

Post reply on HN
Print view View original

21–30 of 95 posts

Re: The end of a myth: Distributed transactions can scale

#21

Post by jeffreygoesto »

jongjong wrote: post #3

That sounds really good. I think reasons for slow adoption are probably a mix of: 1. Lack of developer awareness. 2. Security implications (or perceived implications) of exposing memory directly to a network without passing through CPU or application-level access control mechanisms. The second point may be prohibitive for a lot of general purpose database systems which are intended to be run on shared infrastructure…

Do you have indications why CPU bound? I have never seen Systems other than for example linear algebra or special algorithms to max out a modern CPU. Almost all loads today are in love way or another memory bound.

Re: The end of a myth: Distributed transactions can scale

#22

Post by ikiris »

bjornsing wrote: post #13
withinboredom wrote: post #4

Earlier quoted context omitted.

Not GCP related, but I found some local (to my server) telcos and reached out to their IT department and got access to use their internal NTP servers. So my servers are now stratum 1, connected to GPS and atomic clocks, doing skew via chrony. I’m not paying a dime. I merely had to agree to keep my access low and my servers peer with each other to keep their clocks within a few nanoseconds of each other. It’s amazing…

Time synchronization over a network with unknown latency characteristics is fundamentally impossible.

ptp to the ToR isn't rocket science. If you need something hyper precise and are a large enough customer, you can get just about anything you need built in even the big clouds.

crazy customer needs is what made up most projects I saw or worked with.

Re: The end of a myth: Distributed transactions can scale

#23

Post by atoav »

bjornsing wrote: post #13
withinboredom wrote: post #4

Earlier quoted context omitted.

Not GCP related, but I found some local (to my server) telcos and reached out to their IT department and got access to use their internal NTP servers. So my servers are now stratum 1, connected to GPS and atomic clocks, doing skew via chrony. I’m not paying a dime. I merely had to agree to keep my access low and my servers peer with each other to keep their clocks within a few nanoseconds of each other. It’s amazing…

Time synchronization over a network with unknown latency characteristics is fundamentally impossible.

Really? It is now Tuesday in central europe and I guess most machines¹ managed to figure out that fact.

Most machines will also know it is 8:11.

The question is how much resolution do you need and are your clocks accuratly synced enough for the thing you plan to do.

You are aware that many of the physics experiments that operate at the edge of what is possible use extremely accurate clocks synced over national and sometimes continental borders?

¹ let's ignore the ones that have been configured incorrectly

Re: The end of a myth: Distributed transactions can scale

#24

Post by withinboredom »

readams wrote: post #20
withinboredom wrote: post #4

Earlier quoted context omitted.

Not GCP related, but I found some local (to my server) telcos and reached out to their IT department and got access to use their internal NTP servers. So my servers are now stratum 1, connected to GPS and atomic clocks, doing skew via chrony. I’m not paying a dime. I merely had to agree to keep my access low and my servers peer with each other to keep their clocks within a few nanoseconds of each other. It’s amazing…

You are definitely not within nanoseconds using NTP

I don’t think Chrony is NTP, if we are being pedantic, just NTP compatible. My servers have hardware timestamping on the network interface and chrony uses this.

Re: The end of a myth: Distributed transactions can scale

#25

Post by starfallg »

withinboredom wrote: post #4
jzelinskie wrote: post #2

The last discussion point from this article very much so rings true for me, too. It's been how many years since the Spanner paper and I still can't get a GCP VM with atomic clocks? At least now I can provision Cloud Spanner as a managed service, but is this the future of clouds keeping their services at an advantage?

Not GCP related, but I found some local (to my server) telcos and reached out to their IT department and got access to use their internal NTP servers. So my servers are now stratum 1, connected to GPS and atomic clocks, doing skew via chrony. I’m not paying a dime. I merely had to agree to keep my access low and my servers peer with each other to keep their clocks within a few nanoseconds of each other. It’s amazing…

In real world applications, NTP is accurate to the millisecond, while PTP is accurate to the microsecond.

PTP is often used now in the Telco world as well as for broadcasting applications.

Re: The end of a myth: Distributed transactions can scale

#26

Post by withinboredom »

jpalomaki wrote: post #16
jandrewrogers wrote: post #12

Earlier quoted context omitted.

There are several good ideas in distributed databases that are effectively not deployable in cloud environments because the requisite hardware features don’t exist. Since cloud deployability is a pervasive prerequisite for commercial viability, database designs tend to overfit for the limitations of cloud environments even though we know how to do much better. Basically, we are stuck in a local minima of making datab…

What kind of hardware features are we missing on cloud?

- network packet timestamping via hardware

- this paper

- dedicated bandwidth (Azure gives you bandwidth based on instance size)

- XDP on network interface

Probably more, but those are what I know of from running into their non-existence.

Re: The end of a myth: Distributed transactions can scale

#27

Post by deepGem »

jpalomaki wrote: post #16
jandrewrogers wrote: post #12

Earlier quoted context omitted.

There are several good ideas in distributed databases that are effectively not deployable in cloud environments because the requisite hardware features don’t exist. Since cloud deployability is a pervasive prerequisite for commercial viability, database designs tend to overfit for the limitations of cloud environments even though we know how to do much better. Basically, we are stuck in a local minima of making datab…

What kind of hardware features are we missing on cloud?

The atomic clock is the key to enable distributed transactions and Google has a proprietary lock on their atomic clocks for Spanner.

Re: The end of a myth: Distributed transactions can scale

#28

Post by vlovich123 »

jamesblonde wrote: post #17

This is impressive, but won't have huge impact, IMO. Way back in 2015, MySQL Cluster (NDB Cluster engine) benchmarked 200m transactions/second on commodity hardware [1]. It was read-committed transactions, not snapshot isolation, but still impressive. NDB (or RonDB, the new DB by its author) uses a non-blocking 2-phase commit protocol (failed transaction coordinators are failed over) and is even open-source. Still, i…

How many machines was that? This is doing it across ~35.

Re: The end of a myth: Distributed transactions can scale

#29

Post by deepGem »

jzelinskie wrote: post #2

The last discussion point from this article very much so rings true for me, too. It's been how many years since the Spanner paper and I still can't get a GCP VM with atomic clocks? At least now I can provision Cloud Spanner as a managed service, but is this the future of clouds keeping their services at an advantage?

The atomic clock is not in the VM, it’s a service from what I understand similar to NTP but obviously much more localised.

I in fact think once you offer atomic clock as a service for any public cloud, distributed transactions become a lot easier to implement thereby disrupting Spanner.

Post reply on HN