Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The end of a myth: Distributed transactions can scale

muratbuffalo.blogspot.com

Post reply on HN
Print view View original

11–20 of 95 posts

Re: The end of a myth: Distributed transactions can scale

#11

Post by justinclift »

HopenHeyHi wrote: post #9
jongjong wrote: post #3

That sounds really good. I think reasons for slow adoption are probably a mix of: 1. Lack of developer awareness. 2. Security implications (or perceived implications) of exposing memory directly to a network without passing through CPU or application-level access control mechanisms. The second point may be prohibitive for a lot of general purpose database systems which are intended to be run on shared infrastructure…

The experimental setup involves using a cluster of 56 machines connected by an InfiniBand FDR network That bit might have something to do with it.

Heh Heh Heh

A bulk purchase of ~60 FDR IB cards, cabling, and network switches to support them sounds pretty expensive.

That being said, IB FDR gear is "older tech" now so the cards and switches can commonly be found reasonably cheaply on Ebay. The switches tend to be bloody loud though, so they're not something you'd want nearby if you can help it.

Re: The end of a myth: Distributed transactions can scale

#12

Post by jandrewrogers »

jongjong wrote: post #3

That sounds really good. I think reasons for slow adoption are probably a mix of: 1. Lack of developer awareness. 2. Security implications (or perceived implications) of exposing memory directly to a network without passing through CPU or application-level access control mechanisms. The second point may be prohibitive for a lot of general purpose database systems which are intended to be run on shared infrastructure…

There are several good ideas in distributed databases that are effectively not deployable in cloud environments because the requisite hardware features don’t exist. Since cloud deployability is a pervasive prerequisite for commercial viability, database designs tend to overfit for the limitations of cloud environments even though we know how to do much better.

Basically, we are stuck in a local minima of making databases work well in cloud environments that are not designed to enable efficient distributed databases. It is wasteful and also provides an arbitrage opportunity for cloud companies.

Re: The end of a myth: Distributed transactions can scale

#13

Post by bjornsing »

withinboredom wrote: post #4
jzelinskie wrote: post #2

The last discussion point from this article very much so rings true for me, too. It's been how many years since the Spanner paper and I still can't get a GCP VM with atomic clocks? At least now I can provision Cloud Spanner as a managed service, but is this the future of clouds keeping their services at an advantage?

Not GCP related, but I found some local (to my server) telcos and reached out to their IT department and got access to use their internal NTP servers. So my servers are now stratum 1, connected to GPS and atomic clocks, doing skew via chrony. I’m not paying a dime. I merely had to agree to keep my access low and my servers peer with each other to keep their clocks within a few nanoseconds of each other. It’s amazing…

Time synchronization over a network with unknown latency characteristics is fundamentally impossible.

Re: The end of a myth: Distributed transactions can scale

#15

Post by withinboredom »

bjornsing wrote: post #13
withinboredom wrote: post #4

Earlier quoted context omitted.

Not GCP related, but I found some local (to my server) telcos and reached out to their IT department and got access to use their internal NTP servers. So my servers are now stratum 1, connected to GPS and atomic clocks, doing skew via chrony. I’m not paying a dime. I merely had to agree to keep my access low and my servers peer with each other to keep their clocks within a few nanoseconds of each other. It’s amazing…

Time synchronization over a network with unknown latency characteristics is fundamentally impossible.

Wow. It’s a wonder anyone has a correct time then. /s

Sarcasm aside, their NTP servers are less than 2 hops from my router. I know that because of excellent tools like ping and trace route.

Re: The end of a myth: Distributed transactions can scale

#16

Post by jpalomaki »

jandrewrogers wrote: post #12
jongjong wrote: post #3

That sounds really good. I think reasons for slow adoption are probably a mix of: 1. Lack of developer awareness. 2. Security implications (or perceived implications) of exposing memory directly to a network without passing through CPU or application-level access control mechanisms. The second point may be prohibitive for a lot of general purpose database systems which are intended to be run on shared infrastructure…

There are several good ideas in distributed databases that are effectively not deployable in cloud environments because the requisite hardware features don’t exist. Since cloud deployability is a pervasive prerequisite for commercial viability, database designs tend to overfit for the limitations of cloud environments even though we know how to do much better. Basically, we are stuck in a local minima of making datab…

What kind of hardware features are we missing on cloud?

Re: The end of a myth: Distributed transactions can scale

#17

Post by jamesblonde »

This is impressive, but won't have huge impact, IMO.

Way back in 2015, MySQL Cluster (NDB Cluster engine) benchmarked 200m transactions/second on commodity hardware [1]. It was read-committed transactions, not snapshot isolation, but still impressive. NDB (or RonDB, the new DB by its author) uses a non-blocking 2-phase commit protocol (failed transaction coordinators are failed over) and is even open-source. Still, it hasn't had a big impact outside of the Telecom and real-time gaming worlds.

[1] https://www.slideshare.net/frazerClement/200-million-qps-on-...

Re: The end of a myth: Distributed transactions can scale

#20

Post by readams »

withinboredom wrote: post #4
jzelinskie wrote: post #2

The last discussion point from this article very much so rings true for me, too. It's been how many years since the Spanner paper and I still can't get a GCP VM with atomic clocks? At least now I can provision Cloud Spanner as a managed service, but is this the future of clouds keeping their services at an advantage?

Not GCP related, but I found some local (to my server) telcos and reached out to their IT department and got access to use their internal NTP servers. So my servers are now stratum 1, connected to GPS and atomic clocks, doing skew via chrony. I’m not paying a dime. I merely had to agree to keep my access low and my servers peer with each other to keep their clocks within a few nanoseconds of each other. It’s amazing…

You are definitely not within nanoseconds using NTP
Post reply on HN