The end of a myth: Distributed transactions can scale

The last discussion point from this article very much so rings true for me, too. It's been how many years since the Spanner paper and I still can't get a GCP VM with atomic clocks?

At least now I can provision Cloud Spanner as a managed service, but is this the future of clouds keeping their services at an advantage?

Not GCP related, but I found some local (to my server) telcos and reached out to their IT department and got access to use their internal NTP servers. So my servers are now stratum 1, connected to GPS and atomic clocks, doing skew via chrony. I’m not paying a dime. I merely had to agree to keep my access low and my servers peer with each other to keep their clocks within a few nanoseconds of each other.

It’s amazing what you can get, if you just ask. It’s the latter part you really want, which is doable with modern hardware accelerated time stamping and a peering NTP system (like chrony).

I think the reality is making the SLA guarantees that Spanner in a third party cloud is just... not very practical.

Now do it with shared nothing architecture. When you share all this stuff you are twisting the meaning of "distributed transaction"

    > Is RDMA mature (robust/reliable) enough to use in distributed transactions?

At least on Linux systems, RDMA has been pretty robust/reliable for probably a decade, maybe more.

    > What are the handicaps?

It works differently to TCP/IP, which everyone in IT has at least passing familiarity with. So, it tends to be automatically passed over unless people hit a situation where they're open to "exotic" solutions.

That being said, there's a TCP/IP shim layer (IPoIB) available which can be used by existing software to run on an IB network.

That shim layer though used to have a reputation for flakiness, and it was (or at least used to be) measurably slower than using native IB.

    > What are the reasons for slow uptake on this?

Repeated self-inflicted foot-guns by Mellanox leadership or perhaps their sales and marketing leadership is my best guess.

Mellanox adapters when brand new are priced fairly high for network adapters, or at least they used to be. However, when a network using them is upgraded to the next generation gear, a substantial number of the old ones would commonly end up for ~cheap resale on places like Ebay.

So, *nix DevOps staff ("Sysadmins" back in the day) and anyone else that needed fast networking for their home labs and similar would pick them up and figure out how use them.

Which of course meant over time an increasing group of people familiar with IB, that when figuring out solutions for their work places would then have enough confidence and knowledge to order them.

Sounds like typical organic growth right?

Except Mellanox leadership - or at least their Sales & Marketing people - seemed to be fucking horrified that people were buying their expensive adapters cheaply on Ebay.

So, while the Mellanox technical people were receptive to this ground swell growth in usage by "unofficial" people, and tried to help out, their leadership time and time again did their level best to stamp it out.

Including killing off one of their decent "Community" growth initiatives -> just turned it into a fucking marketing channel for press releases and other crap But also doing things like instructing their support staff to not answer anyone on their forums who seemed to have purchased their gear through unofficial channels, etc. Nor let anyone else do so.

There were many, many examples of this bullshit over the years. And they were always like "Why don't we have huge adoption?"

Gee, I wonder? :( :( :(

Anyway, I gave up on them and moved on a few years afterwards, prior to Nvidia buying them. I still buy older Mellanox ConnectX-3 (VPI) adapters off Ebay occasionally for home gear use (with Linux), and they're still solid. 10/40GbE. :)