I think the idea is great. Since I had really bad experience with NPT and it was not trivial to sync the clocks. BUT just a bit surprised that they raised so much money.
Clockwork raises $21M to keep server clocks in sync
111–120 of 128 posts
Re: Clockwork raises $21M to keep server clocks in sync
#112Earlier quoted context omitted.
None of the services you mentioned need even close to nanosecond precision between servers to operate. Applications that do (other than the ones in the article like Spanner) are concentrated in supercomputing/scientific computing and spend a LOT of money on the problem. So yeah if you are letting your clocks drift by multiple seconds you aren't "using time" in any real sense. If these folks can achieve what they want…
> None of the services you mentioned need even close to nanosecond precision between servers to operate. Applications that do (other than the ones in the article like Spanner) are concentrated in supercomputing/scientific computing and spend a LOT of money on the problem No, not really. I work in finance (HFT, systematic market making) and this industry heavily relies on high precision clock sync. It's even regulated…
especially the Huygens-implementation delivering accuracy in nanoseconds could be some part of its secret sauce (https://en.wikipedia.org/wiki/Clock_synchronization#Huygens)
Re: Clockwork raises $21M to keep server clocks in sync
#113Google Cloud Spanner uses atomic clocks to be able to synchronize using timestamps across distributed DBs. CockroachDB does not require atomic clocks, but I believe there is an "atomic clock mode" available. This approach sounds like it doesn't use atomic clocks, but instead just machine learning algorithms to detect offsets. Would like to understand if the founders consider their approach to be a viable alternative…
I thought even spanner only relied on sync in the 100us-1ms range. Maybe I’m out of date or it’s more about having clocks advancing at a very reliable rate? Usually ‘atomic clocks’ means ‘gps appliance gets time from atomic clocks in gps satellites’ but maybe not in this case.
So Spanner uses TrueTime as an integral part of it's concurrency control algorithm. To preserve external consistency, Spanner sometimes has to wait out an uncertainty interval at transaction commit. The tighter the bounds on timestamp intervals from TrueTime, the less this waiting impacts performance. I've heard through the grape vine that TrueTime currently operates much better than the numbers in the original paper, but can't confirm if that's true.
They use GPS for global sync and a local atomic clock as reference. From conversations over the last few years this seems to be the common setup at big DCs, as the cost for a couple atomic clocks is minuscule compared to the scale of everything else.
Re: Clockwork raises $21M to keep server clocks in sync
#114IEEE 1588 (Precision Time Protocol) has already solved this problem for every use case that matters and offers microsecond precision. https://www.eecis.udel.edu/~mills/ptp.html PTP takes advantage of special Ethernet switches and other devices which can decode or manipulate the time tags in hardware because yes your switches add latency. This just sounds like more horseshit out of SV. I applaud them for convincing so…
A very good overview of the different solutions can be found here: https://ying-zhang.github.io/time/2021-02-revisiting-sync/
seems to be more of a pushing the envelope than snake oil to me, but what do i know
Re: Clockwork raises $21M to keep server clocks in sync
#115Earlier quoted context omitted.
Yup. Going back to the principles of distributed systems, I’m really wondering how “NTP but machine learning” can be used to minimize the clock skew to hundreds of nanoseconds with software only. The RTT between two servers can vary a lot — and way past the calculated minimum RTT. The figures (eg $21M) and names dropped (eg Stanford) are an appeal to authority, which does make me curious. I’d love to see some papers,…
I also haven't seen any work of his using ML, that may just be a buzzword thrown in the PR release. But if anyone's interested, I believe this is the paper alluded to in the article: Clock synchronization: https://www.usenix.org/conference/nsdi18/presentation/geng I think the TechCrunch article doesn't really explain their application of clock synchronization well. Here are the other relevant papers and my attempt at…
I'm from the Clockwork team, thanks for listing the relevant papers.
Accurate clocks sync enables true one-way delay measurements (instead of RTT/2), this allows for edge-based network visibility. We launched Latency Sensei beta – a sensor, monitor and auditor that provides visibility into cloud deployments. The gallery has cloud fitness reports on GCP, AWS and Azure. Some interesting reports include: 1)how VM colocation impairs network bandwidth, and 2) tale of 2 cloud regions, London vs Singapore. Take a look and we'd love to get some feedback https://sensei.clockwork.io/user/gallery/
On Congestion control, an edge-based solution is coming soon. If you're interested in a private beta, email us at hello@clockwork.io
Re: Clockwork raises $21M to keep server clocks in sync
#116Earlier quoted context omitted.
If only these devices had a clock capable of measuring time to within 1 nanosecond. If one has such an accurate clock they might be able to accurately plot their position on earth to within 1 ft merely by checking the timed pulses of orbiting satellites.
I guess you're downvoted for being sarcastic or maybe people don't know 20 dollar GPS chips provide a ~10 nanosecond resolution clock
Haha I have no idea. But yeah just call the GPS algo a Time AI and voila, $21 m in funding to go get some kids in shenzen to put a GPS chip on a USB, PCIe, NVME slot.
Hey look we got a GPU to learn the Kalman filter algo.
Many are actually better than 10 ns basically a 1 foot CEP is 1 ns. If they are fixed and always on the accuracy it can build over a day is incredible.
Re: Clockwork raises $21M to keep server clocks in sync
#117Earlier quoted context omitted.
Precision vs. accuracy of what measurement? For GPS to work at all, you need very accurate knowledge of the offset between each satellite's signal. The only way I can think of to have precision but not accuracy is if you design a system where there's a buffer between the antenna and the signal processing and you don't know how long the buffer is. Is a design like that a practical concern?
They're saying that as a receiver of GPS signals, it doesn't matter if your clock is out of sync with everyone else's clocks, as long as it measures the length of a nanosecond accurately.
But GPS gives you a clock signal.
Re: Clockwork raises $21M to keep server clocks in sync
#118Earlier quoted context omitted.
Precision vs. accuracy of what measurement? For GPS to work at all, you need very accurate knowledge of the offset between each satellite's signal. The only way I can think of to have precision but not accuracy is if you design a system where there's a buffer between the antenna and the signal processing and you don't know how long the buffer is. Is a design like that a practical concern?
Of UTC. Theoretically GPS can work out of sync with respect to UTC as long as all satellites are in sync among themselves.
Oh, you're talking about a theoretical situation where the GPS satellites are reprogrammed to use a different synchronization source? Personally I'd still call that highly accurate, but to a different time standard. But I see where you're coming from.
Re: Clockwork raises $21M to keep server clocks in sync
#119Interesting. CERN has white rabbit for sub-ns timing. Needs special hardware but is now IEEE 1588 (PTP) "High Accuracy" profile. I wonder how this deals with the numerous retimers and DSPs required for really high speed ethernet.
In a simple 100 mbps network where I needed good synchronization, I used the RXC pin of the MII bus as input to a PLL+VCXO and explicitly configured the PHYs in the link to be master-->slave in the desired clock distribution direction. PTP provided phase. Across a small network (a couple switch layers) I was seeing about ~20 ns of time uncertainty. Clocks were locked within about 2 ppb. Worked well enough for my need…
Yeah I think SyncE is what telecom RANs use to get within 5g reqs on timing. White Rabbit operates like 1588 + SyncE + a compensation model for the fiber link. It's pretty slick.
It does need L2-L1 coordination but the ethernet MAC usually has no idea of what's happening on the L1 (it speaks "medium independent interface" or *MII). The CERN people warn against using Base-T since the PHY needs to do pretty complex signal processing which would likely destroy the syntonization.
They then use basic fiber links, but 100g, 400g+ have a gearbox and DSP and FEC in the way.
Re: Clockwork raises $21M to keep server clocks in sync
#120I see this is a problem, but how many companies really need this? In my limited experience I have only seen clock sync issues at scale and simple NTP/Chrony setup solves majority of the problems, unless you're in HFT domain where PTP is needed. Nowadays, many of those atscale systems are getting SaaSified where the companies using them dont have to directly deal with such issues. What am I missing here? Is there really a big market for this?