Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Clockwork raises $21M to keep server clocks in sync

techcrunch.com

Post reply on HN
Print view View original

81–90 of 128 posts

Re: Clockwork raises $21M to keep server clocks in sync

#81

Post by pclmulqdq »

hansvm wrote: post #11

There are strong lower bounds in the error of pure-software time synchronization techniques. Is the proposal to incorporate additional hardware, or are they just probabilistically increasing the accuracy? If the latter, what applications can benefit from maybe being better synchronized, not being able to measure how much better the synchronization is, and maybe still having the same worst-case bounds that other algor…

The lower bounds are based on the use of occasional time measurements. In PTP terms, the clockwork algorithm involves using a high volume (~10k/second) of peer delay requests, and discarding almost all of them. Only the ones that are deemed to be "pure" are used to adjust clocks.

By using a high volume of requests, they can actually average out a lot of the well-behaved jitter sources.

Re: Clockwork raises $21M to keep server clocks in sync

#82

Post by dan-robertson »

pclmulqdq wrote: post #40
paxys wrote: post #9

Earlier quoted context omitted.

None of the services you mentioned need even close to nanosecond precision between servers to operate. Applications that do (other than the ones in the article like Spanner) are concentrated in supercomputing/scientific computing and spend a LOT of money on the problem. So yeah if you are letting your clocks drift by multiple seconds you aren't "using time" in any real sense. If these folks can achieve what they want…

They can't do better than a microsecond or so without hardware timestsmps. There aren't a lot of applications that seem to need anything between milliseconds and nanoseconds. MiFID II did open a nice market in finance, but many of those companies use PTP anyway. I considered starting a competitor in 2018 when I first saw this company, but I don't have the same connections to customers that these guys do. My conclusio…

My understanding is that PTP is neither necessary nor sufficient. The main advantage of it is that some hardware will only support adding hardware timestamps to PTP messages (ie you want to use the time packets left/arrived at your network card rather than the time you processed them in the kernel/userspace). But I think the specialness of PTP can lead to dealing with a lot of bugs from bad implementations.

Re: Clockwork raises $21M to keep server clocks in sync

#83

Post by dan-robertson »

hn_throwaway_99 wrote: post #3

Google Cloud Spanner uses atomic clocks to be able to synchronize using timestamps across distributed DBs. CockroachDB does not require atomic clocks, but I believe there is an "atomic clock mode" available. This approach sounds like it doesn't use atomic clocks, but instead just machine learning algorithms to detect offsets. Would like to understand if the founders consider their approach to be a viable alternative…

I thought even spanner only relied on sync in the 100us-1ms range. Maybe I’m out of date or it’s more about having clocks advancing at a very reliable rate? Usually ‘atomic clocks’ means ‘gps appliance gets time from atomic clocks in gps satellites’ but maybe not in this case.

Re: Clockwork raises $21M to keep server clocks in sync

#84

Post by shiftingleft »

kloading wrote: post #16
galeaspablo wrote: post #8

Earlier quoted context omitted.

Yup. Going back to the principles of distributed systems, I’m really wondering how “NTP but machine learning” can be used to minimize the clock skew to hundreds of nanoseconds with software only. The RTT between two servers can vary a lot — and way past the calculated minimum RTT. The figures (eg $21M) and names dropped (eg Stanford) are an appeal to authority, which does make me curious. I’d love to see some papers,…

I also haven't seen any work of his using ML, that may just be a buzzword thrown in the PR release. But if anyone's interested, I believe this is the paper alluded to in the article: Clock synchronization: https://www.usenix.org/conference/nsdi18/presentation/geng I think the TechCrunch article doesn't really explain their application of clock synchronization well. Here are the other relevant papers and my attempt at…

> I also haven't seen any work of his using ML, that may just be a buzzword thrown in the PR release.

The first article you reference includes something about Support Vector Machines (SVMs).

Re: Clockwork raises $21M to keep server clocks in sync

#85

Post by daniel-cussen »

This startup genuinely sounds awesome.

I played a lot with an asynchronous chip, meaning it didn't have cores, it had computers, and they didn't have a frequency. Then I connected it to an oscillator (it's easy, just lay one wire) and was getting sub-nanosecond overall measurements with no jitter.

And it's like, why can't you hard-code that in? Like in your test suite, test the code and see if it's as fast as it's supposed to be, or so fast it's clearly not doing the work.

Re: Clockwork raises $21M to keep server clocks in sync

#86

Post by hansel_der »

bawolff wrote: post #68

Not trying to be dismissive but honest question: why would machine learning be good in this problem domain? Normally i think of machine learning as a good way to find patterns in high dimensional data, but clock skew doesn't seem like something that is high dimensional.

> why would machine learning be good in this problem domain?

it doesn't have to be, market will decide.

Re: Clockwork raises $21M to keep server clocks in sync

#87

Post by dan-robertson »

Are there any numbers for what accuracy they are hoping to achieve?

If you don’t try hard you are likely to end up with something accurate to a few ms. AWS has some service which can get you synced to around 300 mics in normal conditions. But 300mics is pretty bad.

Doesn’t this company basically require their customers to not use cloud providers or have they figured out how to get good clock sync despite cloud provider networks? It seems limiting if they don’t work in the cloud.

I think the most common problem we have with clock sync (at least the most common problem I see) come from overloaded network cards slowing down timekeeping packets. I wonder if that’s much of a problem with this company’s solution.

Re: Clockwork raises $21M to keep server clocks in sync

#88

Post by dan-robertson »

KennyBlanken wrote: post #70
wmf wrote: post #53

Earlier quoted context omitted.

Then you need a GPS receiver in each server and a bunch of amps and cables to distribute the signal to every receiver.

1PPS signal distribution is not a complex or remotely difficult problem.

For some reason companies that care a lot about clock sync don’t seem to do this so perhaps there is some other reason.

Re: Clockwork raises $21M to keep server clocks in sync

#89

Post by hansel_der »

icelancer wrote: post #50
Uptrenda wrote: post #23

I was actually reading about this recently and learned about the time problem. Some interesting stuff I remember: * On LAN: NTP can get up to 1ms accuracy and on WAN about 10ms accuracy. * People's 'system clocks' use NTP for synchronization but can still be completely off. * When you call DataTime.now() in Javascript it returns the unix timestamp in UTC time. * Since it is UTC it will have the same value anywhere in…

Yeah, until you sync to ~1ms with 99.999% accuracy required for production-specific services, you don't really realize how big of a pain in the ass this particular problem is. And syncing to ~1ms is comparably easy, although annoying as hell. The problem the OP company is trying to solve is quite a bit tougher. Of course it sounds extremely stupid, which is why there are so many drive-by "NTP solved this already" com…

tbf NTP did solve this already, but then came variable clockspeeds & powersaving and bam back to the stoneage and clocks regularly skew in the order of multiple seconds.

Re: Clockwork raises $21M to keep server clocks in sync

#90

Post by dan-robertson »

hansvm wrote: post #11

There are strong lower bounds in the error of pure-software time synchronization techniques. Is the proposal to incorporate additional hardware, or are they just probabilistically increasing the accuracy? If the latter, what applications can benefit from maybe being better synchronized, not being able to measure how much better the synchronization is, and maybe still having the same worst-case bounds that other algor…

Lots of NICs do hardware timestamping of incoming/outgoing packets these days.
Post reply on HN