Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making every (leap) second count with our new public NTP servers

cloudplatform.googleblog.com

Post reply on HN
Print view View original

11–20 of 73 posts

Re: Making every (leap) second count with our new public NTP servers

#11

Post by detaro »

brandmeyer wrote: post #10
detaro wrote: post #7

Earlier quoted context omitted.

Honest question: what's so bad about others' systems running on slightly off time? I get why people care about internal consistency, and why deviations should be quite small, but this?

Slightly contrived example: Lets say that you were running a distributed database, and you had distributed instances across different cloud providers for increased reliability. if your database relies on high-resolution timestamps for distributed conflict resolution, then you're going to have a hard time. Another example: Suppose that a portion of an industrial monitoring system processes remote sensor data in a clou…

first example: ok, yes, if they offer the DB as a service, that would be bad.

If you run it on a VM it's IMHO your responsibility to make sure your time sensitive database nodes have shared time.

Same for the second example, interesting point for SaaS scenario, although it seems like that could break through normal deviations already.

EDIT: ok, the blog post actually mentions "local clocks in sync with VM instances running on Google Compute Engine", my bad. Not sure what to think about that. In comparison, Amazon recommends running NTP on your VMs and their Linux AMIs come with pool.ntp.org configured as default.

Third: It's going to figure out some solution (if Google is only one source it's probably going to drop it as faulty), but you probably should not have added a time source that's officially documented to not strictly follow standards. It's not like Google offered a NTP service for years and now suddenly switched how it works.

I guess I underestimate the amount of trust people put into random time sources: practice is probably messier than theory.

Re: Making every (leap) second count with our new public NTP servers

#12

Post by KayEss »

brandmeyer wrote: post #3

> Instead of adding a single extra second to the end of the day, we'll run the clocks 0.0014% slower across the ten hours before and ten hours after the leap second, and “smear” the extra second across these twenty hours. Holy leaping second, batman! Unilaterally being off by up to a half second from the rest of the world's clocks is a pretty aggressive step. I think I would have preferred to see a resolution made by…

It's far better than what POSIX clocks do. They'll just drop back a second and you'll get the same time twice.

Re: Making every (leap) second count with our new public NTP servers

#13

Post by magicalist »

brandmeyer wrote: post #3

> Instead of adding a single extra second to the end of the day, we'll run the clocks 0.0014% slower across the ten hours before and ten hours after the leap second, and “smear” the extra second across these twenty hours. Holy leaping second, batman! Unilaterally being off by up to a half second from the rest of the world's clocks is a pretty aggressive step. I think I would have preferred to see a resolution made by…

You're going to have a bad time if you assume "the rest of the world" isn't doing their own, different adjustment

https://developers.google.com/time/smear#othersmears

> preferred to see a resolution made by an independent body

Independent bodies have spent the last decade debating if leap seconds should even exist. Agreeing on how to treat them if we keep them is way down the priority list.

Re: Making every (leap) second count with our new public NTP servers

#14

Post by subway »

brandmeyer wrote: post #10
detaro wrote: post #7

Earlier quoted context omitted.

Honest question: what's so bad about others' systems running on slightly off time? I get why people care about internal consistency, and why deviations should be quite small, but this?

Slightly contrived example: Lets say that you were running a distributed database, and you had distributed instances across different cloud providers for increased reliability. if your database relies on high-resolution timestamps for distributed conflict resolution, then you're going to have a hard time. Another example: Suppose that a portion of an industrial monitoring system processes remote sensor data in a clou…

Everywhere Google documents the service, they clearly state you should not mix their smearing NTP servers with non-smearing NTP servers.

Re: Making every (leap) second count with our new public NTP servers

#15

Post by oasisbob »

brandmeyer wrote: post #10
detaro wrote: post #7

Earlier quoted context omitted.

Honest question: what's so bad about others' systems running on slightly off time? I get why people care about internal consistency, and why deviations should be quite small, but this?

Slightly contrived example: Lets say that you were running a distributed database, and you had distributed instances across different cloud providers for increased reliability. if your database relies on high-resolution timestamps for distributed conflict resolution, then you're going to have a hard time. Another example: Suppose that a portion of an industrial monitoring system processes remote sensor data in a clou…

Both of those examples strike me as very contrived.

I can't think of anyone who cares that much about timekeeping who isn't running their own internal NTP infrastructure.

Google's Spanner requires accurate global time, so they deployed GPS and atomic clocks. Same for CDMA. There are some applications for high-resolution time (eg finance), so protocols like PTP exist.

A smeared NTP source in an otherwise normal list of time sources doesn't seem like that big of a deal either - eventually the daemon is just going to mark it as a falseticker and life goes on.

Re: Making every (leap) second count with our new public NTP servers

#16

Post by oasisbob »

subway wrote: post #14
brandmeyer wrote: post #10

Earlier quoted context omitted.

Slightly contrived example: Lets say that you were running a distributed database, and you had distributed instances across different cloud providers for increased reliability. if your database relies on high-resolution timestamps for distributed conflict resolution, then you're going to have a hard time. Another example: Suppose that a portion of an industrial monitoring system processes remote sensor data in a clou…

Everywhere Google documents the service, they clearly state you should not mix their smearing NTP servers with non-smearing NTP servers.

I'm not convinced it would do anything harmful, so long as you have enough NTP sources (which you should have anyway).

From their FAQ:

> We recommend that you do not mix smeared and non-smeared NTP servers. The results during a leap second may be unpredictable.

I read that as a soft SHOULD NOT, not MUST NOT. Would be a fun exercise to try doing it intentionally with common NTP implementations and see what happens.

Re: Making every (leap) second count with our new public NTP servers

#17

Post by jmgao »

ac29 wrote: post #8
JoshTriplett wrote: post #6

Earlier quoted context omitted.

Smearing leap seconds does make sense, but it's an odd step to take unilaterally, rather than coordinating with other NTP servers and with Linux timekeeping (which currently handles leap seconds via a 61-second minute instead).

>Linux timekeeping (which currently handles leap seconds via a 61-second minute instead) Google doesn't think so: "No commonly used operating system is able to handle a minute with 61 seconds"

Bits of pieces of the operating systems might handle leap seconds properly, but it's doubtful that every single component that uses time does the right thing. The last two leap seconds have revealed bugs in the kernel: https://lwn.net/Articles/504744/ for the one in 2015 and https://lwn.net/Articles/648313/ for the one in 2016, and I don't think it's unlikely that the one scheduled for December will reveal another.

Re: Making every (leap) second count with our new public NTP servers

#18

Post by JoshTriplett »

ac29 wrote: post #8
JoshTriplett wrote: post #6

Earlier quoted context omitted.

Smearing leap seconds does make sense, but it's an odd step to take unilaterally, rather than coordinating with other NTP servers and with Linux timekeeping (which currently handles leap seconds via a 61-second minute instead).

>Linux timekeeping (which currently handles leap seconds via a 61-second minute instead) Google doesn't think so: "No commonly used operating system is able to handle a minute with 61 seconds"

Some things did go wrong on a few of the previous leap-second injections, and the Linux timekeeping maintainer had talked about changing the approach to handling them (which has already changed at least once in the past).

I don't, however, think it makes sense to unilaterally change this, without (any obvious signs of) coordination with the timekeeping maintainers and the maintainers of major NTP servers.

Re: Making every (leap) second count with our new public NTP servers

#19

Post by orblivion »

magicalist wrote: post #13
brandmeyer wrote: post #3

> Instead of adding a single extra second to the end of the day, we'll run the clocks 0.0014% slower across the ten hours before and ten hours after the leap second, and “smear” the extra second across these twenty hours. Holy leaping second, batman! Unilaterally being off by up to a half second from the rest of the world's clocks is a pretty aggressive step. I think I would have preferred to see a resolution made by…

You're going to have a bad time if you assume "the rest of the world" isn't doing their own, different adjustment https://developers.google.com/time/smear#othersmears > preferred to see a resolution made by an independent body Independent bodies have spent the last decade debating if leap seconds should even exist. Agreeing on how to treat them if we keep them is way down the priority list.

> You're going to have a bad time...

Hilarious. Was this intentional?

Re: Making every (leap) second count with our new public NTP servers

#20

Post by toast0 »

JoshTriplett wrote: post #6
brandmeyer wrote: post #3

> Instead of adding a single extra second to the end of the day, we'll run the clocks 0.0014% slower across the ten hours before and ten hours after the leap second, and “smear” the extra second across these twenty hours. Holy leaping second, batman! Unilaterally being off by up to a half second from the rest of the world's clocks is a pretty aggressive step. I think I would have preferred to see a resolution made by…

Smearing leap seconds does make sense, but it's an odd step to take unilaterally, rather than coordinating with other NTP servers and with Linux timekeeping (which currently handles leap seconds via a 61-second minute instead).

I think taking this step unilaterally is the only way it's going to be taken. Given that Google doesn't want to deal with leap seconds [1], and that the standards organizations have been debating removing leap seconds for years, at least they're publicizing what they're doing.

[1] for good reasons

Post reply on HN