Live data from Hacker News

Some DNS lookups causing 5xx errors due to leap second bug

cloudflarestatus.com

41–50 of 135 posts

Re: Some DNS lookups causing 5xx errors due to leap second bug

#41

I'm curious what if anything would be problematic if everything just effectively "ignored" leap seconds (i.e. would this outage not have occurred?) --- one minute is always 60 seconds, an hour is always 60 minutes, and a day always 24h. I mean, if you consider the fact that human society has managed to function perfectly well with almost everyone not knowing nor caring what a leap second is, and yet apparently some s…

100 seconds per minute, 100 minutes per hour, 20 hours a day. New seconds are 0.432 old seconds, or whatever ratio they need to make to quit leaping around.

Re: Some DNS lookups causing 5xx errors due to leap second bug

#42
post #28
post #20

Earlier quoted context omitted.

Financial systems need very precise timekeeping. I'm sure other fields do as well.

But financial systems don't care if the position of the sun in the sky is a couple of seconds off from where a model says it should be. Astronomers would care about that, but they already don't use UTC.

But they do care what GPS reference time is. Which are satellites very much dependent on holding an accurate position in the sky (which is dependent on the Earth's rotational speed, which changes, which is why we have leap seconds).

So go figure: which part of this system should be broken because people keep ignoring that leap seconds happen?

Re: Some DNS lookups causing 5xx errors due to leap second bug

#43
post #9
post #7

My CDMA phone dropped service for a few minutes after the leap second. It's absurd that we continue to keep subjecting ourselves to these disruptions and the considerable amount of work that goes into handling leap seconds for the systems that aren't disrupted by them. Leap seconds serve no useful purpose. Applications that care about solar time care usually care about the local solar time, while UT1 is a 'mean solar…

Or maybe we just need to code system that handles leap seconds correctly.

It's difficult to write bug-free code for events that happen very infrequently. Even more so when the distributed nature of the system makes effective testing under real-world conditions nearly impossible.

Re: Some DNS lookups causing 5xx errors due to leap second bug

#44
post #36

Earlier quoted context omitted.

Yes but even here, seconds since Epoch should remain unaltered, and the correction should be made by whatever is rendering a human readable date format (to address every leap second). In most cases, the renderer wouldn't have to address it (since it's only being read by humans, and a second difference does not usually matter) and it's truly a non-issue! The application-layer dev can choose to increment time in whatev…

Unfortunately, epoch time is not literally "seconds since epoch", at least not as implemented/standardized as "Unix time". It skips or repeats itself in case of leap seconds. So it can't save us here. I think if there were such a thing as a different kind of epoch time that literally actually is "seconds since epoch" it would help a lot and work like you suggest.

Yes, I agree -- I am not referring to any specific implementations. Didn't know that about Unix time, but it makes sense for compatibility given the current way we adjust for leap seconds.

Re: Some DNS lookups causing 5xx errors due to leap second bug

#45

I'm curious what if anything would be problematic if everything just effectively "ignored" leap seconds (i.e. would this outage not have occurred?) --- one minute is always 60 seconds, an hour is always 60 minutes, and a day always 24h. I mean, if you consider the fact that human society has managed to function perfectly well with almost everyone not knowing nor caring what a leap second is, and yet apparently some s…

100 seconds per minute, 100 minutes per hour, 20 hours a day. New seconds are 0.432 old seconds, or whatever ratio they need to make to quit leaping around.

Fun fact: the reason we use 60 seconds and 60 minutes is because of the Babylonians who used base-60. IIRC, it's also why we use 360° for a circle.

Re: Some DNS lookups causing 5xx errors due to leap second bug

#47
Once again we're screwed by different people wanting "time" to mean different things. There is no hope for humanity once we start traveling anywhere close to light speed into and out of the solar system.

I propose a new "non-time" time system. It has exactly two real values which range from 0 to tau and an integer, the first real number is radians of earth rotation, and the second is radians of the rotation around the Sun. The integer reflects the number of complete cycles. So lunch time in Greenwich 'pi'.

It has the benefit that its "source" is actually the planet, so we can use a telescope at Greenwich to pick a certain alignment of stars as the "zero", "zero" point and then each time it realigns to that exact point, you can increment the "year" count.

I believe we can build a robust system to support this out of stone. We'll need to create a circle of stones but using a small hole drilled through a stone and a marker on the ground we can always identify 0.0,0.0, 0.0,pi/2, 0.0, pi, and 0.0, 3*pi/2.

Re: Some DNS lookups causing 5xx errors due to leap second bug

#49
What causes real-world problems with leap seconds is actually unrelated to the nasty interactions of metrology and solar time -- it's a specific and avoidable problem with how NTP (and many OSes/languages) represent time -- it's a types issue.

The right way for computers to represent time is with a number that represents the number of constant-rate ticks that have elapsed past a some agreed-upon epoch. If you know what the epoch is and how long each tick is (lots of people use 1 / 9.192 GHz), it is easy to know how many ticks are between any two time values, and you can convert a time value with one epoch to one with a different epoch and tick rate -- you can do everything people expect to do with time. There are no numbers that represent an invalid time value, and for each moment, there is a unique time value that represents it. There's a one-to-one mapping with no nasty edge cases.

Leap seconds are a step function that is added to a constant-rate timescale (whose name is "TAI") in order to generate a discontinuous timescale (whose name is "UTC") that never is too different from solar time. There is nothing fundamentally abhorrent about leap seconds -- there are just good and bad ways to represent, disseminate, and compute with timescales that involve leap seconds.

The right way to handle leap seconds can be seen with many GNSSes and PTP (very high precision hardware-assisted time synchronization over Ethernet). GPS, BeiDou, Galileo, and PTP all involve dissemination and computation on time values -- and with dire consequences for failure/downtime/inaccuracy.

The designers of those systems all somehow converged on the choice to separate out the nice, predictable, constant-rate and discontinuity-free part of UTC from the nasty step function (the leap second offset). Times in all those systems are represented as the tuple (TAI time at t, leap offset at t). This means that the entire system can calculate and work with (discontinuity-free and constant-rate) TAI times but also truck around the leap offsets so when time values need to be presented to a user (or anything that requires a UTC time), the leap offset can be added then. Crucially, all the maths that are done on time values are done on TAI values, so calculating a time difference or a frequency is easy and the result is always correct, regardless of the leap second state of affairs. Representing UTC time as a tuple makes the semantics of that data type easy to reason about -- the "time" bit is in the first element and is completely harmless -- the edge cases have all live in the second half of the tuple.

NTP and Unix (and everything descending and affected by those) have made the mistake of representing and transmitting time as a single integer, TAI(t) + leap offset(t). This is not a data representation that has sensical semantics and it is very hard to reason about it. First of all, the leap second offset is nondeterministic and also unknown -- there is no way to get it from NTP and there is no good way to know the time of the next leap event. Second of all, there are repeated time values for different moments in time (and when a negative leap second will happen, there will be time values that represent no moments in time). Predictably, introducing nondeterministic discontinuities doesn't work so well in the real world. There are a bunch of bugs in NTP software and OS kernels and applications that make themselves shown every time there is a leap second. It's not even just NTP clients that struggle -- 40% of public Stratum-1 NTP servers had erroneous behavior [0] related to the 2015 leap second! Given that level of repeated and widespread failure, the right solution is not to blame programmers -- it should be to blame the standard. The UTC standard and how NTP disseminates UTC are fundamentally not fit for computer timekeeping.

GNSS receivers and PTP hardware get used in mission-critical applications (synchronizing power grids and multi-axis industrial processes, timestamping data from test flights and particle accelerators) all the time -- and even worse, there's no way to conveniently schedule downtime/maintenance windows during leap second events! "Leap smear" isn't an acceptable solution for those applications, either -- you can't lie about how long a second is to the Large Hadron Collider. GNSS and PTP systems handle leap second timescales without a hitch by representing UTC time with the right data type -- a tuple that properly separates two values that have the same unit (seconds) but have vastly different semantics. The NTP and unix timestamp approach of directly baking the discontinuities into the time values reliably causes problems and outages. The leap second debacle is not about solar time vs atomic time; it's about the need for data types that accurately represent the semantics of what they describe.

[0]: http://crin.eng.uts.edu.au/~darryl/Publications/LeapSecond_c...

Post reply on HN