Live data from Hacker News

Leap second causing Linux server crashes?

serverfault.com

41–50 of 122 posts

Re: Leap second causing Linux server crashes?

#41

My Ubuntu servers seem unaffected thus far.

Unfortunately I can confirm that Ubuntu 10.04 is vulnerable. We're proceeding with the fixtime.pl workaround.

I can confirm it too, but didn't catch it in time. A reboot however and everything is back to normal.

Re: Leap second causing Linux server crashes?

#42

Not surprising. In spite of all press that Y2K was just a silly waste of money, its events like these that makes me suspect it would have been a much bigger deal if everyone had ignored it and fixed it after things where shown to break.

Why does everyone always say Y2K wasn't an issue ? I'm sure there were a lot of consultant making too much money with little work - however _alot_ of bug fixes were done, that would have caused problems. So because it was taken seriously , stuff were fixed and issues didn't happen because of that. Personally, I fixed 3 Y2K bugs back then, 2 of them would have brought down a rather critical business support to simply…

"In spite of all press that Y2K was just a silly waste of money, its events like these that makes me suspect it would have been a much bigger deal if everyone had ignored it and fixed it after things where shown to break."

Did you read the parent's post?

Re: Leap second causing Linux server crashes?

#43
post #34

Earlier quoted context omitted.

As part of Google Compute Engine we provide an NTP server to the guest which is based on Google Production time. As such our VMs get to take advantage of this leap second smearing implementation. I was going to mention this at my talk at IO but forgot.

Marco's blog post (linked) had a similar idea - running ntp with -x for a day so it smears time.

In case anyone is looking for the actual link to marco's posts on ntp:

http://my.opera.com/marcomarongiu/blog/index.dml/tag/ntp

Re: Leap second causing Linux server crashes?

#44
It appears to be fixed in Linux 3.4 [1]. According to the original commit [2] it's been broken since 7dffa3c673fbcf835cd7be80bb4aec8ad3f51168 [3], which appeared in 2.6.26.

So, kernels between 2.6.26 and 3.3 (inclusive) are vulnerable.

[1] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2....

[2] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2....

[3] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2....

Re: Leap second causing Linux server crashes?

#45
post #29

Earlier quoted context omitted.

It's harder than leap days, because leap seconds aren't inserted on a regular schedule. Leap days follow a predictable pattern of insertion. Leap seconds are inserted whenever the IERS decides to insert them. The problem of leap seconds is therefore closer to that of time zone definitions -- which are a total mess, because they depend on keeping rapidly changing system tables up to date. I can see why people don't re…

How are systems being notified of the leap seconds now, that wouldn't immediately enable them to update their hypothetical leap second table? It seems like we already have a much bigger lead time for notification than we could possibly need. > I can see why people don't relish the idea of requiring similar tables just to keep system time accurate. But the 'solution' we're using now is to make system time less accurat…

"How are systems being notified of the leap seconds now, that wouldn't immediately enable them to update their hypothetical leap second table?"

I am not an expert, but as far as I know the most automated solutions are doing it via NTP, which just resets the second, then relies on clock drift to bring everything back into synch. Otherwise, I think your only option is to keep the timezone packages up-to-date (which is a non-trivial task for large deployments). A quick search found this:

http://www.novell.com/support/kb/doc.php?id=7001865

"But the 'solution' we're using now is to make system time less accurate, not more accurate."

Yeah, I'm not disputing this. I'm just saying that preserving the assumption that "day == 86400 seconds" probably breaks less code than the alternative. NTP messes with the notion of seconds-since-epoch anyway, so we know that single-second variations in unix time aren't automatically deadly to most unix software.

Re: Leap second causing Linux server crashes?

#47

2012. and we still have problems keeping track of time. This is both fascinating and scary. P.S. for people wanting to know more this video is simple to understand but really amazing http://www.youtube.com/watch?v=xX96xng7sAE

2012 and we still don't really know what time is. The fact that we can keep track of it, even with all the problems, is quite amazing.

Re: Leap second causing Linux server crashes?

#48
post #40

Hey, I'm running Ubuntu 12.04 . Could someone guide me through what I can do to detect/prevent this from crippling my server? Thanks.

Read the linked article.

> The work-around is to just turn off ntpd. If ntpd already issued the adjtimex(2) call, you may need to disable ntpd and reboot to be 100% safe.

Post reply on HN