Live data from Hacker News

Leap second causing Linux server crashes?

serverfault.com

81–90 of 122 posts

Re: Leap second causing Linux server crashes?

#81

I was logged on to a couple of CentOS 6 servers when I saw this happen, and on each one the Java processes went absolutely haywire. Everything else seemed to work fine. I attempted to fix with adjtimex and the script in the linked question, but to no avail, in the end having to restart them all instead. After that, all was good again.

Had the same issue across all our VMs running Java/Tomcat applications.

Re: Leap second causing Linux server crashes?

#82
post #61

It appears to be fixed in Linux 3.4 [1]. According to the original commit [2] it's been broken since 7dffa3c673fbcf835cd7be80bb4aec8ad3f51168 [3], which appeared in 2.6.26. So, kernels between 2.6.26 and 3.3 (inclusive) are vulnerable. [1] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [2] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [3] https://git.kernel.org/?p=linux/kernel/git/t…

Which, in summary, is pretty much every production kernel out there. Spent the last two hours recovering servers, tomorrow will be another interesting day. Whoever figured it'd be a good idea to INSERT[1] the leap-second instead of just slowing/accelerating time... [1] Clock: inserting leap second 23:59:60 UTC

I'm still tryin to understand why all my servers seem to be ok even if they have kernel that should be affected and some of them are running mysql... For example one of them is a debian kernel 2.6.32 running mysql and ntpd, and i see in dmesg Clock: inserting leap second 23:59:60 UTC but the cpu load is ok...

Re: Leap second causing Linux server crashes?

#83
post #75

Fear the Unix 32-bit time-becomes-negative bugs, in 2037. We have 25 years to get ready. I still think we'll be patching at the last minute. (Yeah, lots of systems will be 64-bit by then, but there will still be a lot of embedded crackerbox systems running 32-bit timestamps. It's all the embedded stuff I'm worried about).

It's 2038, not 2037.[1] (Specifically, January 19th, 2038 at 3:14:08am.) And while lots of systems will be 64-bit, many programs still won't be -- and it seems highly likely that this will be a significantly more serious and widespread problem than, say, Y2K or DST. (And certainly more serious than leap seconds, which happen relatively frequently.) Then again, I might be biased: perhaps I'm secretly hoping to spend t…

http://article.gmane.org/gmane.linux.kernel/1184914

Less than a year ago there were already people thinking about your job security. (It's a better explanation than "the glibc maintainers are insane".)

Re: Leap second causing Linux server crashes?

#84
post #80

Earlier quoted context omitted.

> Just set the clock to today (or any day with a leap second), and watch what happens. That won't work. The bug is only triggered when an upstream NTP server reports that a leap second was scheduled. Since leap seconds aren't predictable (and aren't even scheduled very far in advance), just setting the time back to the date of a previous leap second won't do anything.

True, but the question still stands, since you can still test it by just telling the kernel to insert a (fake) leap second.

It also should not be that hard to provide your own upstream ntp server, and have that generate leap seconds at will. Both machines could be VMs, too.

Re: Leap second causing Linux server crashes?

#86

My Ubuntu servers seem unaffected thus far.

I found mysqld (5.5.24) running at 159% CPU on a Ubuntu 11.04 (64-bit) box this morning. ntpd had drifted in the leap second between 1am and 2am (GMT) this morning (NTP drift info is one thing I graph with MRTG).

[EDIT] Ah, covered elsewhere. Fixed by manually setting the date on the box; stopping/restarting mysqld or ntpd doesn't make any difference.

Re: Leap second causing Linux server crashes?

#87
post #55

Ouch! My Debian GNU/Linux 6.0 is still standing Oh well, reading the issue, the machine date is Sat Jun 30 16:11:31 EDT 2012 Stopped ntpd just in case

Same here. Set ntp to restart in 12 hours.

With ntp stopped, no problem whatsoever

Re: Leap second causing Linux server crashes?

#88

POSTMORTEM fix for CPU eating softirqd threads without rebooting: stop ntpd, run ntpdate or sntp, start ntpd /etc/init.d/ntp stop; sntp -s ; /etc/init.d/ntp start Unfortunately sntp / ntpdate wrapper is not shipped with squeeze for example. I've used the binary from SuSE 11.4 just fine on squeeze.

OK this is how it works on squeeze etc.:

apt-get install ntpdate; /etc/init.d/ntp stop; ntpdate pool.ntp.org; /etc/init.d/ntp start

Re: Leap second causing Linux server crashes?

#89
post #61

It appears to be fixed in Linux 3.4 [1]. According to the original commit [2] it's been broken since 7dffa3c673fbcf835cd7be80bb4aec8ad3f51168 [3], which appeared in 2.6.26. So, kernels between 2.6.26 and 3.3 (inclusive) are vulnerable. [1] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [2] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [3] https://git.kernel.org/?p=linux/kernel/git/t…

Which, in summary, is pretty much every production kernel out there. Spent the last two hours recovering servers, tomorrow will be another interesting day. Whoever figured it'd be a good idea to INSERT[1] the leap-second instead of just slowing/accelerating time... [1] Clock: inserting leap second 23:59:60 UTC

> Whoever figured it'd be a good idea to INSERT[1] the leap-second instead of just slowing/accelerating time...

That would be the IERS organisation. There's going to be a vote in 2015 to abolish them entirely.

Re: Leap second causing Linux server crashes?

#90

It appears to be fixed in Linux 3.4 [1]. According to the original commit [2] it's been broken since 7dffa3c673fbcf835cd7be80bb4aec8ad3f51168 [3], which appeared in 2.6.26. So, kernels between 2.6.26 and 3.3 (inclusive) are vulnerable. [1] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [2] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [3] https://git.kernel.org/?p=linux/kernel/git/t…

I have 2.6.27 kernels here (SuSE 11.1) which seem unaffected so breakage might be a little later
Post reply on HN