I was logged on to a couple of CentOS 6 servers when I saw this happen, and on each one the Java processes went absolutely haywire. Everything else seemed to work fine. I attempted to fix with adjtimex and the script in the linked question, but to no avail, in the end having to restart them all instead. After that, all was good again.
Leap second causing Linux server crashes?
81–90 of 122 posts
Re: Leap second causing Linux server crashes?
#82It appears to be fixed in Linux 3.4 [1]. According to the original commit [2] it's been broken since 7dffa3c673fbcf835cd7be80bb4aec8ad3f51168 [3], which appeared in 2.6.26. So, kernels between 2.6.26 and 3.3 (inclusive) are vulnerable. [1] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [2] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [3] https://git.kernel.org/?p=linux/kernel/git/t…
Which, in summary, is pretty much every production kernel out there. Spent the last two hours recovering servers, tomorrow will be another interesting day. Whoever figured it'd be a good idea to INSERT[1] the leap-second instead of just slowing/accelerating time... [1] Clock: inserting leap second 23:59:60 UTC
Re: Leap second causing Linux server crashes?
#83Fear the Unix 32-bit time-becomes-negative bugs, in 2037. We have 25 years to get ready. I still think we'll be patching at the last minute. (Yeah, lots of systems will be 64-bit by then, but there will still be a lot of embedded crackerbox systems running 32-bit timestamps. It's all the embedded stuff I'm worried about).
It's 2038, not 2037.[1] (Specifically, January 19th, 2038 at 3:14:08am.) And while lots of systems will be 64-bit, many programs still won't be -- and it seems highly likely that this will be a significantly more serious and widespread problem than, say, Y2K or DST. (And certainly more serious than leap seconds, which happen relatively frequently.) Then again, I might be biased: perhaps I'm secretly hoping to spend t…
Less than a year ago there were already people thinking about your job security. (It's a better explanation than "the glibc maintainers are insane".)
Re: Leap second causing Linux server crashes?
#84Earlier quoted context omitted.
> Just set the clock to today (or any day with a leap second), and watch what happens. That won't work. The bug is only triggered when an upstream NTP server reports that a leap second was scheduled. Since leap seconds aren't predictable (and aren't even scheduled very far in advance), just setting the time back to the date of a previous leap second won't do anything.
True, but the question still stands, since you can still test it by just telling the kernel to insert a (fake) leap second.
Re: Leap second causing Linux server crashes?
#85Re: Leap second causing Linux server crashes?
#86My Ubuntu servers seem unaffected thus far.
[EDIT] Ah, covered elsewhere. Fixed by manually setting the date on the box; stopping/restarting mysqld or ntpd doesn't make any difference.
Re: Leap second causing Linux server crashes?
#87Re: Leap second causing Linux server crashes?
#88POSTMORTEM fix for CPU eating softirqd threads without rebooting: stop ntpd, run ntpdate or sntp, start ntpd /etc/init.d/ntp stop; sntp -s ; /etc/init.d/ntp start Unfortunately sntp / ntpdate wrapper is not shipped with squeeze for example. I've used the binary from SuSE 11.4 just fine on squeeze.
apt-get install ntpdate; /etc/init.d/ntp stop; ntpdate pool.ntp.org; /etc/init.d/ntp start
Re: Leap second causing Linux server crashes?
#89It appears to be fixed in Linux 3.4 [1]. According to the original commit [2] it's been broken since 7dffa3c673fbcf835cd7be80bb4aec8ad3f51168 [3], which appeared in 2.6.26. So, kernels between 2.6.26 and 3.3 (inclusive) are vulnerable. [1] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [2] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [3] https://git.kernel.org/?p=linux/kernel/git/t…
Which, in summary, is pretty much every production kernel out there. Spent the last two hours recovering servers, tomorrow will be another interesting day. Whoever figured it'd be a good idea to INSERT[1] the leap-second instead of just slowing/accelerating time... [1] Clock: inserting leap second 23:59:60 UTC
That would be the IERS organisation. There's going to be a vote in 2015 to abolish them entirely.
Re: Leap second causing Linux server crashes?
#90It appears to be fixed in Linux 3.4 [1]. According to the original commit [2] it's been broken since 7dffa3c673fbcf835cd7be80bb4aec8ad3f51168 [3], which appeared in 2.6.26. So, kernels between 2.6.26 and 3.3 (inclusive) are vulnerable. [1] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [2] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [3] https://git.kernel.org/?p=linux/kernel/git/t…