Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Leap second causing Linux server crashes?

serverfault.com

Post reply on HN
Print view View original

91–100 of 122 posts

Re: Leap second causing Linux server crashes?

#91

Post by rplnt »

moe wrote: post #61
__david__ wrote: post #44

It appears to be fixed in Linux 3.4 [1]. According to the original commit [2] it's been broken since 7dffa3c673fbcf835cd7be80bb4aec8ad3f51168 [3], which appeared in 2.6.26. So, kernels between 2.6.26 and 3.3 (inclusive) are vulnerable. [1] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [2] https://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.... [3] https://git.kernel.org/?p=linux/kernel/git/t…

Which, in summary, is pretty much every production kernel out there. Spent the last two hours recovering servers, tomorrow will be another interesting day. Whoever figured it'd be a good idea to INSERT[1] the leap-second instead of just slowing/accelerating time... [1] Clock: inserting leap second 23:59:60 UTC

Well, it was a known bug and you had six months to prepare (i.e. update your kernel).

Re: Leap second causing Linux server crashes?

#92

Post by henrikschroder »

boyter wrote: post #41
rarrrrrr wrote: post #33

Earlier quoted context omitted.

Unfortunately I can confirm that Ubuntu 10.04 is vulnerable. We're proceeding with the fixtime.pl workaround.

I can confirm it too, but didn't catch it in time. A reboot however and everything is back to normal.

We didn't catch it in time either. It was oh so much fun to wake up to our service not working at all, all java and mysqld processes spinning like crazy, and having to reboot all servers. :-/

Re: Leap second causing Linux server crashes?

#93

Post by etrain »

ScottBurson wrote: post #64
infinite8s wrote: post #52

Earlier quoted context omitted.

When people say mid-six figures, do they mean 500k? Or 150k?

I think it means 300k -- 3 being a one-digit approximation to sqrt(10). I.e. the geometric mean of 100k and 1M.

Not sure if snark, or if really thst financially sophisticated...

Re: Leap second causing Linux server crashes?

#94

Post by glawatscheck »

glawatscheck wrote: post #88
glawatscheck wrote: post #73

POSTMORTEM fix for CPU eating softirqd threads without rebooting: stop ntpd, run ntpdate or sntp, start ntpd /etc/init.d/ntp stop; sntp -s ; /etc/init.d/ntp start Unfortunately sntp / ntpdate wrapper is not shipped with squeeze for example. I've used the binary from SuSE 11.4 just fine on squeeze.

OK this is how it works on squeeze etc.: apt-get install ntpdate; /etc/init.d/ntp stop; ntpdate pool.ntp.org; /etc/init.d/ntp start

or easier still just date -s "`date`"

without ntpd restart

Re: Leap second causing Linux server crashes?

#95

Post by kabdib »

el_presidente wrote: post #83
bcantrill wrote: post #79

Earlier quoted context omitted.

It's 2038, not 2037.[1] (Specifically, January 19th, 2038 at 3:14:08am.) And while lots of systems will be 64-bit, many programs still won't be -- and it seems highly likely that this will be a significantly more serious and widespread problem than, say, Y2K or DST. (And certainly more serious than leap seconds, which happen relatively frequently.) Then again, I might be biased: perhaps I'm secretly hoping to spend t…

http://article.gmane.org/gmane.linux.kernel/1184914 Less than a year ago there were already people thinking about your job security. (It's a better explanation than "the glibc maintainers are insane".)

But MUCH less than a year ago, many more people were still writing 32-bit-dirty time_t based code.

It's gonna be a fun one.

Re: Leap second causing Linux server crashes?

#97

Post by moe »

rplnt wrote: post #91
moe wrote: post #61

Earlier quoted context omitted.

Which, in summary, is pretty much every production kernel out there. Spent the last two hours recovering servers, tomorrow will be another interesting day. Whoever figured it'd be a good idea to INSERT[1] the leap-second instead of just slowing/accelerating time... [1] Clock: inserting leap second 23:59:60 UTC

Well, it was a known bug and you had six months to prepare (i.e. update your kernel).

Where was it published?

Almost all of my machines run the Debian stable kernel and were still affected.

Re: Leap second causing Linux server crashes?

#98

Post by ralph »

jbeda wrote: post #34
dfc wrote: post #30

Google uses a "leap smear" and slowly accounts for the leap second before it happens.[1] As long as you are not doing any astronomical calculations or constrained by regulatory requirements I think google has the right idea. [1] http://googleblog.blogspot.com/2011/09/time-technology-and-l...

As part of Google Compute Engine we provide an NTP server to the guest which is based on Google Production time. As such our VMs get to take advantage of this leap second smearing implementation. I was going to mention this at my talk at IO but forgot.

So a VM on G's Compute Engine could in turn run an NTP server that exported G's Production Time? Do I also see GPT on App Engine?

Any chance Google could just make a GPT NTP server available as a public service anyway, just as 8.8.8.8 is their public ping responder. ;-)

Re: Leap second causing Linux server crashes?

#99

Post by chmod775 »

If really all of the Linux where affected more than half of the Internet would be still down by now. Could be only a specific combination of kernel/userspace bugs that only exists in some systems.

What a bit sucks is that my VPN was affected to (openvpn) causing my computer to do a poweroff. I replaced the poweroff with

ip route add to 192.168.1.0/24 dev lo

hope that saves me when the next leap second occurs.

Re: Leap second causing Linux server crashes?

#100

Post by btilly »

kabdib wrote: post #75

Fear the Unix 32-bit time-becomes-negative bugs, in 2037. We have 25 years to get ready. I still think we'll be patching at the last minute. (Yeah, lots of systems will be 64-bit by then, but there will still be a lot of embedded crackerbox systems running 32-bit timestamps. It's all the embedded stuff I'm worried about).

If you think that being 64-bit protects you, then you do not understand the problem.

The problem is that 32-bit time is embedded in filesystem representations and related protocols. (eg the POSIX specification for file times in tar.) Therefore even if your machine is 64-bit, it still needs to use 32-bit time for many purposes.

To name a random example, the POSIX specification for times in the tar format is 32-bit. GNU tar has a non-standard extension that already takes care of it. But will everything else that expects to read/write tar files that a GNU tar program implement the same non-standard extension to the format in the same non-standard way? Almost certainly not. And there will be no sign of disaster until the second that we need to start relying on that more precise representation.

Post reply on HN