Live data from Hacker News

Nvidia-smi hangs indefinitely after ~66 days

github.com

21–30 of 54 posts

Re: Nvidia-smi hangs indefinitely after ~66 days

#21
post #20

Earlier quoted context omitted.

66 days + 12 hours are 5,745,600,000,000,000 ns. The log2 of this is 52.351... Javascript and some other languages only have integer precision up to 52 bits then they switch to floating point. Curious.

It's 32 bits of milliseconds, right? Hm, no, that would overflow much sooner (49.7 days).

It's a uint32_t of 750 Hz "jiffies", which does overflow at ~66 days.

Re: Nvidia-smi hangs indefinitely after ~66 days

#24

a pet peeve of mine, (along with people brigading on issues/threads e.g. posting them to unrelated news sites... op....) is woefully incorrect language. > at day 66 all our jobs started randomly failing if there's a definable pattern, you can call it unpredictabily, but you can't call it randomly.

They've meant something like "arbitrary", in its "without any good/justifiable reason" sense. The word "random" is also used in this sense, especially when talking about human-made decisions.

Re: Nvidia-smi hangs indefinitely after ~66 days

#25
post #18

Timestamps should NOT be compared like this. Exactly this is why time_before() or time_after() exist. https://elixir.bootlin.com/linux/v6.15.7/source/include/linu...

Do I understand it correctly that the logic is that if timestamp B is above timestamp A, but the difference is more than half of the unsigned range, B is considered to happen before A?

Re: Nvidia-smi hangs indefinitely after ~66 days

#26
post #13
post #7

Earlier quoted context omitted.

Isn't 32bit counter 49 days? Assuming that one was counting milliseconds, at least. Only remember that because that's the limit for Windows 95…

100ns intervals. My favorite part of that story is how long after Windows 95 was released before anybody discovered the bug.

That's because people actually powered off their computer after work/leisure sessions. Someone on an unlimited night dial-up could had discovered it well before "anybody" but it's not like there was a built-in function to actually send a crash report to Redmond.

https://i.sstatic.net/p9hUgGfg.png

Re: Nvidia-smi hangs indefinitely after ~66 days

#28
post #16

This is only very tangentially related, but I got flashbacks to a time where we had dozens of edge/IoT raspberry pi devices with completely unupgradeable kernels with a bug that would make the whole USB stack shut down after "roughly a week" (7-9 days) of uptime. Once it got shut down, the only way to fix it was to do a full restart, and, at the time, we couldn't really be restarting those devices (not even at night)…

A week? I've had some Pis lose usb in 1-2 days. Fortunately we could afford to make them self restart every couple hours.

Re: Nvidia-smi hangs indefinitely after ~66 days

#29
post #16

This is only very tangentially related, but I got flashbacks to a time where we had dozens of edge/IoT raspberry pi devices with completely unupgradeable kernels with a bug that would make the whole USB stack shut down after "roughly a week" (7-9 days) of uptime. Once it got shut down, the only way to fix it was to do a full restart, and, at the time, we couldn't really be restarting those devices (not even at night)…

I've always been sceptical of the modern tendency of throwing powerful hardware at every embedded projects. In most cases good old atmel AVR or even 8051 would suffice.

Re: Nvidia-smi hangs indefinitely after ~66 days

#30
post #22

Wow, someone in the github comments[1] noticed that one of the bug numbers assigned internally for the issue matches to the day the number of days the driver would stay up. 1: https://github.com/NVIDIA/open-gpu-kernel-modules/issues/971...

You mean number of seconds, but yes, I think everyone looking at this would be converting units to see if there was a particular boundary being met.
Post reply on HN