Nvidia-smi hangs indefinitely after ~66 days • Hacker News Board

Re: Nvidia-smi hangs indefinitely after ~66 days

#31

Post by AceJohnny2 » Sun, Jan 25, 2026, 8:22 AM UTC

pajko wrote: post #18

Timestamps should NOT be compared like this. Exactly this is why time_before() or time_after() exist. https://elixir.bootlin.com/linux/v6.15.7/source/include/linu...

Offtopic...

    * Do this with "=0" to only test the sign of the result. A
    * good compiler would generate better code (and a really good compiler
    * wouldn't care). Gcc is currently neither.

It's funny the love-hate relationship the Linux kernel has with GCC. It's the only supported compiler[1], and yet...

[1] can Clang fully compile Linux yet? I haven't followed the updates in a while.

Re: Nvidia-smi hangs indefinitely after ~66 days

#32

Post by userbinator » Sun, Jan 25, 2026, 8:23 AM UTC

oasisaimlessly wrote: post #21

loeg wrote: post #20

Earlier quoted context omitted.

It's 32 bits of milliseconds, right? Hm, no, that would overflow much sooner (49.7 days).

It's a uint32_t of 750 Hz "jiffies", which does overflow at ~66 days.

While that seems like a convincing explanation, 750Hz is a rather odd value to use for a timer, and more importantly the overflow would be at 66d6h43m43s instead of the reported ~66d12h.

Re: Nvidia-smi hangs indefinitely after ~66 days

#33

Post by EvgeniyZh » Sun, Jan 25, 2026, 8:35 AM UTC

AceJohnny2 wrote: post #31

pajko wrote: post #18

Timestamps should NOT be compared like this. Exactly this is why time_before() or time_after() exist. https://elixir.bootlin.com/linux/v6.15.7/source/include/linu...

Offtopic... * Do this with " =0" to only test the sign of the result. A * good compiler would generate better code (and a really good compiler * wouldn't care). Gcc is currently neither. It's funny the love-hate relationship the Linux kernel has with GCC. It's the only supported compiler[1], and yet... [1] can Clang fully compile Linux yet? I haven't followed the updates in a while.

Yes it can [1].

https://docs.kernel.org/kbuild/llvm.html

Re: Nvidia-smi hangs indefinitely after ~66 days

#34

Post by wvenable » Sun, Jan 25, 2026, 8:38 AM UTC

A few years ago, at my company, we would get random TPM crashes every few months on all our machines. You'd be working and the TPM would just disappear and then any apps that rely on it for key retrieval would error out. Even worse, since the TPM chip is always running, neither a reboot nor a shutdown would fix it -- you literally had to pull the plug.

This went on for months. Then one day we had a power outage. Two months later, every single machine failed at the same time. I checked the logs and it was 49 days and few hours since that outage. It didn't take me too long to figure out what the underlying programming error inside the TPM was. At least we could then describe exactly what the problem was to our PC vendor.

Re: Nvidia-smi hangs indefinitely after ~66 days

#35

Post by rcxdude » Sun, Jan 25, 2026, 9:40 AM UTC

Joker_vD wrote: post #25

pajko wrote: post #18

Timestamps should NOT be compared like this. Exactly this is why time_before() or time_after() exist. https://elixir.bootlin.com/linux/v6.15.7/source/include/linu...

Do I understand it correctly that the logic is that if timestamp B is above timestamp A, but the difference is more than half of the unsigned range, B is considered to happen before A?

Yes. When the timestamps wrap it's fundamentally ambiguous, but this will be correct unless the timestamps are very far apart (and the failure mode is more benign: a really long time difference being considered shorter is better than all time differences being considered zero after the timestamp wraps).

Re: Nvidia-smi hangs indefinitely after ~66 days

#36

Post by spuz » Sun, Jan 25, 2026, 9:57 AM UTC

wvenable wrote: post #34

A few years ago, at my company, we would get random TPM crashes every few months on all our machines. You'd be working and the TPM would just disappear and then any apps that rely on it for key retrieval would error out. Even worse, since the TPM chip is always running, neither a reboot nor a shutdown would fix it -- you literally had to pull the plug. This went on for months. Then one day we had a power outage. Two…

So what was the programming error in the TPM?

Re: Nvidia-smi hangs indefinitely after ~66 days

#37

Post by jcurtis » Sun, Jan 25, 2026, 10:05 AM UTC

spuz wrote: post #36

wvenable wrote: post #34

A few years ago, at my company, we would get random TPM crashes every few months on all our machines. You'd be working and the TPM would just disappear and then any apps that rely on it for key retrieval would error out. Even worse, since the TPM chip is always running, neither a reboot nor a shutdown would fix it -- you literally had to pull the plug. This went on for months. Then one day we had a power outage. Two…

So what was the programming error in the TPM?

49 days is a bit under 2^32 milliseconds... So unsigned int overflow?

Re: Nvidia-smi hangs indefinitely after ~66 days

#38

Post by swinglock » Sun, Jan 25, 2026, 10:09 AM UTC

spuz wrote: post #36

wvenable wrote: post #34

A few years ago, at my company, we would get random TPM crashes every few months on all our machines. You'd be working and the TPM would just disappear and then any apps that rely on it for key retrieval would error out. Even worse, since the TPM chip is always running, neither a reboot nor a shutdown would fix it -- you literally had to pull the plug. This went on for months. Then one day we had a power outage. Two…

So what was the programming error in the TPM?

Something breaking after 49.7 days is a classic. Someone counted milliseconds since start with a 32 bit unsigned int and some code assumed it couldn't wrap.

Re: Nvidia-smi hangs indefinitely after ~66 days

#39

Post by rwmj » Sun, Jan 25, 2026, 10:15 AM UTC

AceJohnny2 wrote: post #31

pajko wrote: post #18

Timestamps should NOT be compared like this. Exactly this is why time_before() or time_after() exist. https://elixir.bootlin.com/linux/v6.15.7/source/include/linu...

Offtopic... * Do this with " =0" to only test the sign of the result. A * good compiler would generate better code (and a really good compiler * wouldn't care). Gcc is currently neither. It's funny the love-hate relationship the Linux kernel has with GCC. It's the only supported compiler[1], and yet... [1] can Clang fully compile Linux yet? I haven't followed the updates in a while.

To be fair this comment predates git history (before 2005) when GCC wasn't a very good compiler. The kernel developers at one point were sticking with a specific version of GCC because later versions would miscompile the kernel. Clang didn't exist then.

GCC is a different beast and far better nowadays.

Re: Nvidia-smi hangs indefinitely after ~66 days

#40

Post by [deleted] » Sun, Jan 25, 2026, 10:27 AM UTC

spuz wrote: post #36

wvenable wrote: post #34

A few years ago, at my company, we would get random TPM crashes every few months on all our machines. You'd be working and the TPM would just disappear and then any apps that rely on it for key retrieval would error out. Even worse, since the TPM chip is always running, neither a reboot nor a shutdown would fix it -- you literally had to pull the plug. This went on for months. Then one day we had a power outage. Two…

So what was the programming error in the TPM?

[deleted]