Nvidia-smi hangs indefinitely after ~66 days • Hacker News Board

Re: Nvidia-smi hangs indefinitely after ~66 days

#51

Post by locao » Sun, Jan 25, 2026, 6:57 PM UTC

nottorp wrote: post #28

jorl17 wrote: post #16

This is only very tangentially related, but I got flashbacks to a time where we had dozens of edge/IoT raspberry pi devices with completely unupgradeable kernels with a bug that would make the whole USB stack shut down after "roughly a week" (7-9 days) of uptime. Once it got shut down, the only way to fix it was to do a full restart, and, at the time, we couldn't really be restarting those devices (not even at night)…

A week? I've had some Pis lose usb in 1-2 days. Fortunately we could afford to make them self restart every couple hours.

I also had the same experience, but I could only make them restart during the night. So I wrote a monitor to check if any of the Pis lost USB before restarting.

When our business grew, even restarting every night, we would get one or two lost USB warnings every day. One day I didn't receive any warnings. I was really happy, I had fix the issue! Three days later a client calls screaming the service is not working for two whole days and we did nothing. After getting every Pi restarted, I went to check the monitor. Shut down. I asked my business partner about it. "The alarms made me anxious, so I decided to shut down the monitor".

Obviously I sold my shares and never looked back.

Re: Nvidia-smi hangs indefinitely after ~66 days

#52

Post by nottorp » Sun, Jan 25, 2026, 7:49 PM UTC

locao wrote: post #51

nottorp wrote: post #28

Earlier quoted context omitted.

A week? I've had some Pis lose usb in 1-2 days. Fortunately we could afford to make them self restart every couple hours.

I also had the same experience, but I could only make them restart during the night. So I wrote a monitor to check if any of the Pis lost USB before restarting. When our business grew, even restarting every night, we would get one or two lost USB warnings every day. One day I didn't receive any warnings. I was really happy, I had fix the issue! Three days later a client calls screaming the service is not working for…

Ah well. Our project was Pis in a crappy mesh network so it lost data occasionally even if they stayed on, and it was not so important to have continous data anyway. We rebooted them every like 3 or 6 hours.

Re: Nvidia-smi hangs indefinitely after ~66 days

#53

Post by userbinator » Sun, Jan 25, 2026, 10:32 PM UTC

reneberlin wrote: post #41

In memory of the "49.7-day bug" Back in the day it was Windows, that had a hard limit on how long it could run in one pass. I forgot when it began and ended, but happily AI helped out to investigate back in time. The bug primarily affected the Windows 9x family of operating systems: Windows 95 (all versions) Windows 98 (original release) Windows 98 Second Edition (SE) While there were separate reports of similar 497-…

This counter was used by the Virtual Device Driver (VMM) to manage system timers.

VMM stands for Virtual Machine Manager. AI slop strikes again.

Re: Nvidia-smi hangs indefinitely after ~66 days

#54

Post by [deleted] » Sun, Jan 25, 2026, 11:28 PM UTC

spuz wrote: post #36

wvenable wrote: post #34

A few years ago, at my company, we would get random TPM crashes every few months on all our machines. You'd be working and the TPM would just disappear and then any apps that rely on it for key retrieval would error out. Even worse, since the TPM chip is always running, neither a reboot nor a shutdown would fix it -- you literally had to pull the plug. This went on for months. Then one day we had a power outage. Two…

So what was the programming error in the TPM?

[deleted]