Nvidia-smi hangs indefinitely after ~66 days

Back in the day it was Windows, that had a hard limit on how long it could run in one pass. I forgot when it began and ended, but happily AI helped out to investigate back in time.

The bug primarily affected the Windows 9x family of operating systems:

Windows 95 (all versions)

Windows 98 (original release)

Windows 98 Second Edition (SE)

While there were separate reports of similar 497-day overflows in Windows NT 4.0 and Windows 2000, the "classic" version of this bug that most people remember is the 49.7-day limit on Windows 95 and 98.

Why 49.7 days? The issue was a classic integer overflow. Windows used a 32-bit counter to track the number of milliseconds since the system started. This counter was used by the Virtual Device Driver (VMM) to manage system timers.

The maximum value for a 32-bit unsigned integer is: 2^32 - 1, which equals: 4,294,967,295 millisec.

If you convert those milliseconds into days: 4,294,967,295 / 1,000 = 4,294,967 seconds 4,294,967 / 60 / 60 / 24 ~ 49.71 days

When the counter hit that maximum value, it would "wrap around" to zero. Because many system services and drivers were waiting for the counter to increase to a certain target time, they would suddenly find themselves waiting for a number that had already passed or was now mathematically impossible to reach in their logic. This caused the "hang"—the mouse might still move, but the OS could no longer process tasks.

When did it start and end? Started: With the release of Windows 95 in August 1995.

Ended: Microsoft officially fixed the bug with a patch in 1999 (Knowledge Base article KB216641). Windows Me (released in 2000) was the first in that specific family to ship with the fix included, and the transition to the Windows NT architecture (Windows XP and later) eventually rendered the specific underlying cause obsolete for home users.

The first version was built pretty much that way, with a tiny microcontroller and extremely optimized code. The problem then became that it was very hard to iterate quickly on it and prototype new features. Every new piece of hardware that was added (or just evaluated) would have to be carefully integrated and it really added to the mess. Maybe it would have been different if the code had been structured with more care from the get-go, who knows (I entered the project already in version 2).

For version 2, the micro-controller was thrown out, and raspberry-pi based solutions were brought in. Sure, it felt like carrying a shotgun to fire at a couple of flies, but having a linux machine with such a vast ecosystem was amazing. On top of that, it was much easier to hire people to work on the project because now they could get by with higher level languages like python and javascript. And it was much, much, much faster to develop on.

The usage of the raspberry pi was, in my view, one of the key details that allowed for what ultimately became an extremely successful product. It was much less energy-efficient, but it was very simple to develop and iterate on. In the span of months we experimented with many hardware addons, as product-market-fit was still being found out, and the plethora of online resources for everything else was a boon.

I'm pretty sure this was _the_ project that really made me realize that more often than not the right solution is the one that lets the right people make the right decisions. And for that particular team, this was, without a doubt, a remarkably successful decision. Most of the problems that typically come with it (such as bloat, and inefficiency) were eventually solved, something which would not have been possible by going slowly at first.