Side note! Nginx was by no means the first web server to use a non-forking mechanism, nor the first open source web server to do so. Certainly Zeus (which was closed source) was earlier and very useful in this sort of application, and so was thttpd (open source, still exists as Merecat). I used thttpd quite a bit for single box applications and at one of my employers, nginx replaced a mixed strategy with Zeus, Apache and thttpd (and we tested one other whose name I can’t recall).
Non-forking httpd servers using select() were a popular little coding challenge for a while in the 90s. Spinner was one of them.
Nginx’s real strength was being able to proxy and cache HTTP using that same mechanism, so you didn’t additionally need to deploy Varnish or some other appliance.
As to whether this is a good mental model for what is coming for local LLMs, I am not sure I am convinced. Apart from more quantisation-aware training, perhaps binary and ternary aware training, custom inference engines per model, and maybe some improvements in diffusion models, the grand challenge in small footprint LLMs is training really small reasoning and tool use models, and so far it’s far from clear they can deliver.
Truly tiny models will not be viable as general coding assistants; even 12B dense is too small and you will find plenty of people who will tell you that 26B/4B or 35B/3B MoE is too. Though perhaps they can be trained for single languages, like just Python or just TS/JS.
More likely is the idea that 30-40B dense models might be good enough for most things once low cost and likely bespoke hardware catches up.
But I don’t think any truly profound advances seem likely in software or training alone. I am no expert but it feels like we’re already a lot closer to efficiency than we were in your analogy, and the gains are perhaps not going to be much more than small increments.
Maybe we will see something like a ternary 60B/10B MoE model turn up. But at the moment at least I am not sure where the incentives are to train these.