Nice, I think this is the second time I see this here on HN, I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time. It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory. Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and p…
You're kinda describing the MoE architecture; you can offload expert layers and stream them as-needed if the experts are small enough and the SSD is fast enough. Dense LLMs typically perform better, but slow down much more than MoE models when you try offloading layers.
What he's envisioning is a dense 1B model that looks at the Python specification and your prompt and goes:
> Ah, I get it now! This is like Darmok and Jalad at Tanagra!
Or at least:
> Android UI development in Python? It's UNIX, I know this!
We do work like this sometimes but in general we rely on internalized knowledge so I don't know to what extent it is a viable strategy.