Earlier quoted context omitted.
You're right, and it's interesting to consider why. It's probably a combination of a few factors: 1) Local LLMs are a relatively new phenomenon and hardware takes years. Apple probably lucked into their unified memory architecture being suitable (in terms of memory size and bandwidth) for local LLMs, but it's only with the newest generations we're hearing about LLMs even being a consideration in their design process.…
There will be a huge market for local inference once it's cheap and widely available. Try to imagine output token speeds of 15,000 tok/s and a time-to-first-token of 200ms. (This has already been done for Llama 8B.) Now imagine gargantuan context windows (2M, 4M, or even bigger); keep in mind the 1M context windows were science fiction a few years ago... now imagine having this on a local model on something like a ph…
I've seen public pronouncements that the RAM shortage could persist for a decade.
And then if consider that the constraint on local LLMs isn't just memory size but bandwidth ...
If you take something like a DGX Spark and increase its memory to 512GB that doesn't even solve the problem. Because the bandwidth of DDR5 just can't manage reasonable speeds for decode. If you take a dense model or an MoE model uses up most of that 128GB in active decode you will only get like 15 tok/sec. "Real" datacentre inference boxes use high bandwidth memory that is 10x the speed.
I think we're unfortunately a long way off, unless people learn to accept working with much less intelligent models locally.
The innovation is going to have to come on the research & software side -- we need to find ways to pack more intelligence into a smaller number of parameters.