21–30 of 101 posts
There needs to be something like this from Ollama. At the moment Ollama has a lot of flaws that prevent it from getting great performance. (My understanding is better GPU/CPU splits, etc). But Ollama is the only way to host an LLM and have it switch out on demand. Sigh.
Earlier quoted context omitted.
Don't post generated/AI-edited comments. HN is for conversation between humans https://news.ycombinator.com/item?id=47340079
That doesn't read like an AI-generated comment to me. He did mention he vibe-coded the project but that's not against the guidelines.
That's a telltale sign of ai written text.
[flagged]
Have you ever generated access frequency statistics for the experts in these models, something like a histogram?
FWIW I never got it to work and did not dig into it much.
Earlier quoted context omitted.
Don't post generated/AI-edited comments. HN is for conversation between humans https://news.ycombinator.com/item?id=47340079
That doesn't read like an AI-generated comment to me. He did mention he vibe-coded the project but that's not against the guidelines.
It certainly has a lot of telltale signs
Intel Optane rolling in its grave.
Still have 4 brand new ones in my storage unit. Just in case these moments. Joke aside (I do have them tho!), I don't think Optane is that much use (not to mention it is only 256GiB for my unit). It is useful legacy crutch if you have legacy software that is not designed to issue multiple reads / writes in parallel. If you do, it is really not faster than NVMe, especially these modern ones.
What makes this approach faster is that the model's access pattern is completely deterministic during
inference. You know exactly which tensors are needed next because transformer layers execute sequentially. So
you can issue large sequential reads and prefetch the next layer while the current one is computing on Metal.
The OS page cache can't do that — it has no concept of "layer N+1 comes after layer N."
For MoE it's even more stark. The OS would page in all 8 experts on the first token that routes to each one,
then evict them under memory pressure with LRU, which has no idea that expert 3 fires 10x more often than
expert 7. The neuron cache here is basically a domain-specific replacement policy.OS paging would be significantly worse here. The kernel's page fault handler is reactive — it doesn't know you're about to read layer 47's FFN weights, so it can't prefetch. You stall on every fault, wait for the 4KB/16KB page to load, then resume. With 80 layers of dense FFN streaming, that's thousands of cold faults per token. What makes this approach faster is that the model's access pattern is completely determin…
man 2 madvise