31–40 of 101 posts
Earlier quoted context omitted.
Don't post generated/AI-edited comments. HN is for conversation between humans https://news.ycombinator.com/item?id=47340079
That doesn't read like an AI-generated comment to me. He did mention he vibe-coded the project but that's not against the guidelines.
Earlier quoted context omitted.
Don't post generated/AI-edited comments. HN is for conversation between humans https://news.ycombinator.com/item?id=47340079
That doesn't read like an AI-generated comment to me. He did mention he vibe-coded the project but that's not against the guidelines.
At least prompt your LLM to dodge the obvious tells when commenting!
You do not explain how any kind of predictor can work for MoE experts.
You do not explain how prediction can even be useful. I can predict the layers used in a dense model (all of them are used in order), but that doesn't help me much. It's still bottlenecked on bandwidth (hint: MoE doesn't change this).
Intel Optane rolling in its grave.
OS paging would be significantly worse here. The kernel's page fault handler is reactive — it doesn't know you're about to read layer 47's FFN weights, so it can't prefetch. You stall on every fault, wait for the 4KB/16KB page to load, then resume. With 80 layers of dense FFN streaming, that's thousands of cold faults per token. What makes this approach faster is that the model's access pattern is completely determin…
OS paging would be significantly worse here. The kernel's page fault handler is reactive — it doesn't know you're about to read layer 47's FFN weights, so it can't prefetch. You stall on every fault, wait for the 4KB/16KB page to load, then resume. With 80 layers of dense FFN streaming, that's thousands of cold faults per token. What makes this approach faster is that the model's access pattern is completely determin…
It will be interesting to compare this to https://news.ycombinator.com/item?id=47476422 and https://news.ycombinator.com/item?id=47490070 . Very similar design except that this is apparently using mmap, which according to the earlier experiment incurs significant overhead.
Intel Optane rolling in its grave.
Headlines like that are misleading.