[dead]
Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
41–50 of 101 posts
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#42Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#43Intel Optane rolling in its grave.
Wouldn't be Intel if they didn't quit halfway through on a good thing.
Still, couldn't one get a RAID 0 card with four drives to saturate a 16x lane? That's already the max one could push through PCIe anyhow.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#44Simon Willison wrote a good post about Dan Woods’ work on “Autoresearching Apple's "LLM in a Flash" to run Qwen 397B locally”.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#45Earlier quoted context omitted.
> but in a 'smart' way so you don't overload the NVMe unnecessarily "overloading NVMe"? What is that about? First time I've heard anything about it. > because putting a ton of stress on your NVMe during generation Really shouldn't "stress your NVMe", something is severely wrong if that's happening. I've been hammering my SSDs forever, and while write operations "hurt" the longevity of the flash cells themselves, the…
I had assumed heat generation on the controller if it's continuously reading. But maybe it's not actually bad.
Just pop a heatsink on it and call it good.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#46[dead]
> for a 1T model youd need to stream something like 2TB of weights per forward pass
Isn't this missing the point of MoE models completely? MoE inference is sparse, you only read a small fraction of the weights per layer. You still have a problem of each individual expert-layer being quite small (a few MiBs each give or take) but those reads are large enough for the NVMe.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#47Are there any 1T parameter open source models?
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#48[dead]
> for a 1T model youd need to stream something like 2TB of weights per forward pass Isn't this missing the point of MoE models completely? MoE inference is sparse, you only read a small fraction of the weights per layer. You still have a problem of each individual expert-layer being quite small (a few MiBs each give or take) but those reads are large enough for the NVMe.
But across a sequence you still have to load most of them.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#49Are there any 1T parameter open source models?
Kimi 2.5?
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#50This is Come on, "Run" is not the right word. "Crawl" is. Headlines like that are misleading.
Yes, and with virtually zero context, which makes an enormous difference for TTFT on the MoE models.