Are there any 1T parameter open source models?
Kimi 2.5?
Thanks, TIL.
Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
51–60 of 101 posts
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#52[dead]
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#53Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#54[dead]
Yes, definitely agree. It's more of a POC than a functional use case. However, for many smaller MoE models this method can actually be useful and capable of achieving multiple tokens/sec.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#55[dead]
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#56[dead]
4K random read with a queue depth of 1 on an M1 Max is about 65MB/s.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#57Where does "1T parameter model" come from? I can only see models with 70B params or less mentioned in the repo.
I'm referencing it as being possible, however I didn't share benchmarks because candidly the performance would be so slow it would only be useful for very specific tasks over long time horizons. The more practical use cases are less flashy but capable of achieving multiple tokens/sec (ie smaller MoE models where not all experts need to be loaded in memory simultaneously)
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#58This is a pretty cool project! Essentially this is like using Swap memory to extend your RAM, but in a 'smart' way so you don't overload the NVMe unnecessarily. I do wonder in practice how the 'smarts' pan out, because putting a ton of stress on your NVMe during generation is probably not the best choice for it's longevity.
This is not putting any stress or wear on the NVMe, it's a pure read workload.
Yes, exactly this.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#59Intel Optane rolling in its grave.
Memristors are also missing in this AI hype even when they were around the corner 10 years back.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#60This is a pretty cool project! Essentially this is like using Swap memory to extend your RAM, but in a 'smart' way so you don't overload the NVMe unnecessarily. I do wonder in practice how the 'smarts' pan out, because putting a ton of stress on your NVMe during generation is probably not the best choice for it's longevity.
> but in a 'smart' way so you don't overload the NVMe unnecessarily "overloading NVMe"? What is that about? First time I've heard anything about it. > because putting a ton of stress on your NVMe during generation Really shouldn't "stress your NVMe", something is severely wrong if that's happening. I've been hammering my SSDs forever, and while write operations "hurt" the longevity of the flash cells themselves, the…
Hypura reads tensor weights from the GGUF file on NVMe into RAM/GPU memory pools, then compute happens entirely in RAM/GPU.
There is no writing to SSDs on inference with this architecture.