This is Come on, "Run" is not the right word. "Crawl" is. Headlines like that are misleading.
Could still be useful; maybe for overnight async workloads? Tell your agent research xyz at night and wake up to a report.
Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
61–70 of 101 posts
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#62Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#63[flagged]
This is interesting work, thank you for sharing. What hardware would you buy today for experimenting? Seems like the new gen of macbook pros are pretty powerful?
Yes definitely. I use a M1 Max with 32gb of RAM daily and it's about on par from a performance standpoint with the new base M5 Pro 24gb. You can check the benchmarks in the repo if you're interested in seeing specific performance metrics, but investing in Apple hardware with as much memory as possible will generally get you furthest in this game.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#64For a lot of local workloads, sub-1 tok/s is useless in foreground and perfectly acceptable in background. If the choice is “this crashes” vs “this finishes overnight,” that’s still a meaningful capability jump.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#65This is Come on, "Run" is not the right word. "Crawl" is. Headlines like that are misleading.
Could still be useful; maybe for overnight async workloads? Tell your agent research xyz at night and wake up to a report.
Assuming 1 token per second and "overnight" being 12 hours, that's 43 200 tokens. I'm not sure what you can meaningfully achieve with that.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#66There needs to be something like this from Ollama. At the moment Ollama has a lot of flaws that prevent it from getting great performance. (My understanding is better GPU/CPU splits, etc). But Ollama is the only way to host an LLM and have it switch out on demand. Sigh.
llama.cpp and llama-swap do this better than Ollama and with far more control.
Don't even need to use llama-swap anymore now that llama-server supports the same functionality.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#67Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#68Earlier quoted context omitted.
> but in a 'smart' way so you don't overload the NVMe unnecessarily "overloading NVMe"? What is that about? First time I've heard anything about it. > because putting a ton of stress on your NVMe during generation Really shouldn't "stress your NVMe", something is severely wrong if that's happening. I've been hammering my SSDs forever, and while write operations "hurt" the longevity of the flash cells themselves, the…
Hypura reads tensor weights from the GGUF file on NVMe into RAM/GPU memory pools, then compute happens entirely in RAM/GPU. There is no writing to SSDs on inference with this architecture.
Even if there was a ton of writing, I'm not sure where NVMe even comes in the picture, write durability is about the flash cells on SSDs, nothing to do with the interface, someone correct me if I'm wrong.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#69This is a pretty cool project! Essentially this is like using Swap memory to extend your RAM, but in a 'smart' way so you don't overload the NVMe unnecessarily. I do wonder in practice how the 'smarts' pan out, because putting a ton of stress on your NVMe during generation is probably not the best choice for it's longevity.
> but in a 'smart' way so you don't overload the NVMe unnecessarily "overloading NVMe"? What is that about? First time I've heard anything about it. > because putting a ton of stress on your NVMe during generation Really shouldn't "stress your NVMe", something is severely wrong if that's happening. I've been hammering my SSDs forever, and while write operations "hurt" the longevity of the flash cells themselves, the…
[flagged]
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#70The MoE point matters here ie sparse activation means you're not reading all 2TB per forward pass, but the access pattern flips from sequential to random which is exactly the worst case for NVMe. Been thinking about this a lot for agent inference workloads where you want consistent latency more than peak throughput.