[dead]
Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
81–90 of 101 posts
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#82[dead]
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#83"As much memory as possible" is right for model capacity but misses bandwidth. Apple Silicon has distinct tiers: M4 Pro at 273 GB/s, M4 Max at 546 GB/s, M4 Ultra at 819 GB/s. Bandwidth determines tok/s once the model fits in memory. An M4 Max gives you 2x the decode speed of an M4 Pro on the same model.
For what Hypura does, the Max is the sweet spot. 64GB loads a 70B at Q4 with room to spare, and double the bandwidth of the Pro means generation is actually usable instead of just technically possible.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#84Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#85[dead]
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#86Earlier quoted context omitted.
Kimi 2.5?
That model is "open weight", not open source. We have no idea what data Moonshot trained on.
I think we lost that terminology war. Open source models mean open weight. There are only a couple examples of fully open source models with open data and code, and the labs are not incentivized to go that far.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#87Nice work on the scheduler. Have you benchmarked
parallel inference across multiple models?
Running GPT, Claude and Gemini simultaneously
on the same input is where latency becomes
a real constraint.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#88Nice work on the scheduler. Have you benchmarked parallel inference across multiple models? Running GPT, Claude and Gemini simultaneously on the same input is where latency becomes a real constraint.
GPT-OSS exists but Claude and Gemini aren't available locally, lol.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#89For a lot of local workloads, sub-1 tok/s is useless in foreground and perfectly acceptable in background. If the choice is “this crashes” vs “this finishes overnight,” that’s still a meaningful capability jump.
How much are you going to spend on electricity though? Is this really going to be more cost-effective than just using openrouter?
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#90Earlier quoted context omitted.
Why would llama with --mmap crash?
This doesn't surprise me all that much, mmap support gets little attention in general and interacts poorly with GPU-side inference. (And that's with it being default, you don't even really need to specify it as a CLI option.) OP has raised a discussion with the llama.cpp folks https://github.com/ggml-org/llama.cpp/discussions/20852 but little interest so far
But if mmap already works why would there be any interest?
Besides, discussions are for users. He didn't open PRs or issues.