Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

github.com

Post reply on HN
Print view View original

91–100 of 101 posts

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#91

Post by austinthetaco »

joelthelion wrote: post #89
vanyaland wrote: post #64

For a lot of local workloads, sub-1 tok/s is useless in foreground and perfectly acceptable in background. If the choice is “this crashes” vs “this finishes overnight,” that’s still a meaningful capability jump.

How much are you going to spend on electricity though? Is this really going to be more cost-effective than just using openrouter?

There are many other reasons someone might want to run a model locally outside of cost savings, ownership of data flow and use in locations without internet to name a couple.

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#93

Post by dev_tools_lab »

zozbot234 wrote: post #88
dev_tools_lab wrote: post #87

Nice work on the scheduler. Have you benchmarked parallel inference across multiple models? Running GPT, Claude and Gemini simultaneously on the same input is where latency becomes a real constraint.

GPT-OSS exists but Claude and Gemini aren't available locally, lol.

True, Claude and Gemini aren’t local yet — I mostly meant running all available local models in parallel.

Even with just open-source LLMs, you can see interesting differences in flagged issues when cross-validating outputs.

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#95

Post by astrange »

zozbot234 wrote: post #78
astrange wrote: post #76

Earlier quoted context omitted.

That works for readahead but it's not good for random access. readv, aio, dispatch_io are better there.

This claim is a bit apples and oranges (no pun intended!). madvise is all about providing hints to the kernel to tune the page cache and readahead (including possibly disabling readahead altogether). it's not about performing reads into private memory buffers, which is actually where the options you mentioned fit in.

Triggering reads is also how you get pages into the page cache, so it helps to know how to do it.

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#99

Post by a7om_com »

simonw wrote: post #71

Suggestion for the maintainers: the comparison table currently lists some pretty old models, Qwen 2.5 14B and Mixtral 8x7B and Llama 3.3 70B. A lot of people are reporting incredible results with the Qwen 3.5 MoE models on Apple hardware right now (streaming experts - see https://simonwillison.net/2026/Mar/24/streaming-experts/ ) - it would be great to get some of those models into that table. Maybe the 1T parameter…

[flagged]
Post reply on HN