Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

github.com

Post reply on HN
Print view View original

71–80 of 101 posts

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#71

Post by simonw »

Suggestion for the maintainers: the comparison table currently lists some pretty old models, Qwen 2.5 14B and Mixtral 8x7B and Llama 3.3 70B.

A lot of people are reporting incredible results with the Qwen 3.5 MoE models on Apple hardware right now (streaming experts - see https://simonwillison.net/2026/Mar/24/streaming-experts/) - it would be great to get some of those models into that table.

Maybe the 1T parameter Kimi K2.5 too if you can get that to work, see https://twitter.com/seikixtc/status/2036246162936910322 and https://twitter.com/danpacary/status/2036480556045836603

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#72

Post by Imustaskforhelp »

simonw wrote: post #71

Suggestion for the maintainers: the comparison table currently lists some pretty old models, Qwen 2.5 14B and Mixtral 8x7B and Llama 3.3 70B. A lot of people are reporting incredible results with the Qwen 3.5 MoE models on Apple hardware right now (streaming experts - see https://simonwillison.net/2026/Mar/24/streaming-experts/ ) - it would be great to get some of those models into that table. Maybe the 1T parameter…

Simon, A little offtopic but it seems that your website isn't working.

> An error occurred in the application and your page could not be served. If you are the application owner, check your logs for details. You can do this from the Heroku CLI with the command

I get this error when I go to simonwillison.net

Any random blog/link works for example though: https://simonwillison.net/2026/Mar/19/openai-acquiring-astra...

(I checked your website because I wanted to see if you had written something about trivy/litellm as well, I highly recommend checking out what has happened within litellm space if possible as I would love to read your thoughts on it)

Have a nice day simon!

Edit: now the website works but I am not sure what had gone wrong previously, (an issue from heroku maybe?) as its working now

Edit-2: after the website working, I am able to see that you have already made a post about it.

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#73

Post by zozbot234 »

maleldil wrote: post #65
feznyng wrote: post #61

Earlier quoted context omitted.

Could still be useful; maybe for overnight async workloads? Tell your agent research xyz at night and wake up to a report.

Assuming 1 token per second and "overnight" being 12 hours, that's 43 200 tokens. I'm not sure what you can meaningfully achieve with that.

Sure, but if long-term throughput is a real limitation there's plenty of ways to address that while still not needing to keep anywhere close to all model weights in RAM (which is still the conventional approach with MoE). So the gain of a smaller memory footprint is quite real.

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#74

Post by abtinf »

simonw wrote: post #71

Suggestion for the maintainers: the comparison table currently lists some pretty old models, Qwen 2.5 14B and Mixtral 8x7B and Llama 3.3 70B. A lot of people are reporting incredible results with the Qwen 3.5 MoE models on Apple hardware right now (streaming experts - see https://simonwillison.net/2026/Mar/24/streaming-experts/ ) - it would be great to get some of those models into that table. Maybe the 1T parameter…

The lack of a token rate metric for the kimi example is disappointing.

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#75

Post by tatef »

simonw wrote: post #71

Suggestion for the maintainers: the comparison table currently lists some pretty old models, Qwen 2.5 14B and Mixtral 8x7B and Llama 3.3 70B. A lot of people are reporting incredible results with the Qwen 3.5 MoE models on Apple hardware right now (streaming experts - see https://simonwillison.net/2026/Mar/24/streaming-experts/ ) - it would be great to get some of those models into that table. Maybe the 1T parameter…

Thanks for sharing this! If you'd be interested in running the benchmark yourself with Hypura I'd happily merge into our stats. Otherwise will add to my todo list :)

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#76

Post by astrange »

zozbot234 wrote: post #30
anshulbasia27 wrote: post #29

OS paging would be significantly worse here. The kernel's page fault handler is reactive — it doesn't know you're about to read layer 47's FFN weights, so it can't prefetch. You stall on every fault, wait for the 4KB/16KB page to load, then resume. With 80 layers of dense FFN streaming, that's thousands of cold faults per token. What makes this approach faster is that the model's access pattern is completely determin…

> The kernel's page fault handler is reactive — it doesn't know you're about to read layer 47's FFN weights, so it can't prefetch. man 2 madvise

That works for readahead but it's not good for random access. readv, aio, dispatch_io are better there.

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#77

Post by astrange »

> Consumer hardware (MacBook Pro, Mac Studio) ships with fast unified memory and NVMe storage, but limited capacity. A 32 GB M1 Max cannot naively load a 40 GB model — the OS will swap-thrash until the OOM killer intervenes.

macOS doesn't have an "OOM killer" in that sense. (It has an out of swap space killer but it's pretty weak.)

So what will happen is, either your memory wiring will fail, or else it will get really slow and panic.

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#78

Post by zozbot234 »

astrange wrote: post #76
zozbot234 wrote: post #30

Earlier quoted context omitted.

> The kernel's page fault handler is reactive — it doesn't know you're about to read layer 47's FFN weights, so it can't prefetch. man 2 madvise

That works for readahead but it's not good for random access. readv, aio, dispatch_io are better there.

This claim is a bit apples and oranges (no pun intended!). madvise is all about providing hints to the kernel to tune the page cache and readahead (including possibly disabling readahead altogether). it's not about performing reads into private memory buffers, which is actually where the options you mentioned fit in.

Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon

#80

Post by zozbot234 »

abtinf wrote: post #74
simonw wrote: post #71

Suggestion for the maintainers: the comparison table currently lists some pretty old models, Qwen 2.5 14B and Mixtral 8x7B and Llama 3.3 70B. A lot of people are reporting incredible results with the Qwen 3.5 MoE models on Apple hardware right now (streaming experts - see https://simonwillison.net/2026/Mar/24/streaming-experts/ ) - it would be great to get some of those models into that table. Maybe the 1T parameter…

The lack of a token rate metric for the kimi example is disappointing.

The latter link says they get ~1.7 tok/s which is quite impressive for a near-SOTA local model running on ordinary hardware.
Post reply on HN