Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
1–10 of 101 posts
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#2[flagged]
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#3Where does "1T parameter model" come from? I can only see models with 70B params or less mentioned in the repo.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#4[flagged]
This is interesting work, thank you for sharing. What hardware would you buy today for experimenting? Seems like the new gen of macbook pros are pretty powerful?
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#5It will be interesting to compare this to https://news.ycombinator.com/item?id=47476422 and https://news.ycombinator.com/item?id=47490070 . Very similar design except that this is apparently using mmap, which according to the earlier experiment incurs significant overhead.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#6[flagged]
Why would llama with --mmap crash?
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#7Where does "1T parameter model" come from? I can only see models with 70B params or less mentioned in the repo.
Yeah title comes from nowhere in the link. No doubt it's possible but all that matters is speed and we learn nothing of that here...
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#8[flagged]
You need to change the title or actually include 1T parameter model content.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#9[flagged]
Don't post generated/AI-edited comments. HN is for conversation between humans
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#10This is a pretty cool project! Essentially this is like using Swap memory to extend your RAM, but in a 'smart' way so you don't overload the NVMe unnecessarily.
I do wonder in practice how the 'smarts' pan out, because putting a ton of stress on your NVMe during generation is probably not the best choice for it's longevity.