[flagged]
Have you ever generated access frequency statistics for the experts in these models, something like a histogram?
Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
11–20 of 101 posts
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#12This is a pretty cool project! Essentially this is like using Swap memory to extend your RAM, but in a 'smart' way so you don't overload the NVMe unnecessarily. I do wonder in practice how the 'smarts' pan out, because putting a ton of stress on your NVMe during generation is probably not the best choice for it's longevity.
This is not putting any stress or wear on the NVMe, it's a pure read workload.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#13[flagged]
Why would llama with --mmap crash?
This doesn't surprise me all that much, mmap support gets little attention in general and interacts poorly with GPU-side inference. (And that's with it being default, you don't even really need to specify it as a CLI option.) OP has raised a discussion with the llama.cpp folks https://github.com/ggml-org/llama.cpp/discussions/20852 but little interest so far
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#14This is a pretty cool project! Essentially this is like using Swap memory to extend your RAM, but in a 'smart' way so you don't overload the NVMe unnecessarily. I do wonder in practice how the 'smarts' pan out, because putting a ton of stress on your NVMe during generation is probably not the best choice for it's longevity.
> but in a 'smart' way so you don't overload the NVMe unnecessarily
"overloading NVMe"? What is that about? First time I've heard anything about it.
> because putting a ton of stress on your NVMe during generation
Really shouldn't "stress your NVMe", something is severely wrong if that's happening. I've been hammering my SSDs forever, and while write operations "hurt" the longevity of the flash cells themselves, the controller interface really shouldn't be affected by this at all, unless I'm missing something here.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#15There needs to be something like this from Ollama. At the moment Ollama has a lot of flaws that prevent it from getting great performance. (My understanding is better GPU/CPU splits, etc). But Ollama is the only way to host an LLM and have it switch out on demand. Sigh.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#16This is a pretty cool project! Essentially this is like using Swap memory to extend your RAM, but in a 'smart' way so you don't overload the NVMe unnecessarily. I do wonder in practice how the 'smarts' pan out, because putting a ton of stress on your NVMe during generation is probably not the best choice for it's longevity.
> but in a 'smart' way so you don't overload the NVMe unnecessarily "overloading NVMe"? What is that about? First time I've heard anything about it. > because putting a ton of stress on your NVMe during generation Really shouldn't "stress your NVMe", something is severely wrong if that's happening. I've been hammering my SSDs forever, and while write operations "hurt" the longevity of the flash cells themselves, the…
I had assumed heat generation on the controller if it's continuously reading. But maybe it's not actually bad.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#17It will be interesting to compare this to https://news.ycombinator.com/item?id=47476422 and https://news.ycombinator.com/item?id=47490070 . Very similar design except that this is apparently using mmap, which according to the earlier experiment incurs significant overhead.
It was written by an LLM, so... yeah.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#18Intel Optane rolling in its grave.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#19There needs to be something like this from Ollama. At the moment Ollama has a lot of flaws that prevent it from getting great performance. (My understanding is better GPU/CPU splits, etc). But Ollama is the only way to host an LLM and have it switch out on demand. Sigh.
llama.cpp and llama-swap do this better than Ollama and with far more control.
Re: Hypura – A storage-tier-aware LLM inference scheduler for Apple Silicon
#20Intel Optane rolling in its grave.
Still have 4 brand new ones in my storage unit. Just in case these moments.
Joke aside (I do have them tho!), I don't think Optane is that much use (not to mention it is only 256GiB for my unit). It is useful legacy crutch if you have legacy software that is not designed to issue multiple reads / writes in parallel. If you do, it is really not faster than NVMe, especially these modern ones.