So when will I be able to "sudo apt-get install llm" ?
Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
11–20 of 61 posts
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#12So when will I be able to "sudo apt-get install llm" ?
Ollama is close...
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#13Earlier quoted context omitted.
Ah the disk swap method
Is there any predictability/patterns for neuron/layer activation? If so, would it be reasonable to have a second tiny model that specifically tries to predict activation and preemptively swap those into memory?
Depends on the architecture, but generally you just move through the layers linearly. Simple iteration.
The number of layers, and the amount of time spent in each of them, makes me think any benefit from pre-loading the layer ahead is negligible.
You really need the entire model on device to consider it performant.
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#14Earlier quoted context omitted.
Ah the disk swap method
Is there any predictability/patterns for neuron/layer activation? If so, would it be reasonable to have a second tiny model that specifically tries to predict activation and preemptively swap those into memory?
This isn't how neural networks work.
For vanilla models, you always use all the weights. That isn't true for mixture-of-experts, though, and in that setting, your approach has merit.
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#15So when will I be able to "sudo apt-get install llm" ?
You can already do it with llamafile, checkout the project, it lets you convert a .gguf model in a portable executable
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#16Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#17Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#18So when will I be able to "sudo apt-get install llm" ?
You can already do it with llamafile, checkout the project, it lets you convert a .gguf model in a portable executable
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#19So when will I be able to "sudo apt-get install llm" ?
I'm not aware of any Debian family distro that packages it, but NixOS has at least ollama and llama-cpp in its repos. Honestly even if the more stable distributions did have these things packaged, I would hesitate to use the packaged versions because all of this stuff is still so quickly moving that you'd be on an old version and it would hurt.
Edit: Arch has ollama in official repos too. OpenSUSE has https://software.opensuse.org/package/ollama .
Re: Serving 70B-scale LLMs efficiently on low-resource edge devices [pdf]
#20Earlier quoted context omitted.
And everything runs way faster on cpu like that
I could run at 1token/s on qwen 2.5 72b q4_k_m on my i7 8750h + alot of ram XD
With this model:
https://huggingface.co/bartowski/Qwen2.5-72B-Instruct-GGUF
Man I need to test the q8 version with llamafiles optimizations, it would be so nice to host it locally with the new ryzens, it could maybe fit my 96GB of ram