Earlier quoted context omitted.
Roughly, at Q8 the model sizes translate to GB, so ~3 and ~70GB.
You mean 8, not 3?
Yes, apologies, can't edit now
Run Llama locally with only PyTorch on CPU
31–35 of 35 posts
Re: Run Llama locally with only PyTorch on CPU
#32If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
> then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly given up on GPU for LLMs. It was a game changer.
First time that I have a "it just works" experience with LLMs on my computer. Amazing. Thanks for the recommendation!
Re: Run Llama locally with only PyTorch on CPU
#33Earlier quoted context omitted.
Ollama is great if you're really in love with the idea of having your multi gigabyte models (likely the majority of your disk space) stored in obfuscated UUID filenames. Ollama also still hasn't addressed the license violations I reported to them back in March. https://github.com/ollama/ollama/issues/3185
Llamafile is great if you don't want to run any meaningful models because it's limited to 4GB.
That's a Windows limitation, though.
Re: Run Llama locally with only PyTorch on CPU
#34Earlier quoted context omitted.
Llamafile is great if you don't want to run any meaningful models because it's limited to 4GB.
That's a Windows limitation, though.
Even on Windows, you just run the binary separate from the model file. I actually run a single binary separate from the model files because I run it with multiple of them, so I kind of forgot that that was even the default way it kind of expects you to hold it.
Re: Run Llama locally with only PyTorch on CPU
#35Fast enough for RPI5 ARM?