Run Llama locally with only PyTorch on CPU
1–10 of 35 posts
Re: Run Llama locally with only PyTorch on CPU
#2Peel back the layers of the onion and other gluey-mess to gain insight into these models.
Re: Run Llama locally with only PyTorch on CPU
#3If your goal is
> I want to peel back the layers of the onion and other gluey-mess to gain insight into these models.
Then this is great.
If your goal is
> Run and explore Llama models locally with minimal dependencies on CPU
then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly given up on GPU for LLMs. It was a game changer.
Re: Run Llama locally with only PyTorch on CPU
#4If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
Ollama (also wrapping llama.cpp) has GPU support, unless you're really in love with the idea of bundling weights into the inference executable probably a better choice for most people.
Re: Run Llama locally with only PyTorch on CPU
#5If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
Ollama (also wrapping llama.cpp) has GPU support, unless you're really in love with the idea of bundling weights into the inference executable probably a better choice for most people.
When I said
> such great performance that I've mostly given up on GPU for LLMs
I mean I used to run ollama on GPU, but llamafile was approximately the same performance on just CPU so I switched. Now that might just be because my GPU is weak by current standards, but that is in fact the comparison I was making.
Edit: Though to be clear, ollama would easily be my second pick; it also has minimal dependencies and is super easy to run locally.
Re: Run Llama locally with only PyTorch on CPU
#6If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
A great place to start is with the LLaMA 3.2 q6 llamafile I posted a few days ago. https://huggingface.co/Mozilla/Llama-3.2-3B-Instruct-llamafi... We have a new CLI chatbot interface that's really fun to use. Syntax highlighting and all. You can also use GPU by passing the -ngl 999 flag.
Re: Run Llama locally with only PyTorch on CPU
#7If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
Ollama (also wrapping llama.cpp) has GPU support, unless you're really in love with the idea of bundling weights into the inference executable probably a better choice for most people.
Ollama is great if you're really in love with the idea of having your multi gigabyte models (likely the majority of your disk space) stored in obfuscated UUID filenames. Ollama also still hasn't addressed the license violations I reported to them back in March. https://github.com/ollama/ollama/issues/3185
Re: Run Llama locally with only PyTorch on CPU
#8With the same mindset, but without even PyTorch as dependency there's a straightforward CPU implementation of llama/gemma in Rust: https://github.com/samuel-vitorino/lm.rs/
It's impressive to realize how little code is needed to run these models at all.
Re: Run Llama locally with only PyTorch on CPU
#9If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
A great place to start is with the LLaMA 3.2 q6 llamafile I posted a few days ago. https://huggingface.co/Mozilla/Llama-3.2-3B-Instruct-llamafi... We have a new CLI chatbot interface that's really fun to use. Syntax highlighting and all. You can also use GPU by passing the -ngl 999 flag.
„On Windows, only the graphics card driver needs to be installed if you own an NVIDIA GPU. On Windows, if you have an AMD GPU, you should install the ROCm SDK v6.1 and then pass the flags --recompile --gpu amd the first time you run your llamafile.”
Looks like there’s a typo, Windows is mentioned twice.
Re: Run Llama locally with only PyTorch on CPU
#10Earlier quoted context omitted.
Ollama (also wrapping llama.cpp) has GPU support, unless you're really in love with the idea of bundling weights into the inference executable probably a better choice for most people.
Ollama is great if you're really in love with the idea of having your multi gigabyte models (likely the majority of your disk space) stored in obfuscated UUID filenames. Ollama also still hasn't addressed the license violations I reported to them back in March. https://github.com/ollama/ollama/issues/3185
I wasn't aware of the license issue, wow. Not a good look especially considering how simple that is to resolve.
The model storage doesn't bother me but I also use Docker so I'm used to having a lot of tool-managed data to deal with. YMMV.
Edit: Removed question about GPU support.