11–20 of 35 posts
If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
Can it use GPU if available, say on Apple silicon Macs
If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
PyTorch has a native llm solution It supports all the LLama models. It supports CPU, MPS and CUDA https://github.com/pytorch/torchchat Getting 4.5 tokens a second using 3.1 8B full precision using CPU only on my M1
Seems like torchchat is exactly what the author was looking for.
> And the 8B model typically gets killed by the OS for using too much memory.
Torchchat also provides some quantization options so you can reduce the model size to fit into memory.
This just imports the Llama reference implementation and patches the device FYI.
There are more robust implementations out there.
Earlier quoted context omitted.
Ollama (also wrapping llama.cpp) has GPU support, unless you're really in love with the idea of bundling weights into the inference executable probably a better choice for most people.
Ollama is great if you're really in love with the idea of having your multi gigabyte models (likely the majority of your disk space) stored in obfuscated UUID filenames. Ollama also still hasn't addressed the license violations I reported to them back in March. https://github.com/ollama/ollama/issues/3185
Even myself I’ve not thought about this so deeply, even though I am also very concerned about honoring other people’s work and that licenses are followed.
I have some command line tools for example that I’ve written in Rust that depend on various libraries. But because I distribute my software in source form mostly, I haven’t really paid attention to how a command-line tool which is distributed as a compiled binary would make sure to include attribution and copies of the licenses of its dependencies.
And so the main place where I’ve given more thought to those concerns is for example in full-blown GUI apps. There they usually have an about menu that will include info about their dependencies. And the other part where I’ve thought about it is in commercial electronics making use of open source software in their firmware. In those physical products they usually include either some printed documents alongside the product where attributions and license texts are sometimes found, and sometimes if the product has a display, or a display output, they have a menu you can find somewhere with that sort of info.
I know that for example Debian is very good at being thorough with details about licenses, but I’ve never looked at what they do with command line tools that compile third-party code into them. Like does Debian package maintainers then for example dig up copies of the licenses from the source and dependencies and put them somewhere in /usr/share/ as plain text files? Or do the .deb files themselves contain license text copies you can view but which are not installed onto the system? Or they work with software authors to add a flag that will show licenses? Or something else?
If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
Do you have a ballpark idea of how much RAM would be necessary to run llama 3.1 8b and 70b on 8-quant?