21–30 of 35 posts
If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
Earlier quoted context omitted.
Can it use GPU if available, say on Apple silicon Macs
> GPU on MacOS ARM64 is supported by compiling a small module using the Xcode Command Line Tools, which need to be installed. This is a one time cost that happens the first time you run your llamafile.
Loved llamafile and used it to build the first version of https://recurse.chat/, but live compilation using XCode Command Line Tool is a no-go for Mac App Store builds (runs in Mac App Sandbox). llama.cpp doesn't need compiling on user's machine fwiw.
Earlier quoted context omitted.
Ollama is great if you're really in love with the idea of having your multi gigabyte models (likely the majority of your disk space) stored in obfuscated UUID filenames. Ollama also still hasn't addressed the license violations I reported to them back in March. https://github.com/ollama/ollama/issues/3185
I think this is also a problem in a lot of tools, that is never talked about. Even myself I’ve not thought about this so deeply, even though I am also very concerned about honoring other people’s work and that licenses are followed. I have some command line tools for example that I’ve written in Rust that depend on various libraries. But because I distribute my software in source form mostly, I haven’t really paid at…
If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
How great is the performance? Tokens/s?
148 tokens predicted, 159 ms per token, 6.27 tokens per secondEarlier quoted context omitted.
How great is the performance? Tokens/s?
Random sample query ("What shape should a kumquat be?") against a 7B model quantised to 4b running on an i7-9750H (so a good CPU, but also a good laptop CPU from 2019) gives: 148 tokens predicted, 159 ms per token, 6.27 tokens per second
If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
Thanks for posting this!
If your goal is > I want to peel back the layers of the onion and other gluey-mess to gain insight into these models. Then this is great. If your goal is > Run and explore Llama models locally with minimal dependencies on CPU then I recommend https://github.com/Mozilla-Ocho/llamafile which ships as a single file with no dependencies and runs on CPU with great performance. Like, such great performance that I've mostly…
Does anyone know what's the easiest way to finetune a model locally is today?
Earlier quoted context omitted.
Ollama (also wrapping llama.cpp) has GPU support, unless you're really in love with the idea of bundling weights into the inference executable probably a better choice for most people.
Ollama is great if you're really in love with the idea of having your multi gigabyte models (likely the majority of your disk space) stored in obfuscated UUID filenames. Ollama also still hasn't addressed the license violations I reported to them back in March. https://github.com/ollama/ollama/issues/3185