but why would you do C++ when its quite clear ML load is highly parallel. the page says vectorized by NEON but no mention whether its autovectorized by gcc or hand optimized. That will have a pretty significant performance impact.
Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
31–40 of 298 posts
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#32Could someone with experience explain: what's the theoretical minimum hardware requirement for llama 7B, 15B, etc, that still provides output on the order of It seems like we can pull some tricks, like using F16, and some kind of quantization, etc. At the end of the day, how much overhead is left that can be reduced? What can I expect to have running on 16gb ram with a 3080 and a midrange AMD processor?
16GB of vram can run the 7B for sure, I'm not sure what the most cutting-edge memory optimization but the 15B is going to be pretty tight I'm not sure that'll fit with what I know of at least, I've got it working at a bit over 20gb of vram I think at 8bit.
If you can't fit it all in vram you can still run it but it'll be slooooow, at least that's been my experience with the 30b.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#33I don't have the hardware to run the 60B model to test this at the moment - How does it perform with programming, for example making a basic python script to scrape a website, or a bash script, etc? I've managed to run the 13B* at 8bit with decent performance on a 4090 - but it's only 24GB of VMRAM so I've been struggling to run the 30B at anything more then a snails pace.
you mean the 13B ?
Yeah my bad, everyone is a bit all over the place with the numbers in this thread.
I'm not exactly sure how these numbers were chosen, they seem a bit odd?
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#34Could someone with experience explain: what's the theoretical minimum hardware requirement for llama 7B, 15B, etc, that still provides output on the order of It seems like we can pull some tricks, like using F16, and some kind of quantization, etc. At the end of the day, how much overhead is left that can be reduced? What can I expect to have running on 16gb ram with a 3080 and a midrange AMD processor?
Well I was able to run the original code with the 7B model on 16GB vram: https://news.ycombinator.com/item?id=35013604 The output I got was underwhelming, though I did not attempt any tuning.
parameter tuning is pretty necessary, according to anecdotes. People on twitter have got good results by changing the default parameters.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#35I'm running 4-bit quantized llamas on torch/cuda with https://github.com/qwopqwop200/GPTQ-for-LLaMa, and I'm seeing significant tokens/second perf degradation compared to 8-bit bitsandbytes mode. I'm very new to this, and understand very little detail, but I thought it would be faster?
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#36This is sort of the polar opposite of how modern high performance ML frameworks are built. Skimming the code, there's a ton of boilerplate for the various operations that could be library-ized and generified, if that makes sense.
I actually really like minimal implementations of state-of-the-art systems because the code is much easier to understand (modern frameworks are super-complex) but I wonder what it means long-term if you don't need frameworks.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#37Don't know anything about ML can someone can explain me what is this hype about?
You can run your own ChatGPT on your Macbook.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#38The performance on Macbook with M1 Pro is said to be 20 tokens/s https://twitter.com/ggerganov/status/1634282694208114690
This is faster than running it on an RTX 4090 I think.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#39I don't have the hardware to run the 60B model to test this at the moment - How does it perform with programming, for example making a basic python script to scrape a website, or a bash script, etc? I've managed to run the 13B* at 8bit with decent performance on a 4090 - but it's only 24GB of VMRAM so I've been struggling to run the 30B at anything more then a snails pace.
The 13b and 30b run quite well on a 4090 at 4-bit quantization.
Ah dang I missed that I was still using the 8bit mode, I'll look into that thanks!
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#40Absolutely love ggerganov's approach with models like this and Whisper. It's just awesome being able to experiment with (what I consider) complex models without needing a billion python/c/cpp dependencies!