Earlier quoted context omitted.
That's GPT3, not ChatGPT.
There's no overhead introduced for the 'final' model inference, is there?
[deleted]
Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
151–160 of 298 posts
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#152How does one learn to do stuff like this? You first learn doing AI algorithms in Python and then transfer the knowledge to C++ or you learn doing them in C++ from the start?
You first need the basic building blocks which is matrix multiplication. You may use libraries here.
Then it’s a matter of loading the model correctly.
If you are just reimplementing and not doing research you don’t need the mathematics.
The algorithm is rather simple too. The issue is more about engineering than machine learning.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#153The performance on Macbook with M1 Pro is said to be 20 tokens/s https://twitter.com/ggerganov/status/1634282694208114690
This is faster than running it on an RTX 4090 I think.
I get 32 tokens/sec on a 4090 using GPTQ 4bit with streaming off, with the model 5x larger than that.
So nowhere close to the 4090, but plenty fast anyway.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#154I have very limited in this domain. Why is it necessary to port LLaMa Into C? Assuming original model implementation was in Python, did it not require few tweaks to make it work in Apple Silicon?
Yes this is a good question. Why did they focus on a specific model rather than a generic solution that makes ANY python based model work on Apple silicon?
Because that's not how machine learning models work. Machine learning as a field goes through a nearly complete revolution annually. Every new major model is a special snowflake of unique cases.
Writing high performance software that handles all of them is next to impossible, because its the special tailoring to the unique features of a given model that provides the high performance.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#155Earlier quoted context omitted.
This is an advanced language model that can now run quickly on consumer grade hardware. You used to need thousands of dollars of GPUs to run a model as sophisticated as this - now it can be done on a laptop,
Wasn't LLaMa official meant to run on consumer grade machine? How does this modify the model to make it work. All of this is confusing.
Yes but it wasn't made to run on a Mac. This project ported LLaMA to Apple Silicon so all the macbook users can finally play with what the rest of us have had access to for the past couple of weeks.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#156Earlier quoted context omitted.
Yes this is a good question. Why did they focus on a specific model rather than a generic solution that makes ANY python based model work on Apple silicon?
Because that's not how machine learning models work. Machine learning as a field goes through a nearly complete revolution annually. Every new major model is a special snowflake of unique cases. Writing high performance software that handles all of them is next to impossible, because its the special tailoring to the unique features of a given model that provides the high performance.
That's not how I think it works. ML is a small number of operations applied to very large blocks of data, tensors. You can build all kinds of complex formulas using those small number of tensor operations, but the (relative) speed is determined by how efficient the small number of operations are implemented, not by how complicated the formulas are (relatively, compared to other operations using the same formula).
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#157Earlier quoted context omitted.
I don't understand this topic well, but given premise that GPT3 and ChatGPT are different only that ChatGPT includes RLHF(Reinforcement Learning from Human Feedback), and LLaMA 7b is comparable to GPT3 on a number of metrics, it would follow that if we were to improve LLaMA 7b with RLHF, the 7b model would be similar to ChatGPT. Is that correct?
You're likely right that applying RLHF (+ fine-tuning with instructions) to LLaMA 7b would produce results similar to ChatGPT, but I think you're implying that that would be feasible today. RLHF requires a large amount of human feedback data and IIRC there's no open data set for that right now.
There are open datasets (see the chatllama harness project and its references). You can of course also cross train it using actual ChatGPT.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#158Earlier quoted context omitted.
Won't the 65b model (almost) fit into 128GB RAM? Or into 128GB RAM and 24GB VRAM?
LLaMA-65B fits in 32GB of VRAM using state of the art GPTQ quantization with no output performance loss. https://github.com/qwopqwop200/GPTQ-for-LLaMa
So if I'm reading this right, 65B at 4bit would consume around 20GB of VRAM and ~130GB of system RAM?
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#159Earlier quoted context omitted.
I don't understand this topic well, but given premise that GPT3 and ChatGPT are different only that ChatGPT includes RLHF(Reinforcement Learning from Human Feedback), and LLaMA 7b is comparable to GPT3 on a number of metrics, it would follow that if we were to improve LLaMA 7b with RLHF, the 7b model would be similar to ChatGPT. Is that correct?
You're likely right that applying RLHF (+ fine-tuning with instructions) to LLaMA 7b would produce results similar to ChatGPT, but I think you're implying that that would be feasible today. RLHF requires a large amount of human feedback data and IIRC there's no open data set for that right now.
There's open-assistant.io, which is doing RLHF directly on the open
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#160I'm getting 56.38 ms per token on my 32GB M1 Max using this code on the 7GB model. Very usable!
what model are you using?
edit: i mean 6B, 13b or 30B?