Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

141–150 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#141

Post by popinman322 »

Kelamir wrote: post #136
redox99 wrote: post #127

Earlier quoted context omitted.

That's GPT3, not ChatGPT.

I don't understand this topic well, but given premise that GPT3 and ChatGPT are different only that ChatGPT includes RLHF(Reinforcement Learning from Human Feedback), and LLaMA 7b is comparable to GPT3 on a number of metrics, it would follow that if we were to improve LLaMA 7b with RLHF, the 7b model would be similar to ChatGPT. Is that correct?

You're likely right that applying RLHF (+ fine-tuning with instructions) to LLaMA 7b would produce results similar to ChatGPT, but I think you're implying that that would be feasible today.

RLHF requires a large amount of human feedback data and IIRC there's no open data set for that right now.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#142

Post by Tepix »

RonnieOwnsLexus wrote: post #133
fwlr wrote: post #104

A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…

But wont it be that in real life no one would want to run a voice command which consumes lot of CPU and battery as opposed to making a network call to a service which has this model hosted ? Agreed that this can always be improved and hardware can get more efficient and better to but at the end of the day, would it ever be better then an API call ?

Privacy concerns are justified.

It's not just that, this can also work completely offline.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#143

Post by Tepix »

gpm wrote: post #92
adeon wrote: post #81

If you are interested in implementing LLaMA yourself or learning, I noticed that the reference code by Facebook is one of the cleaner, easier to read ML code I've seen in a while. https://github.com/facebookresearch/llama/blob/main/llama/mo... It's about 200 lines long. You probably do need a bit of knowledge to understand what you are reading but I was pleasantly surprised. For example in comparison, StableDiffusion…

My code for this is very much not high quality, but I have a CPU + GPU + SSD combination: https://github.com/gmorenz/llama/tree/ssd Usage instructions in the commit message: https://github.com/facebookresearch/llama/commit/5be06e56056... At least with my hardware this runs at "[size of model]/[speed of SSD reads]" tokens per second, which (up to some possible further memory reduction so you can run larger batches at…

Won't the 65b model (almost) fit into 128GB RAM? Or into 128GB RAM and 24GB VRAM?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#144

Post by [deleted] »

recuter wrote: post #60

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…

[deleted]

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#145

Post by Nowado »

redox99 wrote: post #127
doctoboggan wrote: post #120

Earlier quoted context omitted.

According to Meta's benchmarking[0] it is comparable on many metrics. I haven't used it myself so I can't say for sure if that is the case when actually using it. [0]: https://arxiv.org/pdf/2302.13971.pdf

That's GPT3, not ChatGPT.

There's no overhead introduced for the 'final' model inference, is there?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#146

Post by amelius »

mattfrommars wrote: post #51

I have very limited in this domain. Why is it necessary to port LLaMa Into C? Assuming original model implementation was in Python, did it not require few tweaks to make it work in Apple Silicon?

Yes this is a good question. Why did they focus on a specific model rather than a generic solution that makes ANY python based model work on Apple silicon?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#148

Post by MacsHeadroom »

Tepix wrote: post #143
gpm wrote: post #92

Earlier quoted context omitted.

My code for this is very much not high quality, but I have a CPU + GPU + SSD combination: https://github.com/gmorenz/llama/tree/ssd Usage instructions in the commit message: https://github.com/facebookresearch/llama/commit/5be06e56056... At least with my hardware this runs at "[size of model]/[speed of SSD reads]" tokens per second, which (up to some possible further memory reduction so you can run larger batches at…

Won't the 65b model (almost) fit into 128GB RAM? Or into 128GB RAM and 24GB VRAM?

LLaMA-65B fits in 32GB of VRAM using state of the art GPTQ quantization with no output performance loss.

https://github.com/qwopqwop200/GPTQ-for-LLaMa

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#149

Post by [deleted] »

yieldcrv wrote: post #58
0xbadc0de5 wrote: post #42

Earlier quoted context omitted.

The 4-bit GPTQ LLaMA models are the current top-performers. This site has done a lot of the heavy lifting: https://github.com/qwopqwop200/GPTQ-for-LLaMa With 30b-4bit on a RTX 4090, I'm seeing numbers like: Output generated in 4.17 seconds (4.03 tokens/s, 21 tokens) Output generated in 4.38 seconds (4.25 tokens/s, 23 tokens) Output generated in 4.57 seconds (4.25 tokens/s, 24 tokens) Output generated in 3.86 seconds…

And on an M1? I have 64gb available

[deleted]

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#150

Post by toxik »

nl wrote: post #130
toxik wrote: post #128

Earlier quoted context omitted.

I’m pretty sure the code you linked is just simplified for publication. I think it’s interesting to read, I just don’t think it’s what they actually used to train and develop the algorithm.

This is only the model code which defined the shape and how to do a forward pass. It isn't the training code, but it would be unlikely that the model code used then is any different.

There are little hints strewn out through the code that suggests it is indeed “trimmed” from a larger codebase.
Post reply on HN