ggerganov is a genius! I'm using his excellent whisper.cpp in my WisprNote offline private transcribing MacOS app. It 10x the speed of the regular models. He even invented a custom format for the models! The work this guy is doing--first class or beyond!!!
Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
121–130 of 298 posts
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#122I got this working on my 64GB M2 MacBook Pro!
Wrote detailed notes here for anyone else who wants to try this: https://til.simonwillison.net/llms/llama-7b-m2
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#123I got this working on my 64GB M2 MacBook Pro! Wrote detailed notes here for anyone else who wants to try this: https://til.simonwillison.net/llms/llama-7b-m2
Thank you!
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#124Earlier quoted context omitted.
It is using the built-in neural accelerators, that’s why it’s fast, that’s why it’s only supported on Macs so far. The code makes use of official Apple APIs which delegate the necessary BLAS calls to the available hardware.
Confusingly there are 2 mechanisms to do matrix operations on the new apple hardware - AMX ( https://github.com/corsix/amx ) - and the ANE (apple neural engine) - which is enabled by CoreML. This code does not run on the neural engine but the author has a branch for his whisper.cpp project which uses it here: https://github.com/ggerganov/whisper.cpp/pull/566 - so it may not be long before we see it applied here as we…
Three. You can also do it in Metal, which as of recently has cooperative matrix multiplication in the form of the simd_matrix type (this is similar functionality as "tensor cores" in the Nvidia world). I have no idea what the software support is, but I have seen analysis suggesting that the raw tensor multiplication throughput is larger than ANE for the high-end GPUs.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#125> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…
Wonder why AMD/Intel/Nvidia haven’t invented some sort of device that allows the processor and graphics to share memory like Apple has done.
as pointed out, both Jetson and UM do this, but the upcoming Grace does it at higher bandwidth than the apple chip.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#126A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…
The 7b model specifically is not quite "ChatGPT-level" though, is it?
None of the Meta models are RLHF tuned, as far as I know.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#127Earlier quoted context omitted.
The 7b model specifically is not quite "ChatGPT-level" though, is it?
According to Meta's benchmarking[0] it is comparable on many metrics. I haven't used it myself so I can't say for sure if that is the case when actually using it. [0]: https://arxiv.org/pdf/2302.13971.pdf
That's GPT3, not ChatGPT.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#128If you are interested in implementing LLaMA yourself or learning, I noticed that the reference code by Facebook is one of the cleaner, easier to read ML code I've seen in a while. https://github.com/facebookresearch/llama/blob/main/llama/mo... It's about 200 lines long. You probably do need a bit of knowledge to understand what you are reading but I was pleasantly surprised. For example in comparison, StableDiffusion…
I’m pretty sure the code you linked is just simplified for publication. I think it’s interesting to read, I just don’t think it’s what they actually used to train and develop the algorithm.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#129George Hotz already implemented LLaMA 7B and 15B on Twitch yesterday on GPU in Tunygrad llama branch: https://github.com/geohot/tinygrad/tree/llama The only problem is that it's swapping on 16GB Macbook, so you need at least 24GB in practice.
There are two coding sessions streamed so far, very interesting to watch
George Hotz | Programming | can we fit a LLaMA inside a tinygrad? https://www.youtube.com/watch?v=0kRDs9BW2NU
George Hotz | Programming | ChatLLaMA: get in losers we're building a chatbot https://www.youtube.com/watch?v=nctqc8FBJ2U
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#130If you are interested in implementing LLaMA yourself or learning, I noticed that the reference code by Facebook is one of the cleaner, easier to read ML code I've seen in a while. https://github.com/facebookresearch/llama/blob/main/llama/mo... It's about 200 lines long. You probably do need a bit of knowledge to understand what you are reading but I was pleasantly surprised. For example in comparison, StableDiffusion…
I’m pretty sure the code you linked is just simplified for publication. I think it’s interesting to read, I just don’t think it’s what they actually used to train and develop the algorithm.
This is only the model code which defined the shape and how to do a forward pass.
It isn't the training code, but it would be unlikely that the model code used then is any different.