I wrote about why I think LLaMA + llama.cpp means that large language models are having a "Stable Diffusion moment" right now: https://simonwillison.net/2023/Mar/11/llama/
Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
211–220 of 298 posts
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#212A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…
4 words a second doesn't seem fast enough for a voice assistant ?
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#213Earlier quoted context omitted.
But wont it be that in real life no one would want to run a voice command which consumes lot of CPU and battery as opposed to making a network call to a service which has this model hosted ? Agreed that this can always be improved and hardware can get more efficient and better to but at the end of the day, would it ever be better then an API call ?
Privacy concerns are justified. It's not just that, this can also work completely offline.
I'm looking forward to run stuff like this online. Using bigtech corporate souls SaaS AI is just pure dystopia material.
It's even better that we are talking about a relatively low power machine here. Maybe can operate offered.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#214A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…
4 words a second doesn't seem fast enough for a voice assistant ?
I've had difficulty obtaining useful results from the smaller (7B-sized) models. The issue lies in the content, not the speed. If you could stream the text-to-speech, the speed alone would be satisfactory.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#215Earlier quoted context omitted.
4 words a second doesn't seem fast enough for a voice assistant ?
I've had difficulty obtaining useful results from the smaller (7B-sized) models. The issue lies in the content, not the speed. If you could stream the text-to-speech, the speed alone would be satisfactory.
You're right I overestimated how fast we talk!
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#216but why would you do C++ when its quite clear ML load is highly parallel. the page says vectorized by NEON but no mention whether its autovectorized by gcc or hand optimized. That will have a pretty significant performance impact.
Seems like it's extremely high performance for me on an M1 running the 7B model. Totally usable.
just because something is high performance does not means it cannot be improved by say another 2X. My point was not whether its usable or not, it is that if you are going to run on CPU vectorization is rather important part and its odd that the landing page has no remark about it.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#217Earlier quoted context omitted.
Siri is sometimes busy doing laundry or Gods know what. I think the quality of Siri is much better than Google Assistant but I wonder about the lag.
Really? I find Siri can’t understand anything slightly more than basic instructions. Google assistant can seem to do more
I'm very interested in this space. Can you share an example that illustrates the difference in "understanding" between the two?
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#218Earlier quoted context omitted.
llama.cpp/ggml.h // GGML Tensor Library ... // This library implements: // - a set of tensor operations // - automatic differentiation // - basic optimization algorithms
> autodiff There's a lot of // TODO: implement backward in there
Ok thanks for digging deeper, I didn't realize that, and obviously that invalidates the excerpt I posted above
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#219Earlier quoted context omitted.
That's not how I think it works. ML is a small number of operations applied to very large blocks of data, tensors. You can build all kinds of complex formulas using those small number of tensor operations, but the (relative) speed is determined by how efficient the small number of operations are implemented, not by how complicated the formulas are (relatively, compared to other operations using the same formula).
You're half right. First, tensor operations are only a small part of modern ML. Second, how you plug all those small operation together is where all the performance difference is had these days between implementations. Different hardware have a variety of different small operations that do almost the same thing. So when a state of the art model architecture meets a state of the art quantization method and you want to…
Ok, but in the end you're just evaluating a graph, and I suppose that compilers can figure out how to do this in the most efficient way on any type of hardware for which a backend was written. So it makes more sense to work on a backend that you can use for any type of model than to hand-optimize everything.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#220how long before someone creates a simple GUI for this?
That + a small bit of optimisation and everyone with a newer Mac / iPhone will be able to run something akin to chatGPT locally!
Isn't this a pretty crazy development - just weeks ago people said this would be impossible.
From this thread the 13b model runs just as fast as chatGPT on a M2 Macbook Air, and it's not even using the Neural Engine yet so will become significantly faster once that is utilised - wow!