I'm getting 56.38 ms per token on my 32GB M1 Max using this code on the 7GB model.
Very usable!
Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
131–140 of 298 posts
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#132Earlier quoted context omitted.
And 30b: Give me a basic recipe for easy spaghetti carbonara Here is a basic recipe for spaghetti carbonara: Ingredients: 1 lb spaghetti, 3 tbsp butter, 2 tbsp olive oil, 1/4 cup grated Parmesan cheese, 1 egg, 1/2 teaspoon salt, 1/4 teaspoon pepper, 1/2 cup chopped parsley, 1/2 pound bacon, 1 clove garlic, 1/4 cup heavy cream. Directions: Cook spaghetti according to package directions; drain. Heat butter and oil in l…
doesn't say what to do with cream. sprinkling with cheese is a bit wrong. carbonara sauce is more difficult to do than you (or an AI) might think.
Probably because classic Carbonara does not include cream. Including cheese is normal on the other hand.
So, yeah, it's not perfect but it's close to being reasonable.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#133A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…
But wont it be that in real life no one would want to run a voice command which consumes lot of CPU and battery as opposed to making a network call to a service which has this model hosted ?
Agreed that this can always be improved and hardware can get more efficient and better to but at the end of the day, would it ever be better then an API call ?
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#134A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…
> All of the specs indicate an iPhone 14 Pro could achieve similar throughput to an M1 MacBook Pro.
Battery capacity and thermals are different and might be problematic. The phone might throttle performance earlier.
> it looks like we have all the pieces needed to make a ChatGPT-level assistant operate entirely through voice and run entirely on your phone.
As a demo, yes, but would loading the model be fast enough for Siri-like responsiveness? You also would want to run other programs alongside it.
And of course, for Apple to adopt something like this, we would have to get rid of the tendency of these models to derail conversations. Put in something somewhat sexist/racist/…, and it will reply with something a bit more sexist/racist/…)
But yes, it would be a cool demo.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#135Earlier quoted context omitted.
PCIe devices like GPUs can access system memory. Integrated GPUs also access system memory via the same bus as the CPU. It’s not really a new technique. Apple just shipped a highly integrated unit with large memory bandwidth.
That's a pretty important just. They chose to go down this path at this time and shipped, now I can run the 64B Llama on a widely available $3,999 prosumer device. How much would a PC that can do that currently cost me and can I have it by tomorrow?
You can buy a prebuilt pc with a 4090 for less - which is significantly more powerful but still in the 3xxx$.
You could go cheaper with a 3090 which has the same vram and it's just slower.
I think the best combo is a serious Nvidia pc for AI + a cheap MacBook air for portability.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#136Earlier quoted context omitted.
According to Meta's benchmarking[0] it is comparable on many metrics. I haven't used it myself so I can't say for sure if that is the case when actually using it. [0]: https://arxiv.org/pdf/2302.13971.pdf
That's GPT3, not ChatGPT.
I don't understand this topic well, but given premise that GPT3 and ChatGPT are different only that ChatGPT includes RLHF(Reinforcement Learning from Human Feedback), and LLaMA 7b is comparable to GPT3 on a number of metrics, it would follow that if we were to improve LLaMA 7b with RLHF, the 7b model would be similar to ChatGPT. Is that correct?
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#137Earlier quoted context omitted.
That's a pretty important just. They chose to go down this path at this time and shipped, now I can run the 64B Llama on a widely available $3,999 prosumer device. How much would a PC that can do that currently cost me and can I have it by tomorrow?
You can buy a prebuilt pc with a 4090 for less - which is significantly more powerful but still in the 3xxx$. You could go cheaper with a 3090 which has the same vram and it's just slower. I think the best combo is a serious Nvidia pc for AI + a cheap MacBook air for portability.
There's even an Asus laptop with a 4099 for 3999$
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#138Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#139Earlier quoted context omitted.
That's GPT3, not ChatGPT.
I don't understand this topic well, but given premise that GPT3 and ChatGPT are different only that ChatGPT includes RLHF(Reinforcement Learning from Human Feedback), and LLaMA 7b is comparable to GPT3 on a number of metrics, it would follow that if we were to improve LLaMA 7b with RLHF, the 7b model would be similar to ChatGPT. Is that correct?
I’m interested in this as well. Comparatively little attention has been paid to those 7B model results, but they look quite good against 175B GPT-3.
As for ChatGPT, that is GPT-3.5 (same 175B model, but with instruction fine-tuning), plus the RLHF.
Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support
#140Stellar job and it's amazing to have this running in wasm.
I don't get why something like this should be faster than running eg. libtorch in cpu mode, though.
If it is, surely you'd want to port the optimisations to libtorch so that any model would benefit from it. If it's just Mac specific you could even add another target.