Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

181–190 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#181

Post by minxomat »

MacsHeadroom wrote: post #171
minxomat wrote: post #158

Earlier quoted context omitted.

So if I'm reading this right, 65B at 4bit would consume around 20GB of VRAM and ~130GB of system RAM?

LLaMA it doesn't require any system RAM to run. It requires some very minimal system RAM to load the model into VRAM and to compile the 4bit quantized weights. But if you use pre-quantized weights (get them from HuggingFace or a friend) then all you really need is ~32GB of VRAM and maybe around 2GB of system RAM for 65B. (It's 30B which needs 20GB of VRAM.)

The full use case includes quantisation, which the repo points out uses a large amount of system RAM. Of course that’s not required if you skip that step.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#182

Post by snickmy »

fwlr wrote: post #104

A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…

I wish we could start having open source TTS models with similar performance. So far Tortoise TTS is not there yet. Im not sure if Siri neural TTS is offered for 3rd party apps.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#183

Post by snickmy »

gorbypark wrote: post #179

I can confirm that this (7B) runs nicely on a 24GB MacBook Air M2. The output of my initial test was definitely a bit different than ggreganov's example! The first man on the moon was 39 years old on July 16, 1969. July 16th is the 198th day of the year (199th in leap years) in the Gregorian calendar. There are 168 days remaining until the end of the year. 1561 – France is divided into 2535 circles (French: cercles)…

The self attention of this model needs some tuning :) It reminds me how sometimes I have stream of consciousness while talking, and I have ADHD.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#184

Post by gitfan86 »

fwlr wrote: post #104

A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…

I would also expect 10x improvements over the next year due to optimizations found throughout the stack.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#185

Post by gorbypark »

snickmy wrote: post #183
gorbypark wrote: post #179

I can confirm that this (7B) runs nicely on a 24GB MacBook Air M2. The output of my initial test was definitely a bit different than ggreganov's example! The first man on the moon was 39 years old on July 16, 1969. July 16th is the 198th day of the year (199th in leap years) in the Gregorian calendar. There are 168 days remaining until the end of the year. 1561 – France is divided into 2535 circles (French: cercles)…

The self attention of this model needs some tuning :) It reminds me how sometimes I have stream of consciousness while talking, and I have ADHD.

Yeah, definitely. The second run was similarly off the rails..

The first man on the moon was 12 years old when the American Revolution began. As a boy, he was horrified by the sight of his mother being whipped. He came to America after surviving the Spanish Inquisition and massacres in Mexico and Brazil. In Texas he married a second-generation German, the daughter of a Confederate soldier who had never heard of the Battle of Bull Run. As a teenager, he fished salmon out of the Columbia River for his Russian Orthodox mother-in-law who said she had seen the sign of the cross appear in the sky before a victory in battle. He followed the death

Still impressive for a 7B model that's been slimmed down to 4 bits.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#186

Post by return_to_monke »

fwlr wrote: post #104

A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…

i think voice assistants can perform actions on phones (eg "open app, message Alice, call Bob, turn off Bluetooth"). This couldn't do that (I think), which is an obvious drawback

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#187

Post by gorbypark »

stu2b50 wrote: post #50
0xbadc0de5 wrote: post #24

That's all fine and good. But to do anything useful, you're going to want a powerful GPU (RTX 3090, RTX 4090 or A6000) with as much VRAM as possible. Unlike the diffusion models, LLM's are very memory-intensive, even at 4-bit GPTQ. The larger models like llama-13b and llama-30b run quite well at 4-bit on a 24GB GPU. The llama-65b-4bit should run on a dual 3090/4090 rig. Coupled with the leaked Bing prompt and text-ge…

VRAM is the thing that Apple Silicon is going to have in excess compared to anything even close in price. MacBook Airs can have 14-15GB of VRAM if necessary.

Is there an upper limit on the usable VRAM on Airs, or is it just tied to RAM size minus a bit for the OS? Just got a 24GB Air M2 and your comment made me wonder if it was capped at n amount of VRAM, or if it's just that Air's now can have 24GB of ram compared to 16GB with the M1 Airs?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#189

Post by crimjow »

renewiltord wrote: post #117

Quick steps I did to get it working 1. Download the models from here https://github.com/shawwn/llama-dl/blob/main/llama.sh (you'll need to edit the script to make it work in bash) 2. Follow the instructions here. 3. Create an interesting prompt for the AI and put your question at the end. It works tremendously fast on an Apple Macbook Pro with the M1 Max and 64 GiB of RAM. I only downloaded the 7B model because I wan…

It even maintained your spelling errors.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#190

Post by garblegarble »

2Gkashmiri wrote: post #160
nl wrote: post #131

I'm getting 56.38 ms per token on my 32GB M1 Max using this code on the 7GB model. Very usable!

what model are you using? edit: i mean 6B, 13b or 30B?

It'll be 7B they're referring to, on my M1 Max 32GB with a 4000 token output request I get 67ms/token on 7B (4bit) and 154ms/token on 13B (4bit)... I've made a tweak to the code to increase the context size but it doesn't seem to change perf.

  main: mem per token = 22357508 bytes
  main:     load time =  2741.67 ms
  main:   sample time =   156.68 ms
  main:  predict time = 11399.12 ms / 154.04 ms per token
  main:    total time = 14914.39 ms
Post reply on HN