Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

251–260 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#251

Post by Taek »

inawarminister wrote: post #159
popinman322 wrote: post #141

Earlier quoted context omitted.

You're likely right that applying RLHF (+ fine-tuning with instructions) to LLaMA 7b would produce results similar to ChatGPT, but I think you're implying that that would be feasible today. RLHF requires a large amount of human feedback data and IIRC there's no open data set for that right now.

There's open-assistant.io, which is doing RLHF directly on the open

And they've already collected over 100,000 samples, iirc ChatGPT was trained on something like 30,000 samples, so the open models should already be positioned to succeed.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#252

Post by Taek »

MacsHeadroom wrote: post #223
fwlr wrote: post #178

Earlier quoted context omitted.

Oh, it’s probably higher than four words per second, then. I assumed tokens was characters and used the standard “there are five characters in a word” rule of thumb.

It's about 4 charcters per token. So just over 1 token per word. I just round to 1 token per word since text most people generate does not use larger words and because larger common words are still encoded as one token (e.g. HackerNews is probably one token despite being 10 characters).

I typically see people claim 2-3 tokens per word.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#253

Post by Centigonal »

ar9av wrote: post #250
fwlr wrote: post #104

A quick survey of the thread seems to indicate the 7b parameter LLaMA model does about 20 tokens per second (~4 words per second) on a base model M1 Pro, by taking advantage of Apple Silicon’s Neural Engine. Note that the latest model iPhones ship with a Neural Engine of similar performance to latest model M-series MacBooks (both iPhone 14 Pro and M1 MacBook Pro claim 15.8 teraflops on their respective neural engines…

the potential drawbacks of relying entirely on voice-operated assistants like ChatGPT. There are concerns around privacy and the use of personal data, as well as the potential for bias and inaccuracies in the responses generated by these models. It's important to strike a balance between the convenience and benefits of these technologies and the potential risks and limitations they bring. Nonetheless, the advancement…

That's very ChatGPT of you to say!

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#255

Post by fooker »

DennisAleynikov wrote: post #245
Nevermark wrote: post #88

Earlier quoted context omitted.

Given that the RAM is directly accessible by CPU, GPU and DPU/Neural cores, it really is premium RAM. Apple's visionary hardware team has finally caught up with Apple's visionary high RAM prices! :)

Also um... Idk if anyone cares but apple didn't do anything to make unified memory. All arm chips do this lmao Apple just has the best known arm chips with the highest mobile performance (yes faster server arm chips exist too)

It’s amazing how Apple “doesn’t do” anything, but manages to define industry trends over and over again!

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#256

Post by cromka »

andsoitis wrote: post #217
j45 wrote: post #210

Earlier quoted context omitted.

Really? I find Siri can’t understand anything slightly more than basic instructions. Google assistant can seem to do more

I'm very interested in this space. Can you share an example that illustrates the difference in "understanding" between the two?

Just recently Siri would belly-up on “Turn off Living Room lightS” — it would only work if I said “light” (singular). Extremely frustrating. They fixed it, I think, but this arbitrariness and many other make me think Siri is more quirk- and algorithms-based than a true AI.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#257

Post by ngrilly »

hnfong wrote: post #198
ngrilly wrote: post #193

The README says it has been "hacked in an evening"! Have the 10.000 lines of ggml.c been written in an evening, without reusing a lot of pre-existing code? That's quite amazing.

I thought that as well. But then I noticed the same author has an existing project with a ggml.c ( https://github.com/ggerganov/whisper.cpp/blob/master/ggml.c 8000+ lines), so maybe he "just" wrote 2000 lines in an evening. Sounds much more "humanly possible", but still amazing.

Makes more sense. But as you said, still amazing!

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#258

Post by ggerganov »

fwlr wrote: post #247
DennisAleynikov wrote: post #241

Earlier quoted context omitted.

If I may, this library runs LLaMA on CPU. There is no way to run it on the Neural Engine yet. The optimization in this case only seems to refer to the 4bit model loading method (to be friendlier to the arm64 CPU) GeoHot has tinygrad running LLaMA on Metal (but only the 7B model) that's the closest I've seen to taking advantage of apple silicon. Neural Engine implementation would be awesome

Oh shit, I took a closer look and you’re right. The repo was also helpfully updated with a note to this effect: “The Accelerate framework is actually currently unused since I found that for tensor shapes typical for the Decoder, there is no benefit compared to the ARM_NEON intrinsics implementation. Of course, it's possible that I simlpy don't know how to utilize it properly. But in any case, you can even disable it…

Our investigations indicate that it might not be possible to achieve ANE performance improvement over CPU for LLM Decoder inference with batch size of 1 [0]. Just to make it clear - I'm no expert in Core ML / ANE, so these conclusions could be totally wrong.

[0] https://github.com/ggerganov/whisper.cpp/discussions/548#dis...

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#259

Post by 7to2 »

fooker wrote: post #255
DennisAleynikov wrote: post #245

Earlier quoted context omitted.

Also um... Idk if anyone cares but apple didn't do anything to make unified memory. All arm chips do this lmao Apple just has the best known arm chips with the highest mobile performance (yes faster server arm chips exist too)

It’s amazing how Apple “doesn’t do” anything, but manages to define industry trends over and over again!

That's what brand recognition and brand loyalty (and the money they generate) allows you to do. This is not necessarily a bad thing since it forces other competitors - who like to artificially limit their products - to actually get of their greedy asses and compete.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#260

Post by Alifatisk »

jjkeddo199 wrote: post #75

Insanity! This is the same guy who wrote Whisper C++! How does he do this? I feel like I am a side character in some cartoon gasping at the unthinkable power level of the main character.

What’s even more impressive is this part

> This was hacked in an evening

Post reply on HN