Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

271–280 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#271

Post by revscat »

cromka wrote: post #256
andsoitis wrote: post #217

Earlier quoted context omitted.

I'm very interested in this space. Can you share an example that illustrates the difference in "understanding" between the two?

Just recently Siri would belly-up on “Turn off Living Room lightS” — it would only work if I said “light” (singular). Extremely frustrating. They fixed it, I think, but this arbitrariness and many other make me think Siri is more quirk- and algorithms-based than a true AI.

Handling smart home requests is the one thing that Siri seems to do more or less without error, at least for me. I use that multiple times per day per day, and cannot remember the last time that it did not work.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#272

Post by sp332 »

EGreg wrote: post #269

Where do you get the weights? Do you have to fill out the researcher form on FB?

For this model, that is the only "legitimate" route. But since they were leaked, everyone seems to be using them anyway. There are other models you can download from Huggingface.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#273

Post by xen2xen1 »

revscat wrote: post #271
cromka wrote: post #256

Earlier quoted context omitted.

Just recently Siri would belly-up on “Turn off Living Room lightS” — it would only work if I said “light” (singular). Extremely frustrating. They fixed it, I think, but this arbitrariness and many other make me think Siri is more quirk- and algorithms-based than a true AI.

Handling smart home requests is the one thing that Siri seems to do more or less without error, at least for me. I use that multiple times per day per day, and cannot remember the last time that it did not work.

Is Siri better, or does it have you well trained? My smart home stuff works best for me because I know more of the exact labels. I was literally surprised the other day that my wife included an S and it still worked.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#274

Post by circuit10 »

guestbest wrote: post #72
RoboTeddy wrote: post #70

Earlier quoted context omitted.

LLaMA isn't built on RLHF, so it may be necessary to create a more extensive prompt. For example: ``` You are a super intelligent honest question-answering system. Q: What's 2+2? A: 4 Q: What color is the sky? A: ```

This is Commodore 64 tier answers

That's just an example

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#275

Post by lgas »

guestbest wrote: post #72
RoboTeddy wrote: post #70

Earlier quoted context omitted.

LLaMA isn't built on RLHF, so it may be necessary to create a more extensive prompt. For example: ``` You are a super intelligent honest question-answering system. Q: What's 2+2? A: 4 Q: What color is the sky? A: ```

This is Commodore 64 tier answers

That's the point, even C64 tier examples in one shot or few shot learning do wonders for changing the behavior of the model.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#276

Post by EGreg »

sp332 wrote: post #272
EGreg wrote: post #269

Where do you get the weights? Do you have to fill out the researcher form on FB?

For this model, that is the only "legitimate" route. But since they were leaked, everyone seems to be using them anyway. There are other models you can download from Huggingface.

What are the links?

Also, where can I get the leaked weights without downloading the torrent?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#277

Post by smoldesu »

brookst wrote: post #244
KVFinn wrote: post #172

Earlier quoted context omitted.

>How much would a PC that can do that currently cost me and can I have it by tomorrow? At the moment, seems like Apple has an edge here. On PC for single GPU you need an NVIDIA A40, which used prices for is about $2500, and not at retail stores. If you don't mind having two GPUs then two $800 3090 GPUs works, but that's a workstation build you'll have to order from Puget or something. That's probably faster than Appl…

…and for models that require 64GB of VRAM? 120GB of VRAM? You can get a 128GB UMA mac for less than a single 48GB a100, let alone a single 96GB a100. I think Apple got incredibly lucky here, but I don’t see how the PC world catches them any time soon. We’ve all known that UMA is theoretically better for ages, but Apple’s timing couldn’t be better. And scale economies mean they can sell the same chip to people who nee…

> they could snipe ML away from nvidia.

If Nvidia announced tomorrow that they were cancelling every datacenter deal they had, open-sourcing CUDA and publishing their entire patent library to the creative commons, I would still not believe you.

This is a fun project for people with Apple Silicon machines who want to participate in the AI happenings, but I don't think you can warp it into a call for Nvidia's head. Let's wait until Apple pulls the curtains on their rackmount Mac Pros, so we can compare it with Nvidia's ARM server offerings: https://www.nvidia.com/en-us/data-center/grace-cpu/

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#278

Post by DennisAleynikov »

7to2 wrote: post #259
fooker wrote: post #255

Earlier quoted context omitted.

It’s amazing how Apple “doesn’t do” anything, but manages to define industry trends over and over again!

That's what brand recognition and brand loyalty (and the money they generate) allows you to do. This is not necessarily a bad thing since it forces other competitors - who like to artificially limit their products - to actually get of their greedy asses and compete.

Yeah which is why... Unified memory on Chromebooks and android devices as well as surface pro serve the same purpose.

It's marketed as an upsell by apple because it's an advantage of ARM but many people will never care to learn what that means

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#279

Post by DennisAleynikov »

fwlr wrote: post #266
ggerganov wrote: post #258

Earlier quoted context omitted.

Our investigations indicate that it might not be possible to achieve ANE performance improvement over CPU for LLM Decoder inference with batch size of 1 [0]. Just to make it clear - I'm no expert in Core ML / ANE, so these conclusions could be totally wrong. [0] https://github.com/ggerganov/whisper.cpp/discussions/548#dis...

Don’t sell yourself short! (And you have my apologies in advance if my excited comment above has created any extra work for you)

Neural Engine across the M1 and M2 series is also sadly very limited.

I bought one thinking I could exploit it for StableDiffusion and other tasks but found that most libraries say to use GPU for faster generation. What I found is not only is the engine the same on m2 pro (meaning I upgraded for no reason from my m1 basemodel) but it also doesn't scale at all except in the m1 Ultra where it's doubled simply because it's using two dies bridged.

Neural Engine can generate 512x512 images pretty easily but takes a while even compared to using the GPU on a basemodel m1 Mac Mini. It's kinda crazy. Looking into ways to improve it and take advantage of the neural engine in the future but the current situation is very limited. Even apples official implementation and coreML libraries seem to prefer you run them on Metal

Post reply on HN