Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

281–290 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#281

Post by [deleted] »

MacsHeadroom wrote: post #225
gpm wrote: post #197

Earlier quoted context omitted.

Yes (I just don't have that much ram) I have a separate branch that streams weights from ram - at which point I think I was only seeing negligible performance loss compared to storing the weights in vram. The bottleneck was compute, not GPU bandwidth.

The 65B model only needs just over 32GB of VRAM to run. It does not need system RAM to run/use if you use pre-quantized weights which you can find many places already. No need to quantize yourself (besides it takes almost a day to do 4bit GPTQ quantization on 3xA6000).

[deleted]

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#282

Post by int_19h »

fnbr wrote: post #34
TaylorAlexander wrote: post #26

Earlier quoted context omitted.

Well I was able to run the original code with the 7B model on 16GB vram: https://news.ycombinator.com/item?id=35013604 The output I got was underwhelming, though I did not attempt any tuning.

parameter tuning is pretty necessary, according to anecdotes. People on twitter have got good results by changing the default parameters.

For 13b and 30b, it really needs high temperature to produce good outputs.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#283

Post by Tepix »

minxomat wrote: post #158
MacsHeadroom wrote: post #148

Earlier quoted context omitted.

LLaMA-65B fits in 32GB of VRAM using state of the art GPTQ quantization with no output performance loss. https://github.com/qwopqwop200/GPTQ-for-LLaMa

So if I'm reading this right, 65B at 4bit would consume around 20GB of VRAM and ~130GB of system RAM?

Closer to 38-40GB VRAM (and hardly any RAM).

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#284

Post by Tepix »

irusensei wrote: post #213
Tepix wrote: post #142

Earlier quoted context omitted.

Privacy concerns are justified. It's not just that, this can also work completely offline.

I'm looking forward to run stuff like this online. Using bigtech corporate souls SaaS AI is just pure dystopia material. It's even better that we are talking about a relatively low power machine here. Maybe can operate offered.

You mean offline?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#285

Post by wkat4242 »

revscat wrote: post #271
cromka wrote: post #256

Earlier quoted context omitted.

Just recently Siri would belly-up on “Turn off Living Room lightS” — it would only work if I said “light” (singular). Extremely frustrating. They fixed it, I think, but this arbitrariness and many other make me think Siri is more quirk- and algorithms-based than a true AI.

Handling smart home requests is the one thing that Siri seems to do more or less without error, at least for me. I use that multiple times per day per day, and cannot remember the last time that it did not work.

Mine is really really poor at it.

Half the time it responds with "one moment.. One moment.. this is taking too long" or "I have problems connecting to the internet". But there's no internet problems whatsoever and it connects to my home Assistant using local homekit integration which shouldn't even need that.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#286

Post by Tepix »

gpm wrote: post #238
MacsHeadroom wrote: post #225

Earlier quoted context omitted.

The 65B model only needs just over 32GB of VRAM to run. It does not need system RAM to run/use if you use pre-quantized weights which you can find many places already. No need to quantize yourself (besides it takes almost a day to do 4bit GPTQ quantization on 3xA6000).

Quantizing is a lossy process, you can't really claim to be running the 65B model llama at that point (though the 65b qgpt-llama does look like it might be very useful)

The GPTQ paper https://arxiv.org/abs/2210.17323 claims "negligible accuracy degradation relative to the uncompressed baseline".

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#287

Post by Tepix »

MacsHeadroom wrote: post #225
gpm wrote: post #197

Earlier quoted context omitted.

Yes (I just don't have that much ram) I have a separate branch that streams weights from ram - at which point I think I was only seeing negligible performance loss compared to storing the weights in vram. The bottleneck was compute, not GPU bandwidth.

The 65B model only needs just over 32GB of VRAM to run. It does not need system RAM to run/use if you use pre-quantized weights which you can find many places already. No need to quantize yourself (besides it takes almost a day to do 4bit GPTQ quantization on 3xA6000).

Are you sure? I think it took a mere 2 hours to do 4bit GPTQ quantization of LLaMA-65B on 1x RTX 3090. But i may be mistaken.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#288

Post by imtringued »

alphanullmeric wrote: post #73
recuter wrote: post #60

> Currently, only LLaMA-7B is supported since I haven't figured out how to merge the tensors of the bigger models. However, in theory, you should be able to run 65B on a 64GB MacBook Suddenly the choices Apple made with its silicon are looking like pure genius as there will be a lot of apps using this that are essentially exclusive to their platform. Even with the egregious ram pricing. With a lot of fine tuning if y…

Wonder why AMD/Intel/Nvidia haven’t invented some sort of device that allows the processor and graphics to share memory like Apple has done.

Nvidia didn't implement it in their OpenCL drivers because OpenCL's shared virtual memory spec is so terrible that khronos made it optional in OpenCL 3.0 and Intel (the only company actually having high quality OpenCL implementations) dropped shared virtual memory like a hot potato and instead introduced their own unified shared memory extension for OpenCL which they use for SyCL and oneAPI.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#290

Post by brookst »

smoldesu wrote: post #277
brookst wrote: post #244

Earlier quoted context omitted.

…and for models that require 64GB of VRAM? 120GB of VRAM? You can get a 128GB UMA mac for less than a single 48GB a100, let alone a single 96GB a100. I think Apple got incredibly lucky here, but I don’t see how the PC world catches them any time soon. We’ve all known that UMA is theoretically better for ages, but Apple’s timing couldn’t be better. And scale economies mean they can sell the same chip to people who nee…

> they could snipe ML away from nvidia. If Nvidia announced tomorrow that they were cancelling every datacenter deal they had, open-sourcing CUDA and publishing their entire patent library to the creative commons, I would still not believe you. This is a fun project for people with Apple Silicon machines who want to participate in the AI happenings, but I don't think you can warp it into a call for Nvidia's head. Let…

Whoa, who’s calling for nvidia’s head? Not me.

My point was that the PC architecture of separate system and GPU memory is hitting a wall that means inefficiency and higher prices.

I have little doubt that Nvidia’s attempted acquisition of ARM was in part because nvidia recognized this. I expect they are exploring other UMA approaches. But it will be hard in the fragmented, not-vertically-integrated model.

Apple’s advantage here is one platform that can scale: it is hard to imagine Grace and similar running Windows on developer’s desktops. Maybe!

But my point was that, shockingly, Apple has a chance here. A small chance, as I said, but I don’t think anyone (including Apple) saw just how soon UMA was going to become a competitive advantage.

Post reply on HN