Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Is LLaMa.cpp Possible?

finbarr.ca

Post reply on HN
Print view View original

211–220 of 238 posts

Re: How Is LLaMa.cpp Possible?

#211

Post by axus »

class4behavior wrote: post #141
DiggyJohnson wrote: post #136

Earlier quoted context omitted.

I’ve never been a regular Twitter user, and don't really enjoy the platform, but this comment of yours is either abusing the word “shameful” or betraying a major lack of understanding that you can’t expect other people to care deeply about the things you care deeply about. There’s a lot of shameful things in this world, GP using twitter isn’t one of them. Not even a “little.”

It's betraying the word shameful in that it's an utter understatement. If people don't care about supporting companies that enable and spread far-right content and groups, it's they who are a problem. To say nothing of the pure disregard of the human right to privacy (and with AI now IP as well) that is forced on the the rest of the world by the dominance of the US market.

There are a lot of oil producers I consider shameful, but I'm still going to buy gasoline every few weeks.

Re: How Is LLaMa.cpp Possible?

#212

Post by brucethemoose2 »

baby_souffle wrote: post #205
tagyro wrote: post #187

Earlier quoted context omitted.

Can you please share more info on this? I have a 6900xt "gathering dust" in a proxmox server - would like to try to do a passthrough to a vm and use it. Thank you in advance!

Same boat. I’d love to host something local but have been so overwhelmed by the rapid progress and every time I start looking I find a guide that inevitably has a “then plug in your OpenAI api key…” step which is a hard NOPE for me. I have a few decent gpus but I’ve got no idea where to start…

Path of least resistance:

- Download koboldcpp: https://github.com/LostRuins/koboldcpp

- Download your 70B ggml model of choice, for instance airoboros 70B Q3_K_L: https://huggingface.co/models?sort=modified&search=70b+ggml

- Run Koboldcpp with opencl (or rocm) with as many layers as you can manage on the GPU. If you use rocm, you need to install the rocm package from your linux distro (or direct from AMD on Windows).

- Access the UI over http. Switch to instruct mode and copy in the correct prompt formatting from the model download page.

- If you are feeling extra nice, get an AI Horde API key and contribute your idle time to the network, and try out other models on from other hosts: https://lite.koboldai.net/#

Re: How Is LLaMa.cpp Possible?

#213

Post by Tokumei-no-hito »

cameron_b wrote: post #11

I’ve been working through that repo and managed the 13B dataset on a single Pi4 8gig I’ve also replicated the work in OpenMPI ( from a thread on the llama.cpp GitHub repo ) and today I managed to get the 65B dataset operational on three pi4 nodes. I’m not saying this as any achievement of mine, but as a comment on the current reality of reproducible LLM At home on anything you’ve got. It really feels like this techni…

Nice write up man. Thanks for sharing your research

Re: How Is LLaMa.cpp Possible?

#214

Post by jokethrowaway »

class4behavior wrote: post #141
DiggyJohnson wrote: post #136

Earlier quoted context omitted.

I’ve never been a regular Twitter user, and don't really enjoy the platform, but this comment of yours is either abusing the word “shameful” or betraying a major lack of understanding that you can’t expect other people to care deeply about the things you care deeply about. There’s a lot of shameful things in this world, GP using twitter isn’t one of them. Not even a “little.”

It's betraying the word shameful in that it's an utter understatement. If people don't care about supporting companies that enable and spread far-right content and groups, it's they who are a problem. To say nothing of the pure disregard of the human right to privacy (and with AI now IP as well) that is forced on the the rest of the world by the dominance of the US market.

It's both you and right wing people who are the problem.

You're both puppets in the hands of the powerful who wants us divided and weak.

Don't trust authority. Don't trust anyone. In the past left wing people cared and fought about freedom, today is the right wing fighting for freedom.

It's all irrelevant anyway, governments keep growing stronger and stronger during right or left governments. And soon there is not going to be anywhere to run to.

Re: How Is LLaMa.cpp Possible?

#215

Post by lhl »

SleekEagle wrote: post #201

Fantastic article - I would love to see an analysis comparing larger quantized models to smaller unquantized models. e.g. is a 14b quantized model better than a 7b unquantized model?

Basically, the most aggressive quants for a larger sized model is always better than the unquantized model of a smaller model: https://github.com/ggerganov/llama.cpp/pull/1684

Tim Dettmers did a bunch of research on this last year as well: https://arxiv.org/abs/2212.09720

Re: How Is LLaMa.cpp Possible?

#216

Post by throwing_away »

tagyro wrote: post #187
throwing_away wrote: post #178

Earlier quoted context omitted.

ROCm works now! I just set it up tonight on a 6900xt with 16gb vram running wayland at the same time. The trick was using the opencl-amd package (somehow rocm packages don't depend on opencl, but llama does, idk). I'm astonished at the results I can get from the q6_K models.

Can you please share more info on this? I have a 6900xt "gathering dust" in a proxmox server - would like to try to do a passthrough to a vm and use it. Thank you in advance!

Sure thing. There's a bunch of ways to do it, but here's some quick notes on what I did.

* arch linux has tons of `rocm` packages. I installed pretty much all of them: https://archlinux.org/packages/?sort=&q=rocm&maintainer=&fla...

* you also need this one package from AUR: https://aur.archlinux.org/packages/opencl-amd

* llama.cpp now has GPU support including "CLBlast", which is what we need for this, so compile with LLAMA_CLBLAST=ON

* now you can run any model llama.cpp supports, so grab some ggml models that fit on the card from https://huggingface.co/TheBloke.

* Test it out with: ./main -t 30 -ngl 128 -m huginnv1.2.ggmlv3.q6_K.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "### Instruction: Write a story about llamas\n### Response:"

* You should see `BLAS = 1` in the llama.cpp output and you should get maybe 5 tokens per second on a 13b 6bit quantized ggml model.

* You can compile llama-cpp-python with the same arguments and get text-generation-ui to work also, but there's a bit of dependency fighting to do it.

* koboldcpp might be better, I just haven't tried it yet

Hope that helps!

Edit: just tried https://github.com/LostRuins/koboldcpp and it also works great. I should have started here probably.

Compile with `make LLAMA_CLBLAST=1` run with ` python koboldcpp.py --useclblast 0 0 --gpulayers 128 huginnv1.2.ggmlv3.q6_K.bin`

Re: How Is LLaMa.cpp Possible?

#217

Post by chucksmash »

class4behavior wrote: post #141
DiggyJohnson wrote: post #136

Earlier quoted context omitted.

I’ve never been a regular Twitter user, and don't really enjoy the platform, but this comment of yours is either abusing the word “shameful” or betraying a major lack of understanding that you can’t expect other people to care deeply about the things you care deeply about. There’s a lot of shameful things in this world, GP using twitter isn’t one of them. Not even a “little.”

It's betraying the word shameful in that it's an utter understatement. If people don't care about supporting companies that enable and spread far-right content and groups, it's they who are a problem. To say nothing of the pure disregard of the human right to privacy (and with AI now IP as well) that is forced on the the rest of the world by the dominance of the US market.

https://pbfcomics.com/comics/deeply-held-beliefs/

Re: How Is LLaMa.cpp Possible?

#218

Post by enterprise_cog »

Waterluvian wrote: post #14
KerrAvon wrote: post #10

Earlier quoted context omitted.

First, there has to be something capable of matching human performance at a much higher cost. This is still just spicy autocomplete.

Humans just do spicy autocomplete too.

Maybe you do. Luckily, not everyone is quite so simple.

Re: How Is LLaMa.cpp Possible?

#219

Post by brucethemoose2 »

throwing_away wrote: post #216
tagyro wrote: post #187

Earlier quoted context omitted.

Can you please share more info on this? I have a 6900xt "gathering dust" in a proxmox server - would like to try to do a passthrough to a vm and use it. Thank you in advance!

Sure thing. There's a bunch of ways to do it, but here's some quick notes on what I did. * arch linux has tons of `rocm` packages. I installed pretty much all of them: https://archlinux.org/packages/?sort=&q=rocm&maintainer=&fla... * you also need this one package from AUR: https://aur.archlinux.org/packages/opencl-amd * llama.cpp now has GPU support including "CLBlast", which is what we need for this, so compile wit…

Text gen ui is nice. Some specific nicities include pre formatted instruct templates for popular modules, good prompt caching from llama-cpp-python, and integration of a vector db.

But its also finicky, kinda unstable and the dependencies are tricky.

Koboldcpp has other nicities, like some different generation parameters to tweak and some upstream features pulled in from PRs before the official llama.cpp release has them. The UI is nice, predating llamav1. Its standalone, dead simple to compile and has integration with AI Horde, which is (IMO) a huge essential feature.

Re: How Is LLaMa.cpp Possible?

#220

Post by not2b »

dTal wrote: post #100
PaulDavisThe1st wrote: post #89

Earlier quoted context omitted.

one reason why a human is different: just based on word count alone, most LLM's are trained on 3-5 orders of magnitude more input. could be a difference that makes no difference, or ...

Bit of an unfair comparison when humans also have a bunch of senses that LLMs don't have. They might be trained on orders of magnitude more words , but more data? Doubtful.

That's the key. I'm reminded of the Helen Keller story. She was completely blind and deaf. Her teacher spent a very long time signing into her hand. It took a very long time before she realized that the sign for "water" designated the thing she could feel flowing onto her hand; before that breakthrough the signs were meaningless to her. An LLM only knows the structure of language. It doesn't know that there is an external physical world that the language refers to. It only can predict what words follow which other words, and which output is preferred. Without any senses (and the huge bandwidth of information provided by them) an LLM is very crippled.
Post reply on HN