Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Is LLaMa.cpp Possible?

finbarr.ca

Post reply on HN
Print view View original

231–238 of 238 posts

Re: How Is LLaMa.cpp Possible?

#231

Post by loxias »

cameron_b wrote: post #194
loxias wrote: post #107

Earlier quoted context omitted.

> I’ve also replicated the work in OpenMPI... Oh cool! How did it perform? I wonder if this would be an exciting test for Amazon's SRD protocol which appears to be built for HPC. I'm looking for an excuse to play with it...

The objective performance I'm getting is flat poor, mostly because of the network I'm using. On the other hand, simply being able to do it at all with one node on wireless until I can pull another drop, and the rest being on 100 Mbit ... I'm really running a bargain basement cluster. I don't know about SRD, but llama.cpp has MPI configurations built-in. I didn't have to engineer anything or rewrite anything ( I made…

I eagerly await your numbers! Maybe I'll post some of my own if I can get far enough ahead at work.

I was thinking it's time to upgrade from 1GigE anyway, 10GigE is cheap and at work we're ripping that out in favor of 25 and 50...

I'll look at the code, depending on how well the authors used MPI it could be exciting times! It's not that hard (or expensive) to get a bunch of power hungry used servers off ebay and string em together with a cheap 10GigE switch. It would be loud and power hungry but I wonder if I could have a 65B local model in the privacy of my own home, for fractions of the cost of buying a A100....

Edit: Oh, and SRD is a ... network protocol designed to work hand in hand with EFA which can substantially improve the performance of HPC MPI workloads, running on EC2, during network bound phases.

Re: How Is LLaMa.cpp Possible?

#232

Post by loxias »

redox99 wrote: post #58
blovescoffee wrote: post #54

Earlier quoted context omitted.

Am I missing something or how do you know this? Also I think the OP was talking about a single card not multiple but that was just my reading.

Because 175B parameters (350GB for the weights FP16, let's say a bit over 400GB for actual inference), fit very comfortably on 8xA100 (640GB VRAM total). And basically all servers will have 8xA100 ( maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one)

> And basically all servers will have 8xA100 (maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one)

wishing, or guessing something, without actual experience, doesn't make it true.

Re: How Is LLaMa.cpp Possible?

#233

Post by oblio »

lovelyviking wrote: post #228
oblio wrote: post #166

Earlier quoted context omitted.

Because. On a more serious node, your questions seem very... aggressive. Especially the one about the name, is the name offensive or what? To me it sounds quite benign.

As someone unfamiliar with the field I’ve read github page and didn’t find answers to things I was curious about regarding this project. So out of curiosity I wrote simple basic questions: “Why this tool was created? What was the reason for creating such tool? Why it was named this way?“ Suggesting that if I didn’t figured it out by myself perhaps some one else who also wish to join the field and looking for entry po…

Your comment has been flagged and my comment has received a bunch of upvotes. Multiple people consider your wording or tone aggressive.

In case you're a person with social issues, asking so many questions in a succession is considered quite rude.

Space them out, in writing add newlines. Or add more context to your questions.

Your comment history suggests that you do this often and you have a ton of downvoted comments.

Friendly advice :-)

Re: How Is LLaMa.cpp Possible?

#234

Post by Roark66 »

gliptic wrote: post #182
Roark66 wrote: post #138

It is useful to mention running inference on modern cpus that have AVX2 is not that bad. Sure it is slower than on the gpu, but you get the benefit of having a single long continuous region of ram. But there is one huge problem why this is not that popular on x86_64. Having to run in fp32. As far as I know our most common ml libraries (pytorch, tf, onnx etc) do not have an option to quantize to 4 bits and they don't…

Sorry, but the topic of this post, llama.cpp, runs quantized 4/8 bit models just fine on x86_64 with AVX2, or am I missing some requirement you have?

Wait, I wasn't aware llama.cpp even runs on x86_64.i thought it is arm hw only. If what you say is correct that indeed is very interesting. Especially if I can extend it to other models like falcon.

Re: How Is LLaMa.cpp Possible?

#235

Post by eurekin »

ionwake wrote: post #181
eurekin wrote: post #81

Earlier quoted context omitted.

I'd look for llama2 thebloke 70b GGML on hugginface. Update, this might work: https://huggingface.co/TheBloke/Llama-2-70B-GGML Example command mentions the 4bit variant: ./main -m llama-2-70b.ggmlv3.q4_0.bin -gqa 8 -t 13 -p "Llamas are"

I also found this link ( https://blog.lastmileai.dev/run-llama-2-locally-in-7-lines-a... ) but havent verified if it helps, but thanks for your reply I am taking a look now.

Just checked on a m1 w 32GB ram. 7b and 13b work as a charm. 70b doesn't fit, even the 2 bit version

Re: How Is LLaMa.cpp Possible?

#236

Post by gliptic »

Roark66 wrote: post #234
gliptic wrote: post #182

Earlier quoted context omitted.

Sorry, but the topic of this post, llama.cpp, runs quantized 4/8 bit models just fine on x86_64 with AVX2, or am I missing some requirement you have?

Wait, I wasn't aware llama.cpp even runs on x86_64.i thought it is arm hw only. If what you say is correct that indeed is very interesting. Especially if I can extend it to other models like falcon.

It doesn't support Falcon right now, but there's a fork that does (https://github.com/cmp-nct/ggllm.cpp/).

Re: How Is LLaMa.cpp Possible?

#237

Post by simne »

I'm shocked - before I seen it working myself, I was not believe, so large (smart) LLM could run on desktop cpu.

- core i7, 4 cores, 3.3GHz, 64G ddr4-2400, 70b model, ~0.5 tokens/s; 30b model ~1 tokens/s.

Re: How Is LLaMa.cpp Possible?

#238

Post by lovelyviking »

oblio wrote: post #233
lovelyviking wrote: post #228

Earlier quoted context omitted.

As someone unfamiliar with the field I’ve read github page and didn’t find answers to things I was curious about regarding this project. So out of curiosity I wrote simple basic questions: “Why this tool was created? What was the reason for creating such tool? Why it was named this way?“ Suggesting that if I didn’t figured it out by myself perhaps some one else who also wish to join the field and looking for entry po…

Your comment has been flagged and my comment has received a bunch of upvotes. Multiple people consider your wording or tone aggressive. In case you're a person with social issues, asking so many questions in a succession is considered quite rude. Space them out, in writing add newlines. Or add more context to your questions. Your comment history suggests that you do this often and you have a ton of downvoted comments…

Usually friendly advice is given after friendly answering question, which was not demonstrated here.

We are perfectly aware of how to deal with extra-sensitive people and kind approach is the way to go. However on this site I expect certain level of technical training and intelligence in answering direct questions without being offended by search for a deeper understanding of some topic and without inventing emotions that never been there in the first place. I was also under impression that guidelines of this site directly where encouraging answering I expect by declaring : “be kind” “don’t be rude” “suggest a good faith” and unfortunately none of those were demonstrated by your answers even when you have learned the fact that you were wrong and your accusations were completely ungrounded . The fact that such behaviour was encouraged by (as you claim it) “multiple people” including moderation of this site tells a very sad story about this site regarding following own declared principles.

And this example:

“Which crypto? Where can I pay with it? What if I don't want to have a hardware wallet everywhere with me? Can I cancel transactions and get my money back? Does it use more energy per year than Argentina?” (https://news.ycombinator.com/item?id=36656667)

tells another sad story about hypocrisy which means it’s hard to take “friendly“ advice seriously. The next time you wish to give a truly friendly advice try to be kind in the first place, do not suggest a bad faith and do not insist on false ungrounded accusations encouraged by confirmation biases especially when clarification presented.

You could have just apologise for own real aggression contrary to imaginative aggression from my questions and simply give answer while keeping “friendly” advice to yourself.

This way people would learn something about the project rather than learning something about cancelling “culture” which is under the flag of “being nice” demonstrates much worse behaviour in practice than one it claim to oppose. You actually among others should be very aware and disgusted by those tactics after you’ve seen effects of “soviet methods” in Romania.

Post reply on HN