Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Is LLaMa.cpp Possible?

finbarr.ca

Post reply on HN
Print view View original

71–80 of 238 posts

Re: How Is LLaMa.cpp Possible?

#71

Post by croes »

circuit10 wrote: post #32
ggm wrote: post #13

Earlier quoted context omitted.

The irony in your statement is immense. Yes, Kurzeweil has been saying this for decades. No it doesn't mean AGI is close. These llms do nothing to advance AGI. There is no theoretical basis to the belief in emergent intelligence from statistical language models and the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the dee…

By most measures you could think of for intelligence languages models are improving, so I don’t see why you think this wouldn’t lead to something at least almost human-level if you scaled it up enough Of course there could be some wall somewhere but I don’t see why there would be

Because you need more training data for better results and they are running out of new training data.

Re: How Is LLaMa.cpp Possible?

#72

Post by axiom92 »

redox99 wrote: post #58
blovescoffee wrote: post #54

Earlier quoted context omitted.

Am I missing something or how do you know this? Also I think the OP was talking about a single card not multiple but that was just my reading.

Because 175B parameters (350GB for the weights FP16, let's say a bit over 400GB for actual inference), fit very comfortably on 8xA100 (640GB VRAM total). And basically all servers will have 8xA100 ( maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one)

> And basically all servers will have 8xA100

for those wondering: no this is not the norm. My lab at CMU doesn't own any A100s (we have A6000s).

Re: How Is LLaMa.cpp Possible?

#73

Post by andrewstuart »

>> Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers.

So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU.

Sure the GPU i not fast, but if memory is important.....

Re: How Is LLaMa.cpp Possible?

#74

Post by BobbyJo »

andrewstuart wrote: post #73

>> Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU. Sure the GPU i not fast, but if memory is important.....

Most CPU RAM is much slower than GPU RAM. GPUs typically pack RAM 2 generations ahead with a wider bus than anything you'd find on a consumer motherboard.

Re: How Is LLaMa.cpp Possible?

#75

Post by redox99 »

eurekin wrote: post #60
redox99 wrote: post #52

Earlier quoted context omitted.

VRAM is not latency optimized. VRAM has worse latency than your CPU RAM. The reason why it's mounted closer is because of signal integrity because of higher frequencies, not because of latency.

Interesting. Where can I read more about that?

Sorry can't provide any resources right now. If you search a bit I'm sure you'll find some latency comparisons between DDR and GDDR.

But basically GPU memory (GDDR5/6/6X/etc) is optimized for bandwidth (because GPUs need to move a lot of data, have few branches, few unknown data dependencies, high spatial locality). CPU memory is more optimized for latency (because of branchy code).

Re: How Is LLaMa.cpp Possible?

#76

Post by winwang »

BobbyJo wrote: post #74
andrewstuart wrote: post #73

>> Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU. Sure the GPU i not fast, but if memory is important.....

Most CPU RAM is much slower than GPU RAM. GPUs typically pack RAM 2 generations ahead with a wider bus than anything you'd find on a consumer motherboard.

For reference, DDR4-3200 in quad channel is ~100 GB/s while a 3090's VRAM is 960 GB/s. Of course, most consumers only have dual channel.

M1 Pro is 200 and M1 Max is 400. Which is slow for GPU memory, but incredible for main memory -- although I'm not sure how much of that a single core can actually pull.

Re: How Is LLaMa.cpp Possible?

#77

Post by doctorpangloss »

axiom92 wrote: post #72
redox99 wrote: post #58

Earlier quoted context omitted.

Because 175B parameters (350GB for the weights FP16, let's say a bit over 400GB for actual inference), fit very comfortably on 8xA100 (640GB VRAM total). And basically all servers will have 8xA100 ( maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one)

> And basically all servers will have 8xA100 for those wondering: no this is not the norm. My lab at CMU doesn't own any A100s (we have A6000s).

The servers the commenter is talking about are DGX machines from NVIDIA.

It doesn’t really make sense to BTO. What you gain economically you lose in the science you can do.

But nobody could have anticipated this.

Re: How Is LLaMa.cpp Possible?

#78

Post by notuseful »

redox99 wrote: post #56
oars wrote: post #38

Great article. Don't see content like this anywhere else outside of HN.

You can find content like this on Twitter if you follow the right people. In fact I read this article before it was even posted here because @karpathy tweeted about it.

[flagged]

Re: How Is LLaMa.cpp Possible?

#79

Post by ggm »

80/20 rule. Approximations have smaller, less accurate approximates which do in many contexts. If your goal is to reach America, a crude compass and speed reasoning works. If you want to target an ICBM you need better positional accuracy.

Re: How Is LLaMa.cpp Possible?

#80

Post by notuseful »

MichaelZuo wrote: post #53
eurekin wrote: post #28

Earlier quoted context omitted.

Whoa, imagine you get a good base LLM model and save all conversations with it. Run a batch process every night to fine tune a LORA on convo dataset. If I ever came across such a chat bot I'd probably freak out as to why it remembers things outside of the context window, without summarisation

That's a pretty neat idea, I would be surprised if no one is already working on that.

I did it years ago on a lark with a seq2seq model in a matrix chat room.
Post reply on HN