Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Is LLaMa.cpp Possible?

finbarr.ca

Post reply on HN
Print view View original

21–30 of 238 posts

Re: How Is LLaMa.cpp Possible?

#21

Post by Aerroon »

Havoc wrote: post #2

What I find more stunning is what this implies going forward. If tech advances as it tends to do then having a 200bn model fit into consumer hardware isn't that far away. Might not be AGI but I think cliched as it is that would "change everything". If not at 200 then 400 or whatever. Doesn't matter - the direction of travel seems certain.

A 200b 4-bit quantized model could potentially fit into 128 GB of RAM. The inference would just be really slow.

Ie you could technically run something like that today.

I think more VRAM on GPUs isn't necessarily a technical limitation either. I think GPU manufacturers could add a lot more VRAM to their cards if they wanted to. The question is whether it would be worth the price increase.

Re: How Is LLaMa.cpp Possible?

#22

Post by coolspot »

ggm wrote: post #13
gct wrote: post #3

Earlier quoted context omitted.

Basically Ray Kurzweil's argument, he's been saying $1000 worth of compute will be able to match human performance around 2029 for decades now.

The irony in your statement is immense. Yes, Kurzeweil has been saying this for decades. No it doesn't mean AGI is close. These llms do nothing to advance AGI. There is no theoretical basis to the belief in emergent intelligence from statistical language models and the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the dee…

> the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the deep semantic meaning of the language presented. There's no intelligence.

Can say the same about a half of population, tbh

Re: How Is LLaMa.cpp Possible?

#23

Post by catchnear4321 »

Waterluvian wrote: post #14
KerrAvon wrote: post #10

Earlier quoted context omitted.

First, there has to be something capable of matching human performance at a much higher cost. This is still just spicy autocomplete.

Humans just do spicy autocomplete too.

monkeys make monkeys accidentally.

monkeys make meseeks on purpose.

there is a difference, but will it be fun?

Re: How Is LLaMa.cpp Possible?

#24

Post by tysam_and »

_kava wrote: post #8

In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…

Yes, there is a logarithmically-bound (or exponential if you're viewing it from another angle) falloff in the information lost in quantization. This comes from the non-uniform "value" of different weights. We can try to get around them with different methods, but at the end of the day, some parameters just hurt more to squeeze.

What is insane though is how far we've taken it. I remember when INT8 from NVIDIA seemed like a nigh-pipedream!

Re: How Is LLaMa.cpp Possible?

#25

Post by eurekin »

cameron_b wrote: post #11

I’ve been working through that repo and managed the 13B dataset on a single Pi4 8gig I’ve also replicated the work in OpenMPI ( from a thread on the llama.cpp GitHub repo ) and today I managed to get the 65B dataset operational on three pi4 nodes. I’m not saying this as any achievement of mine, but as a comment on the current reality of reproducible LLM At home on anything you’ve got. It really feels like this techni…

How many tokens a second?

Re: How Is LLaMa.cpp Possible?

#26

Post by Waterluvian »

catchnear4321 wrote: post #23
Waterluvian wrote: post #14

Earlier quoted context omitted.

Humans just do spicy autocomplete too.

monkeys make monkeys accidentally. monkeys make meseeks on purpose. there is a difference, but will it be fun?

I am 100% invested in how much ridiculous fun this era is going to be. Right up until the moment when it becomes a horror.

Re: How Is LLaMa.cpp Possible?

#27

Post by tysam_and »

In my best estimation, Finbarr makes pretty great content, he and I have had a number of positive interactions on Twitter. I tend to have a pretty grumpy disposition towards a lot of modern ML and such as I feel it's shovelware, but whenever Finbarr puts out work, I tend to set aside some time to give it a good gander, as I feel like it's generally pretty "meaty" (which I honestly find pretty hard to do past a certain pace). Well worth the subscribe if you have not done so already (I'm not affiliated with him, I just really like his work!).

Re: How Is LLaMa.cpp Possible?

#28

Post by eurekin »

Waterluvian wrote: post #19
not2b wrote: post #17

Earlier quoted context omitted.

No, a human isn't born with a set of knowledge like a freshly trained LLM, keeping the model fixed and responding to input. The analog to the model changes based on the human's experience. Just making bigger and bigger LLMs won't give you this.

So a human is different because it keeps training its neural network?

Whoa, imagine you get a good base LLM model and save all conversations with it. Run a batch process every night to fine tune a LORA on convo dataset. If I ever came across such a chat bot I'd probably freak out as to why it remembers things outside of the context window, without summarisation

Re: How Is LLaMa.cpp Possible?

#29

Post by glimshe »

ggm wrote: post #13
gct wrote: post #3

Earlier quoted context omitted.

Basically Ray Kurzweil's argument, he's been saying $1000 worth of compute will be able to match human performance around 2029 for decades now.

The irony in your statement is immense. Yes, Kurzeweil has been saying this for decades. No it doesn't mean AGI is close. These llms do nothing to advance AGI. There is no theoretical basis to the belief in emergent intelligence from statistical language models and the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the dee…

Why do you say they do nothing to advance AGI? Do you know what it takes to advance AGI? It's hard to state that without knowing how AGI would work yourself.

LLMs would be considered magic just a couple years ago. Sure, not AGI but behaves just like one for certain workloads. I find hard to believe we're not a bit closer now - or maybe even a lot closer.

Re: How Is LLaMa.cpp Possible?

#30

Post by pseudonom- »

ripvanwinkle wrote: post #12
_kava wrote: post #8

In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…

Thank you! Is there a sweet spot with quantization. how much can you quantize for given model type and size and still be useful.

Tim Dettmers recently (https://www.manifold1.com/episodes/ai-on-your-phone-tim-dett...):

"But what we found with these neural networks is, if you use 32 bits, they're just fine. And then you use 16 bits, and they're just fine. And then with eight bits, you need to use a couple of tricks and then it's just fine.

And now we find if you can go to four bits, and for some networks, that's much easier. For some networks, it's much more difficult, but then you need a couple more tricks. And so it seems they're much more robust."

Post reply on HN