Will be interesting to see what people can do with local models, particularly for open source programming tools and PCG models for video games.
They'll probably write fanfic and some Harry Potter ships.
How Is LLaMa.cpp Possible?
41–50 of 238 posts
Re: How Is LLaMa.cpp Possible?
#42Earlier quoted context omitted.
The irony in your statement is immense. Yes, Kurzeweil has been saying this for decades. No it doesn't mean AGI is close. These llms do nothing to advance AGI. There is no theoretical basis to the belief in emergent intelligence from statistical language models and the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the dee…
Why do you say they do nothing to advance AGI? Do you know what it takes to advance AGI? It's hard to state that without knowing how AGI would work yourself. LLMs would be considered magic just a couple years ago. Sure, not AGI but behaves just like one for certain workloads. I find hard to believe we're not a bit closer now - or maybe even a lot closer.
AGI should have morals, opinions, self-reflection, learn continuously from sensor data, reason, realize when they’re proven wrong and update their model of the world, and be creative. So far LLMs exhibit none of those. But LLMs exhibit a digestible distillation of a very large body of data which may be a component of an AGI.
But you can have an AGI that doesn’t have encyclopedic knowledge but it’s still highly intelligent, so I don’t think LLMs have to be an intrinsic component.
Re: How Is LLaMa.cpp Possible?
#43I enjoyed this article, but it seems to me that the latency numbers should have units of nanoseconds or maybe CPU cycles. I feel like the article was a bit sloppy with units. Another question that occurs to me is: why do chipmakers even bother putting so many functional units on the chip if almost all workloads are memory bound? Based of the calculations in this article, you could decrease the number of teraflops a m…
The primary use case for GPUs in devices, graphics, is not as often memory bound. Even for other general data compute purposes that may not always be the case. It's specifically neural nets that have extremely wide batches of extremely simple operations occurring across extremely large chunks of memory where being memory bound is the default case.
Re: How Is LLaMa.cpp Possible?
#44I enjoyed this article, but it seems to me that the latency numbers should have units of nanoseconds or maybe CPU cycles. I feel like the article was a bit sloppy with units. Another question that occurs to me is: why do chipmakers even bother putting so many functional units on the chip if almost all workloads are memory bound? Based of the calculations in this article, you could decrease the number of teraflops a m…
1. I think nanosecond-scale latency numbers on operations taking dozens to hundreds of ms are probably overkill?
2. Inference is only one aspect of what GPUs are used for. Many other workloads are compute-bound. That being said, given the recent rise of these kinds of open-source, pre-trained large language models, I wouldn't be surprised if future Nvidia product launches offered variants with significantly more VRAM. There would probably be a lot of interest in "3080-equivalent compute, but 48GB VRAM" these days — certainly I would take one over a 4090 with 24GB VRAM. (Then again, that's basically an A6000, and those go for nearly $7k...)
Re: How Is LLaMa.cpp Possible?
#45Earlier quoted context omitted.
We want to think we’re exceptional but all we can do is say “human consciousness is special” without having any way of measuring it or disproving the assertion that we’re just really fancy pattern matchers. Take any metaphor you want, it’s the same outcome: we may all be philosophical zombies.
We may "just" be neural networks that run on meat instead of silicon, but that does not mean that we're LLMs.
Why doesn’t it?
Re: How Is LLaMa.cpp Possible?
#46Earlier quoted context omitted.
First, there has to be something capable of matching human performance at a much higher cost. This is still just spicy autocomplete.
Humans just do spicy autocomplete too.
Really? I can guess at what spicy autocomplete might actually mean but I doubt a LLM ... OK ChatGPT did a pretty good job of it (I've just asked it), whilst sidestepping the definition of spicy in this context. It is after all a very good next word guesser, given a context, and its not ... me! I am capable of hallucinating but it was 30 odd years ago since I hunted out certain fungi on Dartmoor, or smoked hemp.
To be fair, we humans do often interrupt each other to second guess a sentence completion. Done correctly it is a brief satisfying collaboration. Done wrong ... I've been married for 18 years and know when to bite my tongue, but I still get it wrong from time to time - sometimes deliberately. Despite that, me and the wiff can autocomplete each other's sentences with uncanny accuracy and end up with perfect harmony or a cough slight disagreement as a result.
We are getting some phenomenal slide rules these days but the darleks are not going to be flying up the stairwell just yet, nor will SkyNet be taking over tomorrow.
That said, you just know that some noddy is trying to sell a nuclear "deterrent" LLM AI thingie somewhere. Thankfully, production military equipment takes quite a while to get to deployment. There is a good chance that we will get to grips with all this stuff before SkyNet is let loose for real 8)
Re: How Is LLaMa.cpp Possible?
#47In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…
I was hoping that link would answer the question that's been bugging me for months: what are the penalties that you pay for using a quantized model? Sadly it didn't. It talked about "perplexities" and showed some floating point numbers. I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."
I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."
The closer we get to assessing human like performance, the tougher it is, because it becomes more subjective and less deterministic by the nature of the task. I don't know the answer, but I know that for the metrics we have it's not so easy to translate them into any idea about the kind of performance on some specific thing you might want to do with the model.
Re: How Is LLaMa.cpp Possible?
#48In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…
I was hoping that link would answer the question that's been bugging me for months: what are the penalties that you pay for using a quantized model? Sadly it didn't. It talked about "perplexities" and showed some floating point numbers. I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."
The problem is that it's not consistent enough for a good demo. Not even two different models, but even two different fine tunes of the same base model may be wildly differently affected by quantization. It can range from making hardly a difference to complete garbage output.
Re: How Is LLaMa.cpp Possible?
#49This project's been a blast to work with. While it's written in C++, it provides a C interface to compile against which makes it especially easy to extend with Go, Python and other runtimes. A few folks and I have been building a tool with it in Go for pulling & running multiple models, and serving them on a REST API: https://github.com/jmorganca/ollama In similar light, you haven't checked it out, llama.cpp also has…
Thank you for the link to your project! I'll be playing around with it.
Re: How Is LLaMa.cpp Possible?
#50In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…
>Still, we are looking at GPT3 level of performance on one server rack. That says something when less than a year ago, such AI was literally magic and only run on a massive datacenter.
I'm not sure what you mean by this. You've always been able to run GPT3 on a single server (your typical 8xA100).