Memory bound token generation is a limitation of transformer decoders.
In the past, hardware has motivation algorithm innovations.
I’m curious how long it will take until we see more hardware friendly models.
How Is LLaMa.cpp Possible?
121–130 of 238 posts
Re: How Is LLaMa.cpp Possible?
#122Earlier quoted context omitted.
GPT-4 solves only 1 problem: what is the most likely stream of tokens to follow what we already have. It is remarkably good at this. But there's absolutely zero reason to believe it can solve any other problem at all.
How do humans write if not by intuiting what word comes after another? Intelligence is the ability of that next word decision procedure to determine a next word that is aligned with our human intuition and model of truth. I believe what you’re getting at is modality , that GPT-4 only provides responses in text. You can’t ask it to drive a car, or paint like Dall-e. And that’s a fair criticism, but it’s mostly just be…
> How do humans write if not by intuiting what word comes after another?
We don't know. It may turn out that we use mechanisms similar to LLMs, or it might be something entirely different.
As for the rest: nobody knows how to make ChatGPT butter a piece of toast, let alone drive a car.
ChatGPT does not reason about text, either.
Re: How Is LLaMa.cpp Possible?
#123>> Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU. Sure the GPU i not fast, but if memory is important.....
You're thinking about the wrong bandwidth here. The article is talking about going from the GPU's RAM GPU cores (i.e. through load/store instructions in a cuda kernel), not from CPU's RAM GPU's RAM. That kind of bandwidth is still important but usually not the bottleneck on most ML workloads.
I'm still confused.
The author (correctly) made a distinction about the two like you said, but at the end when talking about Raspberry Pi 4 they use a number (~4GB/s of memory bandwidth) from an article [1] which I can only assume is NOT about graphics memory or its bandwidth (do Raspberry PIs even have it?).
And how exactly is the bandwidth counted if I use integrated GPU (like i5 13600K)? Or pure CPU?
Re: How Is LLaMa.cpp Possible?
#124In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…
I was hoping that link would answer the question that's been bugging me for months: what are the penalties that you pay for using a quantized model? Sadly it didn't. It talked about "perplexities" and showed some floating point numbers. I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."
I have been using nat.dev to compare quantized models and it works great.
Re: How Is LLaMa.cpp Possible?
#125Re: How Is LLaMa.cpp Possible?
#126In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…
I was hoping that link would answer the question that's been bugging me for months: what are the penalties that you pay for using a quantized model? Sadly it didn't. It talked about "perplexities" and showed some floating point numbers. I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."
I have several sets of quant comparisons posted on my HF spaces, the caveat is my prompts are all "English to code": https://huggingface.co/spaces/mike-ravkine/can-ai-code-compa...
The dropdown at the top selects which comparison: Falcon compares GGML, Vicuna compares bits and bytes. I have some more comparisons planned, feel free to open an issue if you'd like to see something specific: https://github.com/the-crypt-keeper/can-ai-code
Re: How Is LLaMa.cpp Possible?
#127Earlier quoted context omitted.
You have a metric for human brains worth of computing power which hasn't already been exceeded? I can't do infinite precision arithmetic or the RSA algorithm in my head, or index a billion strings into lexical sort order. But I am human, I am conscious and no visible VLSI work or algorithmic model will lead to AGI or a human equivalent computing power by 2029. Let alone for $1000.
well, you could just be hallucinating your own consciousness. By 2029 it seems not unreasonable to expect that the most sophisticated models will carry out visual and auditory interactions which could fool even the most sophisticated viewer. At that point, what really does consciousness mean? If the robot insisted to me it was conscious, how can I really say no?
> it seems not unreasonable
This is where we differ.
Re: How Is LLaMa.cpp Possible?
#128Earlier quoted context omitted.
You could have two models answer 100 questions the same way, and differ on the 101st. They’re unpredictable by nature - if we could accurately predict them we’d just use the predictions instead.
(Stupid question) are models still non-deterministic if you set the temperature to zero? Would setting the temperature to zero degrade the quality of response?
Some GPU operations give different results depending on the order they are done. This happens because floating point numbers are approximations and lose associativity. Requiring a strict order causes a big slowdown.
Re: How Is LLaMa.cpp Possible?
#129Earlier quoted context omitted.
I did it years ago on a lark with a seq2seq model in a matrix chat room.
How did it perform? Was it well received by members?
Poorly! It was a small seq2seq and was gibberish to start with. Although it did tell my friend that it loved him which was nice.