161–170 of 238 posts
This project's been a blast to work with. While it's written in C++, it provides a C interface to compile against which makes it especially easy to extend with Go, Python and other runtimes. A few folks and I have been building a tool with it in Go for pulling & running multiple models, and serving them on a REST API: https://github.com/jmorganca/ollama In similar light, you haven't checked it out, llama.cpp also has…
This leaves a ton of stuff out. - Token generation is serial and bandwidth bound, but prompt ingestion is not and runs in batches of 512+. Short tests are fast on pure CPU llama.cpp, but long prompting (such as with ongoing conversation) is extremely slow compared to other backends. - Llama.cpp now has very good ~4 bit quantization that doesn't affect perplexity much. Q6_K almost has the same perplexity as FP16, but…
This leaves a ton of stuff out. - Token generation is serial and bandwidth bound, but prompt ingestion is not and runs in batches of 512+. Short tests are fast on pure CPU llama.cpp, but long prompting (such as with ongoing conversation) is extremely slow compared to other backends. - Llama.cpp now has very good ~4 bit quantization that doesn't affect perplexity much. Q6_K almost has the same perplexity as FP16, but…
What’s perplexity?
Note that this has nothing to do with whether the answer is objectively correct. It's just measuring how confident it is.
Earlier quoted context omitted.
> And basically all servers will have 8xA100 for those wondering: no this is not the norm. My lab at CMU doesn't own any A100s (we have A6000s).
The servers the commenter is talking about are DGX machines from NVIDIA. It doesn’t really make sense to BTO. What you gain economically you lose in the science you can do. But nobody could have anticipated this.
Earlier quoted context omitted.
Because 175B parameters (350GB for the weights FP16, let's say a bit over 400GB for actual inference), fit very comfortably on 8xA100 (640GB VRAM total). And basically all servers will have 8xA100 ( maybe 4xA100). Nobody bothers with a single A100 (of course in a VM you might have access to only one)
> And basically all servers will have 8xA100 for those wondering: no this is not the norm. My lab at CMU doesn't own any A100s (we have A6000s).
This project's been a blast to work with. While it's written in C++, it provides a C interface to compile against which makes it especially easy to extend with Go, Python and other runtimes. A few folks and I have been building a tool with it in Go for pulling & running multiple models, and serving them on a REST API: https://github.com/jmorganca/ollama In similar light, you haven't checked it out, llama.cpp also has…
[flagged]
On a more serious node, your questions seem very... aggressive.
Especially the one about the name, is the name offensive or what? To me it sounds quite benign.
Earlier quoted context omitted.
Thank you! Is there a sweet spot with quantization. how much can you quantize for given model type and size and still be useful.
Tim Dettmers recently ( https://www.manifold1.com/episodes/ai-on-your-phone-tim-dett... ): "But what we found with these neural networks is, if you use 32 bits, they're just fine. And then you use 16 bits, and they're just fine. And then with eight bits, you need to use a couple of tricks and then it's just fine. And now we find if you can go to four bits, and for some networks, that's much easier. For some networks,…
That will be really interesting for FPGAs, because the current ones are basically oceans of 4-bit computers.
Yes, you can gang together a pair of 4LUTs to make a 5LUT, and a pair of 5LUTs to make a 6LUT, but you halve your parallelism each time you do that. OTOH you can't turn a 4LUT into a pair of 3LUTs on any currently-manufactured FPGA. It's simply the "quantum unit" of currently-available hardware -- and it's been that way for at least 15 years (Altera had 3LUTs back in the 2000s). There's no fundamental reason for the number 4 -- but it is a very, very deep local minimum for the current (non-AI) customers of FPGA vendors.
Earlier quoted context omitted.
You're thinking about the wrong bandwidth here. The article is talking about going from the GPU's RAM GPU cores (i.e. through load/store instructions in a cuda kernel), not from CPU's RAM GPU's RAM. That kind of bandwidth is still important but usually not the bottleneck on most ML workloads.
I'm still confused. The author (correctly) made a distinction about the two like you said, but at the end when talking about Raspberry Pi 4 they use a number (~4GB/s of memory bandwidth) from an article [1] which I can only assume is NOT about graphics memory or its bandwidth (do Raspberry PIs even have it?). And how exactly is the bandwidth counted if I use integrated GPU (like i5 13600K)? Or pure CPU? [1] https://f…
An integrated GPU shares memory with the CPU, so at best it gets the same amount of bandwidth assuming the CPU is not using any (which is rather unlikely).
A dedicated GPU has its own private high-bandwidth memory (an RTX 4090 has a memory bandwidth of over 1000 GB/s), but to get anything in there it needs to be loaded over the PCIe bus (which has a measly 32 GB/s bandwidth for PCIe 4.0 x16).
That said, CPU memory does have one big advantage: it tends to be much larger. You can pair up to 192 GB with a regular Ryzen 7000 CPU, while consumer GPUs don't go above 24 GB of memory (RTX 4090, RX 7900 XTX). (There are bigger GPUs out there, but those are generally intended for datacenters, and if you go that route, an Epyc or Xeon CPU can also support much more memory than a plain desktop Ryzen, although you can also slot multiple GPUs into a single server.)
I believe that for LLM performance, memory bandwidth is key, because all the neural network layers need to be streamed from memory, and very little work is done with it each time, although I guess batching operations could help if you're working at scale, since each weight would be applied multiple times then.
Earlier quoted context omitted.
You could have two models answer 100 questions the same way, and differ on the 101st. They’re unpredictable by nature - if we could accurately predict them we’d just use the predictions instead.
(Stupid question) are models still non-deterministic if you set the temperature to zero? Would setting the temperature to zero degrade the quality of response?
For an llm, a trivial change in wording could produce a big change in answer, same as running it again with a new random seed. "Prompt engineering" is basically overfitting if not approached methodically. For example, it would be interesting to try deliberate permutations of an input that don't change the meaning and see how the answer changes as part of an evaluation.
Earlier quoted context omitted.
No, a human isn't born with a set of knowledge like a freshly trained LLM, keeping the model fixed and responding to input. The analog to the model changes based on the human's experience. Just making bigger and bigger LLMs won't give you this.
Humans are born with millions of years of evolutionary training embedded in their dna and brain. We are not born with nothing.
The million-year-old stuff is not what makes humans interesting.