Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Is LLaMa.cpp Possible?

finbarr.ca

Post reply on HN
Print view View original

221–230 of 238 posts

Re: How Is LLaMa.cpp Possible?

#221

Post by fulafel »

mhh__ wrote: post #87
andrewstuart wrote: post #73

>> Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU. Sure the GPU i not fast, but if memory is important.....

This is basically the appeal of the apple chips in this domain. Apple have fuck-you money so they have a bunch of high-bandwidth decent-latency soldered onto the chip.

It's LPDDR memory with wires going between the chips, the difference to other LPDDR applications is just running the wires inside the same package.

Re: How Is LLaMa.cpp Possible?

#223

Post by Matumio »

ant6n wrote: post #155
brucethemoose2 wrote: post #105

This leaves a ton of stuff out. - Token generation is serial and bandwidth bound, but prompt ingestion is not and runs in batches of 512+. Short tests are fast on pure CPU llama.cpp, but long prompting (such as with ongoing conversation) is extremely slow compared to other backends. - Llama.cpp now has very good ~4 bit quantization that doesn't affect perplexity much. Q6_K almost has the same perplexity as FP16, but…

What’s perplexity?

Perplexity is measured by testing a language model on a known text. The model's output is a probability for every possible next word/token. The model is highly perplexed if it gave a low probability to the actual next token.

Re: How Is LLaMa.cpp Possible?

#224

Post by bobboies »

Vvector wrote: post #203
bobboies wrote: post #186

Earlier quoted context omitted.

Wtf does HGX mean? God enough with the acronyms people. Please take an extra ten seconds to speak in proper human language! You could save on the worlds carbon footprint by reducing the number of times humans have to search for “what is NVIDIA hgx?” or is it “what is AMD HGX” and then subsequently visiting the websites to see if that’s right or not.

What does Wft mean? God enough with the acronyms people. /s

You got me there hahaha

However, there’s a difference between an acronym known to the broader public versus some single shot, context-specific one!

Re: How Is LLaMa.cpp Possible?

#225

Post by bobboies »

cameron_b wrote: post #191
bobboies wrote: post #185

Earlier quoted context omitted.

Wtf does SBC mean? God enough with the acronyms people.

In my experience, it usually means Small Block Chevy, but in certain communities it means Single Board Computer, an older way of referring to devices like the Raspberry Pi. I would elaborate and say, anywhere that your computer is resource constrained ( ram, processing power ) but you still want to make up articles for your Amazon Affiliate blog

Single board computer makes sense. I wish folks would type things out.

Re: How Is LLaMa.cpp Possible?

#226

Post by Matumio »

regularfry wrote: post #163
ant6n wrote: post #155

Earlier quoted context omitted.

What’s perplexity?

Perplexity is a measure of how certain the model is of the next token. It's calculated by looking at the probabilities that the model calculates for the next token in a stream. If there are several choices for the next token with similar probabilities, that's telling you that the model is having a hard time telling what the right answer should be: the model is more perplexed, perplexity is higher. If there's a single…

I think what you describe ("confidence about the next token") is the entropy of the model's output. A model can be very certain about the next token (its output has low entropy) but if it is usually wrong on the text you measure it against, it will have high perplexity. (For example when the model was trained only on children's books and you measure it on Wikipedia.)

Re: How Is LLaMa.cpp Possible?

#227

Post by dTal »

not2b wrote: post #220
dTal wrote: post #100

Earlier quoted context omitted.

Bit of an unfair comparison when humans also have a bunch of senses that LLMs don't have. They might be trained on orders of magnitude more words , but more data? Doubtful.

That's the key. I'm reminded of the Helen Keller story. She was completely blind and deaf. Her teacher spent a very long time signing into her hand. It took a very long time before she realized that the sign for "water" designated the thing she could feel flowing onto her hand; before that breakthrough the signs were meaningless to her. An LLM only knows the structure of language. It doesn't know that there is an ext…

Crippled, yes, but I would disagree that it is fundamentally limited, or that an input stream of human language is inadequate to bootstrap "meaning", or in some way philosophically inferior to native biological senses.

It's very interesting you bring up Helen Keller because she's generally regarded as possessing the same level of sentience - and indeed intelligence - as anyone else, despite the extreme narrowness of her sensory input. It took her much longer to get going, but it's not as if she only understood concepts that directly related to touch. The experience with "water" taught her the concept of a symbol, and from there she could bootstrap everything else. LLMs already work with symbols - that is their sense.

In fact we're all a bit like Helen Keller, in the sense that if sensory input is the basis for our entire world model, then it is a very small foundation supporting an incredibly vast and intricate edifice. There is a considerable abstraction gap between concepts like "capitalism" and any direct sensory input. We all of us, all the time, manipulate concepts without thinking through what they "mean" all the way to something we can see and touch.

Re: How Is LLaMa.cpp Possible?

#228

Post by lovelyviking »

oblio wrote: post #166
lovelyviking wrote: post #161

Earlier quoted context omitted.

[flagged]

Because. On a more serious node, your questions seem very... aggressive. Especially the one about the name, is the name offensive or what? To me it sounds quite benign.

As someone unfamiliar with the field I’ve read github page and didn’t find answers to things I was curious about regarding this project.

So out of curiosity I wrote simple basic questions:

“Why this tool was created? What was the reason for creating such tool? Why it was named this way?“

Suggesting that if I didn’t figured it out by myself perhaps some one else who also wish to join the field and looking for entry point to try something in practice would also be confused and thus answering those simple basic questions would really help to the beginners.

With experience of entering into many new fields I know that usually there is a lot of confusion in the beginning about basic stuff .

I also suggested that those are obvious reasons why people ask questions and didn’t expect to be ‘bullied’ for basic simple questions. There were no one word of aggression in text. Perhaps seeing aggression there requires certain level of misplaced imagination.

> Especially the one about the name, is the name offensive or what? To me it sounds quite benign.

So why the simple question doesn’t sound benign to you? Usually there is a fun story about the name for example GNU is Not Unix etc … and sometimes it holds some clue about functionality and then it’s easier to remember.

> Because

well… it’s still unclear

Re: How Is LLaMa.cpp Possible?

#229

Post by lovelyviking »

jmorgan wrote: post #9

This project's been a blast to work with. While it's written in C++, it provides a C interface to compile against which makes it especially easy to extend with Go, Python and other runtimes. A few folks and I have been building a tool with it in Go for pulling & running multiple models, and serving them on a REST API: https://github.com/jmorganca/ollama In similar light, you haven't checked it out, llama.cpp also has…

> A few folks and I have been building a tool with it in Go for pulling & running multiple models, and serving them on a REST API: https://github.com/jmorganca/ollama

Why this tool was created? What was the reason for creating such tool? Why it was named this way?

Explanation for wishing to flag/block/bully basic simple questions: Questions mean what they ask and nothing else. Any imagination of aggression or anything else related to emotions is misplaced.

As someone unfamiliar with the field I’ve read GitHub page and didn’t find answers to things I was curious about regarding this project. So out of curiosity I wrote simple basic questions:

“Why this tool was created? What was the reason for creating such tool? Why it was named this way?“

Suggesting that if I didn’t figured it out by myself perhaps some one else who also wish to join the field and looking for entry point to try something in practice would also be confused and thus answering those simple basic questions would really help to the beginners.

With experience of entering into many new fields I know that usually there is a lot of confusion in the beginning about basic stuff .

I also suggested that those are obvious reasons why people ask questions and didn’t expect to be ‘bullied’ for basic simple questions. There were no one word of aggression in text. Perhaps seeing aggression there requires certain level of misplaced imagination.

Usually there is a fun story about the name for example GNU is Not Unix etc … This is opportunity to tell it if there is. Sometimes name holds some clue about functionality and then it’s easier to remember. Sharing it can help too to beginners.

Re: How Is LLaMa.cpp Possible?

#230

Post by mlboss »

KirillPanov wrote: post #170
mlboss wrote: post #98

Earlier quoted context omitted.

Humans are born with millions of years of evolutionary training embedded in their dna and brain. We are not born with nothing.

Yeah but chimpanzees and cats and mice have all the same million-year-old stuff that we do. The million-year-old stuff is not what makes humans interesting.

I have been thinking about this lately. Things that are recently developed like language skills are easiest to replicate by machines. But things that too took a time to develop like walking and grabbing are still something that machines struggle with.
Post reply on HN