Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Is LLaMa.cpp Possible?

finbarr.ca

Post reply on HN
Print view View original

61–70 of 238 posts

Re: How Is LLaMa.cpp Possible?

#61

Post by not2b »

tysam_and wrote: post #33
not2b wrote: post #17

Earlier quoted context omitted.

No, a human isn't born with a set of knowledge like a freshly trained LLM, keeping the model fixed and responding to input. The analog to the model changes based on the human's experience. Just making bigger and bigger LLMs won't give you this.

Ah, but I believe you forget the implicit biases of genetic programming. Instincts in my experience are the skeleton, and in a sense the default basis functions for the structure of how we live, see, do, and learn.

No, I don't forget that. There's obviously a starting point, behaviors and abilities that newborns already have. The point is that the model is not static.

Re: How Is LLaMa.cpp Possible?

#62

Post by sharms »

Animats wrote: post #20

Is this single-thread? Or are they putting all available CPUs on the problem?

The problem is memory bandwidth rather than CPU cores: "Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. Anything that reduces the memory requirements for these models makes them much easier to serve"

Re: How Is LLaMa.cpp Possible?

#63

Post by eurekin »

cameron_b wrote: post #55
eurekin wrote: post #25

Earlier quoted context omitted.

How many tokens a second?

The other way around is whole number math. I added the 3-node output from the 13B model to github, the timings are below. The 3-node 65B job hasn't finished yet. llama_print_timings: load time = 17766.29 ms llama_print_timings: sample time = 264.42 ms / 128 runs ( 2.07 ms per token, 484.07 tokens per second) llama_print_timings: prompt eval time = 10146.71 ms / 8 tokens ( 1268.34 ms per token, 0.79 tokens per second)…

This is very interesting and actually in the usable realm, for some use cases

Re: How Is LLaMa.cpp Possible?

#65

Post by cameron_b »

eurekin wrote: post #63
cameron_b wrote: post #55

Earlier quoted context omitted.

The other way around is whole number math. I added the 3-node output from the 13B model to github, the timings are below. The 3-node 65B job hasn't finished yet. llama_print_timings: load time = 17766.29 ms llama_print_timings: sample time = 264.42 ms / 128 runs ( 2.07 ms per token, 484.07 tokens per second) llama_print_timings: prompt eval time = 10146.71 ms / 8 tokens ( 1268.34 ms per token, 0.79 tokens per second)…

This is very interesting and actually in the usable realm, for some use cases

My networking setup is not optimal, but it was quite surprising how easy it was to get it all to work.

Re: How Is LLaMa.cpp Possible?

#66

Post by zarzavat »

shrimpx wrote: post #42
glimshe wrote: post #29

Earlier quoted context omitted.

Why do you say they do nothing to advance AGI? Do you know what it takes to advance AGI? It's hard to state that without knowing how AGI would work yourself. LLMs would be considered magic just a couple years ago. Sure, not AGI but behaves just like one for certain workloads. I find hard to believe we're not a bit closer now - or maybe even a lot closer.

AGI should have morals, opinions, self-reflection, learn continuously from sensor data, reason, realize when they’re proven wrong and update their model of the world, and be creative. So far LLMs exhibit none of those. But LLMs exhibit a digestible distillation of a very large body of data which may be a component of an AGI. But you can have an AGI that doesn’t have encyclopedic knowledge but it’s still highly intell…

That is not what AGI means. AGI = Artificial General Intelligence.

1. Artificial = we made it

2. General = it can solve problems in any field

3. Intelligence = the ability to solve problems

A chess engine is a very strong Artificial Intelligence. But it’s not very General, it can only evaluate chess positions.

GPT-4 is very General, you can ask it about any question and get a somewhat reasonable answer. But it’s not very intelligent, often the answer is wrong.

You’re talking about an Artificial Human. That’s a different problem. Intelligence is not species dependent. Dolphins are intelligent (a bit), aliens can be intelligent and have zero emotions or conception of self. There’s certainly plenty of amoral intelligent serial killers.

Re: How Is LLaMa.cpp Possible?

#67

Post by kimixa »

ggm wrote: post #13
gct wrote: post #3

Earlier quoted context omitted.

Basically Ray Kurzweil's argument, he's been saying $1000 worth of compute will be able to match human performance around 2029 for decades now.

The irony in your statement is immense. Yes, Kurzeweil has been saying this for decades. No it doesn't mean AGI is close. These llms do nothing to advance AGI. There is no theoretical basis to the belief in emergent intelligence from statistical language models and the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the dee…

The lack of concept of "knowledge" is a big one for me - if that's an emergent thing it hasn't even shown hints of this yet. This to me seems a pretty hard line right now, as it limits their capability of things even inexperienced humans can do - namely decide if they actually know something, and identify when they don't know something and attempt to fix that - e.g. asking for clarification on vague inputs, or deciding if something is actually truth or fiction.

That then ties into another limitation right now - how after a training the model is pretty static, so cannot learn and has no state outside it's context buffer. This could just be another point where a few orders of magnitude more computing power can "fix" it, doing whole training steps between each input to actually incorporate new information and corrections into itself instead of relying on a fixed size small context.

But I'm not deep enough into things to say if they're fundamental issues, or current techniques will start displaying those effects as emergent characteristics as the complexity and training increases. There's been a few other examples when "known" techniques start to show unexpected characteristics down the line as they are scaled up, so can't really say for sure they'll /never/ be shown, just that the current examples don't seem to show even the beginnings of that sort of thing.

Re: How Is LLaMa.cpp Possible?

#68

Post by jcims »

jmorgan wrote: post #9

This project's been a blast to work with. While it's written in C++, it provides a C interface to compile against which makes it especially easy to extend with Go, Python and other runtimes. A few folks and I have been building a tool with it in Go for pulling & running multiple models, and serving them on a REST API: https://github.com/jmorganca/ollama In similar light, you haven't checked it out, llama.cpp also has…

Just wanted to say thanks homie for making something so hairy so accessible. Love it!!!

Re: How Is LLaMa.cpp Possible?

#69

Post by pedrovhb »

Given the massive imbalance in the memory bandwidth bottleneck, I wonder why specialized hardware is the way it is. Is there some use case in which processing is the bottleneck, or at least it's more even? Are we expecting some software paradigm shift which will change the balance? Why couldn't they just make a cheaper, more rounded card which isn't heavily underutilized because of a large bottleneck?

Re: How Is LLaMa.cpp Possible?

#70

Post by sbierwagen »

reissbaker wrote: post #44
gautamcgoel wrote: post #34

I enjoyed this article, but it seems to me that the latency numbers should have units of nanoseconds or maybe CPU cycles. I feel like the article was a bit sloppy with units. Another question that occurs to me is: why do chipmakers even bother putting so many functional units on the chip if almost all workloads are memory bound? Based of the calculations in this article, you could decrease the number of teraflops a m…

1. I think nanosecond-scale latency numbers on operations taking dozens to hundreds of ms are probably overkill? 2. Inference is only one aspect of what GPUs are used for. Many other workloads are compute-bound. That being said, given the recent rise of these kinds of open-source, pre-trained large language models, I wouldn't be surprised if future Nvidia product launches offered variants with significantly more VRAM…

Yeah, Nvidia won't do big VRAM consumer cards until AMD forces them to. They're running flat out just trying to keep up with demand for H100s at forty thousand USD each.
Post reply on HN