Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How LLMs work

0xkato.xyz

Post reply on HN
Print view View original

121–130 of 293 posts

Re: How LLMs work

#121

Post by sesm »

malwrar wrote: post #10

Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…

I would argue that those are not emergent property of the model, but a property of how humans find insights in a plausible guess.

Re: How LLMs work

#122

Post by Marha01 »

AltruisticGapHN wrote: post #88

I don't like how most LLM explainer articles and videos say that essentially a LLM " predicts the next word". I'm a developer but not very good at maths and I still don't understand any of it. A LLM clearly has some "visual" capacity. You ask Gemini to build something with Canvas and it's able to reason about the shape of things. Like recently I waanted a checkbox that has like a gradient flowing around the edge. It…

LLMs fundamentally work by predicting the next word (token). But that should not be used to diminish their potential capabilities. It's like saying that human brains "just predict (or produce) the next electrical impulse". Fundamentally correct, but says nothing about the potential emergent capabilities of scaled-up systems that work like that.

Emergent properties of complex systems should not be diminished just because the underlying operating principle is simple.

Re: How LLMs work

#123

Post by qsera »

360MustangScope wrote: post #109
antran22 wrote: post #91

Earlier quoted context omitted.

It's still predicting the next word. Somewhere in the gigantic dataset that the LLM was trained on, there is a phrase that says "gradient border" being in the vicinity of a CSS code that render the stuff. Therefore when you run it on an inference loop there's a good chance it output that CSS code when you tell it to render a "gradient border" Multi-modal models that can understand visual input do exists, but no such…

What about things it wasn’t trained on? For instance I’ve written a few custom languages to learn how to write a VM and the lexer/parser/compiler/etc. that it had never seen before and then just gave it the syntax which is different than what it had ever seen before. Simply due to the fact I made it and it had never been trained on it. After giving it my documentation, it was able to write the language just like a la…

Because in its training data there is information on how to map from documentation of a language to actual programs. This means that following the pattern it can map between documentation for any language to programs in that language.

But I think it will have difficulty in crossing paradigm boundaries, by simply using documentation.

Re: How LLMs work

#126

Post by dist-epoch »

antirez wrote: post #75
malwrar wrote: post #10

Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…

There is a different way to look at this: that is, actually the Transformer is a minimal complication of what the based model is: in theory the neural network could be just a huge FFN, which is anyway the part of the Transformer that does the heavy lifting. But this would be impossibile to train both numerically and computationally, so the Transformer encodes enough priors for it to work: the causal attention, and th…

Isn't that over-simplifying it a bit too much?

You can go another step - a FFN can be simulated on a Turing machine, thus it just exemplifies the incredible semantical power of the Turing machine model of computation. (in fact you don't even need a Turing machine, since there is no looping in one forward pass).

In theory you can run a huge FFN on the tiniest Turing machine, in practice it's much better to run a Transformer on the latest NVIDIA hardware. Or as they say "quantity (performance) has a quality all its own"

Re: How LLMs work

#127

Post by dist-epoch »

golergka wrote: post #107
malwrar wrote: post #10

Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…

After building some toy LLMs on my own I came to realise that architecture is not the hard part. Train is.

That's easy to say AFTER you know the architecture.

Einstein special relativity is taught these days in high-schools. Doesn't mean it wasn't the very hard part at some point in time.

As they say, shoulders of giants.

Re: How LLMs work

#128

Post by dist-epoch »

yukIttEft wrote: post #98

> so the model figures out during training what each token should look for and what it should offer But how does it learn this token-relationship? All it has is many text samples, but still, nowhere it says how the tokens relate to each other, so where does this information come from?

How does evolution learn the form-fitness relationship?

It's the same thing here, you randomly try various token-relationship values and the ones which are slightly better will be favoured.

Re: How LLMs work

#129

Post by zbendefy »

dist-epoch wrote: post #126
antirez wrote: post #75

Earlier quoted context omitted.

There is a different way to look at this: that is, actually the Transformer is a minimal complication of what the based model is: in theory the neural network could be just a huge FFN, which is anyway the part of the Transformer that does the heavy lifting. But this would be impossibile to train both numerically and computationally, so the Transformer encodes enough priors for it to work: the causal attention, and th…

Isn't that over-simplifying it a bit too much? You can go another step - a FFN can be simulated on a Turing machine, thus it just exemplifies the incredible semantical power of the Turing machine model of computation. (in fact you don't even need a Turing machine, since there is no looping in one forward pass). In theory you can run a huge FFN on the tiniest Turing machine, in practice it's much better to run a Trans…

Good point!

There is also the case for Markov chains being theoretically able to do these if tuned well. Or even SAT problem.

Re: How LLMs work

#130

Post by MagicMoonlight »

AltruisticGapHN wrote: post #88

I don't like how most LLM explainer articles and videos say that essentially a LLM " predicts the next word". I'm a developer but not very good at maths and I still don't understand any of it. A LLM clearly has some "visual" capacity. You ask Gemini to build something with Canvas and it's able to reason about the shape of things. Like recently I waanted a checkbox that has like a gradient flowing around the edge. It…

It can’t. It’s like a Redditor, it just repeats what it has seen other people say.

It has read all of stackoverflow, so it has seen your kind of problem before. Try asking it something really unusual and it will shit the bed.

Post reply on HN