Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How LLMs work

0xkato.xyz

Post reply on HN
Print view View original

151–160 of 293 posts

Re: How LLMs work

#151

Post by jmalicki »

otabdeveloper4 wrote: post #39
ai_slop_hater wrote: post #34

Earlier quoted context omitted.

No they are clearly not just scaled up versions of gpt 2; there are different LLM architectures like mixture of experts etc that appeared relatively recently. I am not an expert though, far from it.

MoE and such are basically performance enhancements, they don't make the model smarter.

Performance enhancements are huge though.

If you can make the existing model faster, you can then save your inference budget to then make your model bigger, which then makes it smarter.

A lot of how smart the models can be comes down to budget. If you can make your existing thing cheaper, you can instead make it bigger for the same price.

Re: How LLMs work

#152

Post by coliveira »

malwrar wrote: post #10

Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…

Don't forget the stolen data from books and papers. You'll never get anything intelligent without using the stolen data they had access to.

Re: How LLMs work

#153

Post by dominotw »

malwrar wrote: post #10

Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…

> Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I kept that drawing up for many months after, and would gaze at it often during meetings and idle moments in wonder.

how did you know about the steps and there was math involved. i am curious about your process and you came up with what exactly to learn to unravel the mystery.

Re: How LLMs work

#154

Post by coliveira »

bluerooibos wrote: post #142
2muchcoffeeman wrote: post #110

Earlier quoted context omitted.

I think you need to ask what you actually want to do with the AI. If you want to be a researcher and come out with the next breakthrough, get ready to go back to school and learn some math. If you just need to learn how to use it well and build things with it, then you probably just need to have a high level understanding. Same as programming. I’d bet most programmers have no idea about the physics that makes compute…

> I think you need to ask what you actually want to do with the AI. What about improving the efficiency of token consumption, etc., basically opportunities for improving cost/performance? I keep thinking there has to be a better way to share context with models than dumping entire gigantic skill files of raw text or otherwise into them - I'm betting there's a bunch of low-hanging fruit there.

There may be some low hanging fruit, but they're not available to people without deep understanding of how the math works. Well paid people already spend a lot of time thinking about this.

Re: How LLMs work

#155

Post by hackinthebochs »

Borealid wrote: post #90
AltruisticGapHN wrote: post #88

I don't like how most LLM explainer articles and videos say that essentially a LLM " predicts the next word". I'm a developer but not very good at maths and I still don't understand any of it. A LLM clearly has some "visual" capacity. You ask Gemini to build something with Canvas and it's able to reason about the shape of things. Like recently I waanted a checkbox that has like a gradient flowing around the edge. It…

Your casual understanding is imprecise. At all times the LLM is, indeed, predicting the next token. Anything it does emerges from that. It did not "figure anything out". It predicted that text describing the use of a radial gradient was likely to follow text describing your problem.

>At all times the LLM is, indeed, predicting the next token

The point is that saying they're just "predicting the next token" is not at all explanatory nor providing insight. Saying the brain is just firing action potentials gives you no understanding about how the brain does what it does or what the space of its capabilities are. Similarly, predicting the next token tells you nothing about the capabilities of LLMs.

Re: How LLMs work

#156

Post by LatencyKills »

hackinthebochs wrote: post #150
LatencyKills wrote: post #143

Earlier quoted context omitted.

Not OP but I worked through Sebastian Raschka's "Build a Large Language Model (From Scratch)" [0] and Raj Abhijit Dandekar's "Build a DeepSeek Model (From Scratch)" [1] books. I don't think there is anything in a transformer I couldn't explain in the smallest detail now. [0]: https://www.amazon.com/Build-Large-Language-Model-Scratch/dp... [1]: https://www.amazon.com/Build-DeepSeek-Scratch-Abhijit-Dandek...

>I don't think there is anything in a transformer I couldn't explain in the smallest detail now. If you're up for it I would love to know how and why positional encodings work

Well, as I suggested, working through the implementation yourself will give you that intuition. That said, I think the simplest way to explain why positional encodings are useful is that it gives the transformer just enough information to make attention meaningful without negatively impacting any parallel, content-based comparisons.

A vanilla self-attention layer is just a set of token vectors. Without positional info, swapping two identical embeddings changes very little about what attention can compute. We can "fix" this problem by using positional encodings. Text that has meaning isn't just a set of characters; the location and order of those characters is what provides meaning.

Re: How LLMs work

#157

Post by TheHalfDeafChef »

jmalicki wrote: post #151
otabdeveloper4 wrote: post #39

Earlier quoted context omitted.

MoE and such are basically performance enhancements, they don't make the model smarter.

Performance enhancements are huge though. If you can make the existing model faster, you can then save your inference budget to then make your model bigger, which then makes it smarter. A lot of how smart the models can be comes down to budget. If you can make your existing thing cheaper, you can instead make it bigger for the same price.

Not really “smarter” though? It’s just a big probability engine.

(Not trying to flame bait or anything. I just wouldn’t call LLM as exhibiting intelligence. It is great at making connections based on probability but doesn’t have a semantic understanding of what it is doing)

Re: How LLMs work

#158

Post by klempner »

otabdeveloper4 wrote: post #43
jumploops wrote: post #22

Earlier quoted context omitted.

Those are all just optimizations. We still don’t really know why they work, we just know how to build them.

We do know how they work. They predict the next statistically most likely token. The "bitter lesson" is that fake-it-till-you-make-it is a valid way of doing knowledge work. (Or not make it, then people will just claim you're holding the LLM wrong and it's not the AI's fault.)

Sufficiently good iterated next token prediction is an AI hard problem.

Re: How LLMs work

#159

Post by YeGoblynQueenne »

AltruisticGapHN wrote: post #88

I don't like how most LLM explainer articles and videos say that essentially a LLM " predicts the next word". I'm a developer but not very good at maths and I still don't understand any of it. A LLM clearly has some "visual" capacity. You ask Gemini to build something with Canvas and it's able to reason about the shape of things. Like recently I waanted a checkbox that has like a gradient flowing around the edge. It…

Sorry you're being downvoted for asking a very reasonable question. I don't think any of the replies here address your question either.

If I can do my best to answer, Gemini is a multi-modal system. That means it's trained not only on text but also still images, video and also sound. The training happens in parallel and the representation of each modality is usually different, so the image recognition part is not trained on text tokens but pixels, the video part (probably) on video frames etc. There is some kind of integrated training that goes on so that text can be generated that is correlated to an image and so on, but I don't know the specifics about Gemini in particular. This kind of thing is not exactly new either, you can find systems that captioned images before the rise of LLMs simply by training on examples of images coupled to their textual descriptions.

In that sense it's not entirely correct to call Gemini an "LLM" because it's not only a "language" (or, more precisely, text) model. But LLM I guess becomes a bit of a shorthand for everything based on, or combined with, an LLM.

Anyway that's what's going on: it's not just predicting the next word. It's also predicting the next image frame or the next set of pixels etc associated with the next word.

Re: How LLMs work

#160

Post by root-parent »

malwrar wrote: post #10

Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…

I had the same reaction as you, when I learned in detail, how all this works. But then I also learned about superposition and compressed sensing, and now...I am not so sure anymore...

"Beating Nyquist with Compressed Sensing" - https://youtu.be/A8W1I3mtjp8

Post reply on HN