Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How LLMs work

0xkato.xyz

Post reply on HN
Print view View original

141–150 of 293 posts

Re: How LLMs work

#141

Post by bluerooibos »

malwrar wrote: post #10

Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…

Since you spent a month digging into this, can you recommend any materials/projects to look into to get a decent grasp of how they work?

Re: How LLMs work

#142

Post by bluerooibos »

2muchcoffeeman wrote: post #110
crossroadsguy wrote: post #86

Earlier quoted context omitted.

What hopes/paths does a mere CS bachelor (not deep into stats/maths), and mid level dev (native mobile only; 10-15 years exp.), have about not only understanding it (maybe not fully) but getting possibly into this as a career? Not expecting churning out models and AI systems from the first weeks/months but entry/employment into this field? (If I can be honest, and I am not being disparaging about anything lest it mig…

I think you need to ask what you actually want to do with the AI. If you want to be a researcher and come out with the next breakthrough, get ready to go back to school and learn some math. If you just need to learn how to use it well and build things with it, then you probably just need to have a high level understanding. Same as programming. I’d bet most programmers have no idea about the physics that makes compute…

> I think you need to ask what you actually want to do with the AI.

What about improving the efficiency of token consumption, etc., basically opportunities for improving cost/performance?

I keep thinking there has to be a better way to share context with models than dumping entire gigantic skill files of raw text or otherwise into them - I'm betting there's a bunch of low-hanging fruit there.

Re: How LLMs work

#143

Post by LatencyKills »

bluerooibos wrote: post #141
malwrar wrote: post #10

Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…

Since you spent a month digging into this, can you recommend any materials/projects to look into to get a decent grasp of how they work?

Not OP but I worked through Sebastian Raschka's "Build a Large Language Model (From Scratch)" [0] and Raj Abhijit Dandekar's "Build a DeepSeek Model (From Scratch)" [1] books.

I don't think there is anything in a transformer I couldn't explain in the smallest detail now.

[0]: https://www.amazon.com/Build-Large-Language-Model-Scratch/dp...

[1]: https://www.amazon.com/Build-DeepSeek-Scratch-Abhijit-Dandek...

Re: How LLMs work

#144

Post by LatencyKills »

crossroadsguy wrote: post #86
malwrar wrote: post #10

Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…

What hopes/paths does a mere CS bachelor (not deep into stats/maths), and mid level dev (native mobile only; 10-15 years exp.), have about not only understanding it (maybe not fully) but getting possibly into this as a career? Not expecting churning out models and AI systems from the first weeks/months but entry/employment into this field? (If I can be honest, and I am not being disparaging about anything lest it mig…

I have a BS in CS (and have been in the field for 25 years). I couldn't understand the transformer architecture until I built a few myself. Here are the books I worked through. I now feel I have a very good understanding of modern LLMs.

https://www.amazon.com/Build-Large-Language-Model-Scratch/dp...

https://www.amazon.com/Build-DeepSeek-Scratch-Abhijit-Dandek...

Re: How LLMs work

#145

Post by skydhash »

throw310822 wrote: post #140
skydhash wrote: post #137

Earlier quoted context omitted.

It’s not unknown because that’s what the model computes. It’s matrix multiplication just like shaders.

And how do you know that the model computes it correctly?

Correctness is based on axioms and rules. You need to define your axioms and rules first before you can determine correctness.

If you’re talking about matrix multiplication, I can use mathematical rules and axioms and proves formally that the multiplication is correct. For next token prediction, I can prove that the set of tokens is finite and that the next token is always part of that set.

But things like grammar correctness, or semantic consistency over a few sentences are not hardcoded rules in the model. They’re emergent properties, mostly due to the amount and quality of data available for training. Quantization is mostly about how much we can shed without loosing a particular emergent properties (like dithering or psycho acoustic audio compression)

Re: How LLMs work

#146

Post by zaphirplane »

ctolsen wrote: post #38
10GBps wrote: post #15

Earlier quoted context omitted.

Yep. It's nearly identical to the neural nets we were using in the 90s. Back then even a supercomputer wasn't big enough or fast enough to do what we do today. I have to wonder though. Is this all a human brain is? A similar thing to an LLM just scaled exponentially larger. I mean a brain is not just neurons with simple connections to each other. The neurons, axons, dendrites, , etc in a brain are all holding and pro…

No, it’s definitely not what a human brain is. That makes very little sense. The ways we interact with language (and thus conceptual memory) is completely and fundamentally different.

But … how close a simulation is it. I can see why people are wondering

Re: How LLMs work

#148

Post by Borealid »

layla5alive wrote: post #113
Borealid wrote: post #90

Earlier quoted context omitted.

Your casual understanding is imprecise. At all times the LLM is, indeed, predicting the next token. Anything it does emerges from that. It did not "figure anything out". It predicted that text describing the use of a radial gradient was likely to follow text describing your problem.

Lol, the bird did not 'fly' - it just flapped its wings and generated lift!

No. The how is relevant here because it leads to understanding of the resulting behavior.

If you train the LLM on a corpus that shows people saying the sky is red, you get an LLM that is predisposed to say the sky is red. This is true even if it's also trained on all of the science that explains how and why the sky is blue.

If it were to "figure out" or "reason", it would not have such a predisposition to emit "red" after "the sky is" just because that matches the reward during training.

In other words, the token prediction is important because it both explains the successes AND the failures of the LLM. If there were situations in which a bird could fail to fly, then how it tried to fly would also be crucial knowledge.

Re: How LLMs work

#150

Post by hackinthebochs »

LatencyKills wrote: post #143
bluerooibos wrote: post #141

Earlier quoted context omitted.

Since you spent a month digging into this, can you recommend any materials/projects to look into to get a decent grasp of how they work?

Not OP but I worked through Sebastian Raschka's "Build a Large Language Model (From Scratch)" [0] and Raj Abhijit Dandekar's "Build a DeepSeek Model (From Scratch)" [1] books. I don't think there is anything in a transformer I couldn't explain in the smallest detail now. [0]: https://www.amazon.com/Build-Large-Language-Model-Scratch/dp... [1]: https://www.amazon.com/Build-DeepSeek-Scratch-Abhijit-Dandek...

>I don't think there is anything in a transformer I couldn't explain in the smallest detail now.

If you're up for it I would love to know how and why positional encodings work

Post reply on HN