How LLMs work • Hacker News Board

Re: How LLMs work

#221

Post by layla5alive » Sat, Jun 06, 2026, 8:17 PM UTC

otabdeveloper4 wrote: post #173

Ampersander wrote: post #104

Earlier quoted context omitted.

I do agree bigly. Calling what is basically a superhuman brain inside a computer just a "token predictor" is peak thinkslop.

Inside the magic AI box is literally nothing but this loop: int n_tokens = 0; while (n_tokens If you don't believe me then just download llama.cpp and see for yourself.

decode() looks simple! Wow, obviously intelligence can't live behind that function call! /s

Now, take that for loop, and replace the implementation of decode(context, ++position) and pass it to a human who was bored enough to play along and use a notebook to organize their thoughts and translate them to/from this encoding (you might write a helper function to do this for the human in the front-end of the new decode() impl, but the data flow in and out of decode() will remain the same):

decode(context, position)

{

  cached string_answer = ask_human_question_via_context(context);

  return decode_human_answer_to_tokens(cached string_answer, position);

}

Is the output you get not thinking anymore because it passed through this harness? Did the human's mind somehow get reduced to mere interpolation?

The human mind is still a human mind. Putting a simple harness in front of a mind does not affect its fundamental properties.

In an LLM, decode() is calling into a trillion parameter connectome.

Re: How LLMs work

#222

Post by galaxyLogic » Sat, Jun 06, 2026, 8:24 PM UTC

jumploops wrote: post #35

trollbridge wrote: post #26

Earlier quoted context omitted.

We don't really know why language works with humans, either. If you raise a baby from birth, you kind of observe how it is learning language, but the process is also rather mysterious. My eldest son's first word was to actually imitate a cow mooing, and then after that to imitate a motor noise of a tractor or truck. And then after that a meow. (His first complete sentence was "King Graham fell"...) My next child took…

Completely agree! It’s interesting to me how similar attempting to understand LLMs is to neuroscience. “When we turn this bit off, this other thing happens… if we change these weights the Eiffel Tower is now in Rome” We’re basically just probing around and trying to reverse engineer an emergent system. To your point, this system may be quite different from model to model (human to human) although some similarities li…

Isn't the LLM simply predicting what should be the next sentences after user's input, using its algorithm and data it has exatrcted from existing texts on the internet. The algorithm that does that could have many different designs, some better some worse for the purpose of predicting what output makes most sense next?

So what is it that we don't understand about why theyr work? The algorithm? We have the code. Why the specific algorithm makes such good predictions? I see it as a generalization of trying to predict who wins Kentucky Derby.

Re: How LLMs work

#223

Post by metaquestions » Sat, Jun 06, 2026, 8:26 PM UTC

Thanks for putting this together. I found it very helpful

Re: How LLMs work

#224

Post by galaxyLogic » Sat, Jun 06, 2026, 8:34 PM UTC

Marha01 wrote: post #122

AltruisticGapHN wrote: post #88

I don't like how most LLM explainer articles and videos say that essentially a LLM " predicts the next word". I'm a developer but not very good at maths and I still don't understand any of it. A LLM clearly has some "visual" capacity. You ask Gemini to build something with Canvas and it's able to reason about the shape of things. Like recently I waanted a checkbox that has like a gradient flowing around the edge. It…

LLMs fundamentally work by predicting the next word (token). But that should not be used to diminish their potential capabilities. It's like saying that human brains "just predict (or produce) the next electrical impulse". Fundamentally correct, but says nothing about the potential emergent capabilities of scaled-up systems that work like that. Emergent properties of complex systems should not be diminished just beca…

Game of Life comes to mind: Most simple logic, emerging patterns are hard to believe.

Re: How LLMs work

#225

Post by janalsncm » Sat, Jun 06, 2026, 8:35 PM UTC

blharr wrote: post #167

janalsncm wrote: post #81

Earlier quoted context omitted.

Not just that, I think a lot of people are going to waste their time losing the battle (and make no mistake, they will lose) fighting against AI writing without ever asking themselves what makes writing good in the first place. There’s good AI writing and bad organic writing. But it’s easier to point out a few LLM-isms than to actually identify the problems with text.

> There's good AI writing Sure, but the LLM-isms in AI writing are mentally exhausting to see in every way at this point. The whole point of reading, frankly, is to understand the voice of other people. When you pass that through a distorted filter that makes everyone sound the same... its bad, lossy, frustrating communication It's also dishonest. When you publish something that is direct output without your wording.…

The point of writing is to convey an idea to another person or yourself at a future date. Authenticity has nothing to do with it. I frankly do not care about the “authentic voice” of the author of a random blog. I want to know if they have any interesting ideas.

Re: How LLMs work

#226

Post by galaxyLogic » Sat, Jun 06, 2026, 8:40 PM UTC

hackinthebochs wrote: post #155

Borealid wrote: post #90

Earlier quoted context omitted.

Your casual understanding is imprecise. At all times the LLM is, indeed, predicting the next token. Anything it does emerges from that. It did not "figure anything out". It predicted that text describing the use of a radial gradient was likely to follow text describing your problem.

>At all times the LLM is, indeed, predicting the next token The point is that saying they're just "predicting the next token" is not at all explanatory nor providing insight. Saying the brain is just firing action potentials gives you no understanding about how the brain does what it does or what the space of its capabilities are. Similarly, predicting the next token tells you nothing about the capabilities of LLMs.

True, but that is a great fact to start from, and understand.

Then the next question becomes "HOW do they predict the next token?" There are many ways that can be done, why is this particular algorithm so GOOD?"

When people say "We don't understand how LLM works" isn't it really saying we don't understand how this specific algorithm used to predict the next token works? No, it is not, because "we" do understand how all those algorithms work there are many descriptions of them available.

So the question then really is "Why is the prediction this algorithm makes, so good, as compared to some other statistical algorithms?"

It's not about "Why does AI work so well?". It should be "Why does this particular XYZ algorithm work so well?"

Re: How LLMs work

#227

Post by rhubarbtree » Sat, Jun 06, 2026, 8:45 PM UTC

helloplanets wrote: post #87

The part about positional encoding is not correct. > The intuition: instead of adding position info to each token’s vector, RoPE rotates the vector by an angle that depends on its position You can't rotate the token's entire vector (or all three vectors, whatever is being implied is unclear). You rotate each token's Query and Key vectors only, so dot product can be used to tell how far apart the tokens are when compa…

Yep, you’re correct. I got to that bit and thought that can’t be right. It’s obviously wrong. If you rotate a semantic vector, you change the semantics of it. You don’t want that.

Makes me wonder if the whole thing is just slop.

Is the rest of the article correct?

Anyone suggest an alternative article?

Re: How LLMs work

#228

Post by fizx » Sat, Jun 06, 2026, 8:59 PM UTC

otabdeveloper4 wrote: post #39

ai_slop_hater wrote: post #34

Earlier quoted context omitted.

No they are clearly not just scaled up versions of gpt 2; there are different LLM architectures like mixture of experts etc that appeared relatively recently. I am not an expert though, far from it.

MoE and such are basically performance enhancements, they don't make the model smarter.

Performance enhancements are what allow you to train a bigger model.

Re: How LLMs work

#229

Post by FergusArgyll » Sat, Jun 06, 2026, 9:00 PM UTC

miki123211 wrote: post #166

There's one thing I wish people understood about LLMs, and it doesn't really have anything to do with what's inside the neural network part. It's the fact that LLMs can only write in one direction — forward. When you are writing an essay and realize midway through a sentence that what you've written doesn't make sense, you go back and edit. An LLM can't do that, the only thing it can do is keep on generating. Because…

I get what you're saying but to be slightly pedantic etc.

Why can't an llm tool call `delete(index: int)` or `replace(from_index: int, to_index: int, string: str)` and then it can go back and edit just the way we can?

We also first made the mistake and only afterwards noticed we actually want to change something

Re: How LLMs work

#230

Post by stevenhuang » Sat, Jun 06, 2026, 9:08 PM UTC

perching_aix wrote: post #178

otabdeveloper4 wrote: post #43

Earlier quoted context omitted.

We do know how they work. They predict the next statistically most likely token. The "bitter lesson" is that fake-it-till-you-make-it is a valid way of doing knowledge work. (Or not make it, then people will just claim you're holding the LLM wrong and it's not the AI's fault.)

This "they just predict the next statistically most likely token" is such an handwavey and willfully misleading explanation, it's unreal, and I'm so fucking tired of seeing it so incessantly repeated. It's beyond asinine. You know it perfectly damn well that a typical person's idea of statistics is not some insanely high cardinality stateful prediction, but a "well a coin toss is a 50:50, and a lottery win is a 1:100…

I don't even try anymore. The people who still parrot the stochastic parrot bit this late in the game will simply never understand it.