How LLMs work • Hacker News Board

Re: How LLMs work

#231

Post by otabdeveloper4 » Sat, Jun 06, 2026, 9:09 PM UTC

layla5alive wrote: post #221

otabdeveloper4 wrote: post #173

Earlier quoted context omitted.

Inside the magic AI box is literally nothing but this loop: int n_tokens = 0; while (n_tokens If you don't believe me then just download llama.cpp and see for yourself.

decode() looks simple! Wow, obviously intelligence can't live behind that function call! /s Now, take that for loop, and replace the implementation of decode(context, ++position) and pass it to a human who was bored enough to play along and use a notebook to organize their thoughts and translate them to/from this encoding (you might write a helper function to do this for the human in the front-end of the new decode()…

The LLM predicts next token one at a time. (Stochastically.) This is a literal truth.

Deal with it.

Re: How LLMs work

#232

Post by spaceisballer » Sat, Jun 06, 2026, 9:10 PM UTC

Gareth321 wrote: post #218

spaceisballer wrote: post #207

Solid read especially for someone not in this field. While everything I’ve learned about LLMs has been pretty interesting, all I can say for sure is that I’m more and more skeptical about wide scale adoption. Consumers are being pushed almost to the level of coercion to utilize LLMs. Especially in the case of the government, who in most cases will get a free pass for a year to help build that addiction before the rea…

I’m kind of amazed when I read comments like this, but I have to remind myself that I work in an industry which use these tools at the cutting edge and see what they can really do. In the space of 18 months I changed from skeptic to the belief that our world is going to RAPIDLY change, and soon. I sense that statistics and benchmarks and research and statements from the world’s greatest academics won’t sway you, so m…

I have no doubt there are similar stories to yours or to a lesser extent. And while I’m glad you have gotten some relief, what price are we paying as a society? I’m not just talking about the environmental impact alone, what about the societal impact? Social media has not been around too long but it seems to be a net negative for society. I’m sure there are plenty of anecdotes to the contrary, but the studies are showing more and more of the detrimental impact to kids/teens. The stories and testimonies of how it was engineered to be addictive. Yet we’ve mostly moved on and allowed money more social media to get owned (and now traditional media) and controlled by the rich. But your statement is right, I doubt I’ll get swayed easily. Anecdotes and proclamations work when thinking short term. But taking the long view or looking at recent history I don’t see a positive. We have the hype being brought to us by the people that gave us the dot com bubble, crypto and NFTs. But hey, go fast, break things and never ever think about if what you are doing is really beneficial to society (after us, they are not us.)

Re: How LLMs work

#233

Post by otabdeveloper4 » Sat, Jun 06, 2026, 9:11 PM UTC

stevenhuang wrote: post #230

perching_aix wrote: post #178

Earlier quoted context omitted.

This "they just predict the next statistically most likely token" is such an handwavey and willfully misleading explanation, it's unreal, and I'm so fucking tired of seeing it so incessantly repeated. It's beyond asinine. You know it perfectly damn well that a typical person's idea of statistics is not some insanely high cardinality stateful prediction, but a "well a coin toss is a 50:50, and a lottery win is a 1:100…

I don't even try anymore. The people who still parrot the stochastic parrot bit this late in the game will simply never understand it.

LLMs predict next token one at a time. (Stochastically.) Literally. It's what they do. That's how they literally work.

If you don't believe me, download llama.cpp and see for yourself.

P.S. I write inference backends in C++ every day. The gall of people like you who figured out how to prompt Claude and think they're hot shit now is simply unbelievable.

Re: How LLMs work

#234

Post by jochembrouwer » Sat, Jun 06, 2026, 9:27 PM UTC

ekunazanu wrote: post #76

malwrar wrote: post #10

Back when ChatGPT came out, I was so shocked by how _good_ it was for an “AI” product that I simply had to know how it worked. Over the next month I ended up drawing out a block diagram on a whiteboard I have in my office, with the math involved next to each step in the blackboard. I’d puzzle about each step along the way, and the triumph of completing the drawing was also that of this sense of deep understanding. I…

> This was openai’s entire breakthrough. Making this particular model architecture larger leads to emergent capabilities Basically, the bitter lesson: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson...

So the take-away here is that we (as humans) try to model these AIs like humans, but eventually these AIs get better. Which to me seems like a logical conclusion if they can do "things" (like "learning" or pattern matching) much faster than we can (the compute). Then language in LLMs is a bottleneck, the AI is constrained by the language, and thus if we want to scale further we could let AI create its own language (we would then have to translate whatever it creates back to a language we understand). It is the same for instance if we check the language of Inuit (people who live in the north and make temorary shelters like igloos in the snow) they have multiple words/verbs to describe the snow, while in English we only have one (?): snow. In English we don't need more words (we can explain snow state using multiple words) but for the Inuit language it makes sense to create these new terms (would also make it easier and faster to communicate). So in some sense, all languages are then "newspeak" to whatever a general language is what researches or AI might come up with. If this sounds dumb let me know, but if you know some research in this general language direction (I'd assume general AI research) would love to see it!

Re: How LLMs work

#235

Post by wj » Sat, Jun 06, 2026, 10:42 PM UTC

janalsncm wrote: post #225

blharr wrote: post #167

Earlier quoted context omitted.

> There's good AI writing Sure, but the LLM-isms in AI writing are mentally exhausting to see in every way at this point. The whole point of reading, frankly, is to understand the voice of other people. When you pass that through a distorted filter that makes everyone sound the same... its bad, lossy, frustrating communication It's also dishonest. When you publish something that is direct output without your wording.…

The point of writing is to convey an idea to another person or yourself at a future date. Authenticity has nothing to do with it. I frankly do not care about the “authentic voice” of the author of a random blog. I want to know if they have any interesting ideas.

I think because so much of an idea is shaped by the language used to convey it, it may be hard to separate the person from the LLM.

I think gp may want to know if a has an interesting idea rather than .

Re: How LLMs work

#236

Post by stevenhuang » Sat, Jun 06, 2026, 10:44 PM UTC

otabdeveloper4 wrote: post #233

stevenhuang wrote: post #230

Earlier quoted context omitted.

I don't even try anymore. The people who still parrot the stochastic parrot bit this late in the game will simply never understand it.

LLMs predict next token one at a time. (Stochastically.) Literally. It's what they do. That's how they literally work. If you don't believe me, download llama.cpp and see for yourself. P.S. I write inference backends in C++ every day. The gall of people like you who figured out how to prompt Claude and think they're hot shit now is simply unbelievable.

I help write optimized CUDA kernels for proprietary hardware. They may "literally" work this way, but that is quite besides the point.

If you don't see why then you have exactly demonstrated my point in how practitioners like you simply lack the foundational understanding in philosophy, information theory, human consciousness, human cognition, neuroscience, necessary to bridge this conceptual gap.

(Rather, it is that we know so little of how consciousness or what intelligence even is, that we cannot possibly use first principles to preclude LLMs from possessing these qualities)

You don't understand the argument, so you keep repeating first order mechanistic observations that are irrelevant. If you don't want to understand the argument, don't be surprised when people refuse to engage with you, especially when it's evident to those more knowledgeable the position you hold is the ignorant one.

Re: How LLMs work

#237

Post by sigmoid10 » Sat, Jun 06, 2026, 11:11 PM UTC

redox99 wrote: post #95

sigmoid10 wrote: post #42

Earlier quoted context omitted.

"Attention is all you need" is actually a bad paper if you want to learn about autoregressive LLMs specifically, because it describes a more complicated encoder-decoder architecture while modern LLMs are decoder only. So it's an unnecessarily hard way to get into the subject. "Language Models are Unsupervised Multitask Learners" is probably what you are looking for (aka the GPT-2 paper). This was the first time LLMs…

Chain of Thought was kind of an obvious solution that everybody knew was necessary by the time chatgpt / gpt4 came out. It was just a matter of time that frontier labs actually shipped it. MoE was also pretty straightforward, just a bit surprising how well it worked (that you can get away with just 1/32 active parameters), but most researchers would have come up with it on their own probably. The true ground breaking…

Reasoning is a little bit more than just "baked in" chain of thought prompting. The important takeaway here was that it is not realized at the architecture level of the neural network. And you could say that all these things regarding LLMs were pretty straightforward. But only in hindsight, otherwise there wouldn't have been so much time and effort spent on intermediaries. Breakthroughs mean people simply didn't know stuff before, even if it seems easy with the benefit of hindsight.

Re: How LLMs work

#238

Post by youareartree » Sat, Jun 06, 2026, 11:16 PM UTC

[dead]

Re: How LLMs work

#239

Post by perching_aix » Sat, Jun 06, 2026, 11:18 PM UTC

otabdeveloper4 wrote: post #233

stevenhuang wrote: post #230

Earlier quoted context omitted.

I don't even try anymore. The people who still parrot the stochastic parrot bit this late in the game will simply never understand it.

LLMs predict next token one at a time. (Stochastically.) Literally. It's what they do. That's how they literally work. If you don't believe me, download llama.cpp and see for yourself. P.S. I write inference backends in C++ every day. The gall of people like you who figured out how to prompt Claude and think they're hot shit now is simply unbelievable.

So you work on inference engines, and don't see at all what'd be hilariously disingenuous and reductive about describing how LLMs operate as "just parroting the most statistically likely next token"? It is literally* what they do, yes. And only literally, with a big asterisk of "non-colloquial meaning" after the word "statistically". Like how "significant" means something pretty different, albeit related, in academic writing vs everyday speech.

It's equivalent to professing how you just make apple pies from scratch, while your first step is to always reinvent the universe.

You're further magically blind to this operational fact being weaponized as a trope for furthering anti-ai sentiment (i.e. that it's a political dogwhistle at this point), and to thus you participating in that every time you repeat it?

* Ignoring the decoding caveat I already mentioned, along with the countless ways they're steered. There isn't jack that's likely about some of the responses they produce, and intentionally so. Including the whole chat partner act.

Re: How LLMs work

#240

Post by Borealid » Sat, Jun 06, 2026, 11:23 PM UTC

hackinthebochs wrote: post #155

Borealid wrote: post #90

Earlier quoted context omitted.

Your casual understanding is imprecise. At all times the LLM is, indeed, predicting the next token. Anything it does emerges from that. It did not "figure anything out". It predicted that text describing the use of a radial gradient was likely to follow text describing your problem.

>At all times the LLM is, indeed, predicting the next token The point is that saying they're just "predicting the next token" is not at all explanatory nor providing insight. Saying the brain is just firing action potentials gives you no understanding about how the brain does what it does or what the space of its capabilities are. Similarly, predicting the next token tells you nothing about the capabilities of LLMs.

The point is that the output is text that is statistically correlated with the input.

The capability of the LLM is not to reason, it's to generate text that matches the patterns seen in the training corpus. It's possible that all you need to "reason" is plausible text generation. I'm not saying it's not. But nothing the LLM does fails to be explained by plausible-text-generation.

I contend that the best way to understand an LLM's capabilities is to understand the nature of the probability distribution that produced it. For instance, why does an "angry" prompt tend to produce more help than a "polite" one? Trying to explain that in terms of emotions or reasoning doesn't make sense, but it's readily possible to explain through the connections between text in the training corpus...