Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Illustrated Transformer

jalammar.github.io

Post reply on HN
Print view View original

71–80 of 92 posts

Re: The Illustrated Transformer

#71

Post by libraryofbabel »

crystal_revenge wrote: post #69
libraryofbabel wrote: post #30

I read this article back when I was learning the basics of transformers; the visualizations were really helpful. Although in retrospect knowing how a transformer works wasn't very useful at all in my day job applying LLMs, except as a sort of deep background for reassurance that I had some idea of how the big black box producing the tokens was put together, and to give me the mathematical basis for things like contex…

> massive application of reinforcement learning techniques So sad that "reinforcement learning" is another term whose meaning has been completely destroyed by uneducated hype around LLMs (very similar to "agents"). 5 years ago nobody familiar with RL would consider what these companies are doing as "reinforcement learning". RLHF and similar techniques are much, much closer to traditional fine-tuning than they are rei…

I wasn't referring to RLHF, which people were of course already doing heavily in 2023, but RLVR, aka LLMs solving tons of coding and math problems with a reward function after pre-training. I discussed that in another reply, so I won't repeat it here; instead I'd just refer you to Andrej Karpathy's 2025 LLM Year in Review which discusses it. https://karpathy.bearblog.dev/year-in-review-2025/

> I've found LLMs less useful in 2025 than in 2024.

I really don't know how to reply to this part without sounding insulting, so I won't.

Re: The Illustrated Transformer

#72

Post by edge17 »

D-Machine wrote: post #63
edge17 wrote: post #62

Maybe I'm out of touch, but have transformers replaced all traditional deep learning architectures? (U-nets, etc)?

No, not at all. There is a transformer obsession that is quite possibly not supported by the actual facts (CNNs can still do just as well: https://arxiv.org/abs/2310.16764 ), and CNNs definitely remain preferable for smaller and more specialized tasks (e.g. computer vision on medical data). If you also get into more robust and/or specialized tasks (e.g. rotation invariant computer vision models, graph neural networks…

Is there something I can read to get a better sense of what types of models are most suitable for which problems? All I hear about are transformers nowadays, but what are the types of problems for which transformers are the right architecture choice?

Re: The Illustrated Transformer

#73

Post by [deleted] »

libraryofbabel wrote: post #71
crystal_revenge wrote: post #69

Earlier quoted context omitted.

> massive application of reinforcement learning techniques So sad that "reinforcement learning" is another term whose meaning has been completely destroyed by uneducated hype around LLMs (very similar to "agents"). 5 years ago nobody familiar with RL would consider what these companies are doing as "reinforcement learning". RLHF and similar techniques are much, much closer to traditional fine-tuning than they are rei…

I wasn't referring to RLHF, which people were of course already doing heavily in 2023, but RLVR, aka LLMs solving tons of coding and math problems with a reward function after pre-training. I discussed that in another reply, so I won't repeat it here; instead I'd just refer you to Andrej Karpathy's 2025 LLM Year in Review which discusses it. https://karpathy.bearblog.dev/year-in-review-2025/ > I've found LLMs less us…

[deleted]

Re: The Illustrated Transformer

#74

Post by crystal_revenge »

libraryofbabel wrote: post #71
crystal_revenge wrote: post #69

Earlier quoted context omitted.

> massive application of reinforcement learning techniques So sad that "reinforcement learning" is another term whose meaning has been completely destroyed by uneducated hype around LLMs (very similar to "agents"). 5 years ago nobody familiar with RL would consider what these companies are doing as "reinforcement learning". RLHF and similar techniques are much, much closer to traditional fine-tuning than they are rei…

I wasn't referring to RLHF, which people were of course already doing heavily in 2023, but RLVR, aka LLMs solving tons of coding and math problems with a reward function after pre-training. I discussed that in another reply, so I won't repeat it here; instead I'd just refer you to Andrej Karpathy's 2025 LLM Year in Review which discusses it. https://karpathy.bearblog.dev/year-in-review-2025/ > I've found LLMs less us…

While RLVF is neat, it still is an 'offline' learning model that just borrows a reward function similar to RL.

And did you not read the entire post? Karpathy basically calls out the same point that I am making regarding RL which "of course can be exploited to help move the needle on benchmarks":

> Related to all this is my general apathy and loss of trust in benchmarks in 2025. The core issue is that benchmarks are almost by construction verifiable environments and are therefore immediately susceptible to RLVR and weaker forms of it via synthetic data generation. In the typical benchmaxxing process, teams in LLM labs inevitably construct environments adjacent to little pockets of the embedding space occupied by benchmarks and grow jaggies to cover them. Training on the test set is a new art form

Regarding:

> I really don't know how to reply to this part without sounding insulting, so I won't.

Relevant to citing him: Karpathy has publicly praised some of my past research in LLMs, so please don't hold back your insults. A poster on HN telling me I'm "not using them right!!!" won't shake my confidence terribly. I use LLMs less this year than last year and have been much more productive. I still use them, LLMs are interesting, and very useful. I just don't understand why people have to get into hysterics trying to make them more than that.

I also agree with Karpathy's statement:

> In any case they are extremely useful and I don't think the industry has realized anywhere near 10% of their potential even at present capability.

But magical thinking around them is slowing down progress imho. Your original comment itself is evidence of this:

> I would strongly caution anyone who thinks that they will be able to understand or explain LLM behavior better by studying the architecture closely.

I would say "Rip them open! Start playing around with the internals! Mess around with sampling algorithms! Ignore the 'win market share' hype and benchmark gaming and see just what you can make these models do!" Even if restricted to just open, relatively small models, there's so much more interesting work in this space.

Re: The Illustrated Transformer

#76

Post by throw310822 »

sakesun wrote: post #46
throw310822 wrote: post #14

Earlier quoted context omitted.

Have you tried asking e.g. Claude to explain it to you? None of the usual resources worked for me, until I had a discussion with Claude where I could ask questions about everything that I didn't get.

Perhaps we have already reached ASI. :)

In some respects, yes. There is no single human being with a general knowledge as vast as that of a SOTA LLM, or able to speak as many languages. Claude knows about transformers more than enough to explain them to a layperson, elucidating specific points and resolving doubts. As someone who learns more easily by prodding other people's knowledge rather than from static explanations, I find LLMs extremely useful.

Re: The Illustrated Transformer

#77

Post by menaerus »

crystal_revenge wrote: post #74
libraryofbabel wrote: post #71

Earlier quoted context omitted.

I wasn't referring to RLHF, which people were of course already doing heavily in 2023, but RLVR, aka LLMs solving tons of coding and math problems with a reward function after pre-training. I discussed that in another reply, so I won't repeat it here; instead I'd just refer you to Andrej Karpathy's 2025 LLM Year in Review which discusses it. https://karpathy.bearblog.dev/year-in-review-2025/ > I've found LLMs less us…

While RLVF is neat, it still is an 'offline' learning model that just borrows a reward function similar to RL. And did you not read the entire post? Karpathy basically calls out the same point that I am making regarding RL which "of course can be exploited to help move the needle on benchmarks": > Related to all this is my general apathy and loss of trust in benchmarks in 2025. The core issue is that benchmarks are a…

What do you think about Geoffrey Hinton's concerns about the AI (minus "AGI")? Do you agree with those concerns or do you believe that LLMs are only that much "useful" so they wouldn't impose a risk on our society?

Re: The Illustrated Transformer

#78

Post by menaerus »

D-Machine wrote: post #63
edge17 wrote: post #62

Maybe I'm out of touch, but have transformers replaced all traditional deep learning architectures? (U-nets, etc)?

No, not at all. There is a transformer obsession that is quite possibly not supported by the actual facts (CNNs can still do just as well: https://arxiv.org/abs/2310.16764 ), and CNNs definitely remain preferable for smaller and more specialized tasks (e.g. computer vision on medical data). If you also get into more robust and/or specialized tasks (e.g. rotation invariant computer vision models, graph neural networks…

Using transformers does not mutually exclude other tools in the sleeve.

What about DINOv2 and DINOv3, 1B and 7B, vision transformer models? This paper [1] suggests significant improvements over traditional YOLO-based object detection.

[1] https://arxiv.org/html/2509.20787v2

Re: The Illustrated Transformer

#79

Post by zkmon »

crystal_revenge wrote: post #67
zkmon wrote: post #27

I think the internal of transformers would become less relevant like internal of compilers, as programmers would only care about how to "use" them instead of how to develop them.

Have you written a compiler? I ask because for me writing a compiler was absolutely an inflection point in my journey as a programmer. Being able to look at code and reason about it all the way down to bytecode/IL/asm etc absolutely improved my skill as a programmer and ability to reason about software. For me this was the first time I felt like a real programmer.

Writing a compiler is not a requirement or good use of time for a programmer. Same as why driving a car should not require you to build the car engine. Driver should stick to their role and learn how to drive properly.

Re: The Illustrated Transformer

#80

Post by oedemis »

profsummergig wrote: post #2

Haven't watched it yet... ...but, if you have favorite resources on understanding Q & K, please drop them in comments below... (I've watched the Grant Sanderson/3blue1brown videos [including his excellent talk at TNG Big Tech Day '24], but Q & K still escape me). Thank you in advance.

there is also very good explanation from Luis Serrano, https://youtu.be/fkO9T027an0
Post reply on HN