Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Illustrated Transformer

jalammar.github.io

Post reply on HN
Print view View original

31–40 of 92 posts

Re: The Illustrated Transformer

#31

Post by machinationu »

profsummergig wrote: post #2

Haven't watched it yet... ...but, if you have favorite resources on understanding Q & K, please drop them in comments below... (I've watched the Grant Sanderson/3blue1brown videos [including his excellent talk at TNG Big Tech Day '24], but Q & K still escape me). Thank you in advance.

Q, K and V are a way of filtering the relevant aspects for the task at hand from the token embeddings.

"he was red" - maybe color, maybe angry, the "red" token embedding carries both, but only one aspect is relevant for some particular prompt.

https://ngrok.com/blog/prompt-caching/

Re: The Illustrated Transformer

#32

Post by rvz »

zkmon wrote: post #27

I think the internal of transformers would become less relevant like internal of compilers, as programmers would only care about how to "use" them instead of how to develop them.

Their internals are just as relevant (now even more relevant) as any other technology as they always need to be improved to the SOTA (state of the art) meaning that someone has to understand their internals.

It also means more jobs for the people who understand them at a deeper level to advance the SOTA of specific widely used technologies such as operating systems, compilers, neural network architectures and hardware such as GPUs or TPU chips.

Someone has to maintain and improve them.

Re: The Illustrated Transformer

#33

Post by bonesss »

qoez wrote: post #13
gustavoaca1997 wrote: post #8

I have this book. Really a life savior to help me catching up a few months ago when my team decided to use LLMs in our systems.

Don't really see why you'd need to understand how the transformer works to do LLMs at work. LLMs is just a synthetic human performing reasoning with some failure modes that in-depth knowledge of the transformer interals won't help you predict what they are (just have to use experience with the output to get a sense, or other peoples experiments).

> LLMs is just a synthetic human

1) ‘human’ encompasses behaviours that include revenge cannibalism and recurrent sexual violence —- wish carefully.

2) not even a little bit, and if you want to pretend then pretend they’re a deranged delusional psych patient who will look you in the eye and say genuinely “oops, I guess I was lying, it won’t ever happen again” and then lie to you again, while making sure happens again.

3) don’t anthropomorphize LLMs, they don’t like it.

Re: The Illustrated Transformer

#34

Post by roadside_picnic »

amelius wrote: post #26
roadside_picnic wrote: post #17

Earlier quoted context omitted.

In my experience this is a substantial difference in the ability to really get performance in LLM related engineering work from people who really understand how LLMs work vs people who think it's a magic box. If your mental model of an LLM is: > a synthetic human performing reasoning You are severely overestimating the capabilities of these models and not realizing potential areas of failure (even if your prompt work…

But that AI engineer who is implementing speculative decoding is still just doing basic plumbing that has little to do with the actual reasoning. Yes, he/she might make the process faster, but they will know just as little about why/how the reasoning works as when they implemented a naive, slow version of the inference.

What "actual reasoning" are you referring to? I believe you're making my point for me.

Speculative decoding requires the implementer to understand:

- How the initial prompt is processed by the LLM

- How to retrieve all the probabilities of previously observed tokens in the prompt (this also help people understand things like the probability of the entire prompt itself, the entropy of the prompt etc).

- Details of how the logits generate the distribution of next tokens

- Precise details of the sampling process + the rejection sampling logic for comparing the two models

- How each step of the LLM is run under-the-hood as the response is processed.

Hardly just plumbing, especially since, to my knowledge, there are not a lot of hand-holding tutorials on this topic. You need to really internalize what's going on and how this is going to lead to a 2-5x speed up in inference.

Building all of this yourself gives you a lot of visibility into how the model behaves and how "reasoning" emerges from the sampling process.

edit: Anyone who can perform speculative decoding work also has the ability to inspect the reasoning steps of an LLM and do experiments such as rewinding the thought process of the LLM and substituting a reasoning step to see how it impacts the results. If you're just prompt hacking you're not going to be able to perform these types of experiments to understand exactly how the model is reasoning and what's important to it.

Re: The Illustrated Transformer

#35

Post by esafak »

zkmon wrote: post #27

I think the internal of transformers would become less relevant like internal of compilers, as programmers would only care about how to "use" them instead of how to develop them.

Practitioners already do not need to know about it to run let alone use LLMs. I bet most don't even know the fundamentals of machine learning. Hands up if you know bias from variance...

Re: The Illustrated Transformer

#36

Post by jwitthuhn »

krat0sprakhar wrote: post #12
red2awn wrote: post #3

Earlier quoted context omitted.

Implement transformers yourself (ie in Numpy). You'll never truly understand it by just watching videos.

Do you have a tutorial that I can follow?

If you have 20 hours to spare I highly recommend this youtube playlist from Andrej Karpathy https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...

It starts with the fundamentals of how backpropagation works then advances to building a few simple models and ends with building a GPT-2 clone. It won't taech you everything about AI models but it gives you a solid foundation for branching out.

Re: The Illustrated Transformer

#37

Post by nrhrjrjrjtntbt »

libraryofbabel wrote: post #30

I read this article back when I was learning the basics of transformers; the visualizations were really helpful. Although in retrospect knowing how a transformer works wasn't very useful at all in my day job applying LLMs, except as a sort of deep background for reassurance that I had some idea of how the big black box producing the tokens was put together, and to give me the mathematical basis for things like contex…

It is almost like understanding wood at a molecular level and being a carpenter. It also may help the carpentery, but you cam be a great one without it. And a bad one with the knowledge.

Re: The Illustrated Transformer

#38

Post by gzer0 »

boltzmann_ wrote: post #7

Kudos also to Transformer Explainer team for putting some amazing visualizations https://poloclub.github.io/transformer-explainer/ It really clicked to me after reading this two and watching 3blue1brown videos

This is hands down one of the best visualizations I have ever come across.

Re: The Illustrated Transformer

#39

Post by ozgung »

libraryofbabel wrote: post #30

I read this article back when I was learning the basics of transformers; the visualizations were really helpful. Although in retrospect knowing how a transformer works wasn't very useful at all in my day job applying LLMs, except as a sort of deep background for reassurance that I had some idea of how the big black box producing the tokens was put together, and to give me the mathematical basis for things like contex…

I think the biggest problem is that most tutorials use words to illustrate how the attention mechanism works. In reality, there are no word-associated tokens inside a Transformer. Tokens != word parts. An LLM does not perform language processing inside the Transformer blocks, and a Vision Transformer does not perform image processing. Words and pixels are only relevant at the input. I think this misunderstanding was a root cause of underestimating their capabilities.

Re: The Illustrated Transformer

#40

Post by amelius »

roadside_picnic wrote: post #34
amelius wrote: post #26

Earlier quoted context omitted.

But that AI engineer who is implementing speculative decoding is still just doing basic plumbing that has little to do with the actual reasoning. Yes, he/she might make the process faster, but they will know just as little about why/how the reasoning works as when they implemented a naive, slow version of the inference.

What "actual reasoning" are you referring to? I believe you're making my point for me. Speculative decoding requires the implementer to understand: - How the initial prompt is processed by the LLM - How to retrieve all the probabilities of previously observed tokens in the prompt (this also help people understand things like the probability of the entire prompt itself, the entropy of the prompt etc). - Details of how…

But I can make a similar argument about a simple multiplication:

- You have to know how the inputs are processed.

- You have to left-shift one of the operands by 0, 1, ... N-1 times.

- Add those together, depending on the bits in the other operand.

- Use an addition tree to make the whole process faster.

Does not mean that knowing the above process gives you a good insight in the concept of A*B and all the related math and certainly will not make you better at calculus.

Post reply on HN