Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What are transformer models and how do they work?

txt.cohere.ai

Post reply on HN
Print view View original

11–20 of 114 posts

Re: What are transformer models and how do they work?

#11

Post by antimora »

jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

I am trying to learn more in depth. Could you suggest some good resource for learning transformers?

Re: What are transformer models and how do they work?

#12

Post by quantisan »

antimora wrote: post #11
jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

I am trying to learn more in depth. Could you suggest some good resource for learning transformers?

this one's been mentioned a lot: Let's build GPT: from scratch, in code, spelled out. https://youtu.be/kCc8FmEb1nY

Re: What are transformer models and how do they work?

#13

Post by andai »

antimora wrote: post #11
jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

I am trying to learn more in depth. Could you suggest some good resource for learning transformers?

Here's the original paper: https://arxiv.org/abs/1706.03762

Re: What are transformer models and how do they work?

#14

Post by andai »

quantisan wrote: post #12
antimora wrote: post #11

Earlier quoted context omitted.

I am trying to learn more in depth. Could you suggest some good resource for learning transformers?

this one's been mentioned a lot: Let's build GPT: from scratch, in code, spelled out. https://youtu.be/kCc8FmEb1nY

The whole playlist is fantastic: https://youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9Gv...

Re: What are transformer models and how do they work?

#15

Post by indeedmug »

antimora wrote: post #11
jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

I am trying to learn more in depth. Could you suggest some good resource for learning transformers?

I found these resources to be helpful.

https://jalammar.github.io/illustrated-transformer/ This is a good illustration of the transformer and how the math works.

https://karpathy.ai/zero-to-hero.html If you want a deeper understanding of transform and how they fit in the whole picture of deep learning, this series is far and away the best resource I found. Karpathy goes into transformers by the sixth lecture, the previous lectures give a lot more context how deep learning works.

Re: What are transformer models and how do they work?

#16

Post by leobg »

tomcam wrote: post #4
ttul wrote: post #2

As with most tutorials on Transformers, this one leaves out some essential details: - how are the input encodings generated? - what is in those position vectors? - how are the attention vectors learned? The answer is that these things are all learned as the network is trained; the whole thing is one “thing”. The concept that is most important in understanding neural networks generally is that they start out as just a…

Thank you. This one? https://youtu.be/kCc8FmEb1nY

Is that an actual transformer, though? Like with encoder and decoder layers? That’s the part I never truly understood. Or is it “just” an example of a neural network? Thanks!

Re: What are transformer models and how do they work?

#17

Post by charcircuit »

jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

>and very counterintuitively to me

It's more intuitive if you remember how many dimensions these vectors have.

Re: What are transformer models and how do they work?

#18

Post by chaxor »

jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

I know we don't have access to the details at OpenAI - but it does seem like there have been significant changes to the BPE token size over time. It seems there is a push towards much larger tokens than the previous ~3 char tokens (at least by behavior)

Re: What are transformer models and how do they work?

#19

Post by sillysaurusx »

jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

The positional embedding can be thought of: in the same way you can hear two pieces of music overlaid on each other, you can add both the vocab and pos embedding and it’s able to pick them apart.

If you asked yourself to identify when someone’s playing a high note or low note (pos embedding) and whether they’re playing Beethoven or Lady Gaga (vocab embedding) you could do it.

That’s why it’s additive and why it wouldn’t make much sense for it to be multiplicative.

Re: What are transformer models and how do they work?

#20

Post by [deleted] »

indeedmug wrote: post #15
antimora wrote: post #11

Earlier quoted context omitted.

I am trying to learn more in depth. Could you suggest some good resource for learning transformers?

I found these resources to be helpful. https://jalammar.github.io/illustrated-transformer/ This is a good illustration of the transformer and how the math works. https://karpathy.ai/zero-to-hero.html If you want a deeper understanding of transform and how they fit in the whole picture of deep learning, this series is far and away the best resource I found. Karpathy goes into transformers by the sixth lecture, the pre…

[deleted]
Post reply on HN