What are transformer models and how do they work?
txt.cohere.ai
What are transformer models and how do they work?
1–10 of 114 posts
Re: What are transformer models and how do they work?
#2As with most tutorials on Transformers, this one leaves out some essential details:
- how are the input encodings generated?
- what is in those position vectors?
- how are the attention vectors learned?
The answer is that these things are all learned as the network is trained; the whole thing is one “thing”. The concept that is most important in understanding neural networks generally is that they start out as just a bunch of random numbers and then the numbers are gradually adjusted until the outputs converge closely enough on the desired loss.
I recommend watching Karpathy’s YouTube video where he codes up a Transformer from scratch. It’s the best way to understand these beasts.
Re: What are transformer models and how do they work?
#3I found the blogs written by Jay Alammar to be much more informative and complete. It appears that companies are rehashing and compressing the same content to advertise their products.
Re: What are transformer models and how do they work?
#4As with most tutorials on Transformers, this one leaves out some essential details: - how are the input encodings generated? - what is in those position vectors? - how are the attention vectors learned? The answer is that these things are all learned as the network is trained; the whole thing is one “thing”. The concept that is most important in understanding neural networks generally is that they start out as just a…
Thank you. This one?
Re: What are transformer models and how do they work?
#5Skimming it, there are a few things about this explanation that rub me just slightly the wrong way.
1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such.
2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess.
3. Describing transformers as using a "word embedding". I'm actually not aware of any transformers that use actual word embeddings, except the ones that incidentally fall out of other tokenization approaches sometimes.
4. Describing positional embeddings as multiplicative. They are generally (and very counterintuitively to me, but nevertheless) additive with token embeddings.
5. "what attention does is it moves the words in a sentence (or piece of text) closer in the word embedding" No, that's just incorrect.
6. You don't actually need a softmax layer at the end, since here they're just picking the top token and they can just do that pre-softmax since it won't change. It's also weird how they talked about this here when the most prominent use of softmax in transformers is actually in the attention component.
7. Really shortchanges the feedforward component. It may be simple, but it's really important to making the whole thing work.
8. Nothing about the residual
Re: What are transformer models and how do they work?
#6As with most tutorials on Transformers, this one leaves out some essential details: - how are the input encodings generated? - what is in those position vectors? - how are the attention vectors learned? The answer is that these things are all learned as the network is trained; the whole thing is one “thing”. The concept that is most important in understanding neural networks generally is that they start out as just a…
Thank you. This one? https://youtu.be/kCc8FmEb1nY
I’m sure that’s the one being referred to. It’s great.
Re: What are transformer models and how do they work?
#7I found the blogs written by Jay Alammar to be much more informative and complete. It appears that companies are rehashing and compressing the same content to advertise their products.
I believe Jay actually works at Cohere now (although I’m a little surprised that the post doesn’t state that).
Re: What are transformer models and how do they work?
#8Thanks. ML noob here. I liked the insight that attention adds context, by modifying the distance in an embedding.
Re: What are transformer models and how do they work?
#9Thanks. ML noob here. I liked the insight that attention adds context, by modifying the distance in an embedding.
The way the article presents this is misleading. The attention mechanism builds a new vector as a linear combination of other vectors, but after the first layer these have also all been altered by passing through a transformer layer so it makes less sense to talk about "other tokens" in most cases (it becomes increasingly inaccurate the deeper into the model you go). It's also not really moving closer so much as adding, and what it's adding isn't the embedding-derived-vector but a transform of the embedding-derived-vector after it's been projected into a lower-dimensional-space for that attention head.
It would be more accurate to say that it's integrating information stored in other vectors-derived-from-token-embeddings-at-some-point (which can also entail erasing information)
Re: What are transformer models and how do they work?
#10> Transformers are a new development in machine learning that have been making a lot of noise lately.
The paper on transformers was published 6 years ago.
6 years in ML is an eternity nowadays.