Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What are transformer models and how do they work?

txt.cohere.ai

Post reply on HN
Print view View original

51–60 of 114 posts

Re: What are transformer models and how do they work?

#51

Post by toxik »

hansvm wrote: post #21
jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

> 6 Selecting the likeliest token is only one of many sampling options, and it's extremely poor for most tasks, moreso when you consider the relationships between multiple executions of the model. _Some_ (not necessarily softmax) probability renormalization trained into the model is issential for a lot of techniques.

To expand on this, one of the most common tricks is Nucleus sampling. Roughly, you zero out the lowest probabilities such that the remaining sum to just above some threshold you decide (often around 80%).

The idea is that this is more general than eg changing the temperature of the softmax, or using top-k where you just keep the k most probable outcomes.

Note that if you do Nucleus sampling (aka top-p) with the threshold p=0% you just pick the maximum likelihood estimate.

Re: What are transformer models and how do they work?

#52

Post by seydor »

ttul wrote: post #2

As with most tutorials on Transformers, this one leaves out some essential details: - how are the input encodings generated? - what is in those position vectors? - how are the attention vectors learned? The answer is that these things are all learned as the network is trained; the whole thing is one “thing”. The concept that is most important in understanding neural networks generally is that they start out as just a…

note that embedding and positional vectors are fixed and not trained with the transformer. The attention vectors are also computed in each step, but what is learned is the transformation of inputs to qvk vectors and the fully connected layers. There are some great people on youtube who explain the series of steps. I think this is the most comprehensive: https://www.youtube.com/watch?v=Nw_PJdmydZY

Re: What are transformer models and how do they work?

#53

Post by pankajdoharey »

indeedmug wrote: post #15
antimora wrote: post #11

Earlier quoted context omitted.

I am trying to learn more in depth. Could you suggest some good resource for learning transformers?

I found these resources to be helpful. https://jalammar.github.io/illustrated-transformer/ This is a good illustration of the transformer and how the math works. https://karpathy.ai/zero-to-hero.html If you want a deeper understanding of transform and how they fit in the whole picture of deep learning, this series is far and away the best resource I found. Karpathy goes into transformers by the sixth lecture, the pre…

I agree that Karpathy's YouTube video is an excellent resource for understanding Transformers from scratch. It provides a hands-on experience that can be particularly helpful for those who want to implement the models themselves. Here's the link to the video titled "Let's build GPT: from scratch, in code, spelled out": https://youtu.be/kCc8FmEb1nY

Additionally, for more comprehensive resources on Transformers, you may find these resources useful:

* The Illustrated Transformer by Jay Alammar: http://jalammar.github.io/illustrated-transformer/

* MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention: https://www.youtube.com/watch?v=ySEx_Bqxvvo

* Karpathy's course, Deep Learning and Generative Models (Lecture 6 covers Transformers): https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThs......

These resources cover different aspects of Transformers and can help you grasp the underlying concepts and mechanisms better.

Re: What are transformer models and how do they work?

#54

Post by tarvaina »

leobg wrote: post #16
tomcam wrote: post #4

Earlier quoted context omitted.

Thank you. This one? https://youtu.be/kCc8FmEb1nY

Is that an actual transformer, though? Like with encoder and decoder layers? That’s the part I never truly understood. Or is it “just” an example of a neural network? Thanks!

Not all transformers have separate encoders and decoders. GPTs, for instance, only have the equivalents of decoder layers of the original transformer paper, but they are still considered transformers. Karpathy’s video shows an actual GPT-style transformer.

I think a neural network can be considered a transformer if it contains a stack of attention blocks as its core mechanism.

Re: What are transformer models and how do they work?

#56

Post by radarsat1 »

I've been modifying an LSTM GAN model to use a transformer in the encoder and it seems to do much worse. Or at least the training dynamics are very different. Transformers perform great when they work but it seems to be a lot harder to get them to work in my experience. Can anyone corroborate that or is it likely that I'm doing something fundamentally wrong? To be clear I'm not implementing it myself but using PyTorch's Transformer classes as drop-in replacements for my LSTM-based encoder and decoder. Been trying lots of variations in the hyperparameters and position encoding methods etc but it always either doesn't train at all (generator/discriminator divergence) or it produces blurry images. (The "prenet" and "postnet" remain the same as my reference model so I find this surprising.). Really frustrating when all the latest results say that this should work amazingly well.

Tons of articles like this on "how transformers work", very few on "tips for getting transformers to work in practice."

Re: What are transformer models and how do they work?

#57

Post by hnfong »

riffraff wrote: post #28
1024core wrote: post #10

> Transformers are a new development in machine learning that have been making a lot of noise lately. The paper on transformers was published 6 years ago. 6 years in ML is an eternity nowadays.

Was there any "big idea" after that? It seems most of the user-visible innovation has been "let's use transformers on more data". Perhaps capsule networks? But those are years old too.

Somebody had to invest resources into training those super large models and observe emergent intelligent behavior. It's not like the authors of the original paper knew that transformers would lead to GPT-4 and spark an AGI debate. Nobody expected transformers to get powerful so fast.

Re: What are transformer models and how do they work?

#58

Post by [deleted] »

metanonsense wrote: post #50
antimora wrote: post #11

Earlier quoted context omitted.

I am trying to learn more in depth. Could you suggest some good resource for learning transformers?

When I first tried to understand transformers, I superficially understood most material, but I always felt that I did not really get it on a "I am able to build it and I understand why I am doing it" level. I struggled to get my fingers on what exactly I did not understand. I read the original paper, blog posts, and watched more videos than I care to admit. The one source of information that made it click to me were…

[deleted]

Re: What are transformer models and how do they work?

#59

Post by fulafel »

kleiba wrote: post #49

Oh, come on - this article is from three days ago, but it starts with "Transformers are a new development in machine learning". Transformers have been around for six years now, that's an eternity when you consider how fast this field is moving.

Deep learning (multilayer perceptrons) was 1965... The timeline is pretty long.

Re: What are transformer models and how do they work?

#60

Post by isaacfung »

bomewish wrote: post #48
jaidhyani wrote: post #5

Skimming it, there are a few things about this explanation that rub me just slightly the wrong way. 1. Calling the input token sequence a "command". It probably only makes sense to think of this as a "command" on a model that's been fine-tuned to treat it as such. 2. Skipping over BPE as part of tokenization - but almost every transformer explainer does this, I guess. 3. Describing transformers as using a "word embed…

You seem to know a bunch about this. What’s your rec for best single explainer?

Not the guy you asked, but these are often recommended.

https://jalammar.github.io/illustrated-transformer/

https://nlp.seas.harvard.edu/2018/04/03/attention.html

Post reply on HN