But what is a GPT? Visual intro to Transformers [video]
1–10 of 56 posts
Re: But what is a GPT? Visual intro to Transformers [video]
#2I can't think of anyone better to teach attention mechanism to the masses. This is a dream come true
Re: But what is a GPT? Visual intro to Transformers [video]
#3I can't think of anyone better to teach attention mechanism to the masses. This is a dream come true
Incredible. This 3B1B series was started 6 years ago and keeps going today with chapter 5.
If you haven't seen the first few chapters, I cannot recommend enough.
Re: But what is a GPT? Visual intro to Transformers [video]
#4What an unbelievable salve for all the April Fool's content. Pipe this directly into my veins.
Re: But what is a GPT? Visual intro to Transformers [video]
#5I've just started this video, but already have a question if anyone's familiar with GPT workings - I thought that these models chose the next word based on what's most likely. But if they choose based on "one of the likely" words, could (in general) that not lead to a situation where the list of predictions for the next word are much less likely? Running possibilities of "two words together", then, would be more beneficial if computationally possible (and so on for 3, 4 and n words). Does this exist?
(I realize that choosing the most likely word wouldn't necessarily solve the issue, but choosing the most likely phrase possibly might.)
Edit, post seeing the video and comments: it's beam search, along with temperature to control these things.
Re: But what is a GPT? Visual intro to Transformers [video]
#6I've just started this video, but already have a question if anyone's familiar with GPT workings - I thought that these models chose the next word based on what's most likely. But if they choose based on "one of the likely" words, could (in general) that not lead to a situation where the list of predictions for the next word are much less likely? Running possibilities of "two words together", then, would be more bene…
Something like this does exist, production systems rarely use greedy search but have more holistic search algorithms.
An example is Beam Search:https://www.width.ai/post/what-is-beam-search
Essentially we keep a window of probabilities of predicted tokens to improve the final quality of output.
Re: But what is a GPT? Visual intro to Transformers [video]
#7I've just started this video, but already have a question if anyone's familiar with GPT workings - I thought that these models chose the next word based on what's most likely. But if they choose based on "one of the likely" words, could (in general) that not lead to a situation where the list of predictions for the next word are much less likely? Running possibilities of "two words together", then, would be more bene…
The temperature setting is used to select how rare of a next token is possible. If set to 0 the. The top of the likely list is chosen, if set greater than 0 then some lower probability tokens may be chosen.
Re: But what is a GPT? Visual intro to Transformers [video]
#8I've just started this video, but already have a question if anyone's familiar with GPT workings - I thought that these models chose the next word based on what's most likely. But if they choose based on "one of the likely" words, could (in general) that not lead to a situation where the list of predictions for the next word are much less likely? Running possibilities of "two words together", then, would be more bene…
[deleted]
Re: But what is a GPT? Visual intro to Transformers [video]
#9I've just started this video, but already have a question if anyone's familiar with GPT workings - I thought that these models chose the next word based on what's most likely. But if they choose based on "one of the likely" words, could (in general) that not lead to a situation where the list of predictions for the next word are much less likely? Running possibilities of "two words together", then, would be more bene…
Something like this does exist, production systems rarely use greedy search but have more holistic search algorithms. An example is Beam Search: https://www.width.ai/post/what-is-beam-search Essentially we keep a window of probabilities of predicted tokens to improve the final quality of output.
Thanks, that's exactly what I was looking for! Any idea if it's possible to use beam search on local models like mistral? It sounds like the choice of beam search vs say top-p or top-k should be in the software and not embedded, right?
Re: But what is a GPT? Visual intro to Transformers [video]
#10I've just started this video, but already have a question if anyone's familiar with GPT workings - I thought that these models chose the next word based on what's most likely. But if they choose based on "one of the likely" words, could (in general) that not lead to a situation where the list of predictions for the next word are much less likely? Running possibilities of "two words together", then, would be more bene…
The temperature setting is used to select how rare of a next token is possible. If set to 0 the. The top of the likely list is chosen, if set greater than 0 then some lower probability tokens may be chosen.
Thanks, learnt something new today!