11–20 of 56 posts
I can't think of anyone better to teach attention mechanism to the masses. This is a dream come true
Incredible. This 3B1B series was started 6 years ago and keeps going today with chapter 5. If you haven't seen the first few chapters, I cannot recommend enough.
Earlier quoted context omitted.
Incredible. This 3B1B series was started 6 years ago and keeps going today with chapter 5. If you haven't seen the first few chapters, I cannot recommend enough.
Would you be able to compare them to Andrew Ng's course?
Earlier quoted context omitted.
Something like this does exist, production systems rarely use greedy search but have more holistic search algorithms. An example is Beam Search: https://www.width.ai/post/what-is-beam-search Essentially we keep a window of probabilities of predicted tokens to improve the final quality of output.
Thanks, that's exactly what I was looking for! Any idea if it's possible to use beam search on local models like mistral? It sounds like the choice of beam search vs say top-p or top-k should be in the software and not embedded, right?
(No affiliation)
Earlier quoted context omitted.
Incredible. This 3B1B series was started 6 years ago and keeps going today with chapter 5. If you haven't seen the first few chapters, I cannot recommend enough.
Would you be able to compare them to Andrew Ng's course?
I've just started this video, but already have a question if anyone's familiar with GPT workings - I thought that these models chose the next word based on what's most likely. But if they choose based on "one of the likely" words, could (in general) that not lead to a situation where the list of predictions for the next word are much less likely? Running possibilities of "two words together", then, would be more bene…
The temperature setting is used to select how rare of a next token is possible. If set to 0 the. The top of the likely list is chosen, if set greater than 0 then some lower probability tokens may be chosen.
Earlier quoted context omitted.
Something like this does exist, production systems rarely use greedy search but have more holistic search algorithms. An example is Beam Search: https://www.width.ai/post/what-is-beam-search Essentially we keep a window of probabilities of predicted tokens to improve the final quality of output.
Thanks, that's exactly what I was looking for! Any idea if it's possible to use beam search on local models like mistral? It sounds like the choice of beam search vs say top-p or top-k should be in the software and not embedded, right?
Here is a blog post that describes it: https://huggingface.co/blog/how-to-generate.
I will warn you though that beam search is typically what you do NOT want. Beam search approximately optimizes for the "highest likely sequence at the token level." This is rarely what you need in practice with open-ended generations (e.g. a question-answering chat bot). In practice, you need "highest likely semantic sequence," which is much harder problem.
Of course, various approximations for semantic alignment are currently in the literature, but still a wide open problem.
If you liked that, Andrej karpathy has a few interesting videos on his channels explaining Neural Networks and their inner workings which are aimed at people who know how to program.
I was ignorant enough to try and jump straight in to his videos and despite him recommending I watch his preceeding videos I incorrectly assumed I could figure it out as I went. There is verbiage in there that you simply must know to get the most out of it. After giving up, going away and filling in the gaps though some other learnings, I went back and his videos become (understandably) massively more valueable for me.
I would strongly recommend anyone else wanting to learn neural networks that they learn from my mistake.
I've just started this video, but already have a question if anyone's familiar with GPT workings - I thought that these models chose the next word based on what's most likely. But if they choose based on "one of the likely" words, could (in general) that not lead to a situation where the list of predictions for the next word are much less likely? Running possibilities of "two words together", then, would be more bene…