Those interested in this might also be interested in some notes from the University of Amsterdam:
https://uvadlc-notebooks.readthedocs.io/en/latest/index.html
Tutorial 6 covers transformers.
Neural Networks: Zero to Hero
41–50 of 71 posts
Re: Neural Networks: Zero to Hero
#42I'm doing an ML apprenticeship [1] these weeks and Karpathy's videos are part of it. We've been deep down into them. I found them excellent. All concepts he illustrates are crystal clear in his mind (even though they are complicated concepts themselves) and that shows in his explanations. Also, the way he builds up everything is magnificent. Starting from basic python classes, to derivatives and gradient descent, to…
Do you run the code as you watch? I’ve been simply watching them on a palm from a hammock and I’m worried I’m not getting the full experience.
Something I discovered not so long ago that I wish I had years ago is to watch the video first and then code along after. So simple but makes a world of difference, you can skip errors, fluff and foresee what's next, all around you'd think it's watching two 2 hour videos but it works out in terms of getting the most out of the content and drilling it into your head.
Re: Neural Networks: Zero to Hero
#43My hive mind connection must be good because I literally finished this course yesterday. It was very satisfying to learn how transformers worked, to finally be able to turn the obscure glyphs of the research papers into real code, but I think transformers are too big for what I can do on my own computer. The author mentioned that the toy transformer he was building in the final video took 15 minutes to train on his A…
> his A100 GPU (a $10,000 GPU)
These are available to rent per hour at much lower costs. The author mentions this in the video description.
Re: Neural Networks: Zero to Hero
#44[flagged]
Re: Neural Networks: Zero to Hero
#45[flagged]
How is it relevant here?
Re: Neural Networks: Zero to Hero
#46Re: Neural Networks: Zero to Hero
#47What I appreciate about karpathy's videos is that it doesn't make things any more complicated than they need to be. Simple, engineering language is used. No gatekeeping! It's reassuring, and lets everyone know that anyone can do it. Thanks karpathy!
I just don’t know what he means by logits. Everything else seems like straightforward language.
I'm confused by the comments here, does "logit" here not mean "log odds" like it does in virtually every other context related to machine learning?
Generally I'm a huge fan of not getting too caught up in theory before diving into practice, but I'm seeing multiple responses to this comment without a single mention of "log odds".
The logit function transforms probabilities into the log of the odds (ln P(X)/(1-P(X)), which is important because it makes probabilities linear, which they are not in their standard [0,1] form. It's the foundation of logistic regression, which is, despite much misinformation, quite literally linear regression with a transformed target.
The logistic function is the inverse of the logit: it turns log odds values into probabilities once again. Logistic regression actually transforms the model not the target (most of the time) because the labels 0 and 1 are negative and positive infininity which can't be handled by linear regression (so we transform the model using the inverse instead).
I don't think I can stress enough how important is its to really understand logistic regression (which is also the basic perceptron) before diving into neural networks (which are really just an extension of logistic regression).
Re: Neural Networks: Zero to Hero
#48My hive mind connection must be good because I literally finished this course yesterday. It was very satisfying to learn how transformers worked, to finally be able to turn the obscure glyphs of the research papers into real code, but I think transformers are too big for what I can do on my own computer. The author mentioned that the toy transformer he was building in the final video took 15 minutes to train on his A…
> his A100 GPU (a $10,000 GPU) These are available to rent per hour at much lower costs. The author mentions this in the video description.
True, as much as I enjoy owning and controlling my own hardware, buying an A100 and then letting it sit idle while I procrastinate and play video games probably isn't the best use of resources. He did say "my GPU" (or similar) at one point, and I thought maybe he does enough ML stuff that he bought his own.
Re: Neural Networks: Zero to Hero
#49Earlier quoted context omitted.
I just don’t know what he means by logits. Everything else seems like straightforward language.
Having not watched the series, I can only assume he means logit as in a probability function from 0 to 1 https://deepai.org/machine-learning-glossary-and-terms/logit... .
logit is not a "probability function", quite the opposite. You can see this in the image in the link you posted (the x-axis is from 0-1, the y-axis is from -inf to inf). It transforms probabilities into log odds which is a linear space, and make combining probabilities much nicer.
The inverse logit or logistic function takes log odds and transforms them back into probabilities.
Most machine learning relies heavily on manipulating probabilities, but since probabilities are not linear, the logit/logistic transformations become essential to correctly modeling complex problems involving probabilities.
Re: Neural Networks: Zero to Hero
#50My hive mind connection must be good because I literally finished this course yesterday. It was very satisfying to learn how transformers worked, to finally be able to turn the obscure glyphs of the research papers into real code, but I think transformers are too big for what I can do on my own computer. The author mentioned that the toy transformer he was building in the final video took 15 minutes to train on his A…
> his A100 GPU (a $10,000 GPU) These are available to rent per hour at much lower costs. The author mentions this in the video description.
Approximately 40 cents USD for 15 minutes from cursory research.