Earlier quoted context omitted.
Which GPT Video?
It can be found on the sites home page. Let's build GPT: from scratch, in code, spelled out: https://www.youtube.com/watch?v=kCc8FmEb1nY
Thank you!
Neural Networks: Zero to Hero
31–40 of 71 posts
Re: Neural Networks: Zero to Hero
#32This is really cool and I am so glad my math teacher was a hard ball and I still remember some Calculus. edit: Python really was/is made for this numbers/calculation/visualization thing. Kinda kicking myself now for not investing more in it and sticking with PHP, although PHP has its merits when building different things, Python is a beast with numbers.
I am at graphwiz now and it is getting better and better.
Also using ChatGTP to ask questions where I don't get something.
Wow what a time we live in to learn things.
Re: Neural Networks: Zero to Hero
#33What I appreciate about karpathy's videos is that it doesn't make things any more complicated than they need to be. Simple, engineering language is used. No gatekeeping! It's reassuring, and lets everyone know that anyone can do it. Thanks karpathy!
I just don’t know what he means by logits. Everything else seems like straightforward language.
When people mention logits, they're usually referring to the raw output of the model before it gets transformed/normalised into a probability distribution (i.e. sums to 1, range [0,1]). Logits can take any value. The naming might not be mathematically strict, because it assumes(?) that you're going to apply softmax (which interprets the output of the model as logits), but that's how the term is used.
For example in many classification problems you get a 1D vector of logits from the final layer, you apply softmax to normalise, then argmax to extract the predicted class. It extends to other tasks like semantic segmentation (predict pixel classes) where the "logit" output is the same size as the image with a channel for each class and you apply the same process to get a single channel image with class-per-pixel.
Here's a nice explanation: https://stackoverflow.com/a/66804099/395457
Re: Neural Networks: Zero to Hero
#34Anybody tried the lessons in an Apple MBA/MBP M1/M2?
Is it easily applicable?
Re: Neural Networks: Zero to Hero
#35What I appreciate about karpathy's videos is that it doesn't make things any more complicated than they need to be. Simple, engineering language is used. No gatekeeping! It's reassuring, and lets everyone know that anyone can do it. Thanks karpathy!
I just don’t know what he means by logits. Everything else seems like straightforward language.
He defines it pretty clearly. Logits are the inputs to a softmax layer / calculation, which turn the logits into normalized percentages (the percentages sum to 1.0).
Before going through the softmax layer, the logits will be small numbers around 0, probably. Something like: [2.89, -4.53, 0.24, -1.556, 0.57]. Logits like this are natural outputs of a neural network, because they can be any real number and everything will still work.
The logits become percentage as follows:
julia> x = [2.89, -4.53, 0.24, -1.556, 0.57]
5-element Vector{Float64}:
2.89
-4.53
0.24
-1.556
0.57
julia> x = e.^x
5-element Vector{Float64}:
17.993309601550315
0.010780676072743085
1.2712491503214047
0.2109782988178321
1.768267051433735
julia> x / sum(x)
5-element Vector{Float64}:
0.8465613320288766
0.0005072164987105474
0.05981058503789324
0.009926248902037537
0.08319461753248213
Re: Neural Networks: Zero to Hero
#36My hive mind connection must be good because I literally finished this course yesterday.
It was very satisfying to learn how transformers worked, to finally be able to turn the obscure glyphs of the research papers into real code, but I think transformers are too big for what I can do on my own computer. The author mentioned that the toy transformer he was building in the final video took 15 minutes to train on his A100 GPU (a $10,000 GPU), and the results weren't even that good; the transformer was spelling words correctly using character level tokens, I guess that's something, but it's not GTP4.
Even so, there were a lot of good tips to pick up along the way. This is a great series that I'm thankful to have. The "Backprop Ninja" video was hard work, you manually calculate the gradients and then compare your calculations against PyTorch. It's great to have instant feedback telling you whether your gradients are correct or not.
Re: Neural Networks: Zero to Hero
#37Earlier quoted context omitted.
That program sounds quite impressive, I wonder if any equivalencies exist in the US?
The website doesn't say what—for me—is the best thing about it. The course is peer-led which works like this: once your join, you're part of a team which has one objective: get the best score with your ML recommendation system. There is simulated environment in which all teams of the cohort receive millions of requests per day (and hundreds of thousands of users and items) and you have to build out your infrastructur…
That is really cool and engaging.
Re: Neural Networks: Zero to Hero
#38What I appreciate about karpathy's videos is that it doesn't make things any more complicated than they need to be. Simple, engineering language is used. No gatekeeping! It's reassuring, and lets everyone know that anyone can do it. Thanks karpathy!
I just don’t know what he means by logits. Everything else seems like straightforward language.
Honestly what cracked logits for me was a conversation with ChatGPT in which I gave it my professional background, areas of strength and weakness, and problem context, and had it explain to me. I then went elsewhere to make sure I hadn’t been lied to. I’ve found ChatGPT such an invaluable learning tool when used in this way.
Re: Neural Networks: Zero to Hero
#39I am always lost in these blogs. Is there any gradual progression of understanding/exercises that one can follow to apply ML/NN/DL practically?
Having these disconnected pieces of information with no clear link to one another feels like a lot of noise to me.
Re: Neural Networks: Zero to Hero
#40I am always lost in these blogs. Is there any gradual progression of understanding/exercises that one can follow to apply ML/NN/DL practically? Having these disconnected pieces of information with no clear link to one another feels like a lot of noise to me.
I've been reading The Little Learner, which builds machine learning knowledge on top of Scheme / Racket. After that book, you could watch this series, which will immediately begin explaining how the automatic differentiation works.