Karpathy’s micrograd did it first (and better); start here: https://karpathy.ai/zero-to-hero.html
Karpathy's material is excellent! This was a project I made for fun, and hopefully provides a different perspective on how this can look
MyTorch – Minimalist autograd in 450 lines of Python
11–20 of 20 posts
Re: MyTorch – Minimalist autograd in 450 lines of Python
#12Karpathy’s micrograd did it first (and better); start here: https://karpathy.ai/zero-to-hero.html
Why is it better
Cleaner, more straightforward, more compact code, and considered complete in its scope (i.e. implement backpropagation with a PyTorch-y API and train a neural network with it). MyTorch appears to be an author's self-experiment without concrete vision/plan. This is better for author but worse for outsiders/readers.
P.S. Course goes far beyond micrograd, to makemore (transfomers), minbpe (tokenization), and nanoGPT (LLM training/loading).
Re: MyTorch – Minimalist autograd in 450 lines of Python
#13Earlier quoted context omitted.
Why is it better
Because it's an acclaimed, often cited course by a preeminent AI Researcher (and founding member of OAI) rather than four undocumented python files.
Ironically the reason Karpathy's is better is because he livecoded it and I can be sure it's not some LLM vomit.
Unfortunately, we are now indundated with newbies posting their projects/tutorials/guides in the hopes that doing so will catch the eye of a recuiter and land them a high paying AI job. That's not so bad in itself except for the fact that most of these people are completely clueless and posting AI slop.
Re: MyTorch – Minimalist autograd in 450 lines of Python
#14Better readme would be way to go
Re: MyTorch – Minimalist autograd in 450 lines of Python
#15Earlier quoted context omitted.
Because it's an acclaimed, often cited course by a preeminent AI Researcher (and founding member of OAI) rather than four undocumented python files.
Ironically the reason Karpathy's is better is because he livecoded it and I can be sure it's not some LLM vomit. Unfortunately, we are now indundated with newbies posting their projects/tutorials/guides in the hopes that doing so will catch the eye of a recuiter and land them a high paying AI job. That's not so bad in itself except for the fact that most of these people are completely clueless and posting AI slop.
Haha, couldn't agree with you more. This, however, isn't AI slop. You can see in the commit history that this is from 3 years ago
Re: MyTorch – Minimalist autograd in 450 lines of Python
#16Having written a slightly more involved version of this recently myself I think you did a great job of keeping this compact while still readable. This style of library requires some design for sure.
Supporting higher order derivatives was also something I considered, but it’s basically never needed in production models from what I’ve seen.
Re: MyTorch – Minimalist autograd in 450 lines of Python
#17Better readme would be way to go
In iguana2000's defense, the code is highly self-documenting.
It arguably reads cleaner than Karpathy's in some respects, as he occasionally gets a little ahead of his students with his '1337 Python skillz.
Re: MyTorch – Minimalist autograd in 450 lines of Python
#18Having written a slightly more involved version of this recently myself I think you did a great job of keeping this compact while still readable. This style of library requires some design for sure. Supporting higher order derivatives was also something I considered, but it’s basically never needed in production models from what I’ve seen.
Thanks! I agree about the style
Re: MyTorch – Minimalist autograd in 450 lines of Python
#19Karpathy’s micrograd did it first (and better); start here: https://karpathy.ai/zero-to-hero.html
Karpathy's material is excellent! This was a project I made for fun, and hopefully provides a different perspective on how this can look
I'm very sorry, I should have phrased my original post in a kinder, less dismissive way, and kudos to you for not reacting badly to my rudeness. It is a cool repo and a great accomplishment. Implementing autograd is great as a learning exercise, but my opinion is that you're not going to get the performance or functionality of one of the large, mainstream autograd libraries. Karpathy, for example, throws away micrograd after implementing it and uses pytorch in his later exercises. So it's great that you did this, but for others to learn how autograd works, Karpathy is usually a better route, because the concepts are built up one by one and explained thoroughly.
Re: MyTorch – Minimalist autograd in 450 lines of Python
#20Earlier quoted context omitted.
Karpathy's material is excellent! This was a project I made for fun, and hopefully provides a different perspective on how this can look
I'm very sorry, I should have phrased my original post in a kinder, less dismissive way, and kudos to you for not reacting badly to my rudeness. It is a cool repo and a great accomplishment. Implementing autograd is great as a learning exercise, but my opinion is that you're not going to get the performance or functionality of one of the large, mainstream autograd libraries. Karpathy, for example, throws away microgr…
No worries, you're good, yes Karpathy is for sure the better route