For those interested in simple neural networks to CNN and RNNs implemented with just Numpy (including backprop):
Neural network from scratch
11–20 of 43 posts
Re: Neural network from scratch
#12"from scratch" but uses autograd and glosses over backpropagation.
He doesn't implement matrix operations, floating point addition / multiplication either.
The difference is that autograd isn't something you should already know if you're learning neural networks. Many "from scratch" tutorials implement backprop because this is a key part. I think your comment is a bit facetious and you're not acting in good faith.
Re: Neural network from scratch
#13I remember doing this in PHP(4? 5?) for my undergrad capstone project because I had a looming due date and it was the dev environment I had readily available. No helpful libraries in that decade. Great way to really grok the material, and really lets me appreciate how spoiled we are today in the ML space.
I remember one of the first things I saw on the web (in 1996) was a neural net simulator, written in javascript of all things:
https://web.archive.org/web/19961226105339/http://www.ozemai...
from 1997, a bit more fancy: https://web.archive.org/web/19990117022955/http://www.hav.co...
Re: Neural network from scratch
#14Nice! I made a gpu accelerated backpropagation lib a while ago to learn about NNs, if you are interested check it out here: https://github.com/zbendefy/machine.academy
Re: Neural network from scratch
#15Earlier quoted context omitted.
He doesn't implement matrix operations, floating point addition / multiplication either.
The difference is that autograd isn't something you should already know if you're learning neural networks. Many "from scratch" tutorials implement backprop because this is a key part. I think your comment is a bit facetious and you're not acting in good faith.
I could definitely see an argument that knowing gradient descent is a requisite just as much as knowing matrix multiplication.
Both of these mathematical concepts are being abstracted over and are being used by the author's neutral network implementation.
Now whether you consider using other people's libraries for this as being "from scratch" is up to you.
Re: Neural network from scratch
#16If you are interested in learning what makes a Deep Learning library, and want to code one, for learning experience, you should check out- Minitorch [0].
Re: Neural network from scratch
#17If you actually want to understand and implement neural nets from scratch, look into 3Blue1Brown's videos as well as Andrew Ng's course.
Re: Neural network from scratch
#18"from scratch" but uses autograd and glosses over backpropagation.
Yeah, the autograd choice struck me as odd. Given how simple the model is, it feels like it would have been easy to show how to compute gradients. The whole benefit of having this super simple toy problem is that we can reason about the meaning of individual weights - it's a perfect opportunity to build clear intuition about gradients and weight updates. Switching to torch is just substituting one black box for another - to a novice reader, the torch code is just magical incantations.
Re: Neural network from scratch
#19"from scratch" but uses autograd and glosses over backpropagation.
Yeah, the autograd choice struck me as odd. Given how simple the model is, it feels like it would have been easy to show how to compute gradients. The whole benefit of having this super simple toy problem is that we can reason about the meaning of individual weights - it's a perfect opportunity to build clear intuition about gradients and weight updates. Switching to torch is just substituting one black box for anoth…
This could be the start of a breath-first approach, where you start with very little code, and then dig deep into things like autograd or "backprop" as you get interested in such details.
It seems to me that trying to give explicit formulas for gradients is just swamping the beginner with unnecessary details that don't help to build intuition. I think the author made exactly the right choices.
It used to be that some NN tutorials would swamp the beginner with backprop formulas, which beginners were forced by their professors to memorise. I don't think this succeeded at doing much; it only made the subject seem more complicated than it needed to be; and I think it should all be abstracted away into autograd.
Re: Neural network from scratch
#20I think NNs are going to be a challenge as complexity grows.
I'm trying to make mobs behave autonomously in my 3D action MMO.
The memory (depth) I would need for that to succeed and the processing power to do it in real-time is making my head spin.
Let's hope Raspberry 5 has some hardware to help with this.
At this point I'm probably going to have some state machine AI (think mobs in Minecraft; basically check range / view then target and loop) but instead of deterministic or purely random I'm going to add some NN randomness to the behaviour so that it can be interesting without just adding quantity (more mobs).
So the inputs would be the map topography and entities (mobs and players) and the output whether to engage or not, the backpropagation would be success rate I guess? Or am I thinking about this the wrong way?
I wonder what adding a _how_ to the same network after the _what_ would look like, probably a direction as output instead of just an entity id?