11–18 of 18 posts
I quite like Jeremy's approach: https://nbviewer.org/github/fastai/fastbook/blob/master/17_f...
It shows a very simple "Pythonic" approach to assemble gradient of a composition of functions from the gradients of the components.
Every one should go through this rite of passage work and get to the "Attention is all you need" implementation. It's a world where engineering and the academic papers are very close and reproducible and a must for you to progress in the field. (see also andre karpathys zero to hero nn series on youtube as well its very good and similar to this work)
Edit - it is. Not to talk down on the series. I’m sure it’s good, but it is actually “LLM with PyTorch”.
Edit - I looked again and I was actually not correct. He does ultimately use frameworks, but gives some early talk about how those function under the hood.
Every one should go through this rite of passage work and get to the "Attention is all you need" implementation. It's a world where engineering and the academic papers are very close and reproducible and a must for you to progress in the field. (see also andre karpathys zero to hero nn series on youtube as well its very good and similar to this work)
I would also recommend going through Callum McDougall/Neel Nanda's fantastic Transformer from Scratch tutorial. It takes a different approach to conceptualizing the model (or at least, it implements it in a way which emphasizes different characteristics of Transformers and self-attention), which I found deeply satisfying when I first explored them. https://arena-ch1-transformers.streamlit.app/%5B1.1%5D_Trans...
I did a similar thing for Julia: Llama2.jl contains vanilla Julia code [1] for training small Llama2-style models on the CPU. [1] https://github.com/cafaxo/Llama2.jl/tree/master/src/training
Every one should go through this rite of passage work and get to the "Attention is all you need" implementation. It's a world where engineering and the academic papers are very close and reproducible and a must for you to progress in the field. (see also andre karpathys zero to hero nn series on youtube as well its very good and similar to this work)
Every one should go through this rite of passage work and get to the "Attention is all you need" implementation. It's a world where engineering and the academic papers are very close and reproducible and a must for you to progress in the field. (see also andre karpathys zero to hero nn series on youtube as well its very good and similar to this work)
Is this YouTube series also “from scratch (but not really)” Edit - it is. Not to talk down on the series. I’m sure it’s good, but it is actually “LLM with PyTorch”. Edit - I looked again and I was actually not correct. He does ultimately use frameworks, but gives some early talk about how those function under the hood.