Karpathy's llama2.c ported to pure Python
1–10 of 11 posts
Re: Karpathy's llama2.c ported to pure Python
#2I find these efforts impressive, but what is the value proposition here? (I'm not just talking about this fork, but also Karapathy's llama2.c as well).
Re: Karpathy's llama2.c ported to pure Python
#3I made a jupyter notebook "llama2.ipynb" from the Karpathy project: https://github.com/rbitr/llama2.ipynb
I didn't do a pure python, mine uses numpy, and although I haven't benchmarked, it runs the stories15M model much faster than 1.3 tok/sec on my 2018 macbook. You should try swapping in numpy matrix multiplication, or @ (I actually don't know if that's native or part of another package) for matmul and see what changes.
Re: Karpathy's llama2.c ported to pure Python
#4I find these efforts impressive, but what is the value proposition here? (I'm not just talking about this fork, but also Karapathy's llama2.c as well).
Writing one's own and/or porting every line of code has great value
Re: Karpathy's llama2.c ported to pure Python
#5I made a jupyter notebook "llama2.ipynb" from the Karpathy project: https://github.com/rbitr/llama2.ipynb I didn't do a pure python, mine uses numpy, and although I haven't benchmarked, it runs the stories15M model much faster than 1.3 tok/sec on my 2018 macbook. You should try swapping in numpy matrix multiplication, or @ (I actually don't know if that's native or part of another package) for matmul and see what cha…
1.3 tok / sec is something similar to my Python version port performance, but I tried on M1 Max
Re: Karpathy's llama2.c ported to pure Python
#6I find these efforts impressive, but what is the value proposition here? (I'm not just talking about this fork, but also Karapathy's llama2.c as well).
Personally for me the value was to implement a complex logic from a scientific paper in a pure Python.
It helps to understand the essence of a cutting edge AI technology.
And it's quite fascinating that it would take about 500 lines of core part code to implement inference for such a complex solution.
Re: Karpathy's llama2.c ported to pure Python
#7I find these efforts impressive, but what is the value proposition here? (I'm not just talking about this fork, but also Karapathy's llama2.c as well).
Regarding the original llama2.c as I believe the value proposition is to have simple implementation that can execute the inference locally on wide variety of platforms. What if we can execute fine-tuned Llama7B on our phones?
Re: Karpathy's llama2.c ported to pure Python
#8I find these efforts impressive, but what is the value proposition here? (I'm not just talking about this fork, but also Karapathy's llama2.c as well).
Its educational. It shows a how llama works in a clear, concise, testable way.
Re: Karpathy's llama2.c ported to pure Python
#9I find these efforts impressive, but what is the value proposition here? (I'm not just talking about this fork, but also Karapathy's llama2.c as well).
Regarding the original llama2.c as I believe the value proposition is to have simple implementation that can execute the inference locally on wide variety of platforms. What if we can execute fine-tuned Llama7B on our phones?
> What if we can execute fine-tuned Llama7B on our phones?
7B and 13B are already quite performant with mlc-llm (which uses an Apache TVM Vulkan/Metal backend). Llama.cpp has the potential to perform well too.
These "single file" implementations are not meant to be optimized or feature rich, I dont think.
Re: Karpathy's llama2.c ported to pure Python
#10The llama2.py code defines its own accum, rmsnorm and matmul. Why not use NumPy? A "pure Python" code that is much slower than one using NumPy is less interesting to me.