We changed the URL from https://github.com/likejazz/llama3.np to the article it points to, which gives more background.
Llama 3 implemented in pure NumPy
31–40 of 54 posts
Re: Llama 3 implemented in pure NumPy
#32It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...
So is this the case that the information is in the data set? Or the code is very well defined to be so small? As an outsider it's surprising that such a capable model can be so "simple".
I think with LLMs in general, the algorithms are very refined and require lots of research, despite being "simple" in terms of entropy, or an imagined Kolgomorov complexity for defining algorithms.
So "simple" is a fuzzy term here, but yes, the entropic complexity is in the data, not the algorithms.
Related to the so-called "Bitter lesson".
Edit: the sister comment pointed out what I failed to express: RILHF and training are also algorithms, and their applications and implementations are probably much more complex than the code that evaluates a given prompt.
So basically, "models" (trained NNs) are also an example for the equivalence of code and data.
Fixed data used by code (the trained model) is code in itself, even when it is not directly written by humans or in a human-readable language.
Edit edit: don't forget to count the imported maths code :) but I assume this is not relevant to the "it's just matrix multiplications" overall argument
Re: Llama 3 implemented in pure NumPy
#33It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...
So is this the case that the information is in the data set? Or the code is very well defined to be so small? As an outsider it's surprising that such a capable model can be so "simple".
300 lines of this code is a bit different than 300 lines of typical code where you read files, set up a backend/frontend, or parse data. In the latter case, there are a lot of tedious operations. Sure, the former also has that with reshaping and asserts or wtv.
But in a sense, the 300 lines of Llama code are essentially just lines of math. And reading through any math proof will show you that any particular line can hide large amounts of complexity.
This can be true with code with more tedious operations, but those lines are a smaller fraction of the overall code base by definition.
Even the "tedious" parts of the llama code can hide large complexity. Setting a learning rate with a schedule might require reading a paper or two for your particular architecture.
But yes, once you parse all the math and the theory, the lines are kinda simple matmul and forward lol.
Re: Llama 3 implemented in pure NumPy
#34It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...
Do you know why these are so short? What is the algorithm/magic in all of these?
I tried to make sense of it but cannot
Re: Llama 3 implemented in pure NumPy
#35It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...
Do you know why these are so short? What is the algorithm/magic in all of these? I tried to make sense of it but cannot
The magic is in the billions of learned weights (~synapses). This is just the scaffolding that runs them.
Re: Llama 3 implemented in pure NumPy
#36It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...
Do you know why these are so short? What is the algorithm/magic in all of these? I tried to make sense of it but cannot
Architecturally, LLMs are very simple compared to many software projects.
The crux of their behavior comes from their learned weights which are gigabytes and can cost millions to obtain via training.
Re: Llama 3 implemented in pure NumPy
#37It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...
On line 59, there is a less-than-or-equals comparison between 0 and 1. Curious
https://github.com/meta-llama/llama3/blob/main/llama/model.p...
Re: Llama 3 implemented in pure NumPy
#38Earlier quoted context omitted.
So is this the case that the information is in the data set? Or the code is very well defined to be so small? As an outsider it's surprising that such a capable model can be so "simple".
300 lines of this code is a bit different than 300 lines of typical code where you read files, set up a backend/frontend, or parse data. In the latter case, there are a lot of tedious operations. Sure, the former also has that with reshaping and asserts or wtv. But in a sense, the 300 lines of Llama code are essentially just lines of math. And reading through any math proof will show you that any particular line can…
Sure, knowing the basics of LLM math is necessary. But it's also _enough_ to know this math to fully grasp the code. There are only 4 concepts - attention, feed-forward net, RMS-normalization and rotary embeddings - organized into a clear structure.
Now compare it to the Hugginface implementation [1]. In addition to the aforementioned concepts, you need to understand the hierarchy of `PreTrainedModel`s, 3 types of attention, 3 types of rotary embeddings, HF's definition of attention mask (which is not the same as mask you read about in transformer tutorials), several types of cache class, dozens of flags to control things like output format or serialization, etc.
It's not that Meta's implementation is good and HF's implementation is bad - they pursue different goals in their own optimal way. But if you just want to learn how the model works, Meta's code base is great.
[1]: https://github.com/huggingface/transformers/blob/main/src/tr...
Re: Llama 3 implemented in pure NumPy
#39It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...
Do you know why these are so short? What is the algorithm/magic in all of these? I tried to make sense of it but cannot
The magic is the structure of the model, and the real magic is the billions of weights.
Re: Llama 3 implemented in pure NumPy
#40It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...
On line 59, there is a less-than-or-equals comparison between 0 and 1. Curious https://github.com/meta-llama/llama3/blob/main/llama/model.p...
What's the operator precedence in python?
Is it `assert(0 <= (1 < ndim))` or `assert((0 <= 1) < ndim)`, or something even stranger like `assert(0 <= 1) < ndim`?