> np.sin(freqs)
Didn't we drop 2 pi somewhere?
Llama 3 implemented in pure NumPy
51–54 of 54 posts
Re: Llama 3 implemented in pure NumPy
#52Earlier quoted context omitted.
So is this the case that the information is in the data set? Or the code is very well defined to be so small? As an outsider it's surprising that such a capable model can be so "simple".
The training code is presumably quite a bit more complex than what they've open sourced, but part of the beauty of the GPT-based LLMs is their structural simplicity. Now, that simplicity can be deceiving - there are a lot of conceptual interconnectedness within these models. They've been put together "just so" if you will. If you look at the source code to nanoGPT and compare it to Llama3, the most remarkable thing (…
> beauty of the GPT-based LLMs is their structural simplicity
human brain's structure is also encoded in a short DNA sequence
Re: Llama 3 implemented in pure NumPy
#53Earlier quoted context omitted.
So is this the case that the information is in the data set? Or the code is very well defined to be so small? As an outsider it's surprising that such a capable model can be so "simple".
The training code is presumably quite a bit more complex than what they've open sourced, but part of the beauty of the GPT-based LLMs is their structural simplicity. Now, that simplicity can be deceiving - there are a lot of conceptual interconnectedness within these models. They've been put together "just so" if you will. If you look at the source code to nanoGPT and compare it to Llama3, the most remarkable thing (…
> Goes to show just how much is in the training data.
And in the scale (num_layers, embed_dim, num_heads) of the model of course ;)
Re: Llama 3 implemented in pure NumPy
#54Earlier quoted context omitted.
What's the operator precedence in python? Is it `assert(0 <= (1 < ndim))` or `assert((0 <= 1) < ndim)`, or something even stranger like `assert(0 <= 1) < ndim`?
Python actually does something pretty neat: it chains comparisons so that `x In linked code we can be confident that `0 <= 1`, so only `1 < ndim` should matter. In fact I'd expect peephole optimization to remove most of the code for `0 <= 1`
I did not know that python fact. Thanks for sharing!