Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3 implemented in pure NumPy

docs.likejazz.com

Post reply on HN
Print view View original

21–30 of 54 posts

Re: Llama 3 implemented in pure NumPy

#21

Post by rhdunn »

From the TinyStories dataset card [1] the dataset is generated by GPT-3.5 and GPT-4. Reading the discussions in the community tab [2] it looks like there are a lot of incomplete or misspelled words, incorrect grammar, and even Chinese characters in the dataset.

As such, I'd be weary of using that dataset to train or evaluate models.

[1] https://huggingface.co/datasets/roneneldan/TinyStories

[2] https://huggingface.co/datasets/roneneldan/TinyStories/discu...

Re: Llama 3 implemented in pure NumPy

#22

Post by nwoli »

rhdunn wrote: post #21

From the TinyStories dataset card [1] the dataset is generated by GPT-3.5 and GPT-4. Reading the discussions in the community tab [2] it looks like there are a lot of incomplete or misspelled words, incorrect grammar, and even Chinese characters in the dataset. As such, I'd be weary of using that dataset to train or evaluate models. [1] https://huggingface.co/datasets/roneneldan/TinyStories [2] https://huggingface.co…

It’s just used for checking that the implementation is correct. The dataset is just a toy dataset it doesn’t matter if it has misspelled words

Re: Llama 3 implemented in pure NumPy

#24

Post by ffriend »

lnyan wrote: post #14

`import jax.numpy as np`, then we also get a jax implemention after certain modifications: e.g. remove in-place index assignment, replace unsupported functions, etc

JAX requires a bit more work to maintain fixed-size buffers as required by XLA, especially in case of caching and rotary embeddings. But yeah, overall the code can be pretty similar [1].

[1]: https://github.com/dfdx/fabrique/blob/main/fabrique/llama/mo...

Re: Llama 3 implemented in pure NumPy

#27

Post by blt »

ffriend wrote: post #25

It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...

the simplicity of the transformer is quite refreshing. especially in vision where the Vision Transformer with linear patch encodings replaces complex intertwined decisions about filter size, striding, pooling, #filters, depth, etc., with the simpler decision of how to allocate your FLOPS between dimensionality, #heads, and #layers.

Re: Llama 3 implemented in pure NumPy

#28

Post by blharr »

ffriend wrote: post #25

It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...

So is this the case that the information is in the data set? Or the code is very well defined to be so small? As an outsider it's surprising that such a capable model can be so "simple".

Re: Llama 3 implemented in pure NumPy

#29

Post by jacobn »

blharr wrote: post #28
ffriend wrote: post #25

It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...

So is this the case that the information is in the data set? Or the code is very well defined to be so small? As an outsider it's surprising that such a capable model can be so "simple".

The training code is presumably quite a bit more complex than what they've open sourced, but part of the beauty of the GPT-based LLMs is their structural simplicity.

Now, that simplicity can be deceiving - there are a lot of conceptual interconnectedness within these models. They've been put together "just so" if you will.

If you look at the source code to nanoGPT and compare it to Llama3, the most remarkable thing (when you look past the superficial name changes) is just how similar they are.

If I recall correctly the primary differences are:

  - The MLP: Llama3 uses SwiGLU vs the more "traditional" x = x + proj(gelu(expand(x))) in GPT2
  - The token encoders, which is arguably external to the model
  - Attention: Llama3 uses Grouped Query Attention, vs full Multi-Head Attention in GPT2
  - Normalization: Llama3 uses RMSNorm, vs LayerNorm for GPT2
They were published more than five years apart. On the one hand progress has been breathtaking, truly astounding. On the other hand, it's almost exactly the same model.

Goes to show just how much is in the training data.

Post reply on HN