41–50 of 54 posts
It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...
On line 59, there is a less-than-or-equals comparison between 0 and 1. Curious https://github.com/meta-llama/llama3/blob/main/llama/model.p...
Earlier quoted context omitted.
On line 59, there is a less-than-or-equals comparison between 0 and 1. Curious https://github.com/meta-llama/llama3/blob/main/llama/model.p...
What's the operator precedence in python? Is it `assert(0 <= (1 < ndim))` or `assert((0 <= 1) < ndim)`, or something even stranger like `assert(0 <= 1) < ndim`?
It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...
[1] https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959...
[2] https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959...
Trainable Llama-like transformer (with backpropagation) in numpy only (~600 lines) https://github.com/joennlae/tensorli
It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...
Why is max_seq_len set to 2048 [1] when the model card says the context size is 8k [2]? [1] https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959... [2] https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959...
> All models support sequence length up to 8192 tokens, but we pre-allocate the cache according to max_seq_len and max_batch_size values. So set those according to your hardware.
[0] https://github.com/meta-llama/llama3/tree/14aab0428d3ec3a959...
Earlier quoted context omitted.
So is this the case that the information is in the data set? Or the code is very well defined to be so small? As an outsider it's surprising that such a capable model can be so "simple".
The training code is presumably quite a bit more complex than what they've open sourced, but part of the beauty of the GPT-based LLMs is their structural simplicity. Now, that simplicity can be deceiving - there are a lot of conceptual interconnectedness within these models. They've been put together "just so" if you will. If you look at the source code to nanoGPT and compare it to Llama3, the most remarkable thing (…
It's also worth mentioning that the original implementation by Meta is only 300 lines of very readable code [1]. [1]: https://github.com/meta-llama/llama3/blob/main/llama/model.p...