Funny how the release has a note that the models clearly have ethical problemas that must be addressed, and still the company chooses to publish it.
LLaMA: A foundational, 65B-parameter large language model
101–110 of 209 posts
Re: LLaMA: A foundational, 65B-parameter large language model
#102Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…
> we used 2048 A100-80GB for a period of approximately 5 months
Do we know how much total energy a human consumes from birth to 20 yo? Something like 2000 calories integrated over 20 years. How does it compare to the GPUs above?
Wolfram Alpha:
- human - 17 MW/h ((2000 calories per day) over 20 years in MWh)
- GPUs - 3000 MW/h ((2048 * 400) W over 5 months in MWh)
We still have the edge.
LOL, I'm being downvoted, I wonder way. Some don't like the question.
Re: LLaMA: A foundational, 65B-parameter large language model
#103Funny how the release has a note that the models clearly have ethical problemas that must be addressed, and still the company chooses to publish it.
The old "not my problem how people use the gun, I'm just a manufacturer" defense.
Re: LLaMA: A foundational, 65B-parameter large language model
#104Earlier quoted context omitted.
Presumably, because they trained Copilot on billions of lines of, often licensed, code (without permission), that Copilot has a tendency to regurgitate verbatim, without said license.
For a specific example some variation of "fast inverse square root" will usually get you the exact GPL licensed code from Quake III, comments included.
Do you mean the same code that has its own Wikipedia page where the exact code is written, comments included, and has probably been copy pasted into 100’s of other projects?
Re: LLaMA: A foundational, 65B-parameter large language model
#105Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…
>* 65B model's performance is broadly comparable to PALM-540B. Not a small feat, but also could indicate the benefits of good model-vs-token size ratios [Tables 3,4,5,6]. Their conjecture for underperforming on MMLU (multitask language understanding) compared to PALM-540B and Chinchilla-70B is smaller fraction of books and academic training data.* what do you mean by this ? The OpenAI papers talk roughly about model…
Scaling law is for training till convergence. Both PALM and this model have been undertrained. See the training loss plot in the paper.
Re: LLaMA: A foundational, 65B-parameter large language model
#106"The code is licensed under the GPLv3, which permits commercial use." Yann on Twitter
Re: LLaMA: A foundational, 65B-parameter large language model
#107Earlier quoted context omitted.
The company that tried to kill Linux in the 90s, owned by the world's most famously rich man, is now stealing my code and selling it back to me? Yeah, fuck that.
I mean, that's like saying an author steals the open source alphabet and charges you for reading their ordering of letters, as if the ordering of letters isn't where all the value is.
These models are trained on sequences of words, not told the letters and left to get on with it.
Re: LLaMA: A foundational, 65B-parameter large language model
#108Earlier quoted context omitted.
For a specific example some variation of "fast inverse square root" will usually get you the exact GPL licensed code from Quake III, comments included.
Do you mean the same code that has its own Wikipedia page where the exact code is written, comments included, and has probably been copy pasted into 100’s of other projects? https://en.m.wikipedia.org/wiki/Fast_inverse_square_root
The point is that it's charging having been trained on open source code. What you're saying agrees with that, but your triumphant tone seems to be implying the opposite. Which did you mean?
Re: LLaMA: A foundational, 65B-parameter large language model
#109Newbie asks: what is a 'parameter' here
Re: LLaMA: A foundational, 65B-parameter large language model
#110How does token->embedding lookup work with 1.4T BPE tokens? Since there are more tokens than the 65B parameters it must be doing some sort of interesting thing based on the merge operations. Is it different from what other GPT models with ~100k tokens are doing? At inference, how many of those tokens are used? (they mention most tokens are used only once during training, so the must be very long sequences.)
The 1.4T tokens are what the model was trained on, and not the token range of the embedding.