91–100 of 209 posts
(creator of gpt index / llamaindex here https://github.com/jerryjliu/gpt_index ) Funny that we had just rebranded our tool from GPT Index to LlamaIndex about a week ago to avoid potential trademark issues with OpenAI, and turns out Meta has similar ideas around LLM+llama puns :). Must mean the name is good though! Also very excited to try plugging in the LLaMa model into LlamaIndex, will report the results.
This blog post is terrible at listing the improvements offered by the model, the abstract is better: > We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In par…
> We release all our models to the research community. This is yet more evidence for the "AI isn't a competitive advantage" thesis. State-of-the-Art is a public resource, so competing with AI offers no "moat".
And from the FB blogpost [0] "Request Form Thank you for your interest in Meta AI’s LLaMA (Large Language Model Meta AI) models. To request access to the models, please fill out this form, and we'll review and let you know if your use case is approved. The information you provide below will be used solely to assess eligibility to access these models."
So much for "releasing" the model for research community.
[0] https://ai.facebook.com/blog/large-language-model-llama-meta...
Earlier quoted context omitted.
Yes but fine tuning for RL is not expected to be hard. You're essentially limited by how much human feedback is available, so it's very different from training the foundational model on random bulk data.
At this point I fully expect that someone will release a usable RLHF-fine-tuned language model that can run on consumer hardware, based on the methodology used for LLaMA (and other similar papers e.g. https://github.com/FMInference/FlexGen ), at some point in the next 6-24 months.
"Access to the model will be granted on a case-by-case basis to academic researchers"
They keep saying the word 'release' but I don't think they know what that word means. There are perfectly good words in the English language to describe this situation without abusing "release." They "will begin to grant access to a select few". Nothing about that releases the model, or their control, which they are not doing and shouldn't imply.
The most interesting snippet in the paper I think is this: > For instance, LLaMA-13B outperforms GPT-3 on most bench- marks, despite being 10× smaller. We believe that this model will help democratize the access and study of LLMs, since it can be run on a single GPU.
At inference, how many of those tokens are used? (they mention most tokens are used only once during training, so the must be very long sequences.)
How possible is it to run these models on a gaming GPU?
I was surprised to see this citation in the linked paper > Alan M Turing. 2009. Computing machinery and intelligence Did he invent a time machine too?
Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…
what do you mean by this ? The OpenAI papers talk roughly about model performance scaling by parameters. does this show the other way ?