I hope they make at least the smallest one available for everyone. I find it ironic that they want to prevent misuse, but think allowing access to those affiliated with government organizations will achieve that.
[deleted]
LLaMA: A foundational, 65B-parameter large language model
31–40 of 209 posts
Re: LLaMA: A foundational, 65B-parameter large language model
#32Earlier quoted context omitted.
In terms of medieval warfare, what Facebook is doing here looks like filling the moat with rocks and dirt. OpenAI is worth billions, Microsoft is spending billions to retrofit most of their big offerings with AI, Google is doubtless also spending billions to integrate AI in their products. Their moat in all cases is “a blob of X billion weights trained on Y trillion tokens”. Facebook here is spending mere _millions_…
Which kind of suggests Microsoft made a really bad move antagonizing the open-source community with Gtihub Copilot. They got a few years of lead time in the "AI codes for you" market, but in exchange permanently soured a significant fraction of their potential userbase who will turn to open-source alternatives soon anyway. I wonder if they'd have been better served focusing on selling Azure usage and released Copilot…
Microsoft makes dozens of "really bad moves" every year. This is nothing.
Re: LLaMA: A foundational, 65B-parameter large language model
#33> To maintain integrity and prevent misuse, we are releasing our model under a noncommercial license focused on research use cases. if say i wanted to replicate this paper for commercial use, what would it take and how do i get started? would FB have a basis for objection?
82342 GPU(A-100 80GB) hours for training their 7B(smallest) model according to their paper.
115 GPUs can be used to train it within month.
Re: LLaMA: A foundational, 65B-parameter large language model
#34Earlier quoted context omitted.
I don't see such grand stratagems as being a likely explanation. It seems more likely that a bunch of dorks are running around unsupervised trying their best to make lemonade with whatever budgets they are given as nobody can realistically manage AI researches. Or at least, it seems that much like in political governance, events are suddenly outpacing the ability of corporate to react. In the case of OpenAI it is a "…
this is a pretty biased and uninformed opinion. Pretty condescending to call it a "bunch of dorks...running around unsupervised."
I don't know what my bias is supposed to be but I called them dorks affectionately for one. The other replies are literally arguing that they are very extremely supervised where as I am speculating they are just eager to share their work for the right reasons and the eye of Sauron has yet to turn upon them.
Inside knowledge I never claimed. Anything else I can help you with today? :)
Re: LLaMA: A foundational, 65B-parameter large language model
#35Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef...:
* All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2]
* They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a range of $1 to $4. The 7B model would have cost ~$82-329k and the 65B something in the range of ~$1-4M. They also note their total time spent for all models: "we used 2048 A100-80GB for a period of approximately 5 months" [sec 6, pg 10]
* 65B model's performance is broadly comparable to PALM-540B. Not a small feat, but also could indicate the benefits of good model-vs-token size ratios [Tables 3,4,5,6]. Their conjecture for underperforming on MMLU (multitask language understanding) compared to PALM-540B and Chinchilla-70B is smaller fraction of books and academic training data.
* Math and code tasks: Math tasks they are substantially worse than Minerva (comparing their 65B to Minerva 62B; they hands down fail against Minerva 540B) [Table 7]. Code tasks they are broadly competitive with PALM-540B (HumanEval and MBPP evals) [Table 8]
* Surprising that instruction fine tuning takes such a small part of the paper (sec 4, pg. 7)
Re: LLaMA: A foundational, 65B-parameter large language model
#36I was surprised to see this citation in the linked paper
> Alan M Turing. 2009. Computing machinery and intelligence
Did he invent a time machine too?
Re: LLaMA: A foundational, 65B-parameter large language model
#37Earlier quoted context omitted.
In terms of medieval warfare, what Facebook is doing here looks like filling the moat with rocks and dirt. OpenAI is worth billions, Microsoft is spending billions to retrofit most of their big offerings with AI, Google is doubtless also spending billions to integrate AI in their products. Their moat in all cases is “a blob of X billion weights trained on Y trillion tokens”. Facebook here is spending mere _millions_…
I don't see such grand stratagems as being a likely explanation. It seems more likely that a bunch of dorks are running around unsupervised trying their best to make lemonade with whatever budgets they are given as nobody can realistically manage AI researches. Or at least, it seems that much like in political governance, events are suddenly outpacing the ability of corporate to react. In the case of OpenAI it is a "…
> It seems more likely that a bunch of dorks are running around unsupervised trying their best to make lemonade with whatever budgets they are given
Probably not, Zuck is announcing it.
Re: LLaMA: A foundational, 65B-parameter large language model
#38The most interesting snippet in the paper I think is this:
> For instance, LLaMA-13B outperforms GPT-3 on most bench- marks, despite being 10× smaller. We believe that this model will help democratize the access and study of LLMs, since it can be run on a single GPU.
Re: LLaMA: A foundational, 65B-parameter large language model
#39Earlier quoted context omitted.
> We release all our models to the research community. This is yet more evidence for the "AI isn't a competitive advantage" thesis. State-of-the-Art is a public resource, so competing with AI offers no "moat".
Business orgs have been finetuning open-source models like these on their own internal data to create a moat since BERT in 2018.
Glad to see you still on HN! You've done amazing work in this domain!
I'd argue that this goes further back to the word2vec/glove days too. I was working for a company in 2018 who leveraged my skills for fine-tuning word2vec/fasttext even before BERT/attention is all you need paper.
Re: LLaMA: A foundational, 65B-parameter large language model
#40Earlier quoted context omitted.
How did Microsoft sour developers with Copilot? I know dozens of people that pay for it (including myself) and I feel like it is widely regarded as a "no brainer" for the price that it's offered at. Please help me understand!
The company that tried to kill Linux in the 90s, owned by the world's most famously rich man, is now stealing my code and selling it back to me? Yeah, fuck that.
It's not selling you back your code. It's different code, adapted to a different task; your own code is forever free for you, you don't need anyone to give it to you.
Given the cost of running these models, and the utmost dedication needed to train them, I think it is worth it. GPUs cost money, electricity costs money. They can't serve the world for free and offer good latency.