Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLaMA: A foundational, 65B-parameter large language model

ai.facebook.com

Post reply on HN
Print view View original

161–170 of 209 posts

Re: LLaMA: A foundational, 65B-parameter large language model

#161

Post by a-dub »

fwlr wrote: post #10
mjburgess wrote: post #3

Earlier quoted context omitted.

> We release all our models to the research community. This is yet more evidence for the "AI isn't a competitive advantage" thesis. State-of-the-Art is a public resource, so competing with AI offers no "moat".

In terms of medieval warfare, what Facebook is doing here looks like filling the moat with rocks and dirt. OpenAI is worth billions, Microsoft is spending billions to retrofit most of their big offerings with AI, Google is doubtless also spending billions to integrate AI in their products. Their moat in all cases is “a blob of X billion weights trained on Y trillion tokens”. Facebook here is spending mere _millions_…

i think maybe the LLM team at facebook was bummed out because twitter bullied them on their last public release (and they didn't ignore it), and this time they decided to sit down and nerd flex by doing some undeniably excellent performance work that reduces resource requirements by 10x and limits itself only to publicly available training data.

maybe they care about moats and elon muskcrosoft's closedai or whatever, but i kinda doubt it. again, it feels more like a nerd flex probably for the purposes of raising morale internally and pushing the field as a whole in a good direction by reducing resource requirements.

excellent paper! easy on the eyes and i really like the angle.

Re: LLaMA: A foundational, 65B-parameter large language model

#162

Post by eshack94 »

iandanforth wrote: post #95

"we are publicly releasing LLaMA" "Access to the model will be granted on a case-by-case basis to academic researchers" They keep saying the word 'release' but I don't think they know what that word means. There are perfectly good words in the English language to describe this situation without abusing "release." They "will begin to grant access to a select few". Nothing about that releases the model, or their contro…

Completely agree. Wish they'd stop throwing misleading words around for PR purposes. Just call it what it is.

Re: LLaMA: A foundational, 65B-parameter large language model

#164

Post by pavelstoev »

foobiekr wrote: post #156
SethTro wrote: post #138

Earlier quoted context omitted.

(1022362 + 82432) gpu-hours / 2048gpus / 5 months ~= 15% uptime. That's only 0.08 nines of availability! I remember in one of their old guidebooks a lot of struggle to keep their 64 machine (512 gpu) cluster running this was probably 4x the machines and 4x the number of cluster dropouts.

Poor GPU utilization even when available is the rule. Truly amazing. Staging of data is probably a huge part of it.

At CentML, we profiled GPU utilization on a larger AI/ML research institute cluster. 10% to 45% range, mostly in 10% utilization range. We then offered them software optimizers (which do not affect model accuracy) to get to the 90% utilization for GPUs

Re: LLaMA: A foundational, 65B-parameter large language model

#165

Post by alibero »

rcpt wrote: post #76
recuter wrote: post #34

Earlier quoted context omitted.

I don't know what my bias is supposed to be but I called them dorks affectionately for one. The other replies are literally arguing that they are very extremely supervised where as I am speculating they are just eager to share their work for the right reasons and the eye of Sauron has yet to turn upon them. Inside knowledge I never claimed. Anything else I can help you with today? :)

> I called them dorks affectionately Never in my life have I seen that word used affectionately

As one of the unsupervised dorks working on LLMs at Meta (not one of the authors here) I took it in a positive way :)

Re: LLaMA: A foundational, 65B-parameter large language model

#166

Post by nl »

nhinck2 wrote: post #160
nl wrote: post #151

Earlier quoted context omitted.

This isn't stealing at all. I want my open source code to be used like this.

If only there was some kind of contract like thing you could release your code under so that there was no ambiguity.

Sarcasm doesn't translate well online...

To be clear: there is and it's pretty difficult to argue that MS is violating even the GPL.

Re: LLaMA: A foundational, 65B-parameter large language model

#167

Post by hansvm »

akomtu wrote: post #149
saurabh20n wrote: post #35

Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…

By "parameters" they probably mean float32s, and 65B of those is 0.25 TB of data - more than enough to memorize a 1.5T sequence of "tokens" (3 letter triplets?). This begs the question: are these models better than a fuzzy hash table?

Yes and no. Information theoretically, tokens are pretty well compressed, and you can't get another 6x losslessly.

Moreover, anything even kind of looking like a hash table in the input/output space is ruled out by the observed facts that the models can extremely respond frequently to samples crafted to not be in the training set and that it takes into account many long-range dependencies (i.e., the hash table would have to be exponentially larger than it is to match the model's performance).

That said, they are just statistical party tricks. The magic happens because the lookup tables are in a latent space. That's why you can drop in garbage like "uberworldchefinatormichelingodfoodpleasureorgasmmaestro" when asking for recipes and food recommendations and get an experience planets apart from queries excluding the nonsense phrases. The model is just pulling together some token associations, and throwing in the right tokens can take advantage of those in situations where a thinking person would barely be able to parse what you're asking.

Your question feels like it has a motive though. What are you really asking?

Re: LLaMA: A foundational, 65B-parameter large language model

#168

Post by pgt »

woeirua wrote: post #146
saurabh20n wrote: post #35

Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…

These cost estimates really make me question OpenAI's valuation. Also, they kind of prove to me that most companies are totally incapable of making the investments necessary to get much out of this type of AI.

Financial hurdles to competitors can make the company that has overcome them more defensible.

Re: LLaMA: A foundational, 65B-parameter large language model

#169

Post by programmer_dude »

323 wrote: post #102
saurabh20n wrote: post #35

Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…

> we used 2048 A100-80GB for a period of approximately 5 months Do we know how much total energy a human consumes from birth to 20 yo? Something like 2000 calories integrated over 20 years. How does it compare to the GPUs above? Wolfram Alpha: - human - 17 MW/h ((2000 calories per day) over 20 years in MWh) - GPUs - 3000 MW/h ((2048 * 400) W over 5 months in MWh) We still have the edge. LOL, I'm being downvoted, I wo…

Your units are bad. Did you mean MWh instead of MW/h?

Re: LLaMA: A foundational, 65B-parameter large language model

#170

Post by elbear »

sclarisse wrote: post #158
naasking wrote: post #72

Earlier quoted context omitted.

Facebook maybe can't make money from it, but they arguably could save money from it, for instance in automating some fact checking and moderation activities that they currently spend quite a bit of money on.

If you’re going to ask an AI to do fact checking with today’s technology, I would urge you to start by asking the AI a simple test question: Which is heavier? A pound of feathers or two pounds of lead?

Do you have an explanation for why it fails on this question?
Post reply on HN