Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLaMA: A foundational, 65B-parameter large language model

ai.facebook.com

Post reply on HN
Print view View original

121–130 of 209 posts

Re: LLaMA: A foundational, 65B-parameter large language model

#121

Post by swyx »

causalmodels wrote: post #56
swyx wrote: post #18

> To maintain integrity and prevent misuse, we are releasing our model under a noncommercial license focused on research use cases. if say i wanted to replicate this paper for commercial use, what would it take and how do i get started? would FB have a basis for objection?

I'm going to assume you know how to stand up and manage a distributed training cluster as a simplifying assumption. Note this is an aggressive assumption. You would need to replicate the preprocessing steps. Replicating these steps is going to be tricky as they are not described in detail.Then you would need to implement the model using xformers [1]. Using xformers is going to save you a lot of compute spend. You wil…

thank you! love getting a peek into infra math. hopefully these will come down someday but also my resources rise up to be able to train these inhouse. can foresee foundation model training being a core competency of many tech platform companies in future.

Re: LLaMA: A foundational, 65B-parameter large language model

#122

Post by KarlKemp »

robertlagrant wrote: post #107
mbb70 wrote: post #49

Earlier quoted context omitted.

I mean, that's like saying an author steals the open source alphabet and charges you for reading their ordering of letters, as if the ordering of letters isn't where all the value is.

These models are trained on sequences of words, not told the letters and left to get on with it.

It continues to amaze how people are incapable of following even the most trivial cases of abstract reasoning.

Re: LLaMA: A foundational, 65B-parameter large language model

#123

Post by modeless »

iandanforth wrote: post #95

"we are publicly releasing LLaMA" "Access to the model will be granted on a case-by-case basis to academic researchers" They keep saying the word 'release' but I don't think they know what that word means. There are perfectly good words in the English language to describe this situation without abusing "release." They "will begin to grant access to a select few". Nothing about that releases the model, or their contro…

Someone needs to start a pirate bay for torrents of ML models. Call it ClosedAI, since in AI land these words mean the opposite of what they say.

Re: LLaMA: A foundational, 65B-parameter large language model

#124

Post by mgraczyk »

fwlr wrote: post #10
mjburgess wrote: post #3

Earlier quoted context omitted.

> We release all our models to the research community. This is yet more evidence for the "AI isn't a competitive advantage" thesis. State-of-the-Art is a public resource, so competing with AI offers no "moat".

In terms of medieval warfare, what Facebook is doing here looks like filling the moat with rocks and dirt. OpenAI is worth billions, Microsoft is spending billions to retrofit most of their big offerings with AI, Google is doubtless also spending billions to integrate AI in their products. Their moat in all cases is “a blob of X billion weights trained on Y trillion tokens”. Facebook here is spending mere _millions_…

My guess is that the primary strategic motivation here is recruiting and maintaining a competent org of people who know how to build large models. I don't think that the actual release is as calculated as all that. It's more about proving to themselves and the world that they can train big models.

Re: LLaMA: A foundational, 65B-parameter large language model

#127

Post by melling »

323 wrote: post #102
saurabh20n wrote: post #35

Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…

> we used 2048 A100-80GB for a period of approximately 5 months Do we know how much total energy a human consumes from birth to 20 yo? Something like 2000 calories integrated over 20 years. How does it compare to the GPUs above? Wolfram Alpha: - human - 17 MW/h ((2000 calories per day) over 20 years in MWh) - GPUs - 3000 MW/h ((2048 * 400) W over 5 months in MWh) We still have the edge. LOL, I'm being downvoted, I wo…

Every human requires the same energy, 20+ years, and training.

The trained computer model can be duplicated and used, requiring much less energy.

None of this matters to me, though.

The goal is to build better models. We can worry about the efficiency later.

Re: LLaMA: A foundational, 65B-parameter large language model

#128

Post by Dylan16807 »

323 wrote: post #102
saurabh20n wrote: post #35

Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…

> we used 2048 A100-80GB for a period of approximately 5 months Do we know how much total energy a human consumes from birth to 20 yo? Something like 2000 calories integrated over 20 years. How does it compare to the GPUs above? Wolfram Alpha: - human - 17 MW/h ((2000 calories per day) over 20 years in MWh) - GPUs - 3000 MW/h ((2048 * 400) W over 5 months in MWh) We still have the edge. LOL, I'm being downvoted, I wo…

> We still have the edge.

Depends on what you're doing. A human is much smarter than one of these models, but the model has approximate knowledge of orders of magnitude more things. And the energy costs per word of output are a lot closer.

Re: LLaMA: A foundational, 65B-parameter large language model

#130

Post by invig »

webmaven wrote: post #90
vineyardmike wrote: post #53

Earlier quoted context omitted.

I think there is definitely a component of competition here. Facebook has no real way to monetize this (eg they won’t release an api a la OpenAI and they don’t own a search engine). Since they can’t monetize it… why not provide a bit of kindling to lower the barrier for everyone else to compete with your competitors. This strategy is called “commodize your complement”. If Facebook makes it easier to develop a google…

> Facebook has no real way to monetize this (eg they won’t release an api a la OpenAI and they don’t own a search engine). Since they can’t monetize it… why not provide a bit of kindling to lower the barrier for everyone else to compete with your competitors. This strategy is called “commodize your complement”. Your analysis is good, but that isn't what "commoditize your complement" means. Strictly speaking, a search…

I don't think that's the best interpretation of complement. Think of complement as "is your users experience benefited by the existence of the other thing". I think that Search qualifies as something that enhances the Facebook users experience (e.g. my cousin mentioned thing that they like doing today, how do I find more about thing?)
Post reply on HN