Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLaMA: A foundational, 65B-parameter large language model

ai.facebook.com

Post reply on HN
Print view View original

51–60 of 209 posts

Re: LLaMA: A foundational, 65B-parameter large language model

#51

Post by cardine »

mjburgess wrote: post #3
minimaxir wrote: post #2

This blog post is terrible at listing the improvements offered by the model, the abstract is better: > We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In par…

> We release all our models to the research community. This is yet more evidence for the "AI isn't a competitive advantage" thesis. State-of-the-Art is a public resource, so competing with AI offers no "moat".

And yet there are still no publicly available models that could actually compete with ChatGPT.

I'm not even talking about RLHF (although data like that is also a huge moat) - just simple things like larger context sizes.

There are still plenty of AI advantages to be had if you go just a little bit outside of what is currently possible with off the shelf models.

Re: LLaMA: A foundational, 65B-parameter large language model

#52

Post by igravious »

muglug wrote: post #36

I was surprised to see this citation in the linked paper > Alan M Turing. 2009. Computing machinery and intelligence Did he invent a time machine too?

Absolutely a pet peeve of mine.

What's wrong with (orig. pub. 1950, rev. ed. 2009) or something like that.

With Zotero, the ref manager I use, you need to know the special magic incantation to store the original date of publication (the publishing date refers to the edition you're citing) but it just looks stupid (I think) to see Nietzsche F. 1999, Untimely Meditations (or whatever) and also I'd like to sort texts by original date of publication from oldest to newest because that's interesting. You have to put a special magic code in the extra field and then your tool-chain has to preserve it and transmogrify it correctly in the doc you're making use of the citation.

Re: LLaMA: A foundational, 65B-parameter large language model

#53

Post by vineyardmike »

recuter wrote: post #11
fwlr wrote: post #10

Earlier quoted context omitted.

In terms of medieval warfare, what Facebook is doing here looks like filling the moat with rocks and dirt. OpenAI is worth billions, Microsoft is spending billions to retrofit most of their big offerings with AI, Google is doubtless also spending billions to integrate AI in their products. Their moat in all cases is “a blob of X billion weights trained on Y trillion tokens”. Facebook here is spending mere _millions_…

I don't see such grand stratagems as being a likely explanation. It seems more likely that a bunch of dorks are running around unsupervised trying their best to make lemonade with whatever budgets they are given as nobody can realistically manage AI researches. Or at least, it seems that much like in political governance, events are suddenly outpacing the ability of corporate to react. In the case of OpenAI it is a "…

I think there is definitely a component of competition here.

Facebook has no real way to monetize this (eg they won’t release an api a la OpenAI and they don’t own a search engine). Since they can’t monetize it… why not provide a bit of kindling to lower the barrier for everyone else to compete with your competitors. This strategy is called “commodize your complement”.

If Facebook makes it easier to develop a google alternative, especially by doing something that doesn’t hurt them, then they just weakened a competitor. See Facebook releasing datasets for mapping. Think of the panic ChatGPT caused Google. It only cost a few Million to train but it’s probably costing Google more than that already.

Re: LLaMA: A foundational, 65B-parameter large language model

#55

Post by twoodfin »

fwlr wrote: post #10
mjburgess wrote: post #3

Earlier quoted context omitted.

> We release all our models to the research community. This is yet more evidence for the "AI isn't a competitive advantage" thesis. State-of-the-Art is a public resource, so competing with AI offers no "moat".

In terms of medieval warfare, what Facebook is doing here looks like filling the moat with rocks and dirt. OpenAI is worth billions, Microsoft is spending billions to retrofit most of their big offerings with AI, Google is doubtless also spending billions to integrate AI in their products. Their moat in all cases is “a blob of X billion weights trained on Y trillion tokens”. Facebook here is spending mere _millions_…

Isn't this a classic "commoditize your complements" play? Facebook's value is in their social graph and users' attachment to it via various apps. TikTok and others are not trying to replicate that, they're creating similar volumes of user attachment via models. You can easily imagine LLM's being applied by competitors in a comparable way. If models become commodities, then Facebook continues to hold the one advantage nobody else has.

Re: LLaMA: A foundational, 65B-parameter large language model

#56

Post by causalmodels »

swyx wrote: post #18

> To maintain integrity and prevent misuse, we are releasing our model under a noncommercial license focused on research use cases. if say i wanted to replicate this paper for commercial use, what would it take and how do i get started? would FB have a basis for objection?

I'm going to assume you know how to stand up and manage a distributed training cluster as a simplifying assumption. Note this is an aggressive assumption.

You would need to replicate the preprocessing steps. Replicating these steps is going to be tricky as they are not described in detail.Then you would need to implement the model using xformers [1]. Using xformers is going to save you a lot of compute spend. You will need to manually implement the backwards pass to reduce recomputation of expensive activations.

The model was trained using 2048 A100 GPUs with 80GBs of VRAM. A single 8 A100 GPU machine from Lambda Cloud costs $12.00/hr [2]. The team from meta used 256 such machines giving you a per day cost of $73,728. It takes 21 days to train this model. The upfront lower bound cost estimate of doing this is [(12.00 * 24) * 21 * 256) = ] $1,548,288 dollars assuming everything goes smoothly and your model doesn't bite it during training. You may be able to negotiate bulk pricing for these types of workloads.

That dollar value is just for the compute resources alone. Given the compute costs required you will probably also want a team composed of ML Ops engineers to monitor the training cluster and research scientists to help you with the preprocessing and model pipelines.

[1] https://github.com/facebookresearch/xformers [2] https://lambdalabs.com/service/gpu-cloud

edit: these costs are for the 65B parameter model.

Re: LLaMA: A foundational, 65B-parameter large language model

#58

Post by machinekob »

saurabh20n wrote: post #35

Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…

I hate when people don't include approximation for traning before final hyperparameters are found as its most costly part of whole process most of the time.

Just yes we train it for so long etc. but they never speak about tens or even hundres of runs before they finalize the model parameters and architecture -.-

Re: LLaMA: A foundational, 65B-parameter large language model

#59

Post by adeon »

Has anyone had success being approved to download weights even if you are just a hobbyist with a big GPU? (I'm here about to begrudgingly fill out the Google Form) Asking in general, not just for this particular model.

I've been following these language models and so far went through approval once for a English/Chinese GLM-130B model and it took only ~30 minutes to get approved even though I am a complete nobody.

Post reply on HN