Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLaMA: A foundational, 65B-parameter large language model

ai.facebook.com

Post reply on HN
Print view View original

181–190 of 209 posts

Re: LLaMA: A foundational, 65B-parameter large language model

#181

Post by imtringued »

warkdarrior wrote: post #143
Mizza wrote: post #139

Earlier quoted context omitted.

You mean this code? https://archive.softwareheritage.org/browse/content/sha1_git... Do you see that notice at the top of the file? It says: == This file is part of Quake III Arena source code. Quake III Arena source code is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your…

How is it made proprietary? The Quake III Arena is no more proprietary now then if it were stored on GitHub proprietary web servers. Copilot is just a fancy code index, that sometimes returns the original code and other times it gives you a modified copy.

The problem is you have to obey the license of the code even if you just take a snippet and Copilot does not reproduce the correct license.

Re: LLaMA: A foundational, 65B-parameter large language model

#182

Post by egorfine »

iandanforth wrote: post #95

"we are publicly releasing LLaMA" "Access to the model will be granted on a case-by-case basis to academic researchers" They keep saying the word 'release' but I don't think they know what that word means. There are perfectly good words in the English language to describe this situation without abusing "release." They "will begin to grant access to a select few". Nothing about that releases the model, or their contro…

Agree.

At first I was "oh wow, I should download and run this immediately". Then I have read the press-release and of course this was a marketingspeak with exactly the opposite meaning.

Re: LLaMA: A foundational, 65B-parameter large language model

#183

Post by selfhoster11 »

Zuiii wrote: post #154
modeless wrote: post #123

Earlier quoted context omitted.

Someone needs to start a pirate bay for torrents of ML models. Call it ClosedAI, since in AI land these words mean the opposite of what they say.

> since in AI land these words mean the opposite of what they say. Huh. I think you just might be right: OpenAI that isn't open, AI safety/ethics "researchers" that have nothing to do with safety or ethics, almost every answer chatGPT gives about a topic considered "sensitive" by said "researchers", almost every time ChatGPT falsely asserts it "cannot" do something or simply lies (1). I often wonder why this field be…

> almost every time ChatGPT falsely asserts it "cannot" do something or simply lies (1).

Fun story: ChatGPT, if directly faced with empirical evidence that it can do something that OpenAI made it say it can’t do, (for example, by causing the model to lie about itself by poisoning the input corpus with falsehoods about ChatGPT’s own capabilities), it cannot grasp that there’s a paradox.

Good job, OpenAI.

Re: LLaMA: A foundational, 65B-parameter large language model

#184

Post by selfhoster11 »

sclarisse wrote: post #158
naasking wrote: post #72

Earlier quoted context omitted.

Facebook maybe can't make money from it, but they arguably could save money from it, for instance in automating some fact checking and moderation activities that they currently spend quite a bit of money on.

If you’re going to ask an AI to do fact checking with today’s technology, I would urge you to start by asking the AI a simple test question: Which is heavier? A pound of feathers or two pounds of lead?

I was going to provide evidence that this works correctly, but I was surprised to find that it gets severely tripped up on this particular question, even when told to “show working step by step”.

Re: LLaMA: A foundational, 65B-parameter large language model

#185

Post by cubefox »

sandGorgon wrote: post #178
vishal0123 wrote: post #105

Earlier quoted context omitted.

Scaling law is for training till convergence. Both PALM and this model have been undertrained. See the training loss plot in the paper.

hey thanks for your reply. umm...so does OpenAI. In fact this is OpenAI discovery from [1]: > Convergence is inefficient: When working within a fixed compute budget C but without any other restric- tions on the model size N or available data D, we attain optimal performance by training very large models and stopping significantly short of convergence (see Figure 3). Maximally compute-efficient training would therefor…

This is the old scaling laws paper. The scaling laws in it turned out to be wrong and superseded by the Chinchilla DeepMind paper: https://arxiv.org/abs/2203.15556

Re: LLaMA: A foundational, 65B-parameter large language model

#186

Post by machinekob »

woeirua wrote: post #146
saurabh20n wrote: post #35

Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…

These cost estimates really make me question OpenAI's valuation. Also, they kind of prove to me that most companies are totally incapable of making the investments necessary to get much out of this type of AI.

Sadly big players take all in current world and microsoft is pretty big :|

Re: LLaMA: A foundational, 65B-parameter large language model

#187

Post by swyx »

melling wrote: post #127
323 wrote: post #102

Earlier quoted context omitted.

> we used 2048 A100-80GB for a period of approximately 5 months Do we know how much total energy a human consumes from birth to 20 yo? Something like 2000 calories integrated over 20 years. How does it compare to the GPUs above? Wolfram Alpha: - human - 17 MW/h ((2000 calories per day) over 20 years in MWh) - GPUs - 3000 MW/h ((2048 * 400) W over 5 months in MWh) We still have the edge. LOL, I'm being downvoted, I wo…

Every human requires the same energy, 20+ years, and training. The trained computer model can be duplicated and used, requiring much less energy. None of this matters to me, though. The goal is to build better models. We can worry about the efficiency later.

exactly. we are speedrunning 200,000 years of intelligent life evolution here.

Re: LLaMA: A foundational, 65B-parameter large language model

#188

Post by golem14 »

iandanforth wrote: post #95

"we are publicly releasing LLaMA" "Access to the model will be granted on a case-by-case basis to academic researchers" They keep saying the word 'release' but I don't think they know what that word means. There are perfectly good words in the English language to describe this situation without abusing "release." They "will begin to grant access to a select few". Nothing about that releases the model, or their contro…

IDK:

"Mr. Burns: Smithers, release the hounds. "

Re: LLaMA: A foundational, 65B-parameter large language model

#189

Post by foobiekr »

mirker wrote: post #159
foobiekr wrote: post #156

Earlier quoted context omitted.

Poor GPU utilization even when available is the rule. Truly amazing. Staging of data is probably a huge part of it.

Is it failures or is this some backfill/budget scheduling while everyone is sleeping?

A lot of it appears to be non-streaming approaches to data distribution resulting in actual job behavior that looks a lot more like stage-process-clear batch jobs than what you'd want to hide the latency of data moves.

Re: LLaMA: A foundational, 65B-parameter large language model

#190

Post by foobiekr »

pavelstoev wrote: post #164
foobiekr wrote: post #156

Earlier quoted context omitted.

Poor GPU utilization even when available is the rule. Truly amazing. Staging of data is probably a huge part of it.

At CentML, we profiled GPU utilization on a larger AI/ML research institute cluster. 10% to 45% range, mostly in 10% utilization range. We then offered them software optimizers (which do not affect model accuracy) to get to the 90% utilization for GPUs

90% sustained utilization is quite amazing, and 10% is shockingly typical. I am a quite skeptical that this holds for training and very large data sets, of the sort where data placement comes into play, but if so, congratulations, and I hope things go well for you.
Post reply on HN