Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLaMA: A foundational, 65B-parameter large language model

ai.facebook.com

Post reply on HN
Print view View original

141–150 of 209 posts

Re: LLaMA: A foundational, 65B-parameter large language model

#141

Post by WASDx »

323 wrote: post #115
robbiep wrote: post #112

Earlier quoted context omitted.

It’s because your human math for power output is so far off it’s hard to know where to start to point you in the right direction

Please do tell. Or better provide your estimation. I just took raw calorie intake, no heating/transportation/lighting/computer usage/....

A thing to keep in mind is that 1 MWh of raw calories takes much more than 1 MWh to produce (fuel for tractors, inefficiency of meat etc). The GPU energy is also easier to make renewable.

I did an extremely rough calculation recently that the training of GPT-3 is comparable to one transatlantic flight (all passengers combined) in terms of emissions, very depending on the energy mix of course.

Re: LLaMA: A foundational, 65B-parameter large language model

#142

Post by ntonozzi »

simonw wrote: post #74
zozbot234 wrote: post #70

Earlier quoted context omitted.

Yes but fine tuning for RL is not expected to be hard. You're essentially limited by how much human feedback is available, so it's very different from training the foundational model on random bulk data.

At this point I fully expect that someone will release a usable RLHF-fine-tuned language model that can run on consumer hardware, based on the methodology used for LLaMA (and other similar papers e.g. https://github.com/FMInference/FlexGen ), at some point in the next 6-24 months.

How are they different from models that already exist like FLAN-T5?

Re: LLaMA: A foundational, 65B-parameter large language model

#143

Post by warkdarrior »

Mizza wrote: post #139
theRealMe wrote: post #104

Earlier quoted context omitted.

Do you mean the same code that has its own Wikipedia page where the exact code is written, comments included, and has probably been copy pasted into 100’s of other projects? https://en.m.wikipedia.org/wiki/Fast_inverse_square_root

You mean this code? https://archive.softwareheritage.org/browse/content/sha1_git... Do you see that notice at the top of the file? It says: == This file is part of Quake III Arena source code. Quake III Arena source code is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your…

How is it made proprietary? The Quake III Arena is no more proprietary now then if it were stored on GitHub proprietary web servers. Copilot is just a fancy code index, that sometimes returns the original code and other times it gives you a modified copy.

Re: LLaMA: A foundational, 65B-parameter large language model

#144

Post by yawaramin »

return_to_monke wrote: post #120
carlsborg wrote: post #106

"The code is licensed under the GPLv3, which permits commercial use." Yann on Twitter

The code. They didn't release a model tho, so unless you have a lot of money or get granted access to it, not for you.

LeCun on Twitter: https://twitter.com/ylecun/status/1629189925089296386

    > LLaMA is a new open-source, high-performance large language model from Meta AI - FAIR.
    >
    > Meta is committed to open research and releases all the models the research community under a GPL v3 license.

Re: LLaMA: A foundational, 65B-parameter large language model

#145

Post by Mizza »

warkdarrior wrote: post #143
Mizza wrote: post #139

Earlier quoted context omitted.

You mean this code? https://archive.softwareheritage.org/browse/content/sha1_git... Do you see that notice at the top of the file? It says: == This file is part of Quake III Arena source code. Quake III Arena source code is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your…

How is it made proprietary? The Quake III Arena is no more proprietary now then if it were stored on GitHub proprietary web servers. Copilot is just a fancy code index, that sometimes returns the original code and other times it gives you a modified copy.

Because as you say, it provides original or modified code but doesn't provide provenance or license information. It's copyright laundering. After decades of fighting the community in the courts over shit like this, Microsoft just turns around and says well, it's okay when we do it? Foh.

Re: LLaMA: A foundational, 65B-parameter large language model

#146

Post by woeirua »

saurabh20n wrote: post #35

Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…

These cost estimates really make me question OpenAI's valuation.

Also, they kind of prove to me that most companies are totally incapable of making the investments necessary to get much out of this type of AI.

Re: LLaMA: A foundational, 65B-parameter large language model

#148

Post by charcircuit »

Mizza wrote: post #139
theRealMe wrote: post #104

Earlier quoted context omitted.

Do you mean the same code that has its own Wikipedia page where the exact code is written, comments included, and has probably been copy pasted into 100’s of other projects? https://en.m.wikipedia.org/wiki/Fast_inverse_square_root

You mean this code? https://archive.softwareheritage.org/browse/content/sha1_git... Do you see that notice at the top of the file? It says: == This file is part of Quake III Arena source code. Quake III Arena source code is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your…

That code didn't originate from quake

Re: LLaMA: A foundational, 65B-parameter large language model

#149

Post by akomtu »

saurabh20n wrote: post #35

Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…

By "parameters" they probably mean float32s, and 65B of those is 0.25 TB of data - more than enough to memorize a 1.5T sequence of "tokens" (3 letter triplets?). This begs the question: are these models better than a fuzzy hash table?

Re: LLaMA: A foundational, 65B-parameter large language model

#150

Post by Teever »

323 wrote: post #115
robbiep wrote: post #112

Earlier quoted context omitted.

It’s because your human math for power output is so far off it’s hard to know where to start to point you in the right direction

Please do tell. Or better provide your estimation. I just took raw calorie intake, no heating/transportation/lighting/computer usage/....

That's the entire problem. There's so much more energy that goes into a modern human beyond just what they eat. Beyond physical items you've listed like clothing there's also education and healthcare. Those two institutions are critical in making a modern human and they both have their own dependency chains of physical resource, energy, and the input of even more humans.
Post reply on HN