Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLaMA: A foundational, 65B-parameter large language model

ai.facebook.com

Post reply on HN
Print view View original

111–120 of 209 posts

Re: LLaMA: A foundational, 65B-parameter large language model

#111

Post by isoprophlex »

323 wrote: post #102
saurabh20n wrote: post #35

Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…

> we used 2048 A100-80GB for a period of approximately 5 months Do we know how much total energy a human consumes from birth to 20 yo? Something like 2000 calories integrated over 20 years. How does it compare to the GPUs above? Wolfram Alpha: - human - 17 MW/h ((2000 calories per day) over 20 years in MWh) - GPUs - 3000 MW/h ((2048 * 400) W over 5 months in MWh) We still have the edge. LOL, I'm being downvoted, I wo…

You mean MWh maybe, not MW/h? (which is what, J/s^2 in SI... "Power rate".)

Re: LLaMA: A foundational, 65B-parameter large language model

#112

Post by robbiep »

323 wrote: post #102
saurabh20n wrote: post #35

Quick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-ef... : * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a ra…

> we used 2048 A100-80GB for a period of approximately 5 months Do we know how much total energy a human consumes from birth to 20 yo? Something like 2000 calories integrated over 20 years. How does it compare to the GPUs above? Wolfram Alpha: - human - 17 MW/h ((2000 calories per day) over 20 years in MWh) - GPUs - 3000 MW/h ((2048 * 400) W over 5 months in MWh) We still have the edge. LOL, I'm being downvoted, I wo…

It’s because your human math for power output is so far off it’s hard to know where to start to point you in the right direction

Re: LLaMA: A foundational, 65B-parameter large language model

#113

Post by pps »

stevenhuang wrote: post #75
zozbot234 wrote: post #70

Earlier quoted context omitted.

Yes but fine tuning for RL is not expected to be hard. You're essentially limited by how much human feedback is available, so it's very different from training the foundational model on random bulk data.

Of course, but even still the level of scale, clean data, and human supervision needed may be significant. It's reported OpenAI used an army of humans to generate question answer prompts and rate the model output. They kept the details closely guarded and only hinted at how they did RLHF and transitioned the architecture to self supervised learning.

I'm stupid in this area and have only a very broad understanding of "AI". But isn't this already happening in this project? https://github.com/LAION-AI/Open-Assistant

Re: LLaMA: A foundational, 65B-parameter large language model

#115

Post by 323 »

robbiep wrote: post #112
323 wrote: post #102

Earlier quoted context omitted.

> we used 2048 A100-80GB for a period of approximately 5 months Do we know how much total energy a human consumes from birth to 20 yo? Something like 2000 calories integrated over 20 years. How does it compare to the GPUs above? Wolfram Alpha: - human - 17 MW/h ((2000 calories per day) over 20 years in MWh) - GPUs - 3000 MW/h ((2048 * 400) W over 5 months in MWh) We still have the edge. LOL, I'm being downvoted, I wo…

It’s because your human math for power output is so far off it’s hard to know where to start to point you in the right direction

Please do tell. Or better provide your estimation. I just took raw calorie intake, no heating/transportation/lighting/computer usage/....

Re: LLaMA: A foundational, 65B-parameter large language model

#116

Post by 323 »

isoprophlex wrote: post #111
323 wrote: post #102

Earlier quoted context omitted.

> we used 2048 A100-80GB for a period of approximately 5 months Do we know how much total energy a human consumes from birth to 20 yo? Something like 2000 calories integrated over 20 years. How does it compare to the GPUs above? Wolfram Alpha: - human - 17 MW/h ((2000 calories per day) over 20 years in MWh) - GPUs - 3000 MW/h ((2048 * 400) W over 5 months in MWh) We still have the edge. LOL, I'm being downvoted, I wo…

You mean MWh maybe, not MW/h? (which is what, J/s^2 in SI... "Power rate".)

Right, I used the correct MWh in Wolfram, but for some reason wrote MW/h, I think it was written like that a long time ago on electricity bills.

Re: LLaMA: A foundational, 65B-parameter large language model

#117

Post by blablablub »

CuriouslyC wrote: post #80
blablablub wrote: post #60

So what hardware do we need to run this model?

7 billion can run on 16+ gb GPUs as fp16, 14 billion can be run on 16+ gb if quantized to int8. 14G @ fp16 and 30G at int8 will require one of the 48 gb cards (less, but hardware mostly goes 24 -> 48).

thanks

Re: LLaMA: A foundational, 65B-parameter large language model

#118

Post by UncleEntity »

rsrsrs86 wrote: post #101

Funny how the release has a note that the models clearly have ethical problemas that must be addressed, and still the company chooses to publish it.

Ethical problems that they don’t purge it of Wrong Think?

Why people care that a dodgy AI can be made to say politically incorrect things is beyond me. Don’t use its outputs for anything important and everything will be alright.

Re: LLaMA: A foundational, 65B-parameter large language model

#119

Post by dentalperson »

bitRAKE wrote: post #110
dentalperson wrote: post #97

How does token->embedding lookup work with 1.4T BPE tokens? Since there are more tokens than the 65B parameters it must be doing some sort of interesting thing based on the merge operations. Is it different from what other GPT models with ~100k tokens are doing? At inference, how many of those tokens are used? (they mention most tokens are used only once during training, so the must be very long sequences.)

The 1.4T tokens are what the model was trained on, and not the token range of the embedding.

Ah, that makes more sense, thank you. Since this was mentioned in the tokenizer section and the number of unique tokens wasn't mentioned I misunderstood.
Post reply on HN