Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta Llama 3

llama.meta.com

Post reply on HN
Print view View original

201–210 of 965 posts

Re: Meta Llama 3

#201

Post by tmikaeld »

YetAnotherNick wrote: post #127
sp332 wrote: post #112

Earlier quoted context omitted.

The bottom of https://ai.meta.com/blog/meta-llama-3/ has in-progress results for the 400B model as well. Looks like it's not quite there yet. Llama 3 400B Base / Instruct MMLU 84.8 86.1 GPQA - 48.0 MATH - 57.8 HumanEval - 84.1 DROP 83.5 -

For the still training 400B: Llama 3 GPT 4(Published) BBH 85.3 83.1 MMLU 86.1 86.4 DROP 83.5 80.9 GSM8K 94.1 92.0 MATH 57.8 52.9 HumEv 84.1 74.4 Although it should be noted that the API numbers were generally better than published numbers for GPT4. [1]: https://deepmind.google/technologies/gemini/

Hm, how much VRAM would this take to run?

Re: Meta Llama 3

#203

Post by Davidzheng »

cjbprime wrote: post #120
modeless wrote: post #89

I was curious how the numbers compare to GPT-4 in the paid ChatGPT Plus, since they don't compare directly themselves. Llama 3 8B Llama 3 70B GPT-4 MMLU 68.4 82.0 86.5 GPQA 34.2 39.5 49.1 MATH 30.0 50.4 72.2 HumanEval 62.2 81.7 87.6 DROP 58.4 79.7 85.4 Note that the free version of ChatGPT that most people use is based on GPT-3.5 which is much worse than GPT-4. I haven't found comprehensive eval numbers for the lates…

Has anyone prepared a comparison to Mixtral 8x22B? (Life sure moves fast.)

it's in the official post the comparison with Mixtral 8x22B

Re: Meta Llama 3

#204

Post by simonw »

jph00 wrote: post #160
observationist wrote: post #67

https://github.com/meta-llama/llama3/blob/main/LICENSE Llama is not open source. It's corporate freeware with some generous allowances. Open source licenses are a well defined thing. Meta marketing saying otherwise doesn't mean they get to usurp the meaning of a well understood and commonly used understanding of the term "open source." https://opensource.org/license Nothing about Meta's license is open source. It's a…

> " Nothing about Meta's license is open source. It's a carefully constructed legal agreement intended to prevent any meaningful encroachment by anyone, ever, into any potential Meta profit, and to disavow liability to prevent reputational harm in the case of someone using their freeware for something embarrassing. " You seem to be making claims that have little connection to the actual license. The license states yo…

Those additional restrictions mean it's not an open source license by the OSI definition, which matters if you care about words sometimes having unambiguous meanings.

I call models like this "openly licensed" but not "open source licensed".

Re: Meta Llama 3

#205

Post by zo1 »

Maybe a side-note or off-topic. But am I the only one that's shocked/confused why these giant tech companies have huge models, so much compute to run them on, and they still can't get certain basic things right. Something as simple, for Facebook, as detecting a fake profile that's super-obvious to any human that's been on the net for any appreciable amount of time.

Re: Meta Llama 3

#206

Post by Simon321 »

anvuong wrote: post #198
gliched_robot wrote: post #105

Earlier quoted context omitted.

Wild considering, GPT-4 is 1.8T.

I actually can't wrap my head around this number, even though I have been working on and off with deep learning for a few years. The biggest models we've ever deployed on production still have less than 1B parameters, and the latency is already pretty hard to manage during rush hours. I have no idea how they deploy (multiple?) 1.8T models that serve tens of millions of users a day.

It's a mixture of experts model. Only a small part of those parameters are active at any given time. I believe it's 16x110B

Re: Meta Llama 3

#208

Post by cjbprime »

Davidzheng wrote: post #203
cjbprime wrote: post #120

Earlier quoted context omitted.

Has anyone prepared a comparison to Mixtral 8x22B? (Life sure moves fast.)

it's in the official post the comparison with Mixtral 8x22B

Where? I only see comparisons to Mistral 7B and Mistral Medium, which are totally different models.

Re: Meta Llama 3

#210

Post by cjbprime »

schleck8 wrote: post #196
cjbprime wrote: post #172

Earlier quoted context omitted.

(You can't compare parameter count with a mixture of experts model, which is what the 1.8T rumor says that GPT-4 is.)

You absolutely can since it has a size advantage either way. MoE means the expert model performs better BECAUSE of the overall model size.

Fair enough, although it means we don't know whether a 1.8T MoE GPT-4 will have a "size advantage" over Llama 3 400B.
Post reply on HN