Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.1

llama.meta.com

Post reply on HN
Print view View original

201–210 of 279 posts

Re: Llama 3.1

#201

Post by zargon »

coconut08 wrote: post #165
microtonal wrote: post #132

Earlier quoted context omitted.

You can run the 4-bit GPTQ/AWQ quantized Llama 405B somewhat reasonably on 4x H100 or A100. You will be somewhat limited in how many tokens you can have in flight between requests and you cannot create CUDA graphs for larger batch sizes. You can run 405B well on 8x H100 and A100, either with the mixed BFloat16/FP8 checkpoint that Meta provided or GPTQ/AWQ-quantized models. Note though that the A100 does not have nati…

how much vram do you need for 4-bit llama 405?

405 billion * 4 bits = approximately 200 GB. Plus extra for the amount of context you want.

Re: Llama 3.1

#202

Post by CGamesPlay »

The LMSys Overall leaderboard https://chat.lmsys.org/?leaderboard> can tell us a bit more about how these models will perform in real life, rather than in a benchmark context. By comparing the ELO score against the MMLU benchmark scores, we can see models which outperform / underperform based on their benchmark scores relative to other models. A low score here indicates that the model is more optimized for the benchmark, while a higher score indicates it's more optimized for real-world examples. Using that, we can make some inferences about the training data used, and then extrapolate how future models might perform. Here's a chart: https://docs.getgrist.com/gV2DtvizWtG7/LLMs/p/5?embed=true>

Examples: OpenAI's GPT 4o-mini is second only to 4o on LMSys Overall, but is 6.7 points behind 4o on MMLU. It's "punching above its weight" in real-world contexts. The Gemma series (9B and 27B) are similar, both beating the mean in terms of ELO per MMLU point. Microsoft's Phi series are all below the mean, meaning they have strong MMLU scores but aren't preferred in real-world contexts.

Llama 3 8B previously did substantially better than the mean on LMSys Overall, so hopefully Llama 3.1 8B will be even better! The 70B variant was interestingly right on the mean. Hopefully the 430B variant won't fall below!

Re: Llama 3.1

#203

Post by Davidzheng »

cubefox wrote: post #126
primaprashant wrote: post #50

I have found Claude 3.5 Sonnet really good for coding tasks along with the artifacts feature and seems like it's still the king on the coding benchmarks

I have found it to be better than GPT-4o at math too, despite the latter being better at several math benchmarks.

I personally disagree. But i haven't used sonnet that much

Re: Llama 3.1

#204

Post by Davidzheng »

sroussey wrote: post #143
loudmax wrote: post #80

Earlier quoted context omitted.

Some of those benchmarks show quite significant gains. Going from Llama-3 to Llama-3.1, MMLU scores for 8B are up from 65.3 to 73.0, and 70B are up from 80.9 to 86.0. These scores should always be taken with a grain of salt, but this is encouraging. 405B is hopelessly out of reach for running in a homelab without spending thousands of dollars. For most people wanting to try out the 405B model, the best option is to r…

How much can you quantize that down to run on a Mac Studio with 192GB? Is it possible? Feels like it would have to be 2bit…

Less than 2bit i think. There's this IQ2 quant that fits

Re: Llama 3.1

#205

Post by gantrol »

nomel wrote: post #154
bamboozled wrote: post #127

Earlier quoted context omitted.

This nodel is not “open source”, free to use maybe.

I really wish people would use "open weights" rather than "open source". It's precise and obvious, and leaves an accurate descriptor for actual "open source" models, where the source and methods that that generate the artifact , that is the weights, is open.

Training a model is like automatic programming, and the key of it is having a well-organized dataset.

If some "opensource" model just have the model and training methods but no dataset, it’s like some repo which released an executable file with a detailed design doc. Where is the source code? Do it yourself, please.

NOTE: I understand the difficulty of open-sourcing datasets. I'm just saying that the term "opensource" is getting diluted.

Re: Llama 3.1

#206

Post by lolinder »

nomel wrote: post #154
bamboozled wrote: post #127

Earlier quoted context omitted.

This nodel is not “open source”, free to use maybe.

I really wish people would use "open weights" rather than "open source". It's precise and obvious, and leaves an accurate descriptor for actual "open source" models, where the source and methods that that generate the artifact , that is the weights, is open.

It's not precise. People who want to use "open weights" instead of "open source" are focusing on the wrong thing.

The weights are, for all practical purposes, source code in their own right. The GPL defines "source code" as "the preferred form of the work for making modifications to it". Almost no one would be capable of reproducing them even if given the source + data. At the same time, the weights are exactly what you need for the one type of modification that's within reach of most people: fine-tuning. That they didn't release the surrounding code that produced this "source" isn't that much different than a company releasing a library but not their whole software stack.

I'd argue that "source" vs "weights" is a dangerous distraction from the far more insidious word in "open source" when used to refer to the Llama license: "open".

The Llama 3.1 license [0] specifically forbids its use by very large organizations, by militaries, and by nuclear industries. It also contains a long list of forbidden use cases. This specific list sounds very reasonable to me on its face, but having a list of specific groups of people or fields of endeavor who are banned from participating runs counter to the spirit of open source and opens up the possibility that new "open" licenses come out with different lists of forbidden uses that sound less reasonable.

To be clear, I'm totally fine with them having those terms in their license, but I'm uncomfortable with setting the precedent of embracing the word "open" for it.

Llama is "nearly-open source". That's good enough for me to be able to use it for what I want, but the word "open" is the one that should be called out. "Source" is fine.

[0] https://github.com/meta-llama/llama-models/blob/main/models/...

Re: Llama 3.1

#207

Post by TacticalCoder »

mi_lk wrote: post #191
lelag wrote: post #58

The 405b model is actually competitive against closed source frontier models. Quick comparison with GPT-4o: +----------------+-------+-------+ | Metric | GPT-4o| Llama | | | | 3.1 | | | | 405B | +----------------+-------+-------+ | MMLU | 88.7 | 88.6 | | GPQA | 53.6 | 51.1 | | MATH | 76.6 | 73.8 | | HumanEval | 90.2 | 89.0 | | MGSM | 90.5 | 91.6 | +----------------+-------+-------+

How do you draw/generate such ascii table?

Don't know about OP but I generate such tables using Emacs.

Re: Llama 3.1

#208

Post by sujay1844 »

CGamesPlay wrote: post #202

The LMSys Overall leaderboard https://chat.lmsys.org/?leaderboard > can tell us a bit more about how these models will perform in real life, rather than in a benchmark context. By comparing the ELO score against the MMLU benchmark scores, we can see models which outperform / underperform based on their benchmark scores relative to other models. A low score here indicates that the model is more optimized for the bench…

These days, lmsys elo is the only thing I trust. The other benchmark scores mean nothing to me at this point

Re: Llama 3.1

#209

Post by Lockal »

CGamesPlay wrote: post #202

The LMSys Overall leaderboard https://chat.lmsys.org/?leaderboard > can tell us a bit more about how these models will perform in real life, rather than in a benchmark context. By comparing the ELO score against the MMLU benchmark scores, we can see models which outperform / underperform based on their benchmark scores relative to other models. A low score here indicates that the model is more optimized for the bench…

Something is broken with "meta-llama-3.1-405b-instruct-sp" and "meta-llama-3.1-70b-instruct-sp" there, after few sentences both models switch to infinite random like: "Rotterdam计算 dining counselor/__asan jo Nas было /well-rest esse moltet Grants SL и Four VIHu-turn greatest Morenh elementary(((( parts referralswhich IMOаш ...".

Don't expect any meaningful score there before they wipe results.

Re: Llama 3.1

#210

Post by llm_trw »

nomel wrote: post #154
bamboozled wrote: post #127

Earlier quoted context omitted.

This nodel is not “open source”, free to use maybe.

I really wish people would use "open weights" rather than "open source". It's precise and obvious, and leaves an accurate descriptor for actual "open source" models, where the source and methods that that generate the artifact , that is the weights, is open.

> where the source and methods that that generate the artifact, that is the weights, is open.

When you require the same thing in software, namely the whole stack to run the software in question to be open source, we don't call the license open source.

Post reply on HN