Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama 3.1

llama.meta.com

Post reply on HN
Print view View original

71–80 of 279 posts

Re: Llama 3.1

#71

Post by michaelt »

TheAceOfHearts wrote: post #29

Does anyone know why they haven't released any 30B-ish param models? I was expecting that to happen with this release and have been disappointed once more. They also skipped doing a 30B-ish param model for llama2 despite claiming to have trained one.

I suspect 30B models are in a weird spot, too big for widespread home use, too small for cutting edge performance.

For home users 7B models (which can fit on an 8GB GPU) and 13B models (which can fit on a 16GB GPU) are in far more demand. If you're a researcher, you want a 70B model to get the best performance, and so your benchmarks are comparable to everyone else.

Re: Llama 3.1

#72

Post by gkk »

cchance wrote: post #59
lelag wrote: post #58

The 405b model is actually competitive against closed source frontier models. Quick comparison with GPT-4o: +----------------+-------+-------+ | Metric | GPT-4o| Llama | | | | 3.1 | | | | 405B | +----------------+-------+-------+ | MMLU | 88.7 | 88.6 | | GPQA | 53.6 | 51.1 | | MATH | 76.6 | 73.8 | | HumanEval | 90.2 | 89.0 | | MGSM | 90.5 | 91.6 | +----------------+-------+-------+

Super cool, though sadly 405b will be outside most personal usage without cloud providers which sorta defeats the purpose of opensource to some extent atleast sadly, because .. nvidia's rampup of consumer VRAM is glacial

If you think of open source as a protocol through which the ecosystem of companies loosely collaborate, then it's a big deal. E.g. Groq can work on inference without a complicated negotiations with Meta. Ditto for Huggingface, and smaller startups.

I agree with you on open source in the original, home tinkerer sense.

Re: Llama 3.1

#73

Post by sfblah »

Is there an actual open-source community around this in the spirit of other ones where people outside meta can somehow "contribute" to it? If I wanted to "work on" this somehow, what would I do?

Re: Llama 3.1

#74

Post by foundval »

You can chat with these new models at ultra-low latency at groq.com. 8B and 70B API access is available at console.groq.com. 405B API access for select customers only – GA and 3rd party speed benchmarks soon.

If you want to learn more, there is a writeup at https://wow.groq.com/now-available-on-groq-the-largest-and-m....

(disclaimer, I am a Groq employee)

Re: Llama 3.1

#75

Post by kingsleyopara »

The biggest win here has to be the context length increase to 128k from 8k tokens. Till now my understanding is there hasn't been any open models anywhere close to that.

Re: Llama 3.1

#76

Post by beeboobaa3 »

jermaustin1 wrote: post #42
beeboobaa3 wrote: post #16

Earlier quoted context omitted.

how is this even useful? no one can run it.

You don't use the 405B parameter model at home. I have a lot of luck with 8B and 13B models on a single 3090. You can quantize them down (is that the term) which lowers precision and memory use, but still very usable... most of the time. If you are running a commercial service that uses AI, you buy a few dozen A100s, spend a half million, and you are good for a while. If you are running a commercial inferencing servi…

I can't expect all my users to have 3090s and if we're talking about spending millions there are better things to invest in than a stack of GPUs that will be obsolete in a year or three.

Re: Llama 3.1

#77

Post by Teknomancer »

lolinder wrote: post #61
jcmp wrote: post #36

"Meta AI isn't available yet in your country" Hi from europe :/

Competition is a funny thing—it doesn't just apply to companies competing for customers, it also applies to governments competing for companies to make products available to their citizens. Turns out that if you make compliance with your laws onerous enough they can actually just choose to opt out of your country altogether, or at a minimum delay release in your country until they can check all your boxes. The only s…

Be careful what you wish for.

A Gibsonesque global Turing Police is a sure sign of Dystopia.

Re: Llama 3.1

#78

Post by wuschel »

kingsleyopara wrote: post #68
cchance wrote: post #59

Earlier quoted context omitted.

Super cool, though sadly 405b will be outside most personal usage without cloud providers which sorta defeats the purpose of opensource to some extent atleast sadly, because .. nvidia's rampup of consumer VRAM is glacial

You might be able to get away with running a heavily quantizied 405b model using CPU inference at a blistering fast token every 5 seconds on a 7950x.

OK, I am curious now: What kind of hardware would I need to run such a model for a couple of users with decent performance?

Where could I get a mapping of token / time vs hardware?

Re: Llama 3.1

#80

Post by loudmax »

denz88 wrote: post #23

I'm glad to see the nice incremental gains on the benchmarks for the 8B and 70B models as well.

Some of those benchmarks show quite significant gains. Going from Llama-3 to Llama-3.1, MMLU scores for 8B are up from 65.3 to 73.0, and 70B are up from 80.9 to 86.0. These scores should always be taken with a grain of salt, but this is encouraging.

405B is hopelessly out of reach for running in a homelab without spending thousands of dollars. For most people wanting to try out the 405B model, the best option is to rent compute from a datacenter. Looking forward to seeing what it can accomplish.

Post reply on HN