Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Llama 4 herd

ai.meta.com

Post reply on HN
Print view View original

541–550 of 695 posts

Re: The Llama 4 herd

#541

Post by jamesblonde »

ramshanker wrote: post #320
laborcontract wrote: post #29

General overview below, as the pages don't seem to be working well Llama 4 Models: - Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each. - They are natively multimodal: text + image input, text-only output. - Key achievements include industry-leading context lengths, strong coding/reasoning performance, and improved multilingual capabilities. - Knowledge cuto…

I have a gut feeling, next in line will be 2 or more level of MoE. Further reducing the memory bandwidth and compute requirements. So top level MoE router decides which sub MoE to route.

The solution to all problems in computer science is add a new level of indirection (or abstraction).

Re: The Llama 4 herd

#542

Post by rfoo »

pavelstoev wrote: post #442

Model training observations from both Llama 3 and 4 papers: Meta’s Llama 3 was trained on ~16k H100s, achieving ~380–430 TFLOPS per GPU in BF16 precision, translating to a solid 38 - 43% hardware efficiency [Meta, Llama 3]. For Llama 4 training, Meta doubled the compute, using ~32K H100s and switched to FP8 precision. Despite the precision gain, observed efficiency dropped to about 19.7%, with GPUs delivering ~390 TF…

That's about the same number for DeepSeek-V3. If you count in fp8 MFU is about 20%. MoEs are hard.

That could also be why they did fp8. If we use theoretical performance of bf16 as baseline (I know this makes few sense, but for compare with previous trainings it's convenient) the about 40% MFU, not too bad.

IOW, MoE kills training MFU and they had to do fp8 to make it not looking funny. Both DeepSeek and Meta GenAI.

Re: The Llama 4 herd

#543

Post by tucnak »

EGreg wrote: post #423

Can we somehow load these inside node.js? What is the easiest way to load them remotely? Huggingface Spaces? Google AI Studio? I am teaching a course on AI to non-technical students, and I wanted the students to have a minimal setup: which in this case would be: 1) Browser with JS (simple folder of HTML, CSS) and Tensorflow.js that can run models like Blazeface for face recognition, eye tracking etc. (available since…

You're not qualified to teach a course on AI if you're asking questions like that. Please don't scam students, they're naive and don't know better and you're predating.

Re: The Llama 4 herd

#545

Post by rfoo »

voxgen wrote: post #523
bastawhiz wrote: post #440

I don't really understand how Scout and Maverick are distillations of Behemoth if Behemoth is still training. Maybe I missed or misunderstood this in the post? Did they distill the in-progress Behemoth and the result was good enough for models of those sizes for them to consider releasing it? Or is Behemoth just going through post-training that takes longer than post-training the distilled versions? Sorry if this is…

> Or is Behemoth just going through post-training that takes longer than post-training the distilled versions? This is the likely main explanation. RL fine-tuning repeatedly switches between inference to generate and score responses, and training on those responses. In inference mode they can parallelize across responses, but each response is still generated one token at a time. Likely 5+ minutes per iteration if the…

> see o3-mini's release date & pricing vs R1's

This backfires though, if OAI released o3-mini before DeepSeek-R1, R1 would be a lot less impactful.

Re: The Llama 4 herd

#546

Post by Jensson »

singularity2001 wrote: post #318
ezst wrote: post #201

Earlier quoted context omitted.

> LeCun consistently states LLMs are not the right path yet LLMs are still the main flagship model they are shipping. I really don't see what's controversial about this. If that's to mean that LLMs are inherently flawed/limited and just represent a local maxima in the overall journey towards developing better AI techniques, I thought that was pretty universal understanding by now.

local maximum that keeps rising and no bar/boundary in sight

Even a narrow AI can get better with no bar in sight, but it will never get to AGI. That is the argument here.

Re: The Llama 4 herd

#547

Post by Jensson »

falcor84 wrote: post #344
sroussey wrote: post #260

Earlier quoted context omitted.

He thinks LLMs are a local maxima, not the ultimate one. Doesn't mean that a local maxima can't be useful!

If that's what he said, I'd be happy, but I was more concerned about this: > His belief is so strong that, at a conference last year, he advised young developers, "Don't work on LLMs. [These models are] in the hands of large companies, there's nothing you can bring to the table. You should work on next-gen AI systems that lift the limitations of LLMs." It's ok to say that we'll need to scale other mountains, but I'm…

> but I'm concerned that the "Don't" there would push people away from the engineering that would give them the relevant inspiration.

You have way more yay-sayers than nay-sayers, there is never a risk that we don't go hard enough into the current trends, there is however a risk that we go too hard into it and ignore other paths.

Re: The Llama 4 herd

#549

Post by Jensson »

vmladenov wrote: post #480
huijzer wrote: post #475

Earlier quoted context omitted.

> Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation. Doesn’t explain why roughly half of American voters were not “leaning left” during the election. EDIT: 07:29 UTC changed "Americans" to "American voters".

It is not and has never been half. 2024 voter turnout was 64%

> It is not and has never been half. 2024 voter turnout was 64%

He said half of voters, those who didn't vote aren't voters.

Re: The Llama 4 herd

#550

Post by reissbaker »

lostmsu wrote: post #493
reissbaker wrote: post #346

Earlier quoted context omitted.

Oh, it'll never run on a 4090. 17B is the active parameter count, not the total param count (and "active" doesn't mean you can slice just those params out and put them on the GPU — which parameters are active constantly changes, even per-token. "Active" just means you get tokens faster than a dense model). It's 109B total parameters, so you'd need at least 54.5GB VRAM just for the weights alone. A Framework Desktop,…

Sounds runnable on 2x5090 presumably for $4k if back in stock.

True! A Framework Desktop or mid-tier Mac Studio would also work and would be cheaper — and you could even run Scout at FP8. A maxed-out Mac Studio could even handle Maverick at FP8, albeit at pretty high cost ($10k).

It's still runnable locally. Just not on a 4090.

Post reply on HN