Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Llama 4 herd

ai.meta.com

Post reply on HN
Print view View original

441–450 of 695 posts

Re: The Llama 4 herd

#441

Post by dughnut »

paxys wrote: post #230
vessenes wrote: post #220

Earlier quoted context omitted.

Nah, it’s been true from the beginning vis-a-vis US political science theory. That is, if you deliver something like https://www.pewresearch.org/politics/quiz/political-typology... To models from GPT-3 on you get highly “liberal” per Pew’s designations. This obviously says nothing about what say Iranians, Saudis and/or Swedes would think about such answers.

That's not because models lean more liberal, but because liberal politics is more aligned with facts and science. Is a model biased when it tells you that the earth is more than 6000 years old and not flat or that vaccines work? Not everything needs a "neutral" answer.

[flagged]

Re: The Llama 4 herd

#442

Post by pavelstoev »

Model training observations from both Llama 3 and 4 papers:

Meta’s Llama 3 was trained on ~16k H100s, achieving ~380–430 TFLOPS per GPU in BF16 precision, translating to a solid 38 - 43% hardware efficiency [Meta, Llama 3].

For Llama 4 training, Meta doubled the compute, using ~32K H100s and switched to FP8 precision. Despite the precision gain, observed efficiency dropped to about 19.7%, with GPUs delivering ~390 TFLOPS out of a theoretical 1,979 FP8 TFLOPS [Meta, Llama 4].

I am not the one to critique, and rather, this is a recognition of the enormous complexity of operating GPUs at this scale. Training massive models across tens of thousands of GPUs stretches today’s AI infrastructure to its limit.

Besides accelerating inference workloads, advanced GPU optimizations can be integrated into training and fine-tuning pipelines. From various kernel optimization techniques (over 90) to increasing memory access efficiency and scaling up to cluster-wide resource coordination, efficiency can be maximized with some complex software.

References: [Meta, Llama 3] https://ai.meta.com/research/publications/the-llama-3-herd-o... [Meta, Llama 4] https://ai.meta.com/blog/llama-4-multimodal-intelligence/

Re: The Llama 4 herd

#443

Post by shostack »

hrpnk wrote: post #279

Available on Groq: https://groq.com/llama-4-now-live-on-groq-build-fast-at-the-... Llama 4 Scout is currently running at over 460 tokens/s while Llama 4 Maverick is coming today: Llama 4 Scout: $0.11 / M input tokens and $0.34 / M output tokens Llama 4 Maverick: $0.50 / M input tokens and $0.77 / M output tokens

Maverick looks comparable to Claude 3.7 and Gemini pro 2.5 in terms of quality but orders of magnitude cheaper. Am I missing something?

Is it possible to use Groq to run these new models in Cline or Roo?

Re: The Llama 4 herd

#444

Post by jumpCastle »

highfrequency wrote: post #407

Crazy that there are now five and a half companies that all have roughly state of the art LLMs. > We developed a new training technique which we refer to as MetaP that allows us to reliably set critical model hyper-parameters such as per-layer learning rates and initialization scales. We found that chosen hyper-parameters transfer well across different values of batch size, model width, depth, and training tokens. Th…

It's quite similar to muP

https://github.com/microsoft/mup

Re: The Llama 4 herd

#445

Post by dragonwriter »

jwr wrote: post #439

For those unfamiliar with the "active parameters" terminology, what would be the RAM requirements? E.g.can I run the smallest one on my Macbook Pro (M4 Max, 64GB) like I can run gemma3?

The RAM requirements for storing the parameters are set by the total, not active, parameters. Llama4 Scout is 109B model, so, at Int4 quantization, it will require ~55GB for the model. With 64GB, you could probably run it, but I would imagine not with a very large context size.

Re: The Llama 4 herd

#448

Post by dragonwriter »

reissbaker wrote: post #346
kristopolous wrote: post #303

Earlier quoted context omitted.

17B puts it beyond the reach of a 4090 ... anybody do 4 bit quant on it yet?

Oh, it'll never run on a 4090. 17B is the active parameter count, not the total param count (and "active" doesn't mean you can slice just those params out and put them on the GPU — which parameters are active constantly changes, even per-token. "Active" just means you get tokens faster than a dense model). It's 109B total parameters, so you'd need at least 54.5GB VRAM just for the weights alone. A Framework Desktop,…

Well, Scout should run on the rumored 96GB 4090, since it runs on a single 80GB H100. But, yeah, it'd have to be at sub-2bit quantization to run on a standard 24GB.

Re: The Llama 4 herd

#449

Post by dragonwriter »

voidspark wrote: post #224
accrual wrote: post #184

Earlier quoted context omitted.

Isn't there a vast quantity of relevant information in CJK languages? I remember reading some models even "think" in other languages where there might be more detail before outputting in the target language.

The model wasn't trained on those languages (yet). The only possible explanation is racism. The model is also racist against Russians and Icelanders.

> The model wasn't trained on those languages (yet).

It probably has been trained on them (it was trained on 40 trillion tokens covering 200 languages, they almost certainly didn't avoid CJK languages.

They only have been further fine-tuned on a set of 12 languages. (I wonder if that is the set the base Behemoth model both are distilled from had been trained on when they were distilled; Behemoth is apparently not completely finished, and perhaps there will be further revisions of the distilled models as it is.)

Re: The Llama 4 herd

#450

Post by phire »

phire wrote: post #409
jimmyl02 wrote: post #315

Earlier quoted context omitted.

the most unintuitive part is that from my understanding, individual tokens are routed to different experts. this is hard to comprehend with "experts" as that means two you can have different experts for two sequential tokens right? I think where MoE is misleading is that the experts aren't what we would call "experts" in the normal world but rather they are experts for a specific token . that concept feels difficult…

It's not even per token. The routing happens once per layer, with the same token bouncing between layers. It's more of a performance optimization than anything else, improving memory liquidity. Except it's not an optimization for running the model locally (where you only run a single query at a time, and it would be nice to keep the weights on the disk until they are relevant). It's a performance optimization for lar…

BTW, I'd love to see a large model designed from scratch for efficient local inference on low-memory devices.

While current MoE implementations are tuned for load-balancing over large pools of GPUs, there is nothing stopping you tuning them to only switch expert once or twice per token, and ideally keep the same weights across multiple tokens.

Well, nothing stopping you, but there is the question of if it will actually produce a worthwhile model.

Post reply on HN