Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Llama 4 herd

ai.meta.com

Post reply on HN
Print view View original

481–490 of 695 posts

Re: The Llama 4 herd

#481

Post by az226 »

silverlake wrote: post #465
pavelstoev wrote: post #442

Model training observations from both Llama 3 and 4 papers: Meta’s Llama 3 was trained on ~16k H100s, achieving ~380–430 TFLOPS per GPU in BF16 precision, translating to a solid 38 - 43% hardware efficiency [Meta, Llama 3]. For Llama 4 training, Meta doubled the compute, using ~32K H100s and switched to FP8 precision. Despite the precision gain, observed efficiency dropped to about 19.7%, with GPUs delivering ~390 TF…

I think BF16 and FP16 are 1979 TFPOPs, but FP8 is 2x faster at 3958 TFLOPs. So only 10% efficiency, down from 20%. That’s not good.

That’s with sparsity. So it’s 29% down from 40%.

Re: The Llama 4 herd

#482

Post by az226 »

cavisne wrote: post #467
pavelstoev wrote: post #442

Model training observations from both Llama 3 and 4 papers: Meta’s Llama 3 was trained on ~16k H100s, achieving ~380–430 TFLOPS per GPU in BF16 precision, translating to a solid 38 - 43% hardware efficiency [Meta, Llama 3]. For Llama 4 training, Meta doubled the compute, using ~32K H100s and switched to FP8 precision. Despite the precision gain, observed efficiency dropped to about 19.7%, with GPUs delivering ~390 TF…

The H100 theoretical flops number is just marketing, as it relies on sparsity that LLMs don’t use

And the practical flops always end up lower. As an example a V100 has 125 according to spec, but the ideal case is more like 100 and non-ideal like 60.

Re: The Llama 4 herd

#483

Post by segmondy »

tucnak wrote: post #367
hydroreadsstuff wrote: post #243

This means GPUs are dead for local enthusiast AI. And SoCs with big RAM are in. Because 17B active parameters should reach enough performance on 256bit LPDDR5x.

This has been the case for a while now. 3090 hoarders were always just doing it for street cred or whatever, no way these guys are computing anything of actual value. Tenstorrent is on fire, though. For small businesses this is what matters. If 10M context is not a scam, I think we'll see SmartNIC adoption real soon. I would literally long AMD now because their Xilinx people are probably going to own the space real s…

I was going to buy my first GPU for DL in 2018, but crypto didn't make it easy. I waited for the prices to fall, but demand kept up, then covid happened, then LLM happened and used GPUs now cost more than their original new prices. ... as we can see by the paper launch from Nvidia, lack of competition, and the prices of the 5000 series easily 50% above original MSRP. Demand is still here, now we have tarrif... Folks got reasons to collect, hoard or do whatever you think they are doing, even if it's just for street cred.

Re: The Llama 4 herd

#484

Post by az226 »

mkl wrote: post #372
simonw wrote: post #343

This thread so far (at 310 comments) summarized by Llama 4 Maverick: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-maverick -o max_tokens 20000 Output: https://gist.github.com/simonw/016ea0fd83fc499f046a94827f9b4... And with Scout I got complete junk output for some reason: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-scout -o max_tokens 20000 Junk output here: https://gist.github.com/simonw/d01c…

That Gemini 2.5 one is impressive. I found it interesting that the blog post didn't mention Gemini 2.5 at all. Okay, it was released pretty recently, but 10 days seems like enough time to run the benchmarks, so maybe the results make Llama 4 look worse?

Meta will most likely compare against it when they release the upcoming Llama 4 reasoning model.

Re: The Llama 4 herd

#485

Post by mdp2021 »

knowriju wrote: post #468
mdp2021 wrote: post #328

Earlier quoted context omitted.

> If I ask an LLM how old the Earth is, and it replies ~4.5 billion years old It will have to reply "According to Clair Patterson and further research, the Earth is ~4.5 billion years old". Or some other form that points to the source somewhere.

Pretty sad that the rest of the world needs to pay for the extra tokens because of non-scientific american bias. This is also possibly a big point why countries/regions want sovereign LLMs which will propagate regional biases only.

It's not a matter of «extra tokens»: it's that the fact, the "summary after the protocols", is what I wrote. It is the correct answer. It's what you should expect from a lucid speaker.

Re: The Llama 4 herd

#486

Post by AnthonyMouse »

paxys wrote: post #230
vessenes wrote: post #220

Earlier quoted context omitted.

Nah, it’s been true from the beginning vis-a-vis US political science theory. That is, if you deliver something like https://www.pewresearch.org/politics/quiz/political-typology... To models from GPT-3 on you get highly “liberal” per Pew’s designations. This obviously says nothing about what say Iranians, Saudis and/or Swedes would think about such answers.

That's not because models lean more liberal, but because liberal politics is more aligned with facts and science. Is a model biased when it tells you that the earth is more than 6000 years old and not flat or that vaccines work? Not everything needs a "neutral" answer.

> Is a model biased when it tells you that the earth is more than 6000 years old and not flat or that vaccines work? Not everything needs a "neutral" answer.

That's the motte and bailey.

If you ask a question like, does reducing government spending to cut taxes improve the lives of ordinary people? That isn't a science question about CO2 levels or established biology. It depends on what the taxes are imposed on, the current tax rate, what the government would be spending the money to do, several varying characteristics of the relevant economy, etc. It doesn't have the same answer in all circumstances.

But in politics it does, which is that the right says yes and the left says no. Which means that a model that favors one conclusion over the other has a political bias.

Re: The Llama 4 herd

#487

Post by tomjen3 »

vessenes wrote: post #289
InvOfSmallC wrote: post #266

Earlier quoted context omitted.

For a super ignorant person: Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each Those experts are LLM trained on specific tasks or what?

This was an idea that sounded somewhat silly until it was shown it worked. The idea is that you encourage through training a bunch of “experts” to diversify and “get good” at different things. These experts are say 1/10 to 1/100 of your model size if it were a dense model. So you pack them all up into one model, and you add a layer or a few layers that have the job of picking which small expert model is best for your…

Cool. Those that mean I could just run the query through the router and then load only the required expert? That is could I feasibly run this on my Macbook?

Re: The Llama 4 herd

#488

Post by hijodelsol »

martin82 wrote: post #479
ckrapu wrote: post #68

"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social topics. This is due to the types of training data available on the internet." Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.

No it is not. Right leaning opinions are heavily censored and shunned in all major publishing platforms that bots can scrape. For example, before Trump, if you contested the utterly normal common sense and scientifically sound idea that a trans woman is still a man, you would be banned - therefore, people with common sense will simply disengage, self-censor and get on with life.

Maybe because that position is both scientifically and morally unsound and if held strongly will lead to dehumanization and hate, attributes we should prevent any LLM from having.

Re: The Llama 4 herd

#489

Post by intelkishan »

accrual wrote: post #180
andrewstuart wrote: post #108

How much smaller would such a model be if it discarded all information not related to computers or programming?

I wonder if there will be a market for "old timey" models one day, ones with a cutoff date of 1800 or similar.

I guess the lack of training corpus would be a major issue with such a use case.

Re: The Llama 4 herd

#490

Post by disgruntledphd2 »

azinman2 wrote: post #466
drusepth wrote: post #396

Earlier quoted context omitted.

RAG still has lots of benefits for anyone paying per input token (e.g. over APIs).

Not to mention latency

And grounding for the model. Smaller models with tend to hallucinate a little less (anecdotally).
Post reply on HN