Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Llama 4 herd

ai.meta.com

Post reply on HN
Print view View original

361–370 of 695 posts

Re: The Llama 4 herd

#361

Post by faraaz98 »

vessenes wrote: post #289
InvOfSmallC wrote: post #266

Earlier quoted context omitted.

For a super ignorant person: Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each Those experts are LLM trained on specific tasks or what?

This was an idea that sounded somewhat silly until it was shown it worked. The idea is that you encourage through training a bunch of “experts” to diversify and “get good” at different things. These experts are say 1/10 to 1/100 of your model size if it were a dense model. So you pack them all up into one model, and you add a layer or a few layers that have the job of picking which small expert model is best for your…

I've been calling for this approach for a while. It's kinda similar to how the human brain has areas that are good at specific tasks

Re: The Llama 4 herd

#362

Post by CSMastermind »

mrbonner wrote: post #43

What an electrifying time to be alive! The last era that felt even remotely this dynamic was during the explosive rise of JavaScript frameworks—when it seemed like a new one dropped every quarter. Back then, though, the vibe was more like, “Ugh, another framework to learn?” Fast forward to now, and innovation is sprinting forward again—but this time, it feels like a thrilling ride we can’t wait to be part of.

I lived through the explosion of JavaScript frameworks and this feels way bigger to me. For me at least it feels closer to the rise of the early internet.

Reminds me of 1996.

Re: The Llama 4 herd

#363

Post by parineum »

ipsento606 wrote: post #306
dcsommer wrote: post #302

Earlier quoted context omitted.

> 40% of Americans believe that God created the earth in the last 10,000 years. Citation needed. That claim is not compatible with Pew research findings which put only 18% of Americans as not believing in any form of human evolution. https://www.pewresearch.org/religion/2019/02/06/the-evolutio...

https://news.gallup.com/poll/647594/majority-credits-god-hum...

Only 3 questions that combine two data points.

There's no way to answer that god created humans in their present form without also saying within the last 10000 years.

This is why polling isn't always reliable. This poll should, at the very least, be two questions and there should be significantly more options.

Re: The Llama 4 herd

#364

Post by h8hawk »

mrbonner wrote: post #43

What an electrifying time to be alive! The last era that felt even remotely this dynamic was during the explosive rise of JavaScript frameworks—when it seemed like a new one dropped every quarter. Back then, though, the vibe was more like, “Ugh, another framework to learn?” Fast forward to now, and innovation is sprinting forward again—but this time, it feels like a thrilling ride we can’t wait to be part of.

Comparing JS frameworks to LLMs is like comparing a bike to a spaceship—completely different beasts.

Re: The Llama 4 herd

#365

Post by mrbonner »

vessenes wrote: post #289
InvOfSmallC wrote: post #266

Earlier quoted context omitted.

For a super ignorant person: Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each Those experts are LLM trained on specific tasks or what?

This was an idea that sounded somewhat silly until it was shown it worked. The idea is that you encourage through training a bunch of “experts” to diversify and “get good” at different things. These experts are say 1/10 to 1/100 of your model size if it were a dense model. So you pack them all up into one model, and you add a layer or a few layers that have the job of picking which small expert model is best for your…

So this is kind of an ensemble sort of thing in ML like random forest and GBT?

Re: The Llama 4 herd

#366

Post by bonoboTP »

jimmyl02 wrote: post #315
zamadatix wrote: post #297

Earlier quoted context omitted.

The only thing about this which may be unintuitive from the name is an "Expert" is not something like a sub-llm that's good at math and gets called when you ask a math question. Models like this have layers of networks they run tokens through and each layer is composed of 256 sub-networks, any of which can be selected (or multiple selected and merged in some way) for each layer independently. So the net result is the…

the most unintuitive part is that from my understanding, individual tokens are routed to different experts. this is hard to comprehend with "experts" as that means two you can have different experts for two sequential tokens right? I think where MoE is misleading is that the experts aren't what we would call "experts" in the normal world but rather they are experts for a specific token . that concept feels difficult…

Also note that MoE is a decades old term, predating deep learning. It's not supposed to be interpreted literally.

Re: The Llama 4 herd

#367

Post by tucnak »

hydroreadsstuff wrote: post #243

This means GPUs are dead for local enthusiast AI. And SoCs with big RAM are in. Because 17B active parameters should reach enough performance on 256bit LPDDR5x.

This has been the case for a while now. 3090 hoarders were always just doing it for street cred or whatever, no way these guys are computing anything of actual value.

Tenstorrent is on fire, though. For small businesses this is what matters. If 10M context is not a scam, I think we'll see SmartNIC adoption real soon. I would literally long AMD now because their Xilinx people are probably going to own the space real soon. Infiniband is cool and all, but it's also stupid and their scale-out strategy is non-existent. This is why https://github.com/deepseek-ai/3FS came out but of course nobody had figured it out because they still think LLM's is like, chatbots, or something. I think we're getting to a point where it's a scheduling problem, basically. So you get like like lots of GDDR6 (HBM doesnn't matter anymore) as L0, DDR5 as L1, and NVMe-oF is L2. Most of the time the agents will be running the code anyway...

This is also why Google never really subscribed to "function calling" apis

Re: The Llama 4 herd

#368

Post by tarruda »

simonw wrote: post #343

This thread so far (at 310 comments) summarized by Llama 4 Maverick: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-maverick -o max_tokens 20000 Output: https://gist.github.com/simonw/016ea0fd83fc499f046a94827f9b4... And with Scout I got complete junk output for some reason: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-scout -o max_tokens 20000 Junk output here: https://gist.github.com/simonw/d01c…

> I'm a little unimpressed by its instruction following

Been trying the 109b version on Groq and it seems less capable than Gemma 3 27b

Re: The Llama 4 herd

#369

Post by j_maffe »

falcor84 wrote: post #287
j_maffe wrote: post #233

Earlier quoted context omitted.

> But ... that's not how science works Not sure where this is coming from. Also, it's important to keep in mind the quote "The electric light did not come from the continuous improvement of candles"

Well, having candles and kerosene lamps to work late definitely didn't hurt. But in any case, while these things don't work in a predictable way, the engineering work on lightbulbs in your example led to theoretical advances in our understanding of materials science, vacuum technology, and of course electrical systems. I'm not arguing that LLMs on their own will certainly lead directly to AGI without any additional i…

I don't think anyone ould disagree with what you're saying here, especially LeCun.

Re: The Llama 4 herd

#370

Post by csdvrx »

simonw wrote: post #343

This thread so far (at 310 comments) summarized by Llama 4 Maverick: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-maverick -o max_tokens 20000 Output: https://gist.github.com/simonw/016ea0fd83fc499f046a94827f9b4... And with Scout I got complete junk output for some reason: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-scout -o max_tokens 20000 Junk output here: https://gist.github.com/simonw/d01c…

I have found the Gemini 2.5 Pro summary genuinely interesting: it adequately describes what I've read.

Have you thought about automatizing hn-summaries for say what the 5 top posts are at 8 AM EST?

That would be a simple product to test the market. If successful, it could be easily extended to a weekly newsletter summary.

Post reply on HN