Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Llama 4 herd

ai.meta.com

Post reply on HN
Print view View original

371–380 of 695 posts

Re: The Llama 4 herd

#371

Post by charcircuit »

falcor84 wrote: post #93
ksec wrote: post #54

Interesting this is released literally one hour after another discussions suggesting Meta ( https://news.ycombinator.com/item?id=43562768 ) >at this point it does not matter what you believe about LLMs: in general, to trust LeCun words is not a good idea. Add to this that LeCun is directing an AI lab that as the same point has the following huge issues: 1. Weakest ever LLM among the big labs with similar resources (a…

I don't understand what LeCun is trying to say. Why does he give an interview saying that LLM's are almost obsolete just when they're about to release a model that increases the SotA context length by an order of magnitude? It's almost like a Dr. Jekyll and Mr. Hyde situation.

A company can do R&D into new approaches while optimizing and iterating upon an existing approach.

Re: The Llama 4 herd

#372

Post by mkl »

simonw wrote: post #343

This thread so far (at 310 comments) summarized by Llama 4 Maverick: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-maverick -o max_tokens 20000 Output: https://gist.github.com/simonw/016ea0fd83fc499f046a94827f9b4... And with Scout I got complete junk output for some reason: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-scout -o max_tokens 20000 Junk output here: https://gist.github.com/simonw/d01c…

That Gemini 2.5 one is impressive. I found it interesting that the blog post didn't mention Gemini 2.5 at all. Okay, it was released pretty recently, but 10 days seems like enough time to run the benchmarks, so maybe the results make Llama 4 look worse?

Re: The Llama 4 herd

#373

Post by tucnak »

flawn wrote: post #30

10M Context Window with such a cheap performance WHILE having one of the top LMArena scores is really impressive. The choice to have 128 experts is also unseen as far as I know, right? But seems to have worked pretty good as it seems.

Let's see how that 10M context holds up, 128k pretrain is good indicator is not a scam but we're yet to see any numbers on this "iRoPE" architecture, at 17b active parameters and with 800G fabrics hitting the market, I think it could work, like I'm sure next year it'll be considered idiotic to keep K/V in actual memory.

Re: The Llama 4 herd

#374

Post by Alex-Programs »

hrpnk wrote: post #279

Available on Groq: https://groq.com/llama-4-now-live-on-groq-build-fast-at-the-... Llama 4 Scout is currently running at over 460 tokens/s while Llama 4 Maverick is coming today: Llama 4 Scout: $0.11 / M input tokens and $0.34 / M output tokens Llama 4 Maverick: $0.50 / M input tokens and $0.77 / M output tokens

Brilliant! Incredibly fast.

Re: The Llama 4 herd

#375

Post by [deleted] »

throwaway314155 wrote: post #237
kadushka wrote: post #192

Earlier quoted context omitted.

In our field (AI) nobody can see even 5 months ahead, including people who are training a model today to be released 5 months from now. Predicting something 5 years from now is about as accurate as predicting something 100 years from now.

Which would be nice if LeCun hadn't predicted the success of neural networks more broadly about 30 years before most others.

[deleted]

Re: The Llama 4 herd

#376

Post by reissbaker »

averageRoyalty wrote: post #321
ipsento606 wrote: post #206

Earlier quoted context omitted.

I find it impossible to discuss bias without a shared understanding of what it actually means to be unbiased - or at least, a shared understanding of what the process of reaching an unbiased position looks like. 40% of Americans believe that God created the earth in the last 10,000 years. If I ask an LLM how old the Earth is, and it replies ~4.5 billion years old, is it biased?

40% of Americans is about 2% of the worlds population though. It's hardly biased, it's stating the current scientific stance over a fringe belief with no evidence.

And what percentage of the world's >1B Muslims agree with you? Fundamentalist Christianity may have waned over the last century... But broaden your borders a little bit and I think you'll find Western secular liberalism is hardly the only major world ideology, or even the dominant one.

Re: The Llama 4 herd

#377

Post by Alex-Programs »

CSMastermind wrote: post #362
mrbonner wrote: post #43

What an electrifying time to be alive! The last era that felt even remotely this dynamic was during the explosive rise of JavaScript frameworks—when it seemed like a new one dropped every quarter. Back then, though, the vibe was more like, “Ugh, another framework to learn?” Fast forward to now, and innovation is sprinting forward again—but this time, it feels like a thrilling ride we can’t wait to be part of.

I lived through the explosion of JavaScript frameworks and this feels way bigger to me. For me at least it feels closer to the rise of the early internet. Reminds me of 1996.

I used to feel dismayed that I missed that era of the internet and technology (I'm 19). IRC, forums, work-in-progress gifs on personal websites, etc.

I still wish I were there for that, but I'm glad I get to be here for LLMs and the intelligence explosion. I have absolutely no idea what the world will look like in a few years. It certainly isn't the certain high-paying tech job in a largely static world that it looked like a few years ago.

But whatever happens, it's going to be interesting!

I wonder whether I'm spending my time optimally, working on a little SAAS that happens to use LLMs as a downstream commodity, contributing through a niche benchmark.

Re: The Llama 4 herd

#378

Post by [deleted] »

faraaz98 wrote: post #361
vessenes wrote: post #289

Earlier quoted context omitted.

This was an idea that sounded somewhat silly until it was shown it worked. The idea is that you encourage through training a bunch of “experts” to diversify and “get good” at different things. These experts are say 1/10 to 1/100 of your model size if it were a dense model. So you pack them all up into one model, and you add a layer or a few layers that have the job of picking which small expert model is best for your…

I've been calling for this approach for a while. It's kinda similar to how the human brain has areas that are good at specific tasks

[deleted]

Re: The Llama 4 herd

#379

Post by georgeck »

simonw wrote: post #343

This thread so far (at 310 comments) summarized by Llama 4 Maverick: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-maverick -o max_tokens 20000 Output: https://gist.github.com/simonw/016ea0fd83fc499f046a94827f9b4... And with Scout I got complete junk output for some reason: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-scout -o max_tokens 20000 Junk output here: https://gist.github.com/simonw/d01c…

I tried summarizing the thread so far (339 comments) with a custom system prompt [0] and a user-prompt that captures the structure (hierarchy and upvotes) of the thread [1].

This is the output that we got (based on the HN-Companion project) [2]:

LLama 4 Scout - https://gist.github.com/annjose/9303af60a38acd5454732e915e33...

Llama 4 Maverick - https://gist.github.com/annjose/4d8425ea3410adab2de4fe9a5785...

Claude 3.7 - https://gist.github.com/annjose/5f838f5c8d105fbbd815c5359f20...

The summary from Scout and Maverick both look good (comparable to Claude), and with this structure, Scout seems to follow the prompt slightly better.

In this case, we used the models 'meta-llama/llama-4-maverick' and 'meta-llama/llama-4-scout' from OpenRouter.

--

[0] - https://gist.github.com/annjose/5145ad3b7e2e400162f4fe784a14...

[1] - https://gist.github.com/annjose/d30386aa5ce81c628a88bd86111a...

[2] - https://github.com/levelup-apps/hn-enhancer

edited: To add OpenRouter model details.

Re: The Llama 4 herd

#380

Post by usef- »

faraaz98 wrote: post #361
vessenes wrote: post #289

Earlier quoted context omitted.

This was an idea that sounded somewhat silly until it was shown it worked. The idea is that you encourage through training a bunch of “experts” to diversify and “get good” at different things. These experts are say 1/10 to 1/100 of your model size if it were a dense model. So you pack them all up into one model, and you add a layer or a few layers that have the job of picking which small expert model is best for your…

I've been calling for this approach for a while. It's kinda similar to how the human brain has areas that are good at specific tasks

It's already used a lot — the paper I believe is from 1991, and GPT4 among many others is MoE
Post reply on HN