Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Llama 4 herd

ai.meta.com

Post reply on HN
Print view View original

411–420 of 695 posts

Re: The Llama 4 herd

#411

Post by typewithrhythm »

ckrapu wrote: post #68

"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social topics. This is due to the types of training data available on the internet." Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.

Training data is always filtered, if you want a representative of the population you would need to include conspiracy theories about the Jews, and rants about per capita crime rates... But nobody really wants a model the returns that.

Re: The Llama 4 herd

#412

Post by lordswork »

philsnow wrote: post #348
vessenes wrote: post #289

Earlier quoted context omitted.

This was an idea that sounded somewhat silly until it was shown it worked. The idea is that you encourage through training a bunch of “experts” to diversify and “get good” at different things. These experts are say 1/10 to 1/100 of your model size if it were a dense model. So you pack them all up into one model, and you add a layer or a few layers that have the job of picking which small expert model is best for your…

The idea has also been around for at least 15 years; "ensemble learning" was a topic in my "Data Mining" textbook from around then. Meta calls these individually smaller/weaker models "experts" but I've also heard them referred to as "bozos", because each is not particularly good at anything and it's only together that they are useful. Also bozos has better alliteration with boosting and bagging, two terms that are c…

MOE as an idea specific to neural networks has been around since 1991[1] . OP is probably aware, but adding for others following along, while MoE has roots in ensembling, there are some important differences: Traditional ensembles run all models in parallel and combine their outputs, whereas MoE uses a gating mechanism to activate only a subset of experts per input. This enables efficient scaling via conditional computation and expert specialization, rather than redundancy.

[1]:https://ieeexplore.ieee.org/document/6797059

Re: The Llama 4 herd

#413

Post by khimaros »

georgeck wrote: post #379
simonw wrote: post #343

This thread so far (at 310 comments) summarized by Llama 4 Maverick: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-maverick -o max_tokens 20000 Output: https://gist.github.com/simonw/016ea0fd83fc499f046a94827f9b4... And with Scout I got complete junk output for some reason: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-scout -o max_tokens 20000 Junk output here: https://gist.github.com/simonw/d01c…

I tried summarizing the thread so far (339 comments) with a custom system prompt [0] and a user-prompt that captures the structure (hierarchy and upvotes) of the thread [1]. This is the output that we got (based on the HN-Companion project) [2]: LLama 4 Scout - https://gist.github.com/annjose/9303af60a38acd5454732e915e33... Llama 4 Maverick - https://gist.github.com/annjose/4d8425ea3410adab2de4fe9a5785... Claude 3.7…

as another dateline, Maverick has taken #2 position on LMArena, just behind Gemini 2.5 Pro.

Re: The Llama 4 herd

#415

Post by casey2 »

ipsento606 wrote: post #206
ckrapu wrote: post #68

"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social topics. This is due to the types of training data available on the internet." Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.

I find it impossible to discuss bias without a shared understanding of what it actually means to be unbiased - or at least, a shared understanding of what the process of reaching an unbiased position looks like. 40% of Americans believe that God created the earth in the last 10,000 years. If I ask an LLM how old the Earth is, and it replies ~4.5 billion years old, is it biased?

7% of American adults think chocolate milk comes from brown cows. 48% don't know how it's made.

Bias should be the least of your concerns. Focus on a single target, then when you reach it you can work on being more well rounded.

Re: The Llama 4 herd

#416

Post by greenchair »

paxys wrote: post #230
vessenes wrote: post #220

Earlier quoted context omitted.

Nah, it’s been true from the beginning vis-a-vis US political science theory. That is, if you deliver something like https://www.pewresearch.org/politics/quiz/political-typology... To models from GPT-3 on you get highly “liberal” per Pew’s designations. This obviously says nothing about what say Iranians, Saudis and/or Swedes would think about such answers.

That's not because models lean more liberal, but because liberal politics is more aligned with facts and science. Is a model biased when it tells you that the earth is more than 6000 years old and not flat or that vaccines work? Not everything needs a "neutral" answer.

hooboy, thanks for that laugh!

Re: The Llama 4 herd

#417

Post by Rover222 »

jug wrote: post #387
Rover222 wrote: post #358

Earlier quoted context omitted.

It should be accurate. Adding in DEI to everything is a political bias. Truth is truth.

The problem here and with your comparison is that Gemini (the language model) wasn't creating black vikings because of political bias in the training, but due to how Google augmented the user prompts to force-include diversity. Behind the scenes, you were basically telling Gemini to always remember racial diversity even if you didn't in your prompt. But if you were asking Gemini, vikings were white. This was later re…

Gotcha, thanks for clarifying that

Re: The Llama 4 herd

#418

Post by jjani »

mkl wrote: post #372
simonw wrote: post #343

This thread so far (at 310 comments) summarized by Llama 4 Maverick: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-maverick -o max_tokens 20000 Output: https://gist.github.com/simonw/016ea0fd83fc499f046a94827f9b4... And with Scout I got complete junk output for some reason: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-scout -o max_tokens 20000 Junk output here: https://gist.github.com/simonw/d01c…

That Gemini 2.5 one is impressive. I found it interesting that the blog post didn't mention Gemini 2.5 at all. Okay, it was released pretty recently, but 10 days seems like enough time to run the benchmarks, so maybe the results make Llama 4 look worse?

I'm sure it does, as Gemini 2.5 Pro has been making every other model look pretty bad.

Re: The Llama 4 herd

#419

Post by simplify »

EasyMark wrote: post #401
kubb wrote: post #210

Earlier quoted context omitted.

LLMs are great at cutting through a lot of right (and left) wing rhetorical nonsense. Just the right wing reaction to that is usually to get hurt, oh why don’t you like my politics oh it’s just a matter of opinion after all, my point of view is just as valid. Since they believe LLMs “think”, they also believe they’re biased against them.

I think right wing tends to be much less "tolerant" of live and let live, as religions are often a huge part of their "bias" and those religions often say that others must be punished for not following God's(s') path, up and including destruction of those who don't fall in line.

Everyone has a "religion" – i.e. a system of values they subscribe to.

Secular Americans are annoying because they believe they don't have one, and instead think they're just "good people", calling those who break their core values "bad people".

Re: The Llama 4 herd

#420

Post by bavell »

nrp wrote: post #254
kybernetikos wrote: post #239

Earlier quoted context omitted.

I love the look of it and if I were in the market right now it would be high on the list, but I do understand the confusion here - is it just a cool product you wanted to make or does it somehow link to what I assumed your mission was - to reduce e-waste?

A big part of our mission is accessibility and consumer empowerment. We were able to build a smaller/simpler PC for gamers new to it that still leverages PC standards, and the processor we used also makes local interference of large models more accessible to people who want to tinker with them.

Considering the framework desktop or something like it for a combo homelab / home assistant / HTPC. The new gen of AMD APUs looks to be the sweet spot for a lot of really interesting products.

Love what you guys are doing!!

Post reply on HN