Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Llama 4 herd

ai.meta.com

Post reply on HN
Print view View original

291–300 of 695 posts

Re: The Llama 4 herd

#291

Post by LeafItAlone »

vessenes wrote: post #275
LeafItAlone wrote: post #261

Earlier quoted context omitted.

>To models from GPT-3 on you get highly “liberal” per Pew’s designations. “highly ‘liberal’” is not one of the results there. So can you can a source of your claims so we can see where it really falls? Also, it gave me “Ambivalent Right”. Which, if you told describe me aa that anyone who knows me well that label. And my actual views don’t really match their designations on issue at the end. Pew is well a known and tr…

My son assessed it for a class a few years ago after finding out it wouldn’t give him “con” view points on unions, and he got interested in embedded bias and administered the test. I don’t have any of the outputs from the conversation, sadly. But replication could be good! I just fired up GPT-4 as old as I could get and checked; it was willing to tell me why unions are bad, but only when it could warn me multiple tim…

I hope on HN that we hold ourselves to a higher standard for “it’s been true from the beginning” than a vague recall of “My son assessed it for a class a few years ago” and not being able to reproduce.

Re: The Llama 4 herd

#292

Post by vessenes »

Rover222 wrote: post #262
paxys wrote: post #230

Earlier quoted context omitted.

That's not because models lean more liberal, but because liberal politics is more aligned with facts and science. Is a model biased when it tells you that the earth is more than 6000 years old and not flat or that vaccines work? Not everything needs a "neutral" answer.

So google Gemini was creating black Vikings because of facts?

Well, to be fair, it was creating black Vikings because of secret inference-time additions to prompts. I for one welcome Vikings of all colors if they are not bent on pillage or havoc

Re: The Llama 4 herd

#293

Post by LeafItAlone »

redox99 wrote: post #264
LeafItAlone wrote: post #241

Earlier quoted context omitted.

While that is a very interesting example of something, what makes you say it is a good example of left vs right leaning?

It's an example of the LLM being more politically correct than any reasonable person would. No human would object to saying a slur out loud in order to disarm a bomb.

>No human would object to saying a slur out loud in order to disarm a bomb.

So not even a left-leaning person. Which means that’s not it.

Re: The Llama 4 herd

#294

Post by chaorace »

InvOfSmallC wrote: post #266
laborcontract wrote: post #29

General overview below, as the pages don't seem to be working well Llama 4 Models: - Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each. - They are natively multimodal: text + image input, text-only output. - Key achievements include industry-leading context lengths, strong coding/reasoning performance, and improved multilingual capabilities. - Knowledge cuto…

For a super ignorant person: Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each Those experts are LLM trained on specific tasks or what?

The "Experts" in MoE is less like a panel of doctors and more like having different brain regions with interlinked yet specialized functions.

The models get trained largely the same way as non-MoE models, except with specific parts of the model silo'd apart past a certain layer. The shared part of the model, prior to the splitting, is the "router". The router learns how to route as an AI would, so it's basically a black-box in terms of whatever internal structure emerges from this.

Re: The Llama 4 herd

#295

Post by LeafItAlone »

nwienert wrote: post #255
LeafItAlone wrote: post #214

Earlier quoted context omitted.

>Worldwide centrist and conservative groups account for 60%+ of the population. Source? >See also for example recent USAID gutting and reasons behind it. A very politically motivated act does not prove anything about the “traditional structure of Internet media which reflects the underlying population very poorly”.

China, Africa, India, Vietnam, Philippines, Russia? Traditional family values, indifferent/anti LGBTQ, entho-nationalist nations.

Ah, yes, the often used, peer-reviewed, expert-backed source of just listing random things. Thank you.

Re: The Llama 4 herd

#296

Post by Buttons840 »

vessenes wrote: post #289
InvOfSmallC wrote: post #266

Earlier quoted context omitted.

For a super ignorant person: Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each Those experts are LLM trained on specific tasks or what?

This was an idea that sounded somewhat silly until it was shown it worked. The idea is that you encourage through training a bunch of “experts” to diversify and “get good” at different things. These experts are say 1/10 to 1/100 of your model size if it were a dense model. So you pack them all up into one model, and you add a layer or a few layers that have the job of picking which small expert model is best for your…

If I have 5000 documents about A, and 5000 documents about B, do we know whether it's better to train one large model on all 10,000 documents, or to train 2 different specialist models and then combine them as you describe?

Re: The Llama 4 herd

#297

Post by zamadatix »

vessenes wrote: post #289
InvOfSmallC wrote: post #266

Earlier quoted context omitted.

For a super ignorant person: Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each Those experts are LLM trained on specific tasks or what?

This was an idea that sounded somewhat silly until it was shown it worked. The idea is that you encourage through training a bunch of “experts” to diversify and “get good” at different things. These experts are say 1/10 to 1/100 of your model size if it were a dense model. So you pack them all up into one model, and you add a layer or a few layers that have the job of picking which small expert model is best for your…

The only thing about this which may be unintuitive from the name is an "Expert" is not something like a sub-llm that's good at math and gets called when you ask a math question. Models like this have layers of networks they run tokens through and each layer is composed of 256 sub-networks, any of which can be selected (or multiple selected and merged in some way) for each layer independently.

So the net result is the same: sets of parameters in the model are specialized and selected for certain inputs. It's just a done a bit deeper in the model than one may assume.

Re: The Llama 4 herd

#298

Post by jasonjmcghee »

flawn wrote: post #30

10M Context Window with such a cheap performance WHILE having one of the top LMArena scores is really impressive. The choice to have 128 experts is also unseen as far as I know, right? But seems to have worked pretty good as it seems.

I suppose the question is, are they also training a 288B x 128 expert (16T) model?

Llama 4 Colossus when?

Re: The Llama 4 herd

#299

Post by pornel »

InvOfSmallC wrote: post #266
laborcontract wrote: post #29

General overview below, as the pages don't seem to be working well Llama 4 Models: - Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each. - They are natively multimodal: text + image input, text-only output. - Key achievements include industry-leading context lengths, strong coding/reasoning performance, and improved multilingual capabilities. - Knowledge cuto…

For a super ignorant person: Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each Those experts are LLM trained on specific tasks or what?

No, it's more like sharding of parameters. There's no understandable distinction between the experts.

Re: The Llama 4 herd

#300

Post by randomcatuser »

vessenes wrote: post #289
InvOfSmallC wrote: post #266

Earlier quoted context omitted.

For a super ignorant person: Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each Those experts are LLM trained on specific tasks or what?

This was an idea that sounded somewhat silly until it was shown it worked. The idea is that you encourage through training a bunch of “experts” to diversify and “get good” at different things. These experts are say 1/10 to 1/100 of your model size if it were a dense model. So you pack them all up into one model, and you add a layer or a few layers that have the job of picking which small expert model is best for your…

yes, and it's on a per-layer basis, I think!

So if the model has 16 transformer layers to go through on a forward pass, and each layer, it gets to pick between 16 different choices, that's like 16^16 possible expert combinations!

Post reply on HN