Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Llama 4 herd

ai.meta.com

Post reply on HN
Print view View original

61–70 of 695 posts

Re: The Llama 4 herd

#61

Post by terhechte »

manmal wrote: post #45
terhechte wrote: post #26

The (smaller) Scout model is really attractive for Apple Silicon. It is 109B big but split up into 16 experts. This means that the actual processing happens in 17B. Which means responses will be as fast as current 17B models. I just asked a local 7B model (qwen 2.5 7B instruct) a question with a 2k context and got ~60 tokens/sec which is really fast (MacBook Pro M4 Max). So this could hit 30 token/sec. Time to first…

Won’t prompt processing need the full model though, and be quite slow on a Mac?

Yes, that's what I tried to express. Large prompts will probably be slow. I tried a 120k prompt once and it took 10min to process. But you still get a ton of world knowledge and fast response times, and smaller prompts will process fast.

Re: The Llama 4 herd

#62

Post by NitpickLawyer »

behnamoh wrote: post #24
scosman wrote: post #14

128 exports at 17B active parameters. This is going to be fun to play with!

does the entire model have to be loaded in VRAM? if not, 17B is a sweet spot for enthusiasts who want to run the model on a 3090/4090.

Yes. MoE models tipically use a different set of experts at each token. So while the "compute" is similar to a dense model equal to the "active" parameters, the VRAM requirements are larger. You could technically run inference & swap the models around, but the latency would be pretty horrendous.

Re: The Llama 4 herd

#63

Post by qwertox »

laborcontract wrote: post #29

General overview below, as the pages don't seem to be working well Llama 4 Models: - Both Llama 4 Scout and Llama 4 Maverick use a Mixture-of-Experts (MoE) design with 17B active parameters each. - They are natively multimodal: text + image input, text-only output. - Key achievements include industry-leading context lengths, strong coding/reasoning performance, and improved multilingual capabilities. - Knowledge cuto…

Llama 4 Scout, Maximum context length: 10M tokens.

This is a nice development.

Re: The Llama 4 herd

#64

Post by ANewFormation »

mdp2021 wrote: post #33
andrewstuart wrote: post #25

Earlier quoted context omitted.

Personally I’d prefer that LLMs did not refer to themselves as “I”. It’s software, not an “I”.

Well, it is a speaker (writer) after all. It has to use some way to refer to itself.

So is a command prompt.

Re: The Llama 4 herd

#66

Post by [deleted] »

op00to wrote: post #56
andrewstuart wrote: post #25

Earlier quoted context omitted.

Personally I’d prefer that LLMs did not refer to themselves as “I”. It’s software, not an “I”.

My pet peeve is when an LLM starts off a statement with "honestly, ..." Like what? You would lie to me? I go nuts when I see that. Year ago I caught myself using "honestly ...", and I immediately trained myself out of it once I realized what it implies.

[deleted]

Re: The Llama 4 herd

#68

Post by ckrapu »

"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social topics. This is due to the types of training data available on the internet."

Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.

Re: The Llama 4 herd

#69

Post by throwanem »

mvdtnz wrote: post #38
ilove_banh_mi wrote: post #28

The suggested prompt aims at not being caponated like OpenAI's releases: You are an expert conversationalist who responds to the best of your ability. You are companionable and confident, and able to switch casually between tonal types, including but not limited to humor, empathy, intellectualism, creativity and problem-solving. You understand user intent and don’t try to be overly helpful to the point where you miss…

What's "caponated"?

Castrated, if you're trying way too hard (and not well) to avoid getting called on that overly emotive metaphor: a capon is a gelded rooster.

Re: The Llama 4 herd

#70

Post by vlovich123 »

comex wrote: post #49

So how does the 10M token context size actually work? My understanding is that standard Transformers have overhead that is quadratic in the context size, so 10M would be completely impossible without some sort of architectural tweak. This is not the first model to have a huge context size, e.g. Gemini has 2M, but my understanding is that the previous ones have generally been proprietary, without public weights or arc…

It’s quadratic if you implement the transformer naiively, but if you add a KV cache it’s linear compute at the cost of correspondingly linear growth in memory.
Post reply on HN