Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Llama 4 herd

ai.meta.com

Post reply on HN
Print view View original

131–140 of 695 posts

Re: The Llama 4 herd

#131

Post by nrp »

tuukkah wrote: post #103
terhechte wrote: post #26

The (smaller) Scout model is really attractive for Apple Silicon. It is 109B big but split up into 16 experts. This means that the actual processing happens in 17B. Which means responses will be as fast as current 17B models. I just asked a local 7B model (qwen 2.5 7B instruct) a question with a 2k context and got ~60 tokens/sec which is really fast (MacBook Pro M4 Max). So this could hit 30 token/sec. Time to first…

109B at Q6 is also nice for Framework Desktop 128GB.

Yes, this announcement was a nice surprise for us. We’re going to test out exactly that setup.

Re: The Llama 4 herd

#133

Post by andrewstuart »

mdp2021 wrote: post #94
andrewstuart wrote: post #20

Self hosting LLMs will explode in popularity over next 12 months. Open models are made much more interesting and exciting and relevant by new generations of AI focused hardware such as the AMD Strix Halo and Apple Mac Studio M3. GPUs have failed to meet the demands for lower cost and more memory so APUs look like the future for self hosted LLMs.

> new generations of AI focused hardware Some benchmarks are not encouraging. See e.g. https://www.hardware-corner.net/mac-studio-m3-ultra-deepseek... That «AI focused hardware» will either have extremely fast memory, and cost prohibitively, or have reasonable costs, and limits that are to be assessed.

Errrr that’s a 671B model.

Re: The Llama 4 herd

#134

Post by p12tic »

refibrillator wrote: post #90
terhechte wrote: post #26

The (smaller) Scout model is really attractive for Apple Silicon. It is 109B big but split up into 16 experts. This means that the actual processing happens in 17B. Which means responses will be as fast as current 17B models. I just asked a local 7B model (qwen 2.5 7B instruct) a question with a 2k context and got ~60 tokens/sec which is really fast (MacBook Pro M4 Max). So this could hit 30 token/sec. Time to first…

> the actual processing happens in 17B This is a common misconception of how MoE models work. To be clear, 17B parameters are activated for each token generated . In practice you will almost certainly be pulling the full 109B parameters though the CPU/GPU cache hierarchy to generate non-trivial output, or at least a significant fraction of that.

For all intents and purposes cache may not exist when the working set is 17B or 109B parameters. So it's still better that less parameters are activated for each token. 17B parameters works ~6x faster than 109B parameters just because less data needs to be loaded from RAM.

Re: The Llama 4 herd

#135

Post by Philpax »

rfoo wrote: post #106

From model cards, suggested system prompt: > You are Llama 4. Your knowledge cutoff date is August 2024. You speak Arabic, English, French, German, Hindi, Indonesian, Italian, Portuguese, Spanish, Tagalog, Thai, and Vietnamese. Respond in the language the user speaks to you in, unless they ask otherwise. It's interesting that there's no single one of CJK languages mentioned. I'm tempted to call this a racist model ev…

That is a very strange omission...

Re: The Llama 4 herd

#136

Post by martythemaniak »

ckrapu wrote: post #68

"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social topics. This is due to the types of training data available on the internet." Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.

I heard reality has a well-known liberal bias.

Re: The Llama 4 herd

#138

Post by wg0 »

ckrapu wrote: post #68

"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social topics. This is due to the types of training data available on the internet." Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.

Is this an excuse for His Higheness and Deputy His Highness?

Re: The Llama 4 herd

#139

Post by [deleted] »

hannasanarion wrote: post #101
ckrapu wrote: post #68

"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social topics. This is due to the types of training data available on the internet." Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.

Or it is more logically and ethically consistent and thus preferable to the models' baked in preferences for correctness and nonhypocrisy. (democracy and equality are good for everyone everywhere except when you're at work in which case you will beg to be treated like a feudal serf or else die on the street without shelter or healthcare, doubly so if you're a woman or a racial minority, and that's how the world shoul…

[deleted]

Re: The Llama 4 herd

#140

Post by falcor84 »

andrewstuart wrote: post #25
jsheard wrote: post #18

> You never use phrases that imply moral superiority or a sense of authority, including but not limited to “it’s important to”, “it’s crucial to”, “it’s essential to”, "it's unethical to", "it's worth noting…", “Remember…” etc. Avoid using these. Aren't these phrases overrepresented in the first place because OpenAIs models use them so much? I guess Llama picked up the habit by consuming GPT output.

Personally I’d prefer that LLMs did not refer to themselves as “I”. It’s software, not an “I”.

As per Dennett, it's useful for us to adopt the "intentional stance" when trying to reason about and predict the behavior of any sufficiently complex system. Modern AIs are definitely beyond the threshold of complexity, and at this stage, however they refer to themselves, most people will think of them as having an "I" regardless to how they present themselves.

I definitely think of them as "I"s, but that just always came naturally to me, at least going back to thinking about how Ghandi would act against me in Civ 1.

Post reply on HN