The Llama 4 herd • Hacker News Board

Re: The Llama 4 herd

#191

Post by Baeocystin » Sat, Apr 05, 2025, 7:43 PM UTC

lelandbatey wrote: post #168

qwertox wrote: post #63

Earlier quoted context omitted.

Llama 4 Scout, Maximum context length: 10M tokens. This is a nice development.

Is the recall and reasoning equally good across the entirety of the 10M token window? Cause from what I've seen many of those window claims equate to more like a functional 1/10th or less context length.

I assume they're getting these massive windows via RAG trickery, vectorization, and other tricks behind the curtain, became I've noticed the same as you- things start dipping in quality pretty quickly.

Does anyone know if I am correct in my assumption?

Re: The Llama 4 herd

#192

Post by kadushka » Sat, Apr 05, 2025, 7:44 PM UTC

phren0logy wrote: post #162

sshh12 wrote: post #129

Earlier quoted context omitted.

Not that I agree with all the linked points but it is weird to me that LeCun consistently states LLMs are not the right path yet LLMs are still the main flagship model they are shipping. Although maybe he's using an odd definition for what counts as a LLM. https://www.threads.net/@yannlecun/post/DD0ac1_v7Ij?hl=en

That is how I read it. Transformer based LLMs have limitations that are fundamental to the technology. It does not seem crazy to me that a guy involved in research at his level would say that they are a stepping stone to something better. What I find most interesting is his estimate of five years, which is soon enough that I would guess he sees one or more potential successors.

In our field (AI) nobody can see even 5 months ahead, including people who are training a model today to be released 5 months from now. Predicting something 5 years from now is about as accurate as predicting something 100 years from now.

Re: The Llama 4 herd

#193

Post by jasonjmcghee » Sat, Apr 05, 2025, 7:45 PM UTC

akulbe wrote: post #165

How well do you folks think this would run on this Apple Silicon setup? MacBook Pro M2 Max 96GB of RAM and which model should I try (if at all)? The alternative is a VM w/dual 3090s set up with PCI passthrough.

Depends on quantization. 109B at 4-bit quantization would be ~55GB of ram for parameters in theory, plus overhead of the KV cache which for even modest context windows could jump total to 90GB or something.

Curious to here other input here. A bit out of touch with recent advancements in context window / KV cache ram usage

Re: The Llama 4 herd

#194

Post by krapp » Sat, Apr 05, 2025, 7:45 PM UTC

senderista wrote: post #169

martythemaniak wrote: post #136

Earlier quoted context omitted.

I heard reality has a well-known liberal bias.

I admit that I cannot even imagine the state of mind in which one could attribute parochial, contingent political preferences to the UNIVERSE.

It's a joke made by Steven Colbert at the 2006 White House correspondents' dinner which referenced the Bush Administration's low poll numbers and the tendency of that administration to attribute bad press to "liberal media bias." This is also the administration that brought us the use of the term "reality based community" as an anti-leftist pejorative.

It is not meant to be literally interpreted as attributing contingent political preferences to the universe, but rather to be a (politically biased) statement on the tendency of conservatives to categorically deny reality and reframe it as leftist propaganda whenever it contradicts their narrative. One can extend this "bias" to include the rejection of mainstream scientific and historical narratives as "woke" by the right in a more modern context.

[0] https://en.wikipedia.org/wiki/Stephen_Colbert_at_the_2006_Wh...

[1] https://en.wikipedia.org/wiki/Reality-based_community

Re: The Llama 4 herd

#195

Post by cuuupid » Sat, Apr 05, 2025, 7:45 PM UTC

I think the most important thing to note here, perhaps more so than the context window, is that this exposes some serious flaws in benchmarks. Per benchmarks, Maverick is competitive only with older models like GPT-4o or Gemini 2.0 Flash, and not with anything in the last few months (incl. reasoning models).

However, the LMArena head to head leaderboard ranks this as 2nd place overall: https://lmarena.ai/?leaderboard

This would indicate there is either a gap between user preference and model performance, or between model performance and whatever benchmarks assess.

Either way, it is surely a huge deal that an open source model is now outperforming GPT 4.5.

Re: The Llama 4 herd

#196

Post by jasonjmcghee » Sat, Apr 05, 2025, 7:45 PM UTC

JKCalhoun wrote: post #32

mtharrison wrote: post #9

Might be worth changing url: https://www.llama.com/

From there I have to "request access" to a model?

You do anyway afaict

Re: The Llama 4 herd

#197

Post by mdp2021 » Sat, Apr 05, 2025, 7:46 PM UTC

nrp wrote: post #127

echelon wrote: post #116

Earlier quoted context omitted.

I don't understand Framework's desktop offerings. For laptops their open approach makes sense, but desktops are already about as hackable and DIY as they come.

We took the Ryzen AI Max, which is nominally a high-end laptop processor, and built it into a standard PC form factor (Mini-ITX). It’s a more open/extensible mini PC using mobile technology.

And given that some people are afraid of malicious software in some brands of mini-PCs on the market, to have some more trusted product around will also be an asset.

Re: The Llama 4 herd

#198

Post by ilove_banh_mi » Sat, Apr 05, 2025, 7:46 PM UTC

throwanem wrote: post #69

mvdtnz wrote: post #38

Earlier quoted context omitted.

What's "caponated"?

Castrated, if you're trying way too hard (and not well) to avoid getting called on that overly emotive metaphor: a capon is a gelded rooster.

There is a key distinction and context: caponation has a productive purpose from the pov of farmers and their desired profits.

Re: The Llama 4 herd

#199

Post by jimmyl02 » Sat, Apr 05, 2025, 7:46 PM UTC

lelandbatey wrote: post #168

qwertox wrote: post #63

Earlier quoted context omitted.

Llama 4 Scout, Maximum context length: 10M tokens. This is a nice development.

Is the recall and reasoning equally good across the entirety of the 10M token window? Cause from what I've seen many of those window claims equate to more like a functional 1/10th or less context length.

the needle in a haystack benchmark looks good but at this point I think we need new benchmarks to test actual understanding of content in such a large window.

Re: The Llama 4 herd

#200

Post by joaogui1 » Sat, Apr 05, 2025, 7:47 PM UTC

yusufozkan wrote: post #119

> while pre-training our Llama 4 Behemoth model using FP8 and 32K GPUs I thought they used a lot more GPUs to train frontier models (e.g. xAi training on 100k). Can someone explain why they are using so few?

I don't want to hunt the details on each of theses releases, but

* You can use less GPUs if you decrease batch size and increase number of steps, which would lead to a longer training time

* FP8 is pretty efficient, if Grok was trained with BF16 then LLama 4 should could need less GPUs because of that

* Depends also on size of the model and number of tokens used for training, unclear whether the total FLOPS for each model is the same

* MFU/Maximum Float Utilization can also vary depending on the setup, which also means that if you're use better kernels and/or better sharding you can reduce the number of GPUs needed