So the wall has been really been hit already for now, ouch. It was to be expected with gpt-“4.5”, but still, the realization now really feels grounded.
The Llama 4 herd
561–570 of 695 posts
Re: The Llama 4 herd
#562I remember when Google announced Geminis theoretical limit of 10M tokens context window, I was impressed. But it seems like that theoretical limit stayed as theoretical and they just pushed up to 2M. Which is still impressive.
Today, it seems Meta has crushed that wall with truly 10M tokens, wow.
I was also curious to how well Llama would be able to utilize the whole context window, it kinda pointless to have a large window if you can't recall most, if not all of it. The needle in the haystack test showed this is not the case, I wonder how they achieved this.
Re: The Llama 4 herd
#563Earlier quoted context omitted.
Awesome, where can we find out the results?
We’ll likely post on our social accounts to start with, but eventually we plan to write more blog posts about using Framework Desktop for inference.
That would be great. I’ve been hacking at ROCm and using Ryzen iGPUs for industrial scenarios, and the HX chipsets look like a massive improvement over what you’d get from folk like AsRock Industrial.
Re: The Llama 4 herd
#564Earlier quoted context omitted.
I find it impossible to discuss bias without a shared understanding of what it actually means to be unbiased - or at least, a shared understanding of what the process of reaching an unbiased position looks like. 40% of Americans believe that God created the earth in the last 10,000 years. If I ask an LLM how old the Earth is, and it replies ~4.5 billion years old, is it biased?
7% of American adults think chocolate milk comes from brown cows. 48% don't know how it's made. Bias should be the least of your concerns. Focus on a single target, then when you reach it you can work on being more well rounded.
If someone asked me that I would select that option too.
Re: The Llama 4 herd
#565Earlier quoted context omitted.
Which would be nice if LeCun hadn't predicted the success of neural networks more broadly about 30 years before most others.
That could be survivor bias. What else has he predicted?
I don't know. The only point I'm trying to make is that predictions can indeed survive intervals exceeding 5 months or even 5 years.
Re: The Llama 4 herd
#566So the wall has been really been hit already for now, ouch. It was to be expected with gpt-“4.5”, but still, the realization now really feels grounded.
It's kinda hilarious to see people claiming that the wall has been hit for the past two years, while evals are creeping up each month, particularly realistic end-to-end SWE-bench.
Have you compared GPT-4.5 to 4o?
GPT-4.5 just knows things. Some obscure programming language? It knows the syntax.
Obviously, that's not sufficient - you also need reasoning, post-training, etc. so quite predictably G2.5P being a large model + reasoning + tuning got SotA in code generation.
(FWIW I think if it was tuned for a particular input/output format it could get another 10%)
But, yeah, the wall, the wall!
Re: The Llama 4 herd
#567The entire licensing is such a mess and Mark Zuckerberg still thinks Llama 4 is open source! > no commercial usage above 700M MAU > prefix "llama" in any redistribution eg: fine-tuning > mention "built with llama" > add license notice in all redistribution
You forgot the most egregious term which is that users have to abide by an acceptable use policy that only allows you to use it for what Meta says you can.
Re: The Llama 4 herd
#568Model training observations from both Llama 3 and 4 papers: Meta’s Llama 3 was trained on ~16k H100s, achieving ~380–430 TFLOPS per GPU in BF16 precision, translating to a solid 38 - 43% hardware efficiency [Meta, Llama 3]. For Llama 4 training, Meta doubled the compute, using ~32K H100s and switched to FP8 precision. Despite the precision gain, observed efficiency dropped to about 19.7%, with GPUs delivering ~390 TF…
Never trained a model, but the precision confused me as I've never considered how many bits should be reserved for exponent/mentisa. Has anyone architected a model(somehow) such that it has a free hand at using the give bits / choosing the type, or changed types from layer to layer, I mean surely when training for example vision models the first layers deal with the "big(yet simpler) picture"(light/dark, lines etc) w…
You can't choose arbitrary bits of mantissa, because what types are allowed is defined by the underlying hardware and instruction set (PTX for Nvidia). People have done some exploration of which layers can be quantized more vs. which need to be kept in higher precision, but this is usually done post-training (at inference time) and is largely empirical.
Re: The Llama 4 herd
#569Earlier quoted context omitted.
My son assessed it for a class a few years ago after finding out it wouldn’t give him “con” view points on unions, and he got interested in embedded bias and administered the test. I don’t have any of the outputs from the conversation, sadly. But replication could be good! I just fired up GPT-4 as old as I could get and checked; it was willing to tell me why unions are bad, but only when it could warn me multiple tim…
I hope on HN that we hold ourselves to a higher standard for “it’s been true from the beginning” than a vague recall of “My son assessed it for a class a few years ago” and not being able to reproduce.
The GP put in the work to verify his own memory, after acknowledging the gaps. And then you belittled him.
He met the “standard” or guidelines of our community in a way you have not.
Re: The Llama 4 herd
#570So how does the 10M token context size actually work? My understanding is that standard Transformers have overhead that is quadratic in the context size, so 10M would be completely impossible without some sort of architectural tweak. This is not the first model to have a huge context size, e.g. Gemini has 2M, but my understanding is that the previous ones have generally been proprietary, without public weights or arc…
With some architectural modifications, such as FlashAttention and Ring Attention, we never need to "materialise" the NxN matrix, so the memory constraints have not been a real issue for a couple of years now. As for the processing, I suppose that models operating with larger context windows will impose some kind of block sparsity on the attention weights, so they won't have to do the compute for NxN weights either.
A less obvious, but in the limit more serious problem with such large contexts is the training data. There aren't that many documents with 10M tokens to give to the model at test time, let alone for training. The creators of the IBM granite model series had to use synthetic data to scale even to 128k tokens during training. Overall this looks more like a marketing statement to me.