How are Maverick and Scout distilled from Behemoth if the latter is not done training? Do they distill from some intermediate, "good enough" snapshot?
The Llama 4 herd
451–460 of 695 posts
Re: The Llama 4 herd
#452Earlier quoted context omitted.
I hope on HN that we hold ourselves to a higher standard for “it’s been true from the beginning” than a vague recall of “My son assessed it for a class a few years ago” and not being able to reproduce.
I literally went back to the oldest model I could access and hand verified that in fact it does what I described, which is lecture you if you don't like unions and goes sweetly along if you do like unions. I feel this is a fair and reasonably well researched existence proof for a Saturday afternoon, and propose that it might be on you to find counter examples.
They were referring to your original claim about Pew research assessing the models as highly liberal when that’s apparently not even one of their ratings.
This is clear because they referenced your quote about it being from the beginning.
No one was arguing that you typed in a question about unions.
Re: The Llama 4 herd
#453I guess I have to say thank you Meta? A somewhat sad rant below. Deepseek starts a toxic trend of providing super, super large MoE. And MoE is famous for being parameter-inefficient, which is unfriendly to normal consumer hardware with limited vram. The super large size of LLM also disables nearly every people from doing meaningful development on these models. R1-1776 is the only fine-tune variation of R1 that makes…
People who downvoted this comment, do you guys really have GPUs with 80GB VRAM or M3 ultra with 512GB rams at home?
Re: The Llama 4 herd
#454Earlier quoted context omitted.
Yes loaded from RAM and loaded to RAM are the big distinction here. It will still be slow if portions of the model need to be read from disk to memory each pass, but only having to execute portions of the model for each token is a huge speed improvement.
It's not too expensive of a Macbook to fit 109B 4-bit parameters in RAM.
Is a 64GiB RAM Macbook really that expensive, especially compared against NVidia GPUs?
Re: The Llama 4 herd
#455Re: The Llama 4 herd
#456This thread so far (at 310 comments) summarized by Llama 4 Maverick: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-maverick -o max_tokens 20000 Output: https://gist.github.com/simonw/016ea0fd83fc499f046a94827f9b4... And with Scout I got complete junk output for some reason: hn-summary.sh 43595585 -m openrouter/meta-llama/llama-4-scout -o max_tokens 20000 Junk output here: https://gist.github.com/simonw/d01c…
That Gemini 2.5 one is impressive. I found it interesting that the blog post didn't mention Gemini 2.5 at all. Okay, it was released pretty recently, but 10 days seems like enough time to run the benchmarks, so maybe the results make Llama 4 look worse?
LM Arena ranks it second, just below Gemini 2.5 Pro.
Re: The Llama 4 herd
#457"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social topics. This is due to the types of training data available on the internet." Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.
Except for a some of the population of white countries right now, almost everyone in existence now and throughout the history of our species is and has been extraordinary more conservative—and racist—than western progressives. Even in white countries, progressivism being ascendant is a new trend after decades of propaganda and progressives controlling academia/entertainment/"news".
It genuinely boggles my mind that white progressives in the west think the rest of the world is like them.
Re: The Llama 4 herd
#458Model training observations from both Llama 3 and 4 papers: Meta’s Llama 3 was trained on ~16k H100s, achieving ~380–430 TFLOPS per GPU in BF16 precision, translating to a solid 38 - 43% hardware efficiency [Meta, Llama 3]. For Llama 4 training, Meta doubled the compute, using ~32K H100s and switched to FP8 precision. Despite the precision gain, observed efficiency dropped to about 19.7%, with GPUs delivering ~390 TF…
It's not just scale. Even for single GPU, it is hard to acheive 2x speed improvement as the GPU specs states. Even NVIDIA's own Tensor Engine acheives 28% extra FLOP/s[1].
Re: The Llama 4 herd
#459What an electrifying time to be alive! The last era that felt even remotely this dynamic was during the explosive rise of JavaScript frameworks—when it seemed like a new one dropped every quarter. Back then, though, the vibe was more like, “Ugh, another framework to learn?” Fast forward to now, and innovation is sprinting forward again—but this time, it feels like a thrilling ride we can’t wait to be part of.
on the other hand, i have started getting LLM fatigue. Every time I read one of these announcements, I go like "oh no, not another LLM model. When is this bubble gonna burst?"
Re: The Llama 4 herd
#460"It’s well-known that all leading LLMs have had issues with bias—specifically, they historically have leaned left when it comes to debated political and social topics. This is due to the types of training data available on the internet." Perhaps. Or, maybe, "leaning left" by the standards of Zuck et al. is more in alignment with the global population. It's a simpler explanation.
Call me crazy, but I don't want an AI that bases its reasoning on politics. I want one that is primarily scientific driven, and if I ask it political questions it should give me representative answers. E.g. "The majority view in [country] is [blah] with the minority view being [bleh]." I have no interest in "all sides are equal" answers because I don't believe all information is equally informative nor equally true.
The current crop of AIs can't do science though, they are disconnected from the physical world and can't test hypothesis or gather data.