Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Models Are Getting Dumber on Purpose

w4g1.dev

Post reply on HN
Print view View original

91–100 of 197 posts

Re: Models Are Getting Dumber on Purpose

#91

Post by margalabargala »

COAGULOPATH wrote: post #85

This AI generated post (100% on Pangram) is pretty out of date. >On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%, so the best recall money can buy still misses half the questions. SimpleQA hasn't been updated in a long time. Gemini 2.5 Pro is a sixteen-month-old model, not "the best recall money can buy". >The part I find most promising is what this does t…

I agree with everything you say except this:

> You can make any modern LLM explain its reasoning

You can make any modern LLM create a plausible, self-consistent explanation that looks like reasoning, but it's not "the reasoning it used to arrive at that answer".

Re: Models Are Getting Dumber on Purpose

#92

Post by kennywinker »

alightsoul wrote: post #38
kennywinker wrote: post #4

Ideally what I'd like to see is pluggable knowledge bases. So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. My model doesn't need to know a single line of python. Then when I want to research electronics components, I grab a 15B model of agentic research techniques…

This is possible today with an agent such as OpenClaw and hermes

Only if you fundamentally misunderstand what I am describing

Re: Models Are Getting Dumber on Purpose

#93

Post by malfist »

COAGULOPATH wrote: post #85

This AI generated post (100% on Pangram) is pretty out of date. >On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%, so the best recall money can buy still misses half the questions. SimpleQA hasn't been updated in a long time. Gemini 2.5 Pro is a sixteen-month-old model, not "the best recall money can buy". >The part I find most promising is what this does t…

Seriously, anybody with a passing knowledge of LLMs knows thats not how they function. You can't encode logic in them because that's not how they work. It's a statistical model with useful emergent properties. It doesn't think, it doesn't reason, it isn't aware of facts or the rules of logic.

Re: Models Are Getting Dumber on Purpose

#94

Post by dist-epoch »

margalabargala wrote: post #91
COAGULOPATH wrote: post #85

This AI generated post (100% on Pangram) is pretty out of date. >On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%, so the best recall money can buy still misses half the questions. SimpleQA hasn't been updated in a long time. Gemini 2.5 Pro is a sixteen-month-old model, not "the best recall money can buy". >The part I find most promising is what this does t…

I agree with everything you say except this: > You can make any modern LLM explain its reasoning You can make any modern LLM create a plausible, self-consistent explanation that looks like reasoning, but it's not "the reasoning it used to arrive at that answer".

This was beautifully shown by asking a model to explain how it added two numbers together (something like 45+21), and it told a plausible story, when in fact they showed it was some rotation on a helix living in some internal manifold.

Like asking a human "how did you catch that fast ball coming at you?"

Re: Models Are Getting Dumber on Purpose

#96

Post by 3eb7988a1663 »

KerrAvon wrote: post #47

> the 24GB card that's been sitting in gaming PCs since 2022. I'd wager most people have less. In 2022 a 3080 might have 12 GB if you were lucky, 10 if you weren't -- and you paid for the privilege. A current RTX 5080 is only 16GB.

The most recent Steam Hardware Survey[0] lists the most popular VRAM at 16GB in 25.9% of users. Second most popular build is 8GB in 25.3%. >=24GB is ~7%, which is quite a bit higher than I expected.

[0] https://store.steampowered.com/hwsurvey/En

Re: Models Are Getting Dumber on Purpose

#97

Post by dist-epoch »

kennywinker wrote: post #34
kalb_almas wrote: post #22

Earlier quoted context omitted.

I would love this but I think the General Reasoning and Make No Mistakes modules would be massive.

I mean, I get pretty damn good general reasoning out of Qwen3.8-27B quantized to 4bits, and that knows Swift, Python, Node, Ruby, Rust, etc. etc. etc. And so far even the biggest model doesn't seem to have a working Make No Mistakes module, so maybe that's not needed

Qwen3.6-27B (not 3.8) was a much better coder than Gemma4-31B, yet Gemma4-31B was a much better reasoner and general LLM to talk with.

Sure, 3.8 maybe it's better now, but an accurate comparison would be with a new Gemma4-31B iteration (that doesn't exist).

Re: Models Are Getting Dumber on Purpose

#98

Post by alightsoul »

kennywinker wrote: post #92
alightsoul wrote: post #38

Earlier quoted context omitted.

This is possible today with an agent such as OpenClaw and hermes

Only if you fundamentally misunderstand what I am describing

You cannot have a single purpose LLM. Every single topic contributes to the performance of an LLM on any given area. You cannot have what you want. LLMs do not work like professions, college degrees or people. Using your examples, it is damaging to just know a single programming language, because there are patterns that are more common in say python than in swift, even though the only thing you want is swift, so LLM performance in swift benefits from pythonic patterns. Programming logic is the same in all languages so by having only swift and no other languages, you remove the number of examples the LLM is trained on which degrades performance in Swift which is the only thing you want

Re: Models Are Getting Dumber on Purpose

#99

Post by kentonv »

margalabargala wrote: post #91
COAGULOPATH wrote: post #85

This AI generated post (100% on Pangram) is pretty out of date. >On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%, so the best recall money can buy still misses half the questions. SimpleQA hasn't been updated in a long time. Gemini 2.5 Pro is a sixteen-month-old model, not "the best recall money can buy". >The part I find most promising is what this does t…

I agree with everything you say except this: > You can make any modern LLM explain its reasoning You can make any modern LLM create a plausible, self-consistent explanation that looks like reasoning, but it's not "the reasoning it used to arrive at that answer".

Tangent: This is often true of humans as well.

We often make a decision based on a gut feeling, and then backfill a logical reason supporting our feeling, without even realizing we're doing it -- rationalization.

Re: Models Are Getting Dumber on Purpose

#100

Post by joshheitzman »

3eb7988a1663 wrote: post #96
KerrAvon wrote: post #47

> the 24GB card that's been sitting in gaming PCs since 2022. I'd wager most people have less. In 2022 a 3080 might have 12 GB if you were lucky, 10 if you weren't -- and you paid for the privilege. A current RTX 5080 is only 16GB.

The most recent Steam Hardware Survey[0] lists the most popular VRAM at 16GB in 25.9% of users. Second most popular build is 8GB in 25.3%. >=24GB is ~7%, which is quite a bit higher than I expected. [0] https://store.steampowered.com/hwsurvey/En

As RAM prices are so nuts, I picked up a used gaming laptop early this year with an RTX 3070 (i.e. 8GB) to match the specs for my gaming tower that's run everything I'm interested in just fine. That includes recent Unreal 5.x games (Satisfactory, Fortnite, etc.) with high graphics settings on a 4k TV (60hz). There aren't a ton of games that require more than 8GB of GPU ram.
Post reply on HN