Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Models Are Getting Dumber on Purpose

w4g1.dev

Post reply on HN
Print view View original

101–110 of 197 posts

Re: Models Are Getting Dumber on Purpose

#101

Post by kennywinker »

dist-epoch wrote: post #97
kennywinker wrote: post #34

Earlier quoted context omitted.

I mean, I get pretty damn good general reasoning out of Qwen3.8-27B quantized to 4bits, and that knows Swift, Python, Node, Ruby, Rust, etc. etc. etc. And so far even the biggest model doesn't seem to have a working Make No Mistakes module, so maybe that's not needed

Qwen3.6-27B (not 3.8) was a much better coder than Gemma4-31B, yet Gemma4-31B was a much better reasoner and general LLM to talk with. Sure, 3.8 maybe it's better now, but an accurate comparison would be with a new Gemma4-31B iteration (that doesn't exist).

I guess the question is how much of the coding skill gap between qwen3.6-27b and gemma4-31b is down to the model architecture, and how much is because of the training data.

Assuming you can tweak the training data, regenerate qwen3.6, and get a better coder, then presumably you could have variants - e.g. qwen3.6-swift-27b and qwen3.6-python-27b. Or maybe all coding is too intertwined and you can only get splits like qwen3.6-research-27b and qwen3.6-coding-27b. Which isn't quite my pluggable-models dream, but it's a step closer.

But maybe the difference isn't the training data, it's the architecture, in which case pluggable models is probably not possible.

Re: Models Are Getting Dumber on Purpose

#102

Post by exe34 »

echelon wrote: post #53
pton_xd wrote: post #42

Earlier quoted context omitted.

> My model doesn't need to know a single line of python. If I had to guess, the weights necessary to encode "how to program" are much larger than the final step of "output python."

But the understanding of the language library ecosystem, or even better, your codebase, could let it execute faster and with less context usage.

That's the bit I feel you can do with ragging - keep a large library of well described solutions and then find what you need from it at runtime.

Re: Models Are Getting Dumber on Purpose

#103

Post by vasilvv »

scld wrote: post #24
edgyquant wrote: post #18

Earlier quoted context omitted.

This would defeat the AGI narrative/belief that so many building these models have

On the otherhand, your own brain probably doesn't use your eyes to hear.

Counterintuitively, seeing things can alter the perception of individual sounds you're hearing:

https://en.wikipedia.org/wiki/McGurk_effect

Re: Models Are Getting Dumber on Purpose

#104

Post by kennywinker »

cyanydeez wrote: post #31
kennywinker wrote: post #13

Earlier quoted context omitted.

That kind of setup is super dependent on a search engine, and search keeps getting worse.

the technicals of search did not get worse. the economics of the search provider did. If government wasn't stuck in the 50s mindset, search provider would have been nationalized just like a utility.

I think the techinicals of search did get worse - the ai sloppification of the internet made the needle harder to find in the haystack.

Re: Models Are Getting Dumber on Purpose

#105

Post by kennywinker »

lowsong wrote: post #72
gagan2020 wrote: post #20

Intelligence vs Knowledge LLMs work is being intelligent not having knowledge of everything is ok. But, they have to be intelligent enough (with some degree of knowledge) that where to find the information (search tools or any other tools for that matter)

LLMs are not intelligent. Please don't anthropomorphize.

As the Portal companion cube taught us, we will anthropomorphize whatever the hell we want.

Re: Models Are Getting Dumber on Purpose

#106

Post by locknitpicker »

everforward wrote: post #52
locknitpicker wrote: post #12

Earlier quoted context omitted.

> So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. Aren't you describing RAG or even MCP servers? Heck, nowadays you get that also with agent skills and specialized tool calling.

I don’t think so because those both live in the context window and as such pollute it when they’re not performing optimally. I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts. It sounds to me like some sort of “dynamic MoE” where you can add/create or remove experts on the fly. I think what you’re describing is the closest approximation we reasonabl…

> I don’t think so because those both live in the context window and as such pollute it when they’re not performing optimally.

There is nothing optimal about needing a few billion more parameters to be able to piece together probable answers that can be asserted by querying an oracle.

> I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts.

Those aren't free. The more parameters you add, the higher the computational cost required to train and prompt a mode.

And all for what? To piece together info that you can just query from a data source?

Re: Models Are Getting Dumber on Purpose

#107

Post by andai »

I wonder if Gemini tops QA due to Google Books in training data? You don't need to pirate LibGen if you started scanning all the books decades ago.

(In retrospect it looks like they were pretty forward thinking!)

Re: Models Are Getting Dumber on Purpose

#108

Post by andai »

COAGULOPATH wrote: post #85

This AI generated post (100% on Pangram) is pretty out of date. >On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%, so the best recall money can buy still misses half the questions. SimpleQA hasn't been updated in a long time. Gemini 2.5 Pro is a sixteen-month-old model, not "the best recall money can buy". >The part I find most promising is what this does t…

A true camper doesn't need to check Pangram, Jimbo. He goes by pure animal instinct!

Re: Models Are Getting Dumber on Purpose

#109

Post by nojokepoke »

kennywinker wrote: post #4

Ideally what I'd like to see is pluggable knowledge bases. So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. My model doesn't need to know a single line of python. Then when I want to research electronics components, I grab a 15B model of agentic research techniques…

The LLM is the reasoning engine that uses natural language. You’re describing skills. It’s the natural evolution.

Re: Models Are Getting Dumber on Purpose

#110

Post by andai »

COAGULOPATH wrote: post #85

This AI generated post (100% on Pangram) is pretty out of date. >On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%, so the best recall money can buy still misses half the questions. SimpleQA hasn't been updated in a long time. Gemini 2.5 Pro is a sixteen-month-old model, not "the best recall money can buy". >The part I find most promising is what this does t…

>>You can make any modern LLM explain its reasoning and find sources for its claims.

>The internet is full of wrong information and I cannot magically edit it to make it all correct, so this doesn't help me.

My favorite RAG experience was asking Bart (or whatever they were calling Gemini back then) an answer to a question I knew.

It gave me the opposite of the truth (as was common with LLMs at the time).

But weirdly, it had cited sources for this "fact."

I checked the sources. Two of them, both AI SEO slop.

In this moment, andai was enlightened...

Post reply on HN