Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Recent AI model progress feels mostly like bullshit

lesswrong.com

Post reply on HN
Print view View original

251–260 of 478 posts

Re: Recent AI model progress feels mostly like bullshit

#251

Post by famouswaffles »

hatefulmoron wrote: post #245
famouswaffles wrote: post #240

Earlier quoted context omitted.

I mentioned it because it's the best example. One example is enough to disprove the "not capable of". There are other examples too. >I think AstroBen was pointing out that LLMs, despite having the ability to solve some very impressive mathematics and programming tasks, don't seem to generalize their reasoning abilities to a domain like chess. That's surprising, isn't it? Not really. The LLMs play chess like they have…

> One example is enough to disprove the "not capable of" nonsense. There are other examples too. Gotcha, fair enough. Throw enough chess data in during training, I'm sure they'd be pretty good at chess. I don't really understand what you're trying to say in your next paragraph. LLMs surely have plenty of training data to be familiar with the rules of chess. They also purportedly have the reasoning skills to use their…

Gradient descent is a dumb optimizer. LLM training is not at all like a human reading a book and more like evolution tuning adaptations over centuries. You would not expect either process to be aware of anything they are converging towards. So having lots of books that talk about chess in training will predictably just return a model that knows how to talk about chess really well. I'm not surprised they may know how to talk about the rules but play them poorly.

And that post had a follow-up. Post-training messing things up could well be the issue seeing the impact even a little more examples and/or regurgitation made. https://dynomight.net/more-chess/

Re: Recent AI model progress feels mostly like bullshit

#252

Post by malingo »

iambateman wrote: post #39

The core point in this article is that the LLM wants to report _something_, and so it tends to exaggerate. It’s not very good at saying “no” or not as good as a programmer would hope. When you ask it a question, it tends to say yes. So while the LLM arms race is incrementally increasing benchmark scores, those improvements are illusory. The real challenge is that the LLM’s fundamentally want to seem agreeable, and th…

"when you ask him anything, he never answers 'no' -- he just yesses you to death and then he takes your dough"

Re: Recent AI model progress feels mostly like bullshit

#253

Post by cma »

InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

OpenAI told how they removed it for GPT-4 in its release paper: only exact string matches. So all discussion of bar exam questions from memory on test taking forums etc., that wouldnn't exactly match, made it in.

Re: Recent AI model progress feels mostly like bullshit

#254

Post by cma »

levocardia wrote: post #232
selcuka wrote: post #228

Earlier quoted context omitted.

> within a week How do we know that Gemini 2.5 wasn't specifically trained or fine-tuned with the new questions? I don't buy that a new model could suddenly score 5 times better than the previous state-of-the-art models.

They retrained their model less than a week before its release, just to juice one particular nonstandard eval? Seems implausible. Models get 5x better at things all the time. Challenges like the Winograd schema have gone from impossible to laughably easy practically overnight. Ditto for "Rs in strawberry," ferrying animals across a river, overflowing wine glass, ...

They could have rlhfed or finetuned on user thumbs up responses, which could include users who took the test and asked it to explain problems after

Re: Recent AI model progress feels mostly like bullshit

#255

Post by alexlikeits1999 »

InkCanon wrote: post #248
sanxiyn wrote: post #199

Earlier quoted context omitted.

Nope, no LLMs reported 50~60% performance on IMO, and SOTA LLMs scoring 5% on USAMO is expected. For 50~60% performance on IMO, you are thinking of AlphaProof, but AlphaProof is not a LLM. We don't have the full paper yet, but clearly AlphaProof is a system built on top of LLM with lots of bells and whistles, just like AlphaFold is.

o1 reportedly got 83% on IMO, and 89th percentile on Codeforces. https://openai.com/index/learning-to-reason-with-llms/ The paper tested it on o1-pro as well. Correct me if I'm getting some versioning mixed up here.

I've gone through the link you posted and the o1 system card and can't see any reference to IMO. Are you sure they were referring to IMO or were they referring to AIME?

Re: Recent AI model progress feels mostly like bullshit

#256

Post by cma »

hatefulmoron wrote: post #227
famouswaffles wrote: post #223

Earlier quoted context omitted.

That might be overstating it, at least if you mean it to be some unreplicable feat. Small models have been trained that play around 1200 to 1300 on the eleuther discord. And there's this grandmaster level transformer - https://arxiv.org/html/2402.04494v1 Open AI, Anthropic and the like simply don't care much about their LLMs playing chess. That or post training is messing things up.

> That might be overstating it, at least if you mean it to be some unreplicable feat. I mean, surely there's a reason you decided to mention 3.5 turbo instruct and not.. 3.5 turbo? Or any other model? Even the ones that came after? It's clearly a big outlier, at least when you consider "LLMs" to be a wide selection of recent models. If you're saying that LLMs/transformer models are capable of being trained to play ch…

Reasoning training causes some about of catastrophic forgetting, so unlikely they burn that on mixing in chess puzzles if they want a commercial product, unless it somehow transfers well to other reasoning problems broadly cared about.

Re: Recent AI model progress feels mostly like bullshit

#257

Post by AstroBen »

famouswaffles wrote: post #242
AstroBen wrote: post #226

Earlier quoted context omitted.

My point wasn't chess specific or that they couldn't have specific training for it. It was a more general "here is something that LLMs clearly aren't being trained for currently, but would also be solvable through reasoning skills" Much in the same way a human who only just learnt the rules but 0 strategy would very, very rarely lose here These companies are shouting that their products are passing incredibly hard ex…

>If an LLM is trained for chess then its performance would just come from memorization, not any kind of "reasoning". If you think you can play chess at that level over that many games and moves with memorization then i don't know what to tell you except that you're wrong. It's not possible so let's just get that out of the way. >These companies are shouting that their products are passing incredibly hard exams, solvi…

> Why doesn't it?

It was surprising to me because I would have expected if there was reasoning ability then it would translate across domains at least somewhat, but yeah what you say makes sense. I'm thinking of it in human terms

Re: Recent AI model progress feels mostly like bullshit

#258

Post by famouswaffles »

akoboldfrying wrote: post #250
levocardia wrote: post #232

Earlier quoted context omitted.

They retrained their model less than a week before its release, just to juice one particular nonstandard eval? Seems implausible. Models get 5x better at things all the time. Challenges like the Winograd schema have gone from impossible to laughably easy practically overnight. Ditto for "Rs in strawberry," ferrying animals across a river, overflowing wine glass, ...

>one particular nonstandard eval A particular nonstandard eval that is currently top comment on this HN thread, due to the fact that, unlike every other eval out there, LLMs score badly on it? Doesn't seem implausible to me at all. If I was running that team, I would be "Drop what you're doing, boys and girls, and optimise the hell out of this test! This is our differentiator!"

It's implausible that fine-tuning of a premier model would have anywhere near that turn around time. Even if they wanted to and had no qualms doing so, it's not happening anywhere near that fast.

Re: Recent AI model progress feels mostly like bullshit

#259

Post by airstrike »

bglazer wrote: post #156
InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

Yeah I’m a computational biology researcher. I’m working on a novel machine learning approach to inferring cellular behavior. I’m currently stumped why my algorithm won’t converge. So, I describe the mathematics to ChatGPT-o3-mini-high to try to help reason about what’s going on. It was almost completely useless. Like blog-slop “intro to ML” solutions and ideas. It ignores all the mathematical context, and zeros in o…

I tend to prefer Claude over all things ChatGPT so maybe give the latest model a try -- although in some way I feel like 3.7 is a step down from the prior 3.5 model

Re: Recent AI model progress feels mostly like bullshit

#260

Post by droopyEyelids »

otterley wrote: post #175
freehorse wrote: post #128

Earlier quoted context omitted.

There is nothing wrong with sharing anecdotal experiences. Reading through anecdotal experiences here can help understand how one's own experience are relatable or not. Moreover, if I have X experience it could help to know if it is because of me doing sth wrong that others have figured out. Furthermore, as we are talking about actual impact of LLMs, as is the point of the article, a bunch of anecdotal experiences ma…

Indeed, there’s nothing at all wrong with sharing anecdotes. The problem is when people make broad assumptions and conclusions based solely on personal experience, which unfortunately happens all too often. Doing so is wired into our brains, though, and we have to work very consciously to intercept our survival instincts.

I think you might be caught up in a bit of the rationalist delusion.

People -only!- draw conclusions based on personal experience. At best you have personal experience with truly objective evidence gathered in a statistically valid manner.

But that only happens in a few vanishingly rare circumstances here on earth. And wherever it happens, people are driven to subvert the evidence gathering process.

Often “working against your instincts” to be more rational only means more time spent choosing which unreliable evidence to concoct a belief from.

Post reply on HN