Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Recent AI model progress feels mostly like bullshit

lesswrong.com

Post reply on HN
Print view View original

331–340 of 478 posts

Re: Recent AI model progress feels mostly like bullshit

#331

Post by lm28469 »

stavros wrote: post #64
aerhardt wrote: post #34

My mom told me yesterday that Paul Newman had massive problems with alcohol. I was somewhat skeptical, so this morning I asked ChatGPT a very simple question: "Is Paul Newman known for having had problems with alcohol?" All of the models up to o3-mini-high told me he had no known problems. Here's o3-mini-high's response: "Paul Newman is not widely known for having had problems with alcohol. While he portrayed charact…

LLMs aren't good at being search engines, they're good at understanding things. Put an LLM on top of a search engine, and that's the appropriate tool for this use case. I guess the problem with LLMs is that they're too usable for their own good, so people don't realizing that they can't perfectly know all the trivia in the world, exactly the same as any human.

> Put an LLM on top of a search engine, and that's the appropriate tool for this use case.

Hm nope, now that the web if flooded by LLM generated content it's game over. I can't tell how many times I almost got fooled by recipes &co which seem legit at first but are utter non sense. And now we're feeding that garbage back to where it came from

Re: Recent AI model progress feels mostly like bullshit

#332

Post by throwawayffffas »

HarHarVeryFunny wrote: post #43

The disconnect between improved benchmark results and lack of improvement on real world tasks doesn't have to imply cheating - it's just a reflection of the nature of LLMs, which at the end of the day are just prediction systems - these are language models, not cognitive architectures built for generality. Of course, if you train an LLM heavily on narrow benchmark domains then its prediction performance will improve…

In my view as well it's not really cheating, it's just over fitting.

If a model doesn't do good in the benchmarks it will either be retrained until it does or you won't hear about it.

Re: Recent AI model progress feels mostly like bullshit

#333

Post by stavros »

lm28469 wrote: post #331
stavros wrote: post #64

Earlier quoted context omitted.

LLMs aren't good at being search engines, they're good at understanding things. Put an LLM on top of a search engine, and that's the appropriate tool for this use case. I guess the problem with LLMs is that they're too usable for their own good, so people don't realizing that they can't perfectly know all the trivia in the world, exactly the same as any human.

> Put an LLM on top of a search engine, and that's the appropriate tool for this use case. Hm nope, now that the web if flooded by LLM generated content it's game over. I can't tell how many times I almost got fooled by recipes &co which seem legit at first but are utter non sense. And now we're feeding that garbage back to where it came from

That's not relevant to what we're discussing, though.

Re: Recent AI model progress feels mostly like bullshit

#334

Post by throwawaymaths »

billforsternz wrote: post #238
InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

I asked Google "how many golf balls can fit in a Boeing 737 cabin" last week. The "AI" answer helpfully broke the solution into 4 stages; 1) A Boeing 737 cabin is about 3000 cubic metres [wrong, about 4x2x40 ~ 300 cubic metres] 2) A golf ball is about 0.000004 cubic metres [wrong, it's about 40cc = 0.00004 cubic metres] 3) 3000 / 0.000004 = 750,000 [wrong, it's 750,000,000] 4) We have to make an adjustment because se…

I've seen humans make exactly these sorts of mistakes?

Re: Recent AI model progress feels mostly like bullshit

#335

Post by prawn »

Sunspark wrote: post #264
billforsternz wrote: post #238

Earlier quoted context omitted.

I asked Google "how many golf balls can fit in a Boeing 737 cabin" last week. The "AI" answer helpfully broke the solution into 4 stages; 1) A Boeing 737 cabin is about 3000 cubic metres [wrong, about 4x2x40 ~ 300 cubic metres] 2) A golf ball is about 0.000004 cubic metres [wrong, it's about 40cc = 0.00004 cubic metres] 3) 3000 / 0.000004 = 750,000 [wrong, it's 750,000,000] 4) We have to make an adjustment because se…

It's fascinating to me when you tell one that you'd like to see translated passages of work from authors who never have written or translated the item in question, especially if they passed away before the piece was written. The AI will create something for you and tell you it was them.

"That's impossible because..."

"Good point! Blah blah blah..."

Absolutely shameless!

Re: Recent AI model progress feels mostly like bullshit

#336

Post by ponorin »

gundmc wrote: post #5

This was published the day before Gemini 2.5 was released. I'd be interested if they see any difference with that model. Anecdotally, that is the first model that really made me go wow and made a big difference for my productivity.

There's somehow this belief that "newer models will disprove " despite the "newer" models being... just a scaled-up version of a previous model, or some anciliary features tacked on. An LLM is an LLM is an LLM: I'll believe it when I see otherwise.

Re: Recent AI model progress feels mostly like bullshit

#337

Post by NiloCK »

levocardia wrote: post #232
selcuka wrote: post #228

Earlier quoted context omitted.

> within a week How do we know that Gemini 2.5 wasn't specifically trained or fine-tuned with the new questions? I don't buy that a new model could suddenly score 5 times better than the previous state-of-the-art models.

They retrained their model less than a week before its release, just to juice one particular nonstandard eval? Seems implausible. Models get 5x better at things all the time. Challenges like the Winograd schema have gone from impossible to laughably easy practically overnight. Ditto for "Rs in strawberry," ferrying animals across a river, overflowing wine glass, ...

I'm not generally inclined toward the "they are cheating cheaters" mindset, but I'll point out that fine tuning is not the same as retraining. It can be done cheaply and quickly.

Models getting 5X better at things all the time is at least as easy to interpret as evidence of task-specific tuning than as breakthroughs in general ability, especially when the 'things being improved on' are published evals with history.

Re: Recent AI model progress feels mostly like bullshit

#338

Post by sigmoid10 »

InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

>I'm incredibly surprised no one mentions this

If you don't see anyone mentioning what you wrote that's not surprising at all, because you totally misunderstood the paper. The models didn't suddenly drop to 5% accuracy on math olympiad questions. Instead this paper came up with a human evaluation that looks at the whole reasoning process (instead of just the final answer) and their finding is that the "thoughts" of reasoning models are not sufficiently human understandable or rigorous (at least for expert mathematicians). This is something that was already well known, because "reasoning" is essentially CoT prompting baked into normal responses. But the empirics also tell us it greatly helps for final outputs nonetheless.

Re: Recent AI model progress feels mostly like bullshit

#339

Post by numa7numa7 »

My lived experience is that unless there's some new breakthrough's AI is more akin to a drill to replace a hammer than a tractor to replace the plow or a printing press.

Maybe any AI experts can elaborate on this but it seems there's a limit to the fundamental underlying model of the LLM architecture of transformers and tokens.

LLM's are amazing but we might need something more or some new paradigm to push us towards true AGI.

Re: Recent AI model progress feels mostly like bullshit

#340

Post by posterguy »

mustyoshi wrote: post #244
posterguy wrote: post #225

Earlier quoted context omitted.

Counterpoint: Paul Newman was absolutely a famous drunk, as evidenced by this Wikipedia page.* Any query for "paul newman alcohol" online will return dozens of reputable sources on the topic. Your post is easily interpretable as handwaving apologetics, and it gives big "Its the children who are wrong" energy. * https://en.wikipedia.org/wiki/Newman_Day

"Paul Newman alcohol" is just showing you results where those words are all present, it's not really implying how widely known it is.

What are you, an LLM? Look at the results of the first twenty hits and come back, then tell me that they don't speak to that specific issue.
Post reply on HN