Recent AI model progress feels mostly like bullshit • Hacker News Board

Re: Recent AI model progress feels mostly like bullshit

#381

Post by hyperbovine » Mon, Apr 07, 2025, 2:56 PM UTC

InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

Is that really so surprising given what we know about how these models actually work? I feel vindicated on behalf of myself and all the other commenters who have been mercilessly downvoted over the past three years for pointing out the obvious fact that next token prediction != reasoning.

Re: Recent AI model progress feels mostly like bullshit

#382

Post by immibis » Mon, Apr 07, 2025, 2:57 PM UTC

jerf wrote: post #373

itchyjunk wrote: post #344

Earlier quoted context omitted.

Can you provide sample questions that are "LLM proof" ?

The models have moved on past this working reliably, but an example that I found in the early days of LLMs is asking it "Which is heavier, two pounds of iron or a pound of feathers?" You could very easily trick it into giving the answer about how they're both the same, because of the number of training instances of the well-known question about a pound of each that it encountered. You can still do this to the current…

"Draw a wine glass filled to the brim with wine" worked recently on image generators. They only knew about half-full wine glasses.

If you asked a multimodal system questions about the image it just generated, it would tell you the wine was almost overflowing out of the top of the glass.

But any trick prompt like this is going to start giving expected results once it gets well-known enough.

Late edit: Another one was the farmer/fox/chicken/cabbage/river problem, but you modify the problem in unexpected ways, by stating, for example, that the cabbage will eat the fox, or that the farmer can bring three items per trip. LLMs used to ignore your modifications and answer the original problem.

Re: Recent AI model progress feels mostly like bullshit

#383

Post by aoeusnth1 » Mon, Apr 07, 2025, 2:58 PM UTC

hyperbovine wrote: post #381

InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

Is that really so surprising given what we know about how these models actually work? I feel vindicated on behalf of myself and all the other commenters who have been mercilessly downvoted over the past three years for pointing out the obvious fact that next token prediction != reasoning.

2.5 pro scores 25%.

It’s just a much harder math benchmark which will fall by the end of next year just like all the others. You won’t be vindicated.

Re: Recent AI model progress feels mostly like bullshit

#384

Post by hyperbovine » Mon, Apr 07, 2025, 2:58 PM UTC

bambax wrote: post #322

otabdeveloper4 wrote: post #319

Earlier quoted context omitted.

Anecdotally: schoolkids are at the leading edge of LLM innovation, and nowadays all homework assignments are explicitly made to be LLM-proof. (Well, at least in my son's school. Yours might be different.) This effectively makes LLMs useless for education. (Also sours the next generation on LLMs in general, these things are extremely lame to the proverbial "kids these days".)

How do you make homework assignments LLM-proof? There may be a huge business opportunity if that actually works, because LLMs are destroying education at a rapid pace.

By giving pen and paper exams and telling your students that the only viable preparation strategy is doing the hw assignments themselves :)

Re: Recent AI model progress feels mostly like bullshit

#385

Post by hyperbovine » Mon, Apr 07, 2025, 2:59 PM UTC

aoeusnth1 wrote: post #383

hyperbovine wrote: post #381

Earlier quoted context omitted.

Is that really so surprising given what we know about how these models actually work? I feel vindicated on behalf of myself and all the other commenters who have been mercilessly downvoted over the past three years for pointing out the obvious fact that next token prediction != reasoning.

2.5 pro scores 25%. It’s just a much harder math benchmark which will fall by the end of next year just like all the others. You won’t be vindicated.

Bold claim! Let's see what that 25% is. I guarantee it is the portion of the exam which is trivially answerable if you have a stored database of all previous math exams ever written to consult.

Re: Recent AI model progress feels mostly like bullshit

#386

Post by aoeusnth1 » Mon, Apr 07, 2025, 3:02 PM UTC

billforsternz wrote: post #238

InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

I asked Google "how many golf balls can fit in a Boeing 737 cabin" last week. The "AI" answer helpfully broke the solution into 4 stages; 1) A Boeing 737 cabin is about 3000 cubic metres [wrong, about 4x2x40 ~ 300 cubic metres] 2) A golf ball is about 0.000004 cubic metres [wrong, it's about 40cc = 0.00004 cubic metres] 3) 3000 / 0.000004 = 750,000 [wrong, it's 750,000,000] 4) We have to make an adjustment because se…

2.5 pro nails each of these calculations. I don’t agree with Google’s decision to use a weak model in its search queries, but you can’t say progress on LLMs in bullshit as evidenced by a weak model no one thinks is close to SOTA.

Re: Recent AI model progress feels mostly like bullshit

#387

Post by hyperbovine » Mon, Apr 07, 2025, 3:04 PM UTC

sanxiyn wrote: post #299

TrackerFF wrote: post #298

Earlier quoted context omitted.

What would the average human score be? I.e. if you randomly sampled N humans to take those tests.

The average human score on USAMO (let alone IMO) is zero, of course. Source: I won medals at Korean Mathematical Olympiad.

This is a disappointing answer from an MO alum. Pick a quantile, any quantile...

Re: Recent AI model progress feels mostly like bullshit

#388

Post by sksxihve » Mon, Apr 07, 2025, 3:04 PM UTC

machiaweliczny wrote: post #379

sksxihve wrote: post #375

Earlier quoted context omitted.

The whole MCP hype really shows how much of AI is bullshit. These LLMs have consumed more API documentation than possible for a single human and still need software engineers to write glue layers so they can use the APIs.

Because it’s lossy compression. I also consumed a lot of books and even more movies and I don’t have good memory of it all. But some core facts and intuition from it.

AI is far better at regurgitating facts than me even if it's lossy compression but if someone gives me an api doc I can figure out how to use it without them writing a wrapper library around the parts that I need to use to solve whatever problem I'm working on.

Re: Recent AI model progress feels mostly like bullshit

#389

Post by JKCalhoun » Mon, Apr 07, 2025, 3:05 PM UTC

I've suggested (from my lowly layman vantage point) that an LLM has some idea of the fidelity of their response to a query — even if only in broad strokes like, "This answer is tracking with some high probability numbers" or "This answer has a very low correspondence with training data".

To that end the LLM could convey as much.

(Anecdotal, sorry:) I was using Claude (not paid) recently and noticed Claude hedging quite a bit when it had not before. Examples:

"Let me be careful about this response since we're discussing a very specific technical detail ..."

"Given how specific that technical detail is, I want to be transparent that while I aim to be accurate, I may hallucinate such precise historical specifications."

I confess my initial reaction was to ask ChatGPT since the answers are more self-assured, ha ha. So perhaps corporate AI are not likely to try and solve this problem of the LLM telling the user when it is on shaky ground. Bad for business.

Re: Recent AI model progress feels mostly like bullshit

#390

Post by antonvs » Mon, Apr 07, 2025, 3:07 PM UTC

ramesh31 wrote: post #376

lukev wrote: post #40

This is a bit of a meta-comment, but reading through the responses to a post like this is really interesting because it demonstrates how our collective response to this stuff is (a) wildly divergent and (b) entirely anecdote-driven. I have my own opinions, but I can't really say that they're not also based on anecdotes and personal decision-making heuristics. But some of us are going to end up right and some of us ar…

>"This is a bit of a meta-comment, but reading through the responses to a post like this is really interesting because it demonstrates how our collective response to this stuff is (a) wildly divergent and (b) entirely anecdote-driven." People having vastly different opinions on AI simply comes down to token usage. If you are using millions of tokens on a regular basis, you completely understand the revolutionary poin…

It's a tool and like all tools, it's sensitive to how you use it, and it's better for some purposes than others.

Someone who lacks experience, skill, training, or even the ability to evaluate results may try to use a tool and blame the tool when it doesn't give good results.

That said, the hype around LLMs certainly overstates their capabilities.