Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Recent AI model progress feels mostly like bullshit

lesswrong.com

Post reply on HN
Print view View original

341–350 of 478 posts

Re: Recent AI model progress feels mostly like bullshit

#341

Post by alphabetting »

NiloCK wrote: post #337
levocardia wrote: post #232

Earlier quoted context omitted.

They retrained their model less than a week before its release, just to juice one particular nonstandard eval? Seems implausible. Models get 5x better at things all the time. Challenges like the Winograd schema have gone from impossible to laughably easy practically overnight. Ditto for "Rs in strawberry," ferrying animals across a river, overflowing wine glass, ...

I'm not generally inclined toward the "they are cheating cheaters" mindset, but I'll point out that fine tuning is not the same as retraining. It can be done cheaply and quickly. Models getting 5X better at things all the time is at least as easy to interpret as evidence of task-specific tuning than as breakthroughs in general ability, especially when the 'things being improved on' are published evals with history.

Google team said it was outside the training window fwiw

https://x.com/jack_w_rae/status/1907454713563426883

Re: Recent AI model progress feels mostly like bullshit

#343

Post by tim333 »

billforsternz wrote: post #238
InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

I asked Google "how many golf balls can fit in a Boeing 737 cabin" last week. The "AI" answer helpfully broke the solution into 4 stages; 1) A Boeing 737 cabin is about 3000 cubic metres [wrong, about 4x2x40 ~ 300 cubic metres] 2) A golf ball is about 0.000004 cubic metres [wrong, it's about 40cc = 0.00004 cubic metres] 3) 3000 / 0.000004 = 750,000 [wrong, it's 750,000,000] 4) We have to make an adjustment because se…

A lot of humans are similarly good at some stuff and bad at other things.

Looking up the math ability of the average American this is given as an example for the median (from https://www.wyliecomm.com/2021/11/whats-the-latest-u-s-numer...):

>Review a motor vehicle logbook with columns for dates of trip, odometer readings and distance traveled; then calculate trip expenses at 35 cents a mile plus $40 a day.

Which is ok but easier than golf balls in a 747 and hugely easier than USAMO.

Another question you could try from the easy math end is: Someone calculated the tariff rate for a country as (trade deficit)/(total imports from the country). Explain why this is wrong.

Re: Recent AI model progress feels mostly like bullshit

#344

Post by itchyjunk »

otabdeveloper4 wrote: post #326
bambax wrote: post #322

Earlier quoted context omitted.

How do you make homework assignments LLM-proof? There may be a huge business opportunity if that actually works, because LLMs are destroying education at a rapid pace.

You just (lol) need to give non-standard problems and demand students to provide reasoning and explanations along with the answer. Yeah, LLMs can "reason" too, but it's obvious when the output comes from an LLM here. (Yes, that's a lot of work for a teacher. Gone are the days when you could just assign reports as homework.)

Can you provide sample questions that are "LLM proof" ?

Re: Recent AI model progress feels mostly like bullshit

#345

Post by larodi »

AIPedant wrote: post #118
InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

Yes, here's the link: https://arxiv.org/abs/2503.21934v1 Anecdotally, I've been playing around with o3-mini on undergraduate math questions: it is much better at "plug-and-chug" proofs than GPT-4, but those problems aren't independently interesting, they are explicitly pedagogical. For anything requiring insight, it's either: 1) A very good answer that reveals the LLM has seen the problem before (e.g. naming the theo…

This is a paper by INSAIT researchers - a very young institute which hired most of its PHD staff only in the last 2 years, basically onboarding anyone who wanted to be part of it. They were waiving their BG-GPT on national TV in the country as a major breakthrough, while it was basically was a Mistral fine-tuned model, that was eventually never released to the public, nor the training set.

Not sure whether their (INSAIT's) agenda is purely scientific, as there's a lot of PR on linkedin by these guys, literally celebrating every PHD they get, which is at minimum very weird. I'd take anything they release with a grain of sand if not caution.

Re: Recent AI model progress feels mostly like bullshit

#346

Post by dimitri-vs »

mountainriver wrote: post #36
joelthelion wrote: post #25

I've used gemini 2.5 this weekend with aider and it was frighteningly good. It probably depends a lot on what you are using them for, and in general, I think it's still too early to say exactly where LLMs will lead us.

Yep, and what they are going in cursor either the agentic stuff is really game changing. People who can’t recognize this intentionally have their heads in the sand

I guess you haven't been on /r/cursor or forum.cursor.com lately?

"game changing" isn't exactly the sentiment there the last couple months.

Re: Recent AI model progress feels mostly like bullshit

#347

Post by mrweasel »

bluefirebrand wrote: post #149
iambateman wrote: post #39

The core point in this article is that the LLM wants to report _something_, and so it tends to exaggerate. It’s not very good at saying “no” or not as good as a programmer would hope. When you ask it a question, it tends to say yes. So while the LLM arms race is incrementally increasing benchmark scores, those improvements are illusory. The real challenge is that the LLM’s fundamentally want to seem agreeable, and th…

> The real challenge is that the LLM’s fundamentally want to seem agreeable, and that’s not improving LLMs fundamentally do not want to seem anything But the companies that are training them and making models available for professional use sure want them to seem agreeable

That sound reasonable to me, but the those companies forget that there's different types of agreeable. There's the LLM approach, similar to the coworker who will answer all your questions about .NET but not stop you from coding yourself into a corner, and then there's the "Let's sit down and review what it actually is that you're doing, because you're asking a fairly large number of disjoint questions right now".

I've dropped trying to use LLMs for anything, due to political convictions and because I don't feel like they are particularly useful for my line of work. Where I have tried to use various models in the past is for software development, and the common mistake I see the LLMs make is that they can't pick up on mistakes in my line of thinking, or won't point them out. Most of my problems are often down to design errors or thinking about a problem in a wrong way. The LLMs will never once tell me that what I'm trying to do is an indication of a wrong/bad design. There are ways to be agreeable and still point out problems with previously made decisions.

Re: Recent AI model progress feels mostly like bullshit

#348

Post by yahoozoo »

InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

LLMs are “next token” predictors. Yes, I realize that there’s a bit more to it and it’s not always just the “next” token, but at a very high level that’s what they are. So why are we so surprised when it turns out they can’t actually “do” math? Clearly the high benchmark scores are a result of the training sets being polluted with the answers.

Re: Recent AI model progress feels mostly like bullshit

#349

Post by dimitri-vs »

georgemcbay wrote: post #12
gundmc wrote: post #5

This was published the day before Gemini 2.5 was released. I'd be interested if they see any difference with that model. Anecdotally, that is the first model that really made me go wow and made a big difference for my productivity.

As someone who was wildly disappointed with the hype around Claude 3.7, Gemini 2.5 is easily the best programmer-assistant LLM available, IMO. But it still feels more like a small incremental improvement rather than a radical change, and I still feel its limitations constantly. Like... it gives me the sort of decent but uninspired solution I would expect it to generate without predictably walking me through a bunch o…

This is exactly my sentiment. Sonnet-3.5-latest was the perfect code companion: wrote just the right amount of okay quality code but its strength was it really tried to adhere to your instructions. sonnet-3.7 was the exact opposite, wrote waaay too much code and overengineered things like crazy while having very poor instruction adherence. Gemini 2.5 Pro is basically what I hoped sonnet-3.7 would be: follows instructions well but still softly opinionated, massive (usable) context window, fast response, more biased towards latest best practices and a up to date knowledge cutoff.

I'm wondering how much gemini 2.5 being "amazing" comes from sonnet-3.7 being such a disappointment.

Re: Recent AI model progress feels mostly like bullshit

#350

Post by otabdeveloper4 »

itchyjunk wrote: post #344
otabdeveloper4 wrote: post #326

Earlier quoted context omitted.

You just (lol) need to give non-standard problems and demand students to provide reasoning and explanations along with the answer. Yeah, LLMs can "reason" too, but it's obvious when the output comes from an LLM here. (Yes, that's a lot of work for a teacher. Gone are the days when you could just assign reports as homework.)

Can you provide sample questions that are "LLM proof" ?

It's not about being "LLM-proff", it's about teacher involvement in making up novel questions and grading attentively. There's no magic trick.
Post reply on HN