Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Recent AI model progress feels mostly like bullshit

lesswrong.com

Post reply on HN
Print view View original

361–370 of 478 posts

Re: Recent AI model progress feels mostly like bullshit

#361

Post by nucleogenesis »

bambax wrote: post #324
aezart wrote: post #294

Earlier quoted context omitted.

> I have seen much AI output which is extraordinary it's funny how one serious fail can impact my point of view so dramatically. I feel the same way. It's like discovering for the first time that magicians aren't doing "real" magic, just sleight of hand and psychological tricks. From that point on, it's impossible to be convinced that a future trick is real magic, no matter how impressive it seems. You know it's fake…

I think there is a big divide here. Every adult on earth knows magic is "fake", but some can still be amazed and entertained by it, while others find it utterly boring because it's fake, and the only possible (mildly) interesting thing about it is to try to figure out what the trick is. I'm in the second camp but find it kind of sad and often envy the people who can stay entertained even though they know better.

Idk I don’t think of it as fake - it’s creative fiction paired with sometimes highly skilled performance. I’ve learned a lot about how magic tricks work and I still love seeing performers do effects because it takes so much talent to, say, hold and hide 10 coins in your hands while showing them as empty or to shuffle a deck of cards 5x and have the audience cut it only to pull 4 aces off the top.

Re: Recent AI model progress feels mostly like bullshit

#362

Post by colonial »

> AI lab founders believe they are in a civilizational competition for control of the entire future lightcone, and will be made Dictator of the Universe if they succeed.

Bingo, but I'd argue this is only scratching the surface of how twisted things are.

A lot of the stuff these labs put out (see: Amodei's cult-like blog ramblings) reeks of what I call "sociopathic utopianism" - essentially, the logical extreme of ends-justified means, made worse in the context of AI labs by the singularity pseudo-religion.

They all truly believe that AGI/ASI is possible, imminent, and could lead to utopia... so achieving that goal will surely outweigh any unsavory acts they commit in the pursuit of it.

This is why I think it's possible OpenAI took out a hit on Suchir; getting bogged down in a legal battle could delay the arrival of their machine god messiah. Same for defrauding benchmarks - they just need a "few more rounds" of investor cash, and by the time those run out, they'll surely have AGI on lock!

Fools. I look forward to them all getting prison time.

Re: Recent AI model progress feels mostly like bullshit

#363

Post by swader999 »

billforsternz wrote: post #238
InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

I asked Google "how many golf balls can fit in a Boeing 737 cabin" last week. The "AI" answer helpfully broke the solution into 4 stages; 1) A Boeing 737 cabin is about 3000 cubic metres [wrong, about 4x2x40 ~ 300 cubic metres] 2) A golf ball is about 0.000004 cubic metres [wrong, it's about 40cc = 0.00004 cubic metres] 3) 3000 / 0.000004 = 750,000 [wrong, it's 750,000,000] 4) We have to make an adjustment because se…

It'll get it right next time because they'll hoover up the parent post.

Re: Recent AI model progress feels mostly like bullshit

#364

Post by Workaccount2 »

greenmartian wrote: post #268
billforsternz wrote: post #238

Earlier quoted context omitted.

I asked Google "how many golf balls can fit in a Boeing 737 cabin" last week. The "AI" answer helpfully broke the solution into 4 stages; 1) A Boeing 737 cabin is about 3000 cubic metres [wrong, about 4x2x40 ~ 300 cubic metres] 2) A golf ball is about 0.000004 cubic metres [wrong, it's about 40cc = 0.00004 cubic metres] 3) 3000 / 0.000004 = 750,000 [wrong, it's 750,000,000] 4) We have to make an adjustment because se…

Weird thing is, in Google AI Studio all their models—from the state-of-the-art Gemini 2.5Pro, to the lightweight Gemma 2—gave a roughly correct answer. Most even recognised the packing efficiency of spheres. But Google search gave me the exact same slop you mentioned. So whatever Search is using, they must be using their crappiest, cheapest model. It's nowhere near state of the art.

Google is shooting themselves in the foot with whatever model they use for search. It's probably a 2B or 4B model to keep up with demand, and man is it doing way more harm than good.

Re: Recent AI model progress feels mostly like bullshit

#365

Post by toddmorey »

bambax wrote: post #324
aezart wrote: post #294

Earlier quoted context omitted.

> I have seen much AI output which is extraordinary it's funny how one serious fail can impact my point of view so dramatically. I feel the same way. It's like discovering for the first time that magicians aren't doing "real" magic, just sleight of hand and psychological tricks. From that point on, it's impossible to be convinced that a future trick is real magic, no matter how impressive it seems. You know it's fake…

I think there is a big divide here. Every adult on earth knows magic is "fake", but some can still be amazed and entertained by it, while others find it utterly boring because it's fake, and the only possible (mildly) interesting thing about it is to try to figure out what the trick is. I'm in the second camp but find it kind of sad and often envy the people who can stay entertained even though they know better.

I think the problem-solving / want-to-be-engineer side of my brain lights up in that "how did he do that??" way. To me that's the fun of it... I immediately try to engineer my own solutions to what I just saw happen. So I guess I'm the first camp, but find trying to figure out the trick hugely interesting.

Re: Recent AI model progress feels mostly like bullshit

#366

Post by Workaccount2 »

sigmoid10 wrote: post #338
InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

>I'm incredibly surprised no one mentions this If you don't see anyone mentioning what you wrote that's not surprising at all, because you totally misunderstood the paper. The models didn't suddenly drop to 5% accuracy on math olympiad questions. Instead this paper came up with a human evaluation that looks at the whole reasoning process (instead of just the final answer) and their finding is that the "thoughts" of r…

On top of that, what the model prints out in the CoT window is not necessarily what the model is actually thinking. Anthropic just showed this in their paper from last week where they got models to cheat at a question by "accidentally" slipping them the answer, and the CoT had no mention of answer being slipped to them.

Re: Recent AI model progress feels mostly like bullshit

#367

Post by CamperBob2 »

senordevnyc wrote: post #265
billforsternz wrote: post #238

Earlier quoted context omitted.

I asked Google "how many golf balls can fit in a Boeing 737 cabin" last week. The "AI" answer helpfully broke the solution into 4 stages; 1) A Boeing 737 cabin is about 3000 cubic metres [wrong, about 4x2x40 ~ 300 cubic metres] 2) A golf ball is about 0.000004 cubic metres [wrong, it's about 40cc = 0.00004 cubic metres] 3) 3000 / 0.000004 = 750,000 [wrong, it's 750,000,000] 4) We have to make an adjustment because se…

Just tried with o3-mini-high and it came up with something pretty reasonable: https://chatgpt.com/share/67f35ae9-5ce4-800c-ba39-6288cb4685...

It's just the usual HN sport: ask a low-end, obsolete or unspecified model, get a bad answer, brag about how you "proved" AI is pointless hype, collect karma.

Edit: Then again, maybe they have a point, going by an answer I just got from Google's best current model ( https://g.co/gemini/share/374ac006497d ) I haven't seen anything that ridiculous from a leading-edge model for a year or more.

Re: Recent AI model progress feels mostly like bullshit

#368

Post by toddmorey »

throwawaymaths wrote: post #334
billforsternz wrote: post #238

Earlier quoted context omitted.

I asked Google "how many golf balls can fit in a Boeing 737 cabin" last week. The "AI" answer helpfully broke the solution into 4 stages; 1) A Boeing 737 cabin is about 3000 cubic metres [wrong, about 4x2x40 ~ 300 cubic metres] 2) A golf ball is about 0.000004 cubic metres [wrong, it's about 40cc = 0.00004 cubic metres] 3) 3000 / 0.000004 = 750,000 [wrong, it's 750,000,000] 4) We have to make an adjustment because se…

I've seen humans make exactly these sorts of mistakes?

As another commenter mentioned, LLMs tend to make these bad mistakes with enormous confidence. And because they represent SOTA technology (and can at times deliver incredible results), they have extra credence.

More than even filling the gaps in knowledge / skills, would be a huge advancement in AI for it to admit when it doesn't know the answer or is just wildly guessing.

Re: Recent AI model progress feels mostly like bullshit

#369

Post by xeromal »

itchyjunk wrote: post #344
otabdeveloper4 wrote: post #326

Earlier quoted context omitted.

You just (lol) need to give non-standard problems and demand students to provide reasoning and explanations along with the answer. Yeah, LLMs can "reason" too, but it's obvious when the output comes from an LLM here. (Yes, that's a lot of work for a teacher. Gone are the days when you could just assign reports as homework.)

Can you provide sample questions that are "LLM proof" ?

Part of the proof is knowing your students and forcing an answer that will rat out whether they used an LLM. There is no universal question and it requires personal knowledge of each student. You're looking for something that doesn't exist.

Re: Recent AI model progress feels mostly like bullshit

#370

Post by apercu »

AIPedant wrote: post #118
InkCanon wrote: post #75

The biggest story in AI was released a few weeks ago but was given little attention: on the recent USAMO, SOTA models scored on average 5% (IIRC, it was some abysmal number). This is despite them supposedly having gotten 50%, 60% etc performance on IMO questions. This massively suggests AI models simply remember the past results, instead of actually solving these questions. I'm incredibly surprised no one mentions th…

Yes, here's the link: https://arxiv.org/abs/2503.21934v1 Anecdotally, I've been playing around with o3-mini on undergraduate math questions: it is much better at "plug-and-chug" proofs than GPT-4, but those problems aren't independently interesting, they are explicitly pedagogical. For anything requiring insight, it's either: 1) A very good answer that reveals the LLM has seen the problem before (e.g. naming the theo…

In my experience LLMs can't get basic western music theory right, there's no way I would use an LLM for something harder than that.
Post reply on HN