Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Recent AI model progress feels mostly like bullshit

lesswrong.com

Post reply on HN
Print view View original

451–460 of 478 posts

Re: Recent AI model progress feels mostly like bullshit

#451

Post by airstrike »

pdimitar wrote: post #442
airstrike wrote: post #259

Earlier quoted context omitted.

I tend to prefer Claude over all things ChatGPT so maybe give the latest model a try -- although in some way I feel like 3.7 is a step down from the prior 3.5 model

What do you find inferior in 3.7 compared to 3.5 btw? I only recently started using Claude so I don't have a point of reference.

It's hard to say, super subjective. It's just wrong more often and sometimes it goes off in tangents wrt. what I asked. Also I might ask a question and it starts coding an entire React project. Every once in a while it will literally max out its response tokens because it can't stop writing code.

Just feels less "stable" or "tight" overall.

Re: Recent AI model progress feels mostly like bullshit

#452

Post by code_for_monkey »

apercu wrote: post #370
AIPedant wrote: post #118

Earlier quoted context omitted.

Yes, here's the link: https://arxiv.org/abs/2503.21934v1 Anecdotally, I've been playing around with o3-mini on undergraduate math questions: it is much better at "plug-and-chug" proofs than GPT-4, but those problems aren't independently interesting, they are explicitly pedagogical. For anything requiring insight, it's either: 1) A very good answer that reveals the LLM has seen the problem before (e.g. naming the theo…

In my experience LLMs can't get basic western music theory right, there's no way I would use an LLM for something harder than that.

music theory is a really good test because in my experience the AI is extremely bad at it

Re: Recent AI model progress feels mostly like bullshit

#453

Post by pdimitar »

airstrike wrote: post #451
pdimitar wrote: post #442

Earlier quoted context omitted.

What do you find inferior in 3.7 compared to 3.5 btw? I only recently started using Claude so I don't have a point of reference.

It's hard to say, super subjective. It's just wrong more often and sometimes it goes off in tangents wrt. what I asked. Also I might ask a question and it starts coding an entire React project. Every once in a while it will literally max out its response tokens because it can't stop writing code. Just feels less "stable" or "tight" overall.

I see. I have a similar feeling; as if they made it to quickly force you to pay (quickly maxing out one conversation in my case). I'm quite cynical and paranoid in this regard and I try hard not to be ruled by those two... but I can't shake the feeling that they're right this time.

Re: Recent AI model progress feels mostly like bullshit

#454

Post by airstrike »

pdimitar wrote: post #453
airstrike wrote: post #451

Earlier quoted context omitted.

It's hard to say, super subjective. It's just wrong more often and sometimes it goes off in tangents wrt. what I asked. Also I might ask a question and it starts coding an entire React project. Every once in a while it will literally max out its response tokens because it can't stop writing code. Just feels less "stable" or "tight" overall.

I see. I have a similar feeling; as if they made it to quickly force you to pay (quickly maxing out one conversation in my case). I'm quite cynical and paranoid in this regard and I try hard not to be ruled by those two... but I can't shake the feeling that they're right this time.

I hear you but FWIW I don't think it's on purpose as it feels like an inferior product to me as a paid user

Re: Recent AI model progress feels mostly like bullshit

#455

Post by dsign »

pdimitar wrote: post #446
dsign wrote: post #301

Earlier quoted context omitted.

You want to block subjectivity? Write some formulas. There are three questions to consider: a) Have we, without any reasonable doubt, hit a wall for AI development? Emphasis on "reasonable doubt". There is no reasonable doubt that the Earth is roughly spherical. That level of certainty. b) Depending on your answer for (a), the next question to consider is if we the humans have motivations to continue developing AI. c…

A lot of people judge by the lack of their desired outcome. Calling that fear and denial is disingenuous and unfair.

That's actually a valid point. I stand corrected.

Re: Recent AI model progress feels mostly like bullshit

#456

Post by summerlight »

fire_lake wrote: post #441
summerlight wrote: post #300

Earlier quoted context omitted.

AFAIK they got 15% of unseen queries everyday, so it might be not very simple to design an effective cache layer on that. Semantic-aware clustering of natural language queries and projecting them into a cache-able low rank dimension is a non-trivial problem. Of course, LLM can effectively solve that, but then what's the point of using cache when you need LLM for clustering queries...

Not a search engineer, but wouldn’t a cache lookup to a previous LLM result be faster than a conventional free text search over the indexed websites? Seems like this could save money whilst delivering better results?

Yes, that's what Google's doing for AI overview IIUC. From what I've seen from my experiences, this is working okay and improving over time but not close to perfection. The results are stale for developing stories, some bad results are kept there for a long time, effectively same queries are returning different caches etc etc...

Re: Recent AI model progress feels mostly like bullshit

#457

Post by motorest »

apercu wrote: post #370
AIPedant wrote: post #118

Earlier quoted context omitted.

Yes, here's the link: https://arxiv.org/abs/2503.21934v1 Anecdotally, I've been playing around with o3-mini on undergraduate math questions: it is much better at "plug-and-chug" proofs than GPT-4, but those problems aren't independently interesting, they are explicitly pedagogical. For anything requiring insight, it's either: 1) A very good answer that reveals the LLM has seen the problem before (e.g. naming the theo…

In my experience LLMs can't get basic western music theory right, there's no way I would use an LLM for something harder than that.

> In my experience LLMs can't get basic western music theory right, there's no way I would use an LLM for something harder than that.

This take is completely oblivious, and frankly sounds like a desperate jab. There are a myriad of activities whose core requirement is a) derive info from a complex context which happens to be supported by a deep and plentiful corpus, b) employ glorified template and rule engines.

LLMs excel at what might be described as interpolating context following input and output in natural language. As in a chatbot that is extensivey trained in domain-specific tasks, which can also parse and generate content. There is absolutely zero lines of intellectual work that do not benefit extensively from this sort of tool. Zero.

Re: Recent AI model progress feels mostly like bullshit

#458

Post by motorest »

otabdeveloper4 wrote: post #319
AIPedant wrote: post #118

Earlier quoted context omitted.

Yes, here's the link: https://arxiv.org/abs/2503.21934v1 Anecdotally, I've been playing around with o3-mini on undergraduate math questions: it is much better at "plug-and-chug" proofs than GPT-4, but those problems aren't independently interesting, they are explicitly pedagogical. For anything requiring insight, it's either: 1) A very good answer that reveals the LLM has seen the problem before (e.g. naming the theo…

Anecdotally: schoolkids are at the leading edge of LLM innovation, and nowadays all homework assignments are explicitly made to be LLM-proof. (Well, at least in my son's school. Yours might be different.) This effectively makes LLMs useless for education. (Also sours the next generation on LLMs in general, these things are extremely lame to the proverbial "kids these days".)

> This effectively makes LLMs useless for education.

No. You're only arguing LLMs are useless at regurgitating homework assignments to allow students to avoid doing it.

The point of education is not mindless doing homework.

Re: Recent AI model progress feels mostly like bullshit

#459

Post by namaria »

squiggleblaz wrote: post #359
mrweasel wrote: post #347

Earlier quoted context omitted.

That sound reasonable to me, but the those companies forget that there's different types of agreeable. There's the LLM approach, similar to the coworker who will answer all your questions about .NET but not stop you from coding yourself into a corner, and then there's the "Let's sit down and review what it actually is that you're doing, because you're asking a fairly large number of disjoint questions right now". I'v…

I think it's your responsibility to control the LLM. Sometimes, I worry that I'm beginning to code myself into a corner, and I ask if this is the dumbest idea it's ever heard and it says there might be a better way to do it. Sometimes I'm totally sceptical and ask that question first thing. (Usually it hallucinates when I'm being really obtuse though, and in a bad case that's the first time I notice it.)

> I think it's your responsibility to control the LLM.

Yes. The issue here is control and NLP is a poor interface to exercise control over the computer. Code on the other hand is a great way. That is the whole point of skepticism around LLM in software development.

Re: Recent AI model progress feels mostly like bullshit

#460

Post by namaria »

boesboes wrote: post #393
iambateman wrote: post #39

The core point in this article is that the LLM wants to report _something_, and so it tends to exaggerate. It’s not very good at saying “no” or not as good as a programmer would hope. When you ask it a question, it tends to say yes. So while the LLM arms race is incrementally increasing benchmark scores, those improvements are illusory. The real challenge is that the LLM’s fundamentally want to seem agreeable, and th…

This rings true. What I notice is that the longer i let Claude work on some code for instance, the more bullshit it invents. I usually can delete about 50-60% of the code & tests it came up with. And when you ask it to 'just write a test' 50/50 it will try to run it, fail on some trivial issues, delete 90% of your test code and start to loop deeper and deeper into the rabit hole of it's own halliciations. Or maybe I…

> Or maybe I just suck at prompting hehe

Every time someone argues for the utility of LLMs in software development by saying you need to be better at prompting, or add more rules for the LLM on the repository, they are making an argument against using NLP in software development.

The whole point of code is that it is a way to be very specific and exact and to exercise control over the computer behavior. The entire value proposition of using an LLM is that it is easier because you don't need to be so specific and exact. If then you say you need to be more specific and exact with the prompting, you are slowly getting at the fact that using NLP for coding is a bad idea.

Post reply on HN