Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Recent AI model progress feels mostly like bullshit

lesswrong.com

Post reply on HN
Print view View original

401–410 of 478 posts

Re: Recent AI model progress feels mostly like bullshit

#401

Post by aezart »

bambax wrote: post #324
aezart wrote: post #294

Earlier quoted context omitted.

> I have seen much AI output which is extraordinary it's funny how one serious fail can impact my point of view so dramatically. I feel the same way. It's like discovering for the first time that magicians aren't doing "real" magic, just sleight of hand and psychological tricks. From that point on, it's impossible to be convinced that a future trick is real magic, no matter how impressive it seems. You know it's fake…

I think there is a big divide here. Every adult on earth knows magic is "fake", but some can still be amazed and entertained by it, while others find it utterly boring because it's fake, and the only possible (mildly) interesting thing about it is to try to figure out what the trick is. I'm in the second camp but find it kind of sad and often envy the people who can stay entertained even though they know better.

It's still entertaining, that's true. I like magic tricks.

The point is the analogy to LLMs. A lot of people are very optimistic about their capabilities, while other people who have "seen behind the curtain" are skeptical, and feel that the fundamental flaws are still there even if they're better-hidden.

Re: Recent AI model progress feels mostly like bullshit

#402

Post by consumer451 »

behnamoh wrote: post #397
consumer451 wrote: post #395

Earlier quoted context omitted.

I would go even further than TFA. In my personal experience using Windsurf daily, Sonnet 3.5 is still my preferred model. 3.7 makes many more changes that I did not ask for, often breaking things. This is an issue with many models, but it got worse with 3.7.

3.7 is like a wild horse. you really must ground it with clear instructions. it sucks that it doesn't automatically know that but it's tameable.

Could you share any successful prompting techniques for grounding 3.7, even just a project-specific example?

Re: Recent AI model progress feels mostly like bullshit

#403

Post by AstroBen »

famouswaffles wrote: post #261
AstroBen wrote: post #257

Earlier quoted context omitted.

> Why doesn't it? It was surprising to me because I would have expected if there was reasoning ability then it would translate across domains at least somewhat, but yeah what you say makes sense. I'm thinking of it in human terms

Transfer Learning during LLM training tends to be 'broader' than that. Like how - Training LLMs on code makes them solve reasoning problems better - Training Language Y alongside X makes them much better at Y than if they were trained on language Y alone and so on. Probably because well gradient descent is a dumb optimizer and training is more like evolution than a human reading a book. Also, there is something genui…

It seems to be fairly nuanced in how abilities transfer: https://arxiv.org/html/2310.16937v2

Very hard for me to wrap my head around the idea that an LLM being able to discuss, even perhaps teach high level chess strategy wouldn't transfer at all to its playing performance

Re: Recent AI model progress feels mostly like bullshit

#404

Post by waffletower »

apercu wrote: post #370
AIPedant wrote: post #118

Earlier quoted context omitted.

Yes, here's the link: https://arxiv.org/abs/2503.21934v1 Anecdotally, I've been playing around with o3-mini on undergraduate math questions: it is much better at "plug-and-chug" proofs than GPT-4, but those problems aren't independently interesting, they are explicitly pedagogical. For anything requiring insight, it's either: 1) A very good answer that reveals the LLM has seen the problem before (e.g. naming the theo…

In my experience LLMs can't get basic western music theory right, there's no way I would use an LLM for something harder than that.

While I may be mistaken, but I don't believe that LLMs are trained on a large corpus of machine readable music representations, which would arguably be crucial to strong performance in common practice music theory. I would also surmise that most music theory related datasets largely arrive without musical representations altogether. A similar problem exists for many other fields, particularly mathematics, but it is much more profitable to invest the effort to span such representation gaps for them. I would not gauge LLM generality on music theory performance, when its niche representations are likely unavailable in training and it is widely perceived as having miniscule economic value.

Re: Recent AI model progress feels mostly like bullshit

#405

Post by maccard »

sksxihve wrote: post #375
maccard wrote: post #22

My experience as someone who uses LLMs and a coding assist plugin (sometimes), but is somewhat bearish on AI is that GPT/Claude and friends have gotten worse in the last 12 months or so, and local LLMs have gone from useless to borderline functional but still not really usable for day to day. Personally, I think the models are “good enough” that we need to start seeing the improvements in tooling and applications tha…

The whole MCP hype really shows how much of AI is bullshit. These LLMs have consumed more API documentation than possible for a single human and still need software engineers to write glue layers so they can use the APIs.

I don't think I agree, entirely.

The problem is that up until _very_ recently, it's been possible to get LLMs to generate interesting and exciting results (as a result of all the API documentation and codebases they've inhaled), but it's been very hard to make that usable. I think we need to be able to control the output format of the LLMs in a better way before we can work on what's in the output. I don't konw if MCP is the actual solution to that, but it's certainly an attempt at it...

Re: Recent AI model progress feels mostly like bullshit

#406

Post by bambax »

hyperbovine wrote: post #384
bambax wrote: post #322

Earlier quoted context omitted.

How do you make homework assignments LLM-proof? There may be a huge business opportunity if that actually works, because LLMs are destroying education at a rapid pace.

By giving pen and paper exams and telling your students that the only viable preparation strategy is doing the hw assignments themselves :)

You wish. I used to think that too. But it turns out, nowadays, every single exam in person is done with a phone hidden somewhere, with various efficiency, and you can't really strip students before they enter the room.

Some teachers try to collect the phones beforehand, but then students simply give out older phones and keep their active ones with them.

You could try to verify that the phones they're giving out are working by calling them, but that would take an enormous amount of time and it's impractical for simple exams.

We really have no idea how much AI is ruining education right now.

Re: Recent AI model progress feels mostly like bullshit

#407

Post by maccard »

sksxihve wrote: post #388
machiaweliczny wrote: post #379

Earlier quoted context omitted.

Because it’s lossy compression. I also consumed a lot of books and even more movies and I don’t have good memory of it all. But some core facts and intuition from it.

AI is far better at regurgitating facts than me even if it's lossy compression but if someone gives me an api doc I can figure out how to use it without them writing a wrapper library around the parts that I need to use to solve whatever problem I'm working on.

> but if someone gives me an api doc I can figure out how to use it without them writing a wrapper library around the parts that I need to use to solve whatever problem I'm working on.

I think this is where AI is faling short hugely. AI _should_ be able to integrate with IDEs and tooling (e.g. LSP, Treesitter, Editorconfig) to make sure that it's contextually doing the right thin.

But it's not.

Re: Recent AI model progress feels mostly like bullshit

#408

Post by lukev »

dsign wrote: post #301
lukev wrote: post #40

This is a bit of a meta-comment, but reading through the responses to a post like this is really interesting because it demonstrates how our collective response to this stuff is (a) wildly divergent and (b) entirely anecdote-driven. I have my own opinions, but I can't really say that they're not also based on anecdotes and personal decision-making heuristics. But some of us are going to end up right and some of us ar…

You want to block subjectivity? Write some formulas. There are three questions to consider: a) Have we, without any reasonable doubt, hit a wall for AI development? Emphasis on "reasonable doubt". There is no reasonable doubt that the Earth is roughly spherical. That level of certainty. b) Depending on your answer for (a), the next question to consider is if we the humans have motivations to continue developing AI. c…

Invalid expression: value of type "probability distribution" cannot be cast to type "boolean".

Re: Recent AI model progress feels mostly like bullshit

#409

Post by sksxihve »

maccard wrote: post #405
sksxihve wrote: post #375

Earlier quoted context omitted.

The whole MCP hype really shows how much of AI is bullshit. These LLMs have consumed more API documentation than possible for a single human and still need software engineers to write glue layers so they can use the APIs.

I don't think I agree, entirely. The problem is that up until _very_ recently, it's been possible to get LLMs to generate interesting and exciting results (as a result of all the API documentation and codebases they've inhaled), but it's been very hard to make that usable. I think we need to be able to control the output format of the LLMs in a better way before we can work on what's in the output. I don't konw if MC…

That's reasonable along with your comment below too, but when you have the ceo of anthropic saying "AI will write all code for software engineers within a year" last month I would say that is pretty hard to believe given how it performs without user intervention (MCP etc...). It feels like bullshit just like the self driving car stuff did ~10 years ago.

Re: Recent AI model progress feels mostly like bullshit

#410

Post by otterley »

freehorse wrote: post #352
otterley wrote: post #175

Earlier quoted context omitted.

Indeed, there’s nothing at all wrong with sharing anecdotes. The problem is when people make broad assumptions and conclusions based solely on personal experience, which unfortunately happens all too often. Doing so is wired into our brains, though, and we have to work very consciously to intercept our survival instincts.

People "make conclusions" because they have to take decisions day to day. We cannot wait for the perfect bulletproof evidence before that. Data is useful to take into account, but if I try to use X llm that has some perfect objective benchmark backing it, while I cannot make it be useful to me while Y llm has better results, it would be stupid not to base my decision on my anecdotal experience. Or vice versa, if I ha…

Agreed 100%. When insufficient data exists, you have to fall back to other sources like analogies, personal observations, secondhand knowledge, etc. However, I’ve seen too many instances of people claiming their own limited experience is the truth when overwhelming and easily attainable evidence and data exists that proves it to be false.
Post reply on HN