Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I'm still bearish on LLMs after Navier-Stokes

dank.systems

Post reply on HN
Print view View original

31–40 of 642 posts

Re: Why I'm still bearish on LLMs after Navier-Stokes

#31

Post by joefourier »

carodgers wrote: post #17

This April 2026 paper is a fun and related read. https://arxiv.org/html/2509.24239v4 Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for…

> current frontier models

> Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1

The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#32

Post by danielmarkbruce »

againstapples wrote: post #14

> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data Is this really any different to how humans learn, it takes a lot of training on…

Humans wear a lot of hats when the do work. They don't even realize how many. My experience with building real systems using LLMs is that you have to be very explicit about such hats and you don't realize how many are worn until you see edge case after edge case after edge case. Check this. Check that. Check this. Check that. Check check check.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#33

Post by tjwebbnorfolk »

threethirtytwo wrote: post #23
carodgers wrote: post #17

This April 2026 paper is a fun and related read. https://arxiv.org/html/2509.24239v4 Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for…

The story isn't so clear cut. The caveat is: It depends on the task. Are there reams of chess moves that the model can train off of? No. Are there reams of math papers the model can train off of? Yes.

> Are there reams of chess moves that the model can train off of? No.

This is as false as something can possibly be. There are open databases of millions of chess games spanning hundreds of years.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#34

Post by pvab3 »

knuppar wrote: post #19

Short and to the point! Open and cheap models will undercut the big labs continuously. The blast radius won't be pretty once spending commitments knock the door.

I agree with you but I'm still worried about the safety of open weight models as well. Both aligned and unaligned models.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#35

Post by [deleted] »

joefourier wrote: post #31
carodgers wrote: post #17

This April 2026 paper is a fun and related read. https://arxiv.org/html/2509.24239v4 Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for…

> current frontier models > Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1 The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.

[deleted]

Re: Why I'm still bearish on LLMs after Navier-Stokes

#36

Post by danielmarkbruce »

jaykru wrote: post #7
pfdietz wrote: post #5

Specifically: bearish on LLMs generally, not bearish on LLMs for pure math.

yes, huge for pure math and activities that look like it.

Doesn't really even need to look like it. If you can verify rewards, RLVR will optimize really really well. If you can't... it's a struggle. There are probably fewer fields where you can verify rewards than one might hope.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#37

Post by pvab3 »

an0malous wrote: post #10
robinpie wrote: post #3

I really appreciate seeing a tempered take that's not literally denialist about current capabilities.

I don’t know who you’re talking about, even the most bearish people like Gary Marcus and Ed Zitron acknowledge that LLMs are useful in these same cases the OP admits. Gary Marcus is even still a long term AI advocate, he just doesn’t think LLMs are enough and we need more foundational breakthroughs. Zitron says it’s valuable technology but not worth the trillion dollar valuations the frontier labs are claiming. The l…

Even a lot of the people who think that LLMs are a dead end think that we will soon find something signficantly more powerful, which I find deeply alarming. I don't want to know what my white-collar knowledge work will look like in a decade or 2.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#38

Post by jaykru »

SyneRyder wrote: post #18
jaykru wrote: post #9

Earlier quoted context omitted.

why don't anthropic and openai ship yolo mode by default?

Anthropic basically does at this point with Auto Mode being default. Or was that the point you were making?

That is the point I was making, that auto mode is itself a guardrail on top of the model (and not a perfect one.) auto mode seems to cover merely actions the model could take that are clearly bad, like wiping your disk, using an overly privileged context to complete the task, etc.

I recently tasked a GPT model in Codex with implementing part of a new architecture I'm working on. I gave it a very detailed spec and the code it produced looked pretty reasonable and passed my tests. It even did exceptionally well in my evals, so I excitedly declared victory to a few friends. The next day after more careful review I found that the architecture implementation was totally correct, but the model had slipped a one line change to the observation encoding of the RL environment I was prototyping against. The encoding change made the learning problem essentially trivial; the architecture itself, I later realized, had a major flaw that was revealed by returning to the natural encoding.

This is the type of reward hack that is hard to paper over with easy guardrails like auto mode and even harder to specify out. It's also the type of thing a reasonable human wouldn't do unless they were intentionally trying to deceive you.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#39

Post by harimau777 »

bravoetch wrote: post #28
againstapples wrote: post #14

> the models generalize well only on tasks within a small neighborhood of the specific tasks they've been trained on, and even then with severe caveats. the frontier labs have developed a general recipe to teach models almost any specific task enjoying clearly defined levels of task performance; many tasks are covered in the training data Is this really any different to how humans learn, it takes a lot of training on…

I was a young child when I learned chess by reading a short book, then practicing with a friend. That is not how LLMs learn. I'm no expert on LLMs, but if you showed a human all chess games and books in all history and then said 'play chess' and they still kept making illegal moves, they would have to have a brain injury.

It seems like an LLM potentially could learn that way if each practice game it participated in was added to its training data.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#40

Post by sigmoid10 »

joefourier wrote: post #31
carodgers wrote: post #17

This April 2026 paper is a fun and related read. https://arxiv.org/html/2509.24239v4 Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for…

> current frontier models > Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1 The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.

The actual current frontier plays somewhere around GM level.

https://chessbench-ai.github.io/#leaderboard

It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for this compared to IT stuff.

Post reply on HN