Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I'm still bearish on LLMs after Navier-Stokes

dank.systems

Post reply on HN
Print view View original

191–200 of 642 posts

Re: Why I'm still bearish on LLMs after Navier-Stokes

#191

Post by nalekberov »

uncivilized wrote: post #98
minraws wrote: post #64

Earlier quoted context omitted.

I know HN readers and posters just read numbers and can't be bothered to read, but please read the methodology before making any claims. > About their ELO ratings from their own website: > A field-relative rating calculated within ChessBench. It compares performance among the tested models and is not a direct equivalent of a human chess rating. I am around 1600 elo in over the board I can mop up Astra Fable etc even…

HN is no different than Reddit, or any social media for that matter, in that commenters pretend to read articles.

[flagged]

Re: Why I'm still bearish on LLMs after Navier-Stokes

#192

Post by jibal »

sigmoid10 wrote: post #40
joefourier wrote: post #31

Earlier quoted context omitted.

> current frontier models > Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1 The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.

The actual current frontier plays somewhere around GM level. https://chessbench-ai.github.io/#leaderboard It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for…

Please do not post misinformation. They are not playing anywhere near GM level.

"Elo is relative to the ChessBench field."

Re: Why I'm still bearish on LLMs after Navier-Stokes

#193

Post by jibal »

echelon wrote: post #78
minraws wrote: post #64

Earlier quoted context omitted.

I know HN readers and posters just read numbers and can't be bothered to read, but please read the methodology before making any claims. > About their ELO ratings from their own website: > A field-relative rating calculated within ChessBench. It compares performance among the tested models and is not a direct equivalent of a human chess rating. I am around 1600 elo in over the board I can mop up Astra Fable etc even…

The AI can write a chess bot program that will beat you. You're thinking about this the wrong way. The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior. We shouldn't ask the multibillion dollar automated software generation system to play games with us any more than we should ask…

First, you're moving the goalposts. Second, it's not actually true that any existing frontier AI can write a chess bot program that can beat a 1600 player ... not unless the program is derived from Stockfish or some other leading engine that has been in development for decades.

> The system is built and delivered as it is because that's how the providers make the most money. If they cared to have it perform well in chess games, you'd see a different shape and behavior.

These comments indicate a complete failure to understand the technology.

I won't respond again.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#194

Post by hackinthebochs »

diehunde wrote: post #149
famouswaffles wrote: post #119

Earlier quoted context omitted.

If all you do is read chess books, you'll be a shit player. Training and practice is what it takes to be great.

Oh right. But if all you do is reading programming books you are an amazing programmer? Where is all the training and practice LLMs did to become so good at coding?

>Where is all the training and practice LLMs did to become so good at coding?

Coding is a matter of translating the natural language description of a problem to the code specification while keeping the semantics fixed (and imputing the unspecified semantics as necessary). It is not considerably more difficult than translating between two dissimilar natural languages. Chess isn't a matter of language translation, but a compute heavy game of finding the best move out of many possibilities with wide variation in the quality of each move. Chess takes directed practice and reinforcement whereas language translation does not.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#195

Post by Certhas »

WhitneyLand wrote: post #59
carodgers wrote: post #17

This April 2026 paper is a fun and related read. https://arxiv.org/html/2509.24239v4 Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for…

1. It’s hard to trust a 2026 paper that’s showing results for such old models. 2. Chess seems to be a poor benchmark for generalized strategic reasoning. People who are good at it rely more on experience and deep domain expertise than on skills that generalize to make them experts at unrelated tasks. 3. The study sounds like proving humans will never fly because they don’t have wings. In reality, humans do fly, and C…

Good science, properly digested and presented takes time.

The idea that anything other than a breathless blog post about the latest model snapshot is useless is really poisonous to proper debate on AI issues

Re: Why I'm still bearish on LLMs after Navier-Stokes

#196

Post by TrackerFF »

The challenge with estimating abilities, is that we don’t know what the models can achieve if we just burn enough money. The navier-stokes shows us what mathematical problem can be solved when $10m worth of compute is thrown at something.

It also makes one wonder: What could AI solve if we managed to orchestrate billions worth of agents to take on a specific problem?

IMO the very best case scenario / potential for these are likely better than we think, but right now hidden due to logistical and financial reasons.

But if we assume that the model costs will continue to drop by a factor of 5-10 annually, there will always be a latency of a couple of years between what is completely out of reach, and what is financially viable.

Basically: If you knew AI could be affordable enough in 3-5 years so that even the most underfunded researchers could use it to solve cancer, how much would you value it now?

Re: Why I'm still bearish on LLMs after Navier-Stokes

#197

Post by jibal »

sashank_1509 wrote: post #68
sigmoid10 wrote: post #40

Earlier quoted context omitted.

The actual current frontier plays somewhere around GM level. https://chessbench-ai.github.io/#leaderboard It's also worth noting that the very latest models (GPT-6 and Fable 5.1) actually play worse than their immediate predecessors, so it is likely that the labs are not benchmaxxing for this yet. If they did, I'm sure they could come up with something superior to humans. But there is probably very little demand for…

These ratings seems very wrong, i have beaten GPT Astra max thinking in chess and my rating is close to 1500. The ratings here seem more accurate: https://chessbenchllm.onrender.com/ GPT-6 almost never suggests an illegal move anymore while even Sol still did so time to time

"Elo is relative to the ChessBench field."

They are of course "wrong" if you don't read the faint fine print and sensibly interpret them as FIDE or similar ratings.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#198

Post by melvinroest »

Yea I get the bearishness from my own personal experience.

Personally, I use LLMs for a lot of things. Oftentimes, I'm a think out loud type of person so even having something that feels like a rubber duck, but more competent, is already amazing for me. And LLMs are a lot more competent than a rubber duck.

But especially sometimes I've noticed that LLMs can be unbelievably stupid. It recently happened a few times with Fable 5.1 as well. Ultimately, I think it comes down to that LLMs can't think broadly. In software development one can usually see this too. For example, a whole app might be built by an LLM and it didn't spend a single token thinking about security because the prompter is at the level of "build a dating app for dogs, make no mistakes". Now you have a dating app for dogs that is insecure.

Since I prompt for almost everything in my life to have an LLM as a sounding board, I'm usually not an expert either. I've noticed LLMs are amazing at "bulk search engine information aggregation" (or whatever you want to call it). So if I need something from the Dutch government, I can find it way more quickly. But oftentimes I've noticed that going for a walk and thinking about a particular thing I'm facing is a more effective way of finding a good solution.

Other times times they are not incredibly stupid, but can't form a strong opinion. This usually happens when I'm tackling a wicked problem [1]. When that's the case, prepare for LLMs to sway with you for every small change in your opinion that you ever will experience.

So I agree: drop in replacement for knowledge workers? No. Rigorous specification is usually needed yes. Though, the small win here is that it doesn't always need to be as rigorous as programming is and it can happen in natural language. It depends on the topic/problem being tackled.

I really like them as UX tools though. Amazing for interactive prototyping and requirements elicitation. And that also corresponds with what the author is saying. Though I find it a bit of a disservice saying "just 3". You know how hard requirements elicitation is? It became a whole lot easier thanks to LLMs (I might change this opinion in a year, haha, but this is the opinion I hold now).

[1] https://en.wikipedia.org/wiki/Wicked_problem

Re: Why I'm still bearish on LLMs after Navier-Stokes

#199

Post by nimbleal »

carodgers wrote: post #117
WhitneyLand wrote: post #59

Earlier quoted context omitted.

1. It’s hard to trust a 2026 paper that’s showing results for such old models. 2. Chess seems to be a poor benchmark for generalized strategic reasoning. People who are good at it rely more on experience and deep domain expertise than on skills that generalize to make them experts at unrelated tasks. 3. The study sounds like proving humans will never fly because they don’t have wings. In reality, humans do fly, and C…

> Claude Fable would destroy any human at chess by coding a strong enough engine on the fly. A bash script can clone and build stockfish, feed in human moves, and reply. By your standard, this bash script would "destroy any human at chess." Are you interested in assessing the intelligence of the model, or the intelligence of the tools the model can use?

Maybe practically it doesn’t matter? Perhaps AGI is not the model but the model plus everything it’s got access to. If we’re modelling intelligence in the way we seem to have to to have any coherent definition of AGI, it seems to me is always going to be more “intelligent” than alone.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#200

Post by 1dom »

carodgers wrote: post #17

This April 2026 paper is a fun and related read. https://arxiv.org/html/2509.24239v4 Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for…

The last post on HN I read was about someone using LLMs to reverse engineer an Apple GPU driver for linux in a month. The top comment points out how the poster must have had specialist internal domain specific contact with Apple. But then the thread concludes that wasn't the case and that this would take domain experts years to do.

> "current frontier models need laborious oversight and guardrails on even the simplest tasks"

I feel this statement is extreme. I can't personally reconcile it with any of the projects we're regularly seeing get delivered largely by LLMs now.

What are you thoughts? Like, what's your position here? Even if you sincerely believe frontier models need laborious oversight on even the simplest of tasks, do you think that accurately captures and reflects the current state and progress of frontier LLMs?

Don't get me wrong, there's lots of things LLMs can't do well, but the idea that they're basically not helpful for even the simplest of tasks seems... disingenuous?

Post reply on HN