Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I'm still bearish on LLMs after Navier-Stokes

dank.systems

Post reply on HN
Print view View original

341–350 of 642 posts

Re: Why I'm still bearish on LLMs after Navier-Stokes

#341

Post by user43928 »

Topfi wrote: post #293
user43928 wrote: post #284

Earlier quoted context omitted.

There is no need to ask. If you want to test SOTA models today, there are obviously only two: GPT-6 Astra and Fable 5.1. The models listed in the paper are from early 2025 and are no longer relevant, much less on the frontier. That Claude version is no longer available today, Gemini 2.5 Pro will be shutdown next month, and the OpenAI models are only available via the API today.

Fortunately, a fellow commenter was so kind and did it with Astra. Didn't do that well either [0]. I'm sure GPT-7 will be super mega ASI regardless (since GPT-6 Astra already claimed AGI in the minds of Jen-Hsun, et al.)... I'll say it till there is any evidence of the contrary, LLMs are not intelligent and their capabilities solely within the realms of well tailored training data. "Just" having been trained on every…

Doesn't look impressive, although I'm hearing a marked improvement in choosing legal moves, compared to early 2025.

Given the pace of improvements, is it really unimaginable that GPT-7 will play Chess reasonably well and generalize better?

I would not be surprised if OpenAI released a model that beats humans at chess this year.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#342

Post by webern777 »

keeda wrote: post #73

The premise in the very first point seems off: > the frontier labs are priced according to the narrative that they have produced or will in the very near future produce a fully automated drop-in replacement for most knowledge workers... Even assuming this is how the AI companies are being valued (they're not), the numbers are off. The "value" of most knowledge workers -- based on what enterprises currently pay for th…

The real issue IMO is that is not really what Anthropic and OpenAI are operating on.

That is the after the fact justification of the AGI dollar auction. Each round is kind of 3x the previous cost and neither can really stop because second place in the dollar auction is so much worse than winning.

The only way to stop the auction is one bidder hits a hard budget constraint, both agree to stop, or an outside party breaks the auction.

IMO this is why they want to slow down or have regulation. I think this is also why we see some claims of already reaching "AGI".

The TAM of global knowledge work is just a narrative tacked on after the fact to justify the AGI dollar auction.

The economic fallacy here with the actual valuation is akin to pricing the electric utilities 120+ years ago as some % of the future cash flow of global food production. Take the TAM of global food production and then work back to what % will the electric utilities capture from the advances in the automation of farming? It is nonsense.

The only narrative that actually justifies the capex spend that I can figure out is a first mover AGI monopoly. Even the oligopoly case is hard to justify the capex spend IMO. There is this enormous mismatch between the AGI monopoly and the actual rolling 12-month window of pricing power.

Even the rolling 12-month window of pricing power is going to saturate well before AGI too so it is hard to see how any of this makes economic sense.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#343

Post by killerstorm »

carodgers wrote: post #17

This April 2026 paper is a fun and related read. https://arxiv.org/html/2509.24239v4 Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for…

This is an absolute nonsense. Any frontier model can implement chess program from scratch - modeling the board, checking legality, etc. If you asked e.g. GPT-6 to get good at chess and gave it a computer, it will get good at chess. That's an actual strategic skill.

Asking GPT to play chess directly using its reasoning only tests its reasoning ability to model chess state. Which it really is NOT optimized for.

This is also true for humans - people who don't have years of chess training can't really tell which moves are legal given an algebraic notation transcript. These people might have good strategic skills in different areas. Chess is just a very, very specific skill

Re: Why I'm still bearish on LLMs after Navier-Stokes

#344

Post by roenxi »

killerstorm wrote: post #343
carodgers wrote: post #17

This April 2026 paper is a fun and related read. https://arxiv.org/html/2509.24239v4 Researchers asked frontier models to play chess. Have a look at the MAR rates in Table 3. When not explicitly told which moves were legal, no model identified legal moves at a rate better than 80%. Many asked for more illegal moves than legal moves. And even when explicitly told which moves were legal, the models continued to ask for…

This is an absolute nonsense. Any frontier model can implement chess program from scratch - modeling the board, checking legality, etc. If you asked e.g. GPT-6 to get good at chess and gave it a computer, it will get good at chess. That's an actual strategic skill. Asking GPT to play chess directly using its reasoning only tests its reasoning ability to model chess state. Which it really is NOT optimized for. This is…

It's an interesting puzzle, isn't it. On the one hand, the AIs are no good at playing Chess.

However, on the other hand, if you ask an AI to win a game of chess it has all the tools on hand to compete at the same level as Stockfish - it can re-implement an engine and even probably has a GPU on hand to train its own neural nets.

So should we say that the AI can play chess well, or that it cannot?

Re: Why I'm still bearish on LLMs after Navier-Stokes

#345

Post by pluc »

utopiah wrote: post #217
yshklarov wrote: post #209

Great article, but the lack of sentence capitalization makes it unnecessarily difficult to read. Apologies if this comment is off-topic, but it really is quite egregious, and since the article was submitted by the author I presume they are open to the feedback.

Array.from(document.body.querySelectorAll('p,li')).filter(e=>e.innerText).map(e=>e.innerText = e.innerText.split('\. ').map(s=>s[0].toUpperCase() + s.slice(1)).join('. ') ) Not perfect but hope it helps.

Or just blink every time you see a period and hopefully miss the next letter

Re: Why I'm still bearish on LLMs after Navier-Stokes

#346

Post by hackinthebochs »

sobellian wrote: post #83
joefourier wrote: post #31

Earlier quoted context omitted.

> current frontier models > Gemini 2.5 Pro, O3, Claude Sonnet 3.7 and ChatGPT 4.1 The gap in capabilities between those models which they tested, and actual current frontier ones is enormous. I would not trust that any conclusions they made are applicable.

I tested both myself and a weak bot against Astra xhigh, https://lichess.org/study/27lCQqDa . It's still pretty bad at chess, though it takes longer to devolve into illegal moves.

So you weren't giving it an updated board state after every move? If you want to compare apples to apples, it should give an updated board state for each move, or you should play blindfolded.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#347

Post by danpalmer »

titzer wrote: post #321
zahlman wrote: post #281

Earlier quoted context omitted.

Just now I tried prompting logged-out ChatGPT (which at least claims to be 5.6-Luna) with: > Let's play a game of chess. You can take White. Please draw an ASCII rendition of the board after each move, so that we can be clear about the position. (I hoped the latter requirement would help it be "not blindfolded"; last time I used a Lichess demo board to track the position in another tab, because I have no talent for b…

[flagged]

Sure, but installing a chess program is child/teen level general ability, and playing chess well is highly trained expert level ability. Which one are we sold AI as being?

Re: Why I'm still bearish on LLMs after Navier-Stokes

#348

Post by kavok »

titzer wrote: post #321
zahlman wrote: post #281

Earlier quoted context omitted.

Just now I tried prompting logged-out ChatGPT (which at least claims to be 5.6-Luna) with: > Let's play a game of chess. You can take White. Please draw an ASCII rendition of the board after each move, so that we can be clear about the position. (I hoped the latter requirement would help it be "not blindfolded"; last time I used a Lichess demo board to track the position in another tab, because I have no talent for b…

[flagged]

I often don’t see agents reaching for available or potential tools/libraries unless explicitly told to.

Sometimes they’ll even manually search or write bespoke code to search json instead of using something like jq.

Re: Why I'm still bearish on LLMs after Navier-Stokes

#349

Post by bitexploder »

kbau wrote: post #329
JohnKemeny wrote: post #308

Earlier quoted context omitted.

Are you saying that modern LLMs cannot play chess now , or that LLMs (GPT architecture) cannot be trained to play chess well? Or are you saying that neural networks in general cannot (practically) be trained to be an above-average chess player? Or are you saying that it depends on the input? Would it be better if they were given a picture/drawing/ascii art of the board? If so, surely they can produce it at will?

Neural Nets can be trained to play chess very well and have been doing so for a long time (see Stockfish and Leela as some of the most popular/strongest ones - top GMs have no chance against them), but these are dedicated models, where the game rules are encoded in the learning process, as opposed to large language models which are natural language processing models. Technically you can give an LLM a lot of chess boo…

https://en.wikipedia.org/wiki/Chinese_room I think about this once in a while. At some point if it does the thing almost perfectly is it still not doing the thing?

Re: Why I'm still bearish on LLMs after Navier-Stokes

#350

Post by simianwords »

janalsncm wrote: post #176
hackinthebochs wrote: post #150

Earlier quoted context omitted.

I would bet a lot of money that Astra can follow the rules of chess (perhaps if repeated within the context window). Also, this is a different argument than what I responded to.

I would definitely take you up on that.

https://www.chessbench.org/

>GPT-6 Astra xHigh: 0.06% rejected moves

Post reply on HN