Is reward hacking really as unsolvable as they claim?
Why I'm still bearish on LLMs after Navier-Stokes
541–550 of 642 posts
Re: Why I'm still bearish on LLMs after Navier-Stokes
#542Earlier quoted context omitted.
They're called helicopters.
And what since then?
This has the smell of "Why don't I have a faster horse".
Why no flying cars. Because objects have mass and inertia and people are incredibly stupid. Making a flying car has been done. Making a flying car not be a weapon of mass destruction is very, very hard.
Also:
https://www.txdot.gov/about/newsroom/statewide/air-taxi-test...
Re: Why I'm still bearish on LLMs after Navier-Stokes
#543This is the most grounded and coherent take I’ve seen on the actual realizable value of LLMs.. pretty much since they came out. > the classes of firms that can accept the use of fully autonomous LLMs are few, by my count just three: 1. those who can accept failure cheaply: firms that would otherwise hire interns, firms involved in rapid prototyping work, etc. 2. those who need done a small set of narrowly defined tas…
4. those who do reversible work, where cost of error is tolerable
Re: Why I'm still bearish on LLMs after Navier-Stokes
#544Earlier quoted context omitted.
> Claude Fable would destroy any human at chess by coding a strong enough engine on the fly. Delusional, but then Claude fable also isn’t beating any human at chess, the engine is.
But does that matter? If the goal for buyers of AI is “replace this knowledge worker”, how much does it matter that the model in a simple loop can’t do it, but the model with a strong general purpose harness and a little time to gather resources and knowledge to augment the harness going forward, plus tool calls, plus custom built tools, etc, can replace the knowledge worker? Probably the only thing saving many jobs…
Tests are often conducted under restricted conditions. For example, elementary school students aren't given calculators in math class, or during an interview, you are asked what encapsulation is and aren't allowed to use Google. The chess test effectively demonstrates the reasoning capabilities of an LLM without relying on brute force, because a human is incapable of calculating trillions of combinations yet plays chess successfully. This test is necessary because many complex problems cannot be solved by brute force, such as managing a business or playing Heroes 3. Therefore, we can make the assumption that if an LLM can play chess at a grandmaster level without brute force, it means it will be able to command an army or manage production.
Re: Why I'm still bearish on LLMs after Navier-Stokes
#545Earlier quoted context omitted.
lol. wish it was. fwiw i was pretty much bearish on AI for years because I did go deep into this shit since i got access to GPT 3. but since astra i have been quite bullish. 98% of people think AI means what gemini tells them when they do a google search. of that 2% who go beyond... maybe 20% of those are using AI to code. most SWE still think "using AI" to code means the copilot pane they open on the side in VSC. th…
>like i said, if you are still having quality issues in 2026, that's a skill gap. So what ever shit the "AI" gets wrong, it is the user's fault, right?
yes, if you use a tool improperly that's on you. i am specifically talking about people who don't use skills (the most basic shit) and then get sloppy code.
Re: Why I'm still bearish on LLMs after Navier-Stokes
#546This is the most grounded and coherent take I’ve seen on the actual realizable value of LLMs.. pretty much since they came out. > the classes of firms that can accept the use of fully autonomous LLMs are few, by my count just three: 1. those who can accept failure cheaply: firms that would otherwise hire interns, firms involved in rapid prototyping work, etc. 2. those who need done a small set of narrowly defined tas…
I think using AI for customer service is really, really ineffective, and it's plain to anybody that has interacted with it.
It's a shame that we've had decades of shitty customer service from companies that have the most responsibility and resources to do it, and people in tech have shrugged their shoulders saying "it's unreasonable to expect people to scale up their customer service! They gotta make money!" Now AI customer service has arrived and things are worse, not better. Now we can't even get in contact with anyone because an agent redirects us to documentation instead of directing us to a human being.
The future, when a company steals your money, is making viral posts online trying to get enough public shame going that you will maybe get some of that money back.
Re: Why I'm still bearish on LLMs after Navier-Stokes
#547Earlier quoted context omitted.
It's a tech bro thing. Altman does it too, and I've worked with people in the past who do it. I read it as "I'll take literally any conscience for myself no matter how minor, at any cost for you no matter how big".
it's an internet thing. people were typing lowercase on IRC in the 90s.
It's not derived from that. Most people don't use IRC anymore and there are basically no open chat platforms left. Most people now use phones to chat, where the keyboard automatically changes case for you after finishing a sentence.
So typing in all lowercase is a choice. I think it's a way to signal "I don't follow norms". It signals membership in a group.
Re: Why I'm still bearish on LLMs after Navier-Stokes
#548Earlier quoted context omitted.
"Transatlantic flight will never be commercially viable, we conclude based on careful study of several aircraft designs from the 1920s"
How about supersonic flight?
I don't think that's a useful comparison. Supersonic military planes have been common for decades. We don't have supersonic passenger planes because the FAA has banned supersonic flight over land since 1973, though the agency is planning on replacing it with a noise standard. Also the original supersonic passenger aircraft were government-sponsored tech demos, not financially sustainable products. With updated laws & modern technology (cameras instead of tilting noses, more efficient engines without afterburners, lighter materials), we could have viable supersonic passenger flight.
Re: Why I'm still bearish on LLMs after Navier-Stokes
#549This is the most grounded and coherent take I’ve seen on the actual realizable value of LLMs.. pretty much since they came out. > the classes of firms that can accept the use of fully autonomous LLMs are few, by my count just three: 1. those who can accept failure cheaply: firms that would otherwise hire interns, firms involved in rapid prototyping work, etc. 2. those who need done a small set of narrowly defined tas…
In my own experience as a software developer, I see no justification for the hype. Demand for software is practically infinite, and agents aren't mind readers so they'll always need workers to turn needs into prompts. Despite how much work they get done on their own, Parkinson's Law ( https://en.wikipedia.org/wiki/Parkinson%27s_Law ) is still in effect. You could even expand "Work expands so as to fill the time avail…
I think the fear is that softwaring engineering becomes a low-skill profession. If the models get good enough you won't need years of experience to be a decent programmer.
Re: Why I'm still bearish on LLMs after Navier-Stokes
#550Great article, but the lack of sentence capitalization makes it unnecessarily difficult to read. Apologies if this comment is off-topic, but it really is quite egregious, and since the article was submitted by the author I presume they are open to the feedback.
Counter-point: Doesn't make it hard to read at all, definitely not "egregious" as it's a very common style to find for informal writing all over the internet. Author's own style is certainly refreshing and welcome over LLM slop that dominates most HN posts now.
> as it's a very common style to find for informal writing all over the internet
No it's not. It's common in chats where those people don't bother with punctuation at all, often writing a single sentence over multiple lines.
It's not common for blogs or long form content.