Can LLMs write better code if you keep asking them to “write better code”? • Hacker News Board

Re: Can LLMs write better code if you keep asking them to “write better code”?

#341

Post by CharlesW » Fri, Jan 03, 2025, 10:09 PM UTC

_yb2s wrote: post #298

The headline question here alone gets at what is the biggest widespread misunderstanding of LLMs, which causes people to systematically doubt and underestimate their ability to exhibit real creativity and understanding based problem solving. At it's core an LLM is a sort of "situation specific simulation engine." You setup a scenario, and it then plays it out with it's own internal model of the situation, trained on…

> At it's core an LLM is a sort of "situation specific simulation engine."

"Sort of" is doing Sisisyphian levels of heavy lifting here. LLMs are statistical models trained on vast amounts of symbols to predict the most likely next symbol, given a sequence of previous symbols. LLMs may appear to exhibit "real creativity", "understand" problem solving (or anything else), or serve as "simulation engines", but it's important to understand that they don't currently do any of those things.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#342

Post by simonw » Fri, Jan 03, 2025, 10:11 PM UTC

lostmsu wrote: post #323

simonw wrote: post #261

Earlier quoted context omitted.

A non-engineer would not be able to interpret ANY of what he did here, or fix any of the bugs.

A non-engineer by definition would not be able to fix bugs. But why does it matter that they won't be able to interpret anything? Just like with real engineers you can ask AI to provide an explanation digestible by an eloi.

By the time a non-engineer has waded through all of those necessary explanations they'll deserve to call themselves an engineer.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#343

Post by chirau » Fri, Jan 03, 2025, 10:15 PM UTC

Deepseek writes some good code, at least in my experience with it

Re: Can LLMs write better code if you keep asking them to “write better code”?

#344

Post by _yb2s » Fri, Jan 03, 2025, 10:27 PM UTC

CharlesW wrote: post #341

_yb2s wrote: post #298

The headline question here alone gets at what is the biggest widespread misunderstanding of LLMs, which causes people to systematically doubt and underestimate their ability to exhibit real creativity and understanding based problem solving. At it's core an LLM is a sort of "situation specific simulation engine." You setup a scenario, and it then plays it out with it's own internal model of the situation, trained on…

> At it's core an LLM is a sort of "situation specific simulation engine." "Sort of" is doing Sisisyphian levels of heavy lifting here. LLMs are statistical models trained on vast amounts of symbols to predict the most likely next symbol, given a sequence of previous symbols. LLMs may appear to exhibit "real creativity", "understand" problem solving (or anything else), or serve as "simulation engines", but it's impor…

I'm not sure if you read the entirety of my comment? Increasingly accurately predicting the next symbol given a sequence of previous symbols, when the symbols represent a time series of real world events, requires increasingly accurately modeling- aka understanding- the real world processes that lead to the events described in them. There is provably no shortcut there- per Solomonoff's theory of inductive inference.

It is a misunderstanding to think of them as fundamentally separate and mutually exclusive, and believing that to be true makes people convince themselves that they cannot possibly ever do things which they can already provably do.

Noam Chomsky (embarrassingly) wrote a NYT article on how LLMs could never, with any amount of improvements be able to answer certain classes of questions - even in principle. This was days before GPT-4 came out, and it could indeed correctly answer the examples he said could not be ever answered- and any imaginable variants thereof.

Receiving symbols and predicting the next one is simply a way of framing input and output that enables training and testing- but doesn't specify or imply any particular method of predicting the symbols, or any particular level of correct modeling or understanding of the underlying process generating the symbols. We are both doing exactly that right now, by talking online.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#345

Post by HarHarVeryFunny » Fri, Jan 03, 2025, 10:37 PM UTC

famouswaffles wrote: post #250

HarHarVeryFunny wrote: post #144

This seems like anthromorphizing the model ... Occam's Razor says that the improvement coming from iterative requests to improve the code comes from the incremental iteration, not incentivizing the model to do it's best. If the latter were the case then one could get the best version on first attempt by telling it your grandmother's life was on the line or whatever. Reasoning is known weakness of these models, so jum…

>If the latter were the case then one could get the best version on first attempt by telling it your grandmother's life was on the line or whatever. Setting aside the fact that "best" is ambiguous, why would this get you the best version ? If you told a human this, you wouldn't be guaranteed to get the best version at all. You would probably get a better version sure but that would be the case for LLMs as well. You w…

The thesis of the article is that the code keeps betting better because the model keeps getting told to do better - that it needs more motivation/criticism. A logical conclusion of this, if it were true, is that the model would generate it's best version on first attempt if only we could motivate it to do so! I'm not sure what motivations/threats work best with LLMs - there was a time when offering to pay the LLM was popular, but "my grandma will die if you don't" was also another popular genre of prompts.

If it's not clear, I disagree with the idea that ANY motivational prompt (we can disagree over what would be best to try) could get the model to produce a solution of the same quality as it will when allowed to iterate on it a few times and make incremental improvements. I think it's being allowed to iterate that is improving the solution, not the motivation to "do better!".

Re: Can LLMs write better code if you keep asking them to “write better code”?

#346

Post by CapeTheory » Fri, Jan 03, 2025, 10:43 PM UTC

mulmen wrote: post #318

bennythomsson wrote: post #287

Earlier quoted context omitted.

Amazing. Next step would be to propose hardcoding 99930-3999 as the O(1) result and live with the output just being wrong sometimes. The bug rate is then in the ballpark of most modern software, including LLMs', so I'd say ship it.

Doesn’t this line of thinking constantly redefine success until all software is only bugs?

Always has been

Re: Can LLMs write better code if you keep asking them to “write better code”?

#347

Post by yubrshen » Fri, Jan 03, 2025, 10:45 PM UTC

When asking LLM repeated improving or adding a new feature in a codebase, the most frustration risk is that LLM might wipe out already working code!

What are your strategies to prevent such destructions of LLM?

Re: Can LLMs write better code if you keep asking them to “write better code”?

#348

Post by CharlesW » Fri, Jan 03, 2025, 10:49 PM UTC

_yb2s wrote: post #344

CharlesW wrote: post #341

Earlier quoted context omitted.

> At it's core an LLM is a sort of "situation specific simulation engine." "Sort of" is doing Sisisyphian levels of heavy lifting here. LLMs are statistical models trained on vast amounts of symbols to predict the most likely next symbol, given a sequence of previous symbols. LLMs may appear to exhibit "real creativity", "understand" problem solving (or anything else), or serve as "simulation engines", but it's impor…

I'm not sure if you read the entirety of my comment? Increasingly accurately predicting the next symbol given a sequence of previous symbols, when the symbols represent a time series of real world events, requires increasingly accurately modeling- aka understanding- the real world processes that lead to the events described in them. There is provably no shortcut there- per Solomonoff's theory of inductive inference.…

> I'm not sure if you read the entirety of my comment?

I did, and I tried my best to avoid imposing preconceived notions while reading. You seem to be equating "being able to predict the next symbol in a sequence" with "possessing a deep causal understanding of the real-world processes that generated that sequence", and if that's an inaccurate way to characterize your beliefs I welcome that feedback.

Before you judge my lack of faith too harshly, I am a fan of LLMs, and I find this kind of anthropomorphism even among technical people who understand the mechanics of how LLMs work super-interesting. I just don't know that it bodes well for how this boom ends.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#349

Post by mvdtnz » Fri, Jan 03, 2025, 10:52 PM UTC

_yb2s wrote: post #298

The headline question here alone gets at what is the biggest widespread misunderstanding of LLMs, which causes people to systematically doubt and underestimate their ability to exhibit real creativity and understanding based problem solving. At it's core an LLM is a sort of "situation specific simulation engine." You setup a scenario, and it then plays it out with it's own internal model of the situation, trained on…

> At it's core an LLM is a sort of "situation specific simulation engine." You setup a scenario, and it then plays it out with it's own internal model of the situation, trained on predicting text in a huge variety of situations. This includes accurate real world models of, e.g. physical systems and processes, that are not going to be accessed or used by all prompts, that don't correctly instruct it to do so.

You have simply invented total nonsense about what an LLM is "at it's core". Confidently stating this does not make it true.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#350

Post by _yb2s » Fri, Jan 03, 2025, 11:06 PM UTC

CharlesW wrote: post #348

_yb2s wrote: post #344

Earlier quoted context omitted.

I'm not sure if you read the entirety of my comment? Increasingly accurately predicting the next symbol given a sequence of previous symbols, when the symbols represent a time series of real world events, requires increasingly accurately modeling- aka understanding- the real world processes that lead to the events described in them. There is provably no shortcut there- per Solomonoff's theory of inductive inference.…

> I'm not sure if you read the entirety of my comment? I did, and I tried my best to avoid imposing preconceived notions while reading. You seem to be equating "being able to predict the next symbol in a sequence" with "possessing a deep causal understanding of the real-world processes that generated that sequence", and if that's an inaccurate way to characterize your beliefs I welcome that feedback. Before you judge…

> You seem to be equating "being able to predict the next symbol in a sequence" with "possessing a deep causal understanding of the real-world processes that generated that sequence"

More or less, but to be more specific I would say that increasingly accurately predicting the next symbols in a massive set of diverse sequences, which explain a huge diversity of real world events described in sequential order, requires increasingly accurate models of the underlying processes of said events. When constrained with a lot of diversity and a small model size, it must eventually become something of a general world model.

I am not understanding why you would see that as anthropomorphism- I see it as quite the opposite. I would expect something non-human that can accurately predict outcomes of a huge diversity of real world situations based purely on some type of model that spontaneously develops by optimization- to do so in an extremely alien and non-human way that is likely incomprehensible in structure to us. Having an extremely alien but accurate way of predicatively modeling events that is not subject to human limitations and biases would be, I think, incredibly useful for escaping limitations of human thought processes, even if replacing them with other different ones.

I am using modeling/predicting accurately in a way synonymous with understanding, but I could see people objecting to the word 'understanding' as itself anthropomorphic... although I disagree. It would require a philosophical debate on what it means to understand something I suppose, but my overall point still stands without using that word at all.