Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

Post reply on HN
Print view View original

231–240 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#231

Post by nyrikki »

nyrikki wrote: post #221
pama wrote: post #195

Earlier quoted context omitted.

To the downvoters: I am curious if the downvoting is because of my speculation, or because of the difference in understanding of decoder transformer models. Thanks!

Because you cite is about: > in-context learning LLMs have no concept of the symantic meaning of what they do, they just are dealing with next token prediction. "in-context learning" is the problem, not the solution to general programming tasks. Memoryless, ergodic, sub Turing complete problems are a very tiny class. Think about how the Entscheidungsproblem relates to halting or the frame problem and the specificatio…

Here is a fairly good lecture series on graduate level complexity theory that will help understand parts. At least why multiple iterations help but why they also aren't the answer to super human results.

https://youtube.com/playlist?list=PLm3J0oaFux3b8Gg1DdaJOzYNs...

Re: Can LLMs write better code if you keep asking them to “write better code”?

#232

Post by sitkack »

ActivePattern wrote: post #203
sitkack wrote: post #199

Earlier quoted context omitted.

Why limit its ability to write separate code?

Because otherwise we are talking about LLMs augmented with external tools (i.e. Python interpreters). My original comment was pointing to the limitations of LLMs in writing code by themselves.

You wouldn't ask a programmer to solve a problem and then also not let them write down the source or debug the program as you write it?

Are you asking it to not write down an algorithm that is general? They are doing a pretty good job on mathematical proofs.

I still don't understand why you wouldn't let its full reasoning abilities by letting it write down code or even another agent. We should be testing towards the result not the methods.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#233

Post by williamdclt »

stavros wrote: post #132
lolinder wrote: post #110

Earlier quoted context omitted.

It depends a lot on the language. I recently tried this with Aider, Claude, and Rust, and after writing one function and its tests the model couldn't even get the code compiling, much less the tests passing. After 6-8 rounds with no progress I gave up. Obviously, that's Rust, which is famously difficult to get compiling. It makes sense that it would have an easier time with a dynamic language like Python where it onl…

I've found something similar, when you keep telling the LLM what the compiler says, it keeps adding more and more complexity to try to fix the error, and it either works by chance (leaving you with way overengineered code) or it just never works. I've very rarely seen it simplify things to get the code to work.

I have the same observation, looks like LLMs are highly biased to add complexity to solve problems: for example add explicit handling of the edge-cases I pointed out rather than rework the algorithm to eliminate edge-cases altogether. Almost everytime it starts with something that's 80% correct, then iterate into something that's 90% correct while being super complex, unmaintainable and having no chance to ever cover the last 10%

Re: Can LLMs write better code if you keep asking them to “write better code”?

#234

Post by evilduck »

emseetech wrote: post #200
qup wrote: post #188

Earlier quoted context omitted.

People should try to switch to a more code-focused interface, like aider. Copy and pasting code it gives you just means your workflow is totally borked, and it's no wonder you wouldn't want to try to let it generate code, because it's such a pain in your ass to try it, diff it, etc.

The workflow isn’t the problem. The code is.

The code that ChatGPT and Claude will output via their chat interfaces is a night and day difference from what will be output from tools built around their APIs.

You "can" get the web UI to behave similarly but it's both tedious and slow to manually copy and paste all of that into your context during each interaction and the output will be unfriendly towards human interaction to paste it back out to your project. But that's like saying you "can" browse the internet with a series of CURL commands and pasting the output into files you save locally and then viewing them locally from your browser, nobody is advised to do that because it's a painfully bad experience compared to just having your browser fetch a site's files directly and rendering them directly.

Just go check out Aider or Cline's project repos and look at the dramatically different amounts of code, repo and task specific context they can automatically inject for you as part of their interface, or how much different the built in system prompts are from whatever the default web UIs use, or even the response structures and outputs and how those are automatically applied to your work instead. I've never once exhausted my daily API limits just treating their APIs as Chat interface backends (via Open WebUI and other chat options), but I exhausted my Claude API token limits _the very first day_ I tried Cline. The volume of information you can easily provide through tooling is impossible to do in the same timeframe by hand.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#235

Post by jmathai »

ThrowawayR2 wrote: post #212
btbuildem wrote: post #140

I've noticed this with GPT as well -- the first result I get is usually mediocre and incomplete, often incorrect if I'm working on something a little more obscure (eg, OpenSCAD code). I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". The next part is a little strange - it arose out of frustration, but it also seems to improve results. Let's call it "negative incent…

We've entered the voodoo witch doctor phase of LLM usage: " Enter thee this arcane incantation along with thy question into the idol and, lo, the ineffable machine spirits wilt be appeased and deign to grant thee the information thou hast asked for. "

We use the approaching of feeding mistakes from LLM generated code back to the LLM until it produces working code [1].

I might have to try some more aggressive prompting :).

[1] https://withlattice.com

Re: Can LLMs write better code if you keep asking them to “write better code”?

#236

Post by draculero »

btbuildem wrote: post #140

I've noticed this with GPT as well -- the first result I get is usually mediocre and incomplete, often incorrect if I'm working on something a little more obscure (eg, OpenSCAD code). I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". The next part is a little strange - it arose out of frustration, but it also seems to improve results. Let's call it "negative incent…

I tried to update some files using Claude. I tried to use a combination of positive and negative reinforcement, telling that I was going to earn a coin for each file converted and I was going to use that money to adopt a stray kitten, but for every unsuccessful file, a poor kitten was going to suffer a lot.

I had the impression that it got a little better. After every file converted, it said something along the lines of “Great! We saved another kitten!" It was hilarious.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#237

Post by empath75 »

btbuildem wrote: post #140

I've noticed this with GPT as well -- the first result I get is usually mediocre and incomplete, often incorrect if I'm working on something a little more obscure (eg, OpenSCAD code). I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". The next part is a little strange - it arose out of frustration, but it also seems to improve results. Let's call it "negative incent…

> I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try".

I think having the mediocre first pass in the context is probably essential to it creating the improved version. I don't think you can really skip the iteration process and get a good result.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#238

Post by strongpigeon »

btbuildem wrote: post #140

I've noticed this with GPT as well -- the first result I get is usually mediocre and incomplete, often incorrect if I'm working on something a little more obscure (eg, OpenSCAD code). I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". The next part is a little strange - it arose out of frustration, but it also seems to improve results. Let's call it "negative incent…

What has been your experience with using ChatGPT for OpenSCAD? I tried it (o1) recently for a project and it was pretty bad. I was trying to model a 2 color candy cane and the code it would give me was ridden with errors (e.g.: using radians for angles while OpenSCAD uses degrees) and the shape it produced looked nothing like what I had hoped.

I used it in another project to solve some trigonometry problems for me and it did great, but for OpenSCAD, damn it was awful.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#239

Post by pama »

nyrikki wrote: post #221
pama wrote: post #195

Earlier quoted context omitted.

To the downvoters: I am curious if the downvoting is because of my speculation, or because of the difference in understanding of decoder transformer models. Thanks!

Because you cite is about: > in-context learning LLMs have no concept of the symantic meaning of what they do, they just are dealing with next token prediction. "in-context learning" is the problem, not the solution to general programming tasks. Memoryless, ergodic, sub Turing complete problems are a very tiny class. Think about how the Entscheidungsproblem relates to halting or the frame problem and the specificatio…

My main concern with the simplification of memorization or near neighbor interpolation that is commonly assumed for LLMs is that these methods are ineffective at scale and unlikely to be used by decoder transformers in practice. That paper shows that the decoder transformer somehow came up with a better decision tree fitting algorithm for low data cases than any of the conventional or boosted tree solutions humans typically use from XGBoost or similar libraries. It also matched the best known algorithms for sparse linear systems. All this while training on sequences of random x1, y1, x2, y2,.. with y for each sequence generated by a new random function of a high-dimensional input x every time. The authors show that KNN does not cut it, and even suboptimal algorithms do not suffice. Not sure what else you need as evidence that decoder transformers can use programs to compress information.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#240

Post by hollywood_court »

I've had great luck with Cursor by simply cursing at it when it makes repeated mistakes.

I'll speak to it like a DI would speak to a recruit a basic training.

And it works.

I was speaking to some of the Cursor dev team on Discord, and they confirmed that being aggressive with the AI can lead to better results.

Post reply on HN