Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

Post reply on HN
Print view View original

171–180 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#171

Post by afiodorov »

dgacmu wrote: post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

Another speed-up is to skip the sum of digits check if n % 9 != 30 % 9. Sum of digits have the same remainder divided by 9 as the number. This rules out 8/9 = 88% candidates.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#172

Post by dotancohen »

btbuildem wrote: post #140

I've noticed this with GPT as well -- the first result I get is usually mediocre and incomplete, often incorrect if I'm working on something a little more obscure (eg, OpenSCAD code). I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". The next part is a little strange - it arose out of frustration, but it also seems to improve results. Let's call it "negative incent…

  > I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try".
Is that actually how you're prompting it? Does that actually give better results?

Re: Can LLMs write better code if you keep asking them to “write better code”?

#173

Post by withinboredom »

Yoric wrote: post #53
evrimoztamur wrote: post #50

Earlier quoted context omitted.

The new Cursor agent is able to check the linter output for warnings and errors, and will continue to iterate (for a reasonable number of steps) until it has cleared them up. It's not quite executing, but it does improve output quality. It can even back itself out of a corner by restoring a previous checkpoint. It works remarkably well with typed Python, but struggles miserably with Rust despite having better error r…

> It seems like with Rust it's not quite aware of which patterns to use, especially when the actual changes required may span multiple files due to the way memory management is structured. What do you mean? Memory management is not related to files in Rust (or most languages).

When did they say that?

Re: Can LLMs write better code if you keep asking them to “write better code”?

#174

Post by withinboredom »

IanCal wrote: post #52
mhmmmmmm wrote: post #22

Earlier quoted context omitted.

Is reliably detecting if code has any infinite loops feasible? Sounds like the halting problem.

It depends how you define reliably. The halting problem isn't so relevant in most development, and nothing stops you having a classifier that says "yes", "no" or "maybe". You can identify code that definitely finishes, and you can identify code that definitely doesn't. You can also identify some risky code that probably might. Under condition X, it would go into an infinite loop - even if you're not sure if condition…

The problem is that you can do this for specific functions/methods, but you cannot do this for a PROGRAM. All programs are "maybe", by definition. You want it to run until you tell it to stop, but you may never tell it to stop. Ergo, all programs have some sort of infinite loop in them somewhere, even if it is buried in your framework or language runtime.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#175

Post by sweezyjeezy »

gjm11 wrote: post #137
dgacmu wrote: post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

There's another, arguably even simpler, optimization that makes me smile. (Because it's silly and arises only from the oddity of the task, and because it's such a huge performance gain.) You're picking 1,000,000 random numbers from 1 to 100,000. That means that any given number is much more likely to appear than not. In particular, it is very likely that the list contains both 3999 (which is the smallest number with…

for 10^5, to get the same collision probability (~2 * exp(-10)), you would just need to compute the 10 maximum/minimum candidates and check against those.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#176

Post by ants_everywhere »

dgacmu wrote: post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

It's decent when you prompt it to find easy-to-miss but substantial improvements around corner cases, which is something I've taken to doing.

Basically you just have to put it in the mode that's looking for such things

Re: Can LLMs write better code if you keep asking them to “write better code”?

#177

Post by pwdisswordfishz »

btbuildem wrote: post #140

I've noticed this with GPT as well -- the first result I get is usually mediocre and incomplete, often incorrect if I'm working on something a little more obscure (eg, OpenSCAD code). I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". The next part is a little strange - it arose out of frustration, but it also seems to improve results. Let's call it "negative incent…

I suspect that all it does is prime it to reach for the part of the training set that was sourced from rude people who are less tolerant of beginners and beginners' mistakes – and therefore less likely to commit them.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#178

Post by scosman »

nahnahno wrote: post #145
scosman wrote: post #118

By iterating it 5 times the author is using ~5x the compute. It’s kinda a strange chain of thought. Also: premature optimization is evil. I like the first iteration most. It’s not “beginner code”, it’s simple. Tell sonnet to optimize it IF benchmarks show it’s a pref problem. But a codebase full of code like this, even when unnecessary, would be a nightmare.

This is not what premature optimization is the root of all evil means. It’s a tautological indictment of doing unnecessary things. It’s not in support of making obviously naive algorithms. And if it were it wouldn’t be a statement worth focusing on. As the point of the article is to see if Claude can write better code from further prompting so it is completely appropriate to “optimize” a single implementation.

I have to disagree. Naive algorithms are absolutely fine if they aren’t performance issues.

The comment you are replying to is making the point that “better” is context dependent. Simple is often better.

> There is no doubt that the grail of efficiency leads to abuse. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%. - Donald Knuth

Re: Can LLMs write better code if you keep asking them to “write better code”?

#179

Post by qup »

ActivePattern wrote: post #155
gjm11 wrote: post #137

Earlier quoted context omitted.

There's another, arguably even simpler, optimization that makes me smile. (Because it's silly and arises only from the oddity of the task, and because it's such a huge performance gain.) You're picking 1,000,000 random numbers from 1 to 100,000. That means that any given number is much more likely to appear than not. In particular, it is very likely that the list contains both 3999 (which is the smallest number with…

This is actually a great example of an optimization that would be extremely difficult for an LLM to find. It requires a separate computation to find the smallest /largest numbers in the range with digits summing to 30. Hence, an LLM is unlikely to be able to generate them accurately on-the-fly.

Are you sure it would be hard?

Maybe it only requires asking the LLM to be creative when designing the algorithm. The parent poster spent some time thinking about it, obviously--he didn't generate it accurately "on the fly," either. But he's able to direct his own attention.

I don't see why the LLM couldn't come up with this logic, if prompted to think about a clever algorithm that was highly specific to this problem.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#180

Post by visarga »

xrd wrote: post #19

Wow, what a great post. I came in very skeptical but this changed a lot of misconceptions I'm holding. One question: Claude seems very powerful for coding tasks, and now my attempts to use local LLMs seem misguided, at least when coding. Any disagreements from the hive mind on this? I really dislike sending my code into a for profit company if I can avoid it. Second question: I really try to avoid VSCode (M$ concerns…

> Claude seems very powerful for coding tasks

> I really dislike sending my code into a for profit company if I can avoid it

I see a link between them - maybe the model got good because it used chat logs to improve?

Post reply on HN