Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

Post reply on HN
Print view View original

281–290 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#281

Post by ryao »

afiodorov wrote: post #171
dgacmu wrote: post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

Another speed-up is to skip the sum of digits check if n % 9 != 30 % 9. Sum of digits have the same remainder divided by 9 as the number. This rules out 8/9 = 88% candidates.

Would someone write a mathematical proof showing this is always true?

Re: Can LLMs write better code if you keep asking them to “write better code”?

#282

Post by viraptor »

fragmede wrote: post #61
viraptor wrote: post #36

Earlier quoted context omitted.

There's no clear threshold with an universal answer. Sometimes prompting will be easier, sometimes writing things yourself. You'll have to add some debugging time to both sides in practice. Also, you can be opportunistic - you're going to write a commit anyway, right? A good commit message will be close to the prompt anyway, so why not start with that and see if you want to write your own or not? > I also get to do a…

why leave the commit message for the human to write? have the LLM start off and add relevant details it missed.

Because the commit message is pure signal. You can reformat it or as useless info, but otherwise, generating it will require writing it. Generating it from code is a waste, because you're trying to distil that same signal from messy code.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#283

Post by bennythomsson »

dgacmu wrote: post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

Or the other obvious optimization to hard-code the lookup in code as a huge list, instead of creating it first.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#284

Post by afiodorov »

ryao wrote: post #281
afiodorov wrote: post #171

Earlier quoted context omitted.

Another speed-up is to skip the sum of digits check if n % 9 != 30 % 9. Sum of digits have the same remainder divided by 9 as the number. This rules out 8/9 = 88% candidates.

Would someone write a mathematical proof showing this is always true?

  a = [int(x) for x in str(n)][::-1]
  assert n == sum(d*(10**i) for i, d in enumerate(a))
Now when you're operating mod 9, 10 == 1 % 9, thus

  10**i == 1 % 9
Comes from the fact that

  (a*b) % 9 == (a % 9) * (b % 9)
Now using

  (a+b) % 9 == (a % 9) + (b % 9)
We get that that sum(a) and n are same mod 9.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#285

Post by sitkack »

ActivePattern wrote: post #255
sitkack wrote: post #232

Earlier quoted context omitted.

You wouldn't ask a programmer to solve a problem and then also not let them write down the source or debug the program as you write it? Are you asking it to not write down an algorithm that is general? They are doing a pretty good job on mathematical proofs. I still don't understand why you wouldn't let its full reasoning abilities by letting it write down code or even another agent. We should be testing towards the…

I'm simply pointing out the limitations of LLMs as code writers. Hybrid systems like ChatGPT-o1 that augment LLMs with tools like Python interpreters certainly have the potential to improve their performance. I am in full agreement! It is worth noting that even ChatGPT-o1 doesn't seem capable of finding this code optimization, despite having access to a Python interpreter.

> y = sum([x for x in range(1,n)] > Write an efficient program that given a number, find the integer n that satisfies the above constraints

Goal: Find n where sum of integers from 1 to n-1 is ≤ 30

This is a triangular number problem: (n-1)(n)/2 ≤ 30

... code elided ...

> Ok, now make an find_n_for_sum(s=30)

def find_n_for_sum(s: int) -> int: return int((-(-1) + (1 + 8s)*0.5) / 2)

# Tests assert sum(range(1, find_n_for_sum(30))) 30

Re: Can LLMs write better code if you keep asking them to “write better code”?

#286

Post by pama »

nyrikki wrote: post #271
pama wrote: post #257

Earlier quoted context omitted.

Thanks for the tip, though I’m not sure how complexity theory will explain the impossibility of superhuman results. The main advantage ML methods have over humans is that they train much faster. Just like humans, they get better with more training. When they are good enough, they can be used to generate synthetic data, especially for cases like software optimization, when it is possible to verify the ground truth. A…

ML is better than biological neurons in some tasks, they are different contexts. Almost all the performance of say college tests are purely from the pre-training, pattern finding and detection. Transformers are limited to DLOGTIME-uniform TC0, they can't even do the Boolean circuit value problem. The ability to use the properties of BPP, does help. Understanding the power of, and limitations of iteration and improvin…

I recall early, incomplete speculation about transformers not solving Boolean circuit value problems; what did you think of this work? https://arxiv.org/abs/2402.12875v3

Re: Can LLMs write better code if you keep asking them to “write better code”?

#287

Post by bennythomsson »

cultureulterior wrote: post #265
ActivePattern wrote: post #155

Earlier quoted context omitted.

This is actually a great example of an optimization that would be extremely difficult for an LLM to find. It requires a separate computation to find the smallest /largest numbers in the range with digits summing to 30. Hence, an LLM is unlikely to be able to generate them accurately on-the-fly.

O1 found it. https://chatgpt.com/share/67782b6b-6248-8012-882d-238b600ef9...

Amazing.

Next step would be to propose hardcoding 99930-3999 as the O(1) result and live with the output just being wrong sometimes. The bug rate is then in the ballpark of most modern software, including LLMs', so I'd say ship it.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#288

Post by minimaxir »

shahzaibmushtaq wrote: post #258
lostmsu wrote: post #182

Earlier quoted context omitted.

TBH I'm not sure how he arrived at "won’t replace software engineers anytime soon" The LLM solved his task. With his "improved prompt" the code is good. The LLM in his setup was not given a chance to actually debug its code. It only took him 5 "improve this code" commands to get to the final optimized result, which means the whole thing was solved (LLM execution time) in under 1 minute.

Did you read the two paragraphs written above and the one where he made that statement? My comments on "what you are not sure" is that Max is a software engineer (I am sure a good one) and he kept iterating the code until it reached close to 100x faster code because he knew what "write better code" looked like. Now ask yourself this question: Is there any chance a no-code/low-code developer will come to a conclusion…

> Max is a software engineer (I am sure a good one)

Opinions are mixed.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#289

Post by qup »

ActivePattern wrote: post #197
qup wrote: post #179

Earlier quoted context omitted.

Are you sure it would be hard? Maybe it only requires asking the LLM to be creative when designing the algorithm. The parent poster spent some time thinking about it, obviously--he didn't generate it accurately "on the fly," either. But he's able to direct his own attention. I don't see why the LLM couldn't come up with this logic, if prompted to think about a clever algorithm that was highly specific to this problem…

I suspect that it would be unlikely to come up with it because it requires execution of a fairly lengthy algorithm (or sophisticated mathematical reasoning) to find the smallest/largest valid numbers in the range. You can verify this for yourself with the following ChatGPT prompt: "What is the smallest number in the range (1, 100000) whose digits sum to 30? Do not execute separate code."

O1 did find the optimization in a sibling comment (sibling to my GP)

So probably time to update your expectations

Re: Can LLMs write better code if you keep asking them to “write better code”?

#290

Post by [deleted] »

brabel wrote: post #270
afiodorov wrote: post #171

Earlier quoted context omitted.

Another speed-up is to skip the sum of digits check if n % 9 != 30 % 9. Sum of digits have the same remainder divided by 9 as the number. This rules out 8/9 = 88% candidates.

Did you measure it? I would expect using % would ruin your performance as it's slow, even if it allows you to avoid doing a bunch of sums (which are fast).

[deleted]
Post reply on HN