Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

Post reply on HN
Print view View original

271–280 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#271

Post by nyrikki »

pama wrote: post #257
nyrikki wrote: post #231

Earlier quoted context omitted.

Here is a fairly good lecture series on graduate level complexity theory that will help understand parts. At least why multiple iterations help but why they also aren't the answer to super human results. https://youtube.com/playlist?list=PLm3J0oaFux3b8Gg1DdaJOzYNs...

Thanks for the tip, though I’m not sure how complexity theory will explain the impossibility of superhuman results. The main advantage ML methods have over humans is that they train much faster. Just like humans, they get better with more training. When they are good enough, they can be used to generate synthetic data, especially for cases like software optimization, when it is possible to verify the ground truth. A…

ML is better than biological neurons in some tasks, they are different contexts.

Almost all the performance of say college tests are purely from the pre-training, pattern finding and detection.

Transformers are limited to DLOGTIME-uniform TC0, they can't even do the Boolean circuit value problem.

The ability to use the properties of BPP, does help.

Understanding the power of, and limitations of iteration and improving approximations requires descriptive complexity theory IMHO.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#272

Post by [deleted] »

asah wrote: post #181
ActivePattern wrote: post #155

Earlier quoted context omitted.

This is actually a great example of an optimization that would be extremely difficult for an LLM to find. It requires a separate computation to find the smallest /largest numbers in the range with digits summing to 30. Hence, an LLM is unlikely to be able to generate them accurately on-the-fly.

You guys are picking on the problem statement. Here's a revised prompt, which also skips the silliness of single threading: Write __fully parallelized__ Python code to solve this problem: __Generate__ 1 million random integers between 1 and 10,000,000, find the difference between the smallest and the largest numbers whose digits sum up to 30.

[deleted]

Re: Can LLMs write better code if you keep asking them to “write better code”?

#273

Post by xrd »

simonw wrote: post #154
xrd wrote: post #19

Wow, what a great post. I came in very skeptical but this changed a lot of misconceptions I'm holding. One question: Claude seems very powerful for coding tasks, and now my attempts to use local LLMs seem misguided, at least when coding. Any disagreements from the hive mind on this? I really dislike sending my code into a for profit company if I can avoid it. Second question: I really try to avoid VSCode (M$ concerns…

Making the decision to trust companies like Anthropic with your data when they say things like "we won't train on your data" is the ultimate LLM productivity hack. It unlocks access to the currently best available coding models. That said, there are increasingly great coding models you can run locally. Qwen2.5-Coder-32B impressed me a lot a few months ago: https://simonwillison.net/2024/Nov/12/qwen25-coder/ The probl…

I have a 24gb Nvidia on my desktop machine and a tailscale/headscale network from my laptop. Unless I'm on a plane without Wi-Fi, I'm usually in a great place.

Thanks for your comment! I'm going to try out qwen.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#274

Post by menacingly »

dotancohen wrote: post #172
btbuildem wrote: post #140

I've noticed this with GPT as well -- the first result I get is usually mediocre and incomplete, often incorrect if I'm working on something a little more obscure (eg, OpenSCAD code). I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". The next part is a little strange - it arose out of frustration, but it also seems to improve results. Let's call it "negative incent…

> I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". Is that actually how you're prompting it? Does that actually give better results?

stuff like this working is why you get odd situations like "don't hallucinate" actually producing fewer hallucinations. it's to me one of the most interesting things about llms

Re: Can LLMs write better code if you keep asking them to “write better code”?

#275

Post by gcanyon »

masfuerte wrote: post #220
gcanyon wrote: post #97

As far as I can see, all the proposed solutions calculate the sums by doing division, and badly. This is in LiveCode, which I'm more familiar with than Python, but it's roughly twice as fast as the mod/div equivalent in LiveCode: repeat with i = 0 to 9 put i * 10000 into ip repeat with j = 0 to 9 put j * 1000 into jp repeat with k = 0 to 9 put k * 100 into kp repeat with l = 0 to 9 put l * 10 into lp repeat with m =…

I had a similar idea iterating over the previously calculated sums. I implemented it in C# and it's a bit quicker taking about 78% of the time to run yours. int[] sums = new int[100000]; for (int i = 9; i >= 0; --i) { sums[i] = i; } int level = 10; while (level = 0; --p) { int sum = sums[p]; for (int i = 9; i > 0; --i) { sums[level * i + p] = i + sum; } } level *= 10; }

Yep, I had a vague notion that I was doing too much work, but I was headed out the door so I wrote the naive/better than the original solution, benchmarked it quickly, and posted it before leaving. Yours also has the advantage of being scalable to ranges other than 1-100,000 without having to write more loop code.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#276

Post by ryao »

gjm11 wrote: post #137
dgacmu wrote: post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

There's another, arguably even simpler, optimization that makes me smile. (Because it's silly and arises only from the oddity of the task, and because it's such a huge performance gain.) You're picking 1,000,000 random numbers from 1 to 100,000. That means that any given number is much more likely to appear than not. In particular, it is very likely that the list contains both 3999 (which is the smallest number with…

With this trick you can test while generating the random numbers and if you see both values, you can short circuit the generation of random numbers.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#277

Post by freehorse »

cultureulterior wrote: post #268
Vetch wrote: post #242

Earlier quoted context omitted.

The information on the creative step which you provided to o1, was also the key step and contained almost all the difficulty. The hope is that 2025 models could eventually come up with solutions like this given enough time, but this is also a toy problem. The question is how much clever answers will cost for real world complex problems. At present it looks like, very much.

For me O1 found this by telling it "There is a further significant optimization possible."

What if you keep telling it that "there is a further significant optimization possible"?

Re: Can LLMs write better code if you keep asking them to “write better code”?

#279

Post by ryao »

brabel wrote: post #270
afiodorov wrote: post #171

Earlier quoted context omitted.

Another speed-up is to skip the sum of digits check if n % 9 != 30 % 9. Sum of digits have the same remainder divided by 9 as the number. This rules out 8/9 = 88% candidates.

Did you measure it? I would expect using % would ruin your performance as it's slow, even if it allows you to avoid doing a bunch of sums (which are fast).

You can do this “without” using the modulus operation by storing the numbers in a boolean array. Start at 3999 and keep adding 9 to find the minimum. Then start at 99930 and keep subtracting 9 to find the maximum. You would need to check if the number is in the array and then if the number’s digits sum to 30.

Note that the conversion of numbers to base 10 to check the digits typically involves doing division and modulus operations, so you are already doing those even if you remove the modulus operation from this check. That is unless you find a clever way of extracting the digits using the modular multiplicative inverse to calculate x/10^k.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#280

Post by bennythomsson »

ryao wrote: post #276
gjm11 wrote: post #137

Earlier quoted context omitted.

There's another, arguably even simpler, optimization that makes me smile. (Because it's silly and arises only from the oddity of the task, and because it's such a huge performance gain.) You're picking 1,000,000 random numbers from 1 to 100,000. That means that any given number is much more likely to appear than not. In particular, it is very likely that the list contains both 3999 (which is the smallest number with…

With this trick you can test while generating the random numbers and if you see both values, you can short circuit the generation of random numbers.

The input generation is outside the scope of this. Otherwise you could directly choose the output values with the apropriate distribution and just skip all the rest.

(Arguably, this criticism applies to exchanging random.randint for a numpy equivalent as well, since that doesn't optimize the solution but only how quickly the question is being generated.)

Post reply on HN