Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

Post reply on HN
Print view View original

261–270 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#261

Post by simonw »

lostmsu wrote: post #182
shahzaibmushtaq wrote: post #165

2 lessons to learn from this blog: > these LLMs won’t replace software engineers anytime soon, because it requires a strong engineering background to recognize what is actually a good idea, along with other constraints that are domain specific. > One issue with my experiments is that I’m benchmarking code improvement using Python, which isn’t the coding language developers consider when hyperoptimizing performance.

TBH I'm not sure how he arrived at "won’t replace software engineers anytime soon" The LLM solved his task. With his "improved prompt" the code is good. The LLM in his setup was not given a chance to actually debug its code. It only took him 5 "improve this code" commands to get to the final optimized result, which means the whole thing was solved (LLM execution time) in under 1 minute.

A non-engineer would not be able to interpret ANY of what he did here, or fix any of the bugs.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#262

Post by Someone »

gadders wrote: post #150
dgacmu wrote: post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

I had a scan of the code examples, but one other idea that occurred to me is that you could immediately drop any numbers below 999 (probably slightly higher, but that would need calculation rather than being intuitive).

> probably slightly higher, but that would need calculation rather than being intuitive

I think it’s easy to figure out that 3999 is the smallest positive integer whose decimal digits add up to 30 (can’t get there with 3 digits, and for 4, you want the first digit to be as small as possible. You get that by making the other 3 as high as possible)

Re: Can LLMs write better code if you keep asking them to “write better code”?

#263

Post by sneak »

jmartinpetersen wrote: post #49

> "As LLMs drastically improve, the generated output becomes more drastically average" Thanks, that really made it click for me.

Average software developers producing average code cost high five to low six figures per year.

LLMs are a tiny tiny fraction of that.

For a majority of software, average code that does the CRUD thing or whatever is fine.

Even if LLMs never get better or cheaper than they are today, our entire industry is forever changed (for the better).

Re: Can LLMs write better code if you keep asking them to “write better code”?

#264

Post by Terr_ »

b112 wrote: post #82
ben_w wrote: post #65

Earlier quoted context omitted.

Even as someone with plenty of experience, this can still be a problem: I use them for stuff outside my domain, but where I can still debug the results. In my case, this means I use it for python and web frontend, where my professional experience has been iOS since 2010. ChatGPT has, for several generations, generally made stuff that works, but the libraries it gives me are often not the most appropriate, and are som…

I wish people would understand what a large language model is. There is no thinking. No comprehension. No decisions. Instead, think of your queries as super human friendly SQL. The database? Massive amounts of data boiled down to unique entries with probabilities. This is a simplistic, but accurate way to think of LLMs. So how much code is on the web for a particular problem solve? 10k blog entries, stackoverflow res…

> Instead, think of your queries as super human friendly SQL.

I feel that comparison oversells things quite a lot.

The user is setting up a text document which resembles a question-and-response exchange, and executing a make-any-document-bigger algorithm.

So it's less querying for data and more like shaping a sleeping dream of two fictional characters in conversation, in the hopes that the dream will depict one character saying something superficially similar to mostly-vanished data.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#265

Post by cultureulterior »

ActivePattern wrote: post #155
gjm11 wrote: post #137

Earlier quoted context omitted.

There's another, arguably even simpler, optimization that makes me smile. (Because it's silly and arises only from the oddity of the task, and because it's such a huge performance gain.) You're picking 1,000,000 random numbers from 1 to 100,000. That means that any given number is much more likely to appear than not. In particular, it is very likely that the list contains both 3999 (which is the smallest number with…

This is actually a great example of an optimization that would be extremely difficult for an LLM to find. It requires a separate computation to find the smallest /largest numbers in the range with digits summing to 30. Hence, an LLM is unlikely to be able to generate them accurately on-the-fly.

O1 found it.

https://chatgpt.com/share/67782b6b-6248-8012-882d-238b600ef9...

Re: Can LLMs write better code if you keep asking them to “write better code”?

#266

Post by nyrikki »

pama wrote: post #239
nyrikki wrote: post #221

Earlier quoted context omitted.

Because you cite is about: > in-context learning LLMs have no concept of the symantic meaning of what they do, they just are dealing with next token prediction. "in-context learning" is the problem, not the solution to general programming tasks. Memoryless, ergodic, sub Turing complete problems are a very tiny class. Think about how the Entscheidungsproblem relates to halting or the frame problem and the specificatio…

My main concern with the simplification of memorization or near neighbor interpolation that is commonly assumed for LLMs is that these methods are ineffective at scale and unlikely to be used by decoder transformers in practice. That paper shows that the decoder transformer somehow came up with a better decision tree fitting algorithm for low data cases than any of the conventional or boosted tree solutions humans ty…

Littlestone and Warmuth make the connection to compression in1986, which was later shown to be equivalent to VC dimensionally or PAC learnablilty.

Look into DBScan, OPTICs for far closer lenses on how clustering works in modern ML commercial ML, KNN not the only form of clustering.

But it is still in-context, additional compression that depends on a decider function, or equivalently a composition linearized set shattering parts.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#267

Post by Der_Einzige »

Normies discover that inference time scaling works. More news at 11!

BTW - prompt optimization is a supported use-case of several frameworks, like dspy and textgrad, and is in general something that you should be doing yourself anyway on most tasks.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#268

Post by cultureulterior »

Vetch wrote: post #242
batterseapower wrote: post #224

Earlier quoted context omitted.

I tried it in OpenAI's O1. If I give it minimaxir's original prompt it writes the obvious loop, even if I include the postamble "Look for tricks that will make this function run as fast as possible in the common case". However, if I then simply ask "What is the most probable result for this function to return?" it figures out the answer and a very good approximation of the probability (4.5e-5). From there it's easily…

The information on the creative step which you provided to o1, was also the key step and contained almost all the difficulty. The hope is that 2025 models could eventually come up with solutions like this given enough time, but this is also a toy problem. The question is how much clever answers will cost for real world complex problems. At present it looks like, very much.

For me O1 found this by telling it "There is a further significant optimization possible."

Re: Can LLMs write better code if you keep asking them to “write better code”?

#269

Post by Der_Einzige »

jmartinpetersen wrote: post #49

> "As LLMs drastically improve, the generated output becomes more drastically average" Thanks, that really made it click for me.

I don't know how many times I'm going to have to post just one of the papers which debunk this tired trope. As models become more intelligent, they also become more plural, more like multiplicities, and yes, much more (super humanely) creative. You can unlock creativity in today's LLMs by doing intelligent sampling on high temperature outputs.

https://openreview.net/forum?id=FBkpCyujtS

Re: Can LLMs write better code if you keep asking them to “write better code”?

#270

Post by brabel »

afiodorov wrote: post #171
dgacmu wrote: post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

Another speed-up is to skip the sum of digits check if n % 9 != 30 % 9. Sum of digits have the same remainder divided by 9 as the number. This rules out 8/9 = 88% candidates.

Did you measure it? I would expect using % would ruin your performance as it's slow, even if it allows you to avoid doing a bunch of sums (which are fast).
Post reply on HN