Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

Post reply on HN
Print view View original

371–380 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#371

Post by thequux »

ryao wrote: post #368
ryao wrote: post #279

Earlier quoted context omitted.

You can do this “without” using the modulus operation by storing the numbers in a boolean array. Start at 3999 and keep adding 9 to find the minimum. Then start at 99930 and keep subtracting 9 to find the maximum. You would need to check if the number is in the array and then if the number’s digits sum to 30. Note that the conversion of numbers to base 10 to check the digits typically involves doing division and modu…

It turns out that there is no modular multiplicative inverse for this, so that trick cannot be used to avoid the modulus and division when getting the base 10 digits: https://extendedeuclideanalgorithm.com/calculator.php?mode=2...

Indeed there isn't; 10 is not relatively prime to 2^32. However, 5 is (and therefore has a multiplicative inverse), so you can right shift and then multiply by the inverse.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#372

Post by raybb »

Fizzadar wrote: post #247
dgacmu wrote: post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

This exactly highlights my fear of widespread use of LLMs for code - missing the actual optimisations because we’re stuck in a review, rather than create, mode of thinking. But maybe that’s a good thing for those of us not dependent on LLMs :)

Well if you or anyone else that has good optimization and performance chops http://openlibrary.org/ has been struggling with performance a bit lately and it's hard to track down the cause. CPU load is low and nothing too much has changed lately so it's unlikely to be a bad query or something.

Main thing I've suggested is upgrading the DB from Postgres 9, which isn't an easy task but like 15 years of DB improvements probably would give some extra performance.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#373

Post by Dylan16807 »

afiodorov wrote: post #362
Dylan16807 wrote: post #357

Earlier quoted context omitted.

In that comment, you are generating your own random numbers and then optimizing away the actual generation. It can't take an input array. While clever, I think that strays too far from the initial prompt.

All I need is the proportion of the qualifying numbers to the input array to run the algorithm and the number of samples. Then we can sample min, max index of the qualifying array and return their difference without having to sample many times if we can derive the joined min max distribution conditional on the Bernoulli. In other words the procedure can take any input array and qualifying criteria. The joint distribu…

Sampling doesn't give you the actual answer for an actual array. If the program uses the array for multiple things, such as organizing the numbers after allocating the correct number of buckets, your method will cause logic errors and crashes.

The O(1) method based on statistics only works when the function making this calculation can hide the array (or lack of array) behind a curtain the entire time. If it has to take an array as input, or share its array as output, the facade crumbles.

The prompt is not "generate this many random numbers and then say max qualifying minus min qualifying". If it was, your method would give valid solutions. But the prompt starts with "Given a list".

In the article, we let ChatGPT generate the random numbers as a matter of convenience. But the timing results are only valid as long as it keeps that part intact and isolated. We have to be able to swap it out for any other source of random numbers. If it invents a method that can't do that, it has failed.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#374

Post by evilduck »

emseetech wrote: post #244
evilduck wrote: post #234

Earlier quoted context omitted.

The code that ChatGPT and Claude will output via their chat interfaces is a night and day difference from what will be output from tools built around their APIs. You "can" get the web UI to behave similarly but it's both tedious and slow to manually copy and paste all of that into your context during each interaction and the output will be unfriendly towards human interaction to paste it back out to your project. But…

I give every AI tool a college try and have since the copilot beta. I’m simply not interested in having these tools type for me. Typing is nowhere near the hardest part of my job and I find it invaluable as a meditative state for building muscle memory for the context of what I’m building. Taking shortcuts has a cost I’m not willing to pay.

If your position is to find fault then that’s what you will accomplish.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#375

Post by dgacmu »

raybb wrote: post #372
Fizzadar wrote: post #247

Earlier quoted context omitted.

This exactly highlights my fear of widespread use of LLMs for code - missing the actual optimisations because we’re stuck in a review, rather than create, mode of thinking. But maybe that’s a good thing for those of us not dependent on LLMs :)

Well if you or anyone else that has good optimization and performance chops http://openlibrary.org/ has been struggling with performance a bit lately and it's hard to track down the cause. CPU load is low and nothing too much has changed lately so it's unlikely to be a bad query or something. Main thing I've suggested is upgrading the DB from Postgres 9, which isn't an easy task but like 15 years of DB improvements p…

It might not be as awful as feared? That big a jump probably requires a dump and restore, but maybe it could still be done in place. pg_upgrade is pretty solid. But I agree - it's likely a cheap and easy perf win.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#377

Post by blueaquilae »

cultureulterior wrote: post #265
ActivePattern wrote: post #155

Earlier quoted context omitted.

This is actually a great example of an optimization that would be extremely difficult for an LLM to find. It requires a separate computation to find the smallest /largest numbers in the range with digits summing to 30. Hence, an LLM is unlikely to be able to generate them accurately on-the-fly.

O1 found it. https://chatgpt.com/share/67782b6b-6248-8012-882d-238b600ef9...

Did it found it before the HN comment? O1 has access to the web so I'm just asking

Re: Can LLMs write better code if you keep asking them to “write better code”?

#378

Post by [deleted] »

btbuildem wrote: post #140

I've noticed this with GPT as well -- the first result I get is usually mediocre and incomplete, often incorrect if I'm working on something a little more obscure (eg, OpenSCAD code). I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". The next part is a little strange - it arose out of frustration, but it also seems to improve results. Let's call it "negative incent…

[deleted]

Re: Can LLMs write better code if you keep asking them to “write better code”?

#379

Post by joshka »

> Of course, these LLMs won’t replace software engineers anytime soon, because it requires a strong engineering background to recognize what is actually a good idea, along with other constraints that are domain specific. Even with the amount of code available on the internet, LLMs can’t discern between average code and good, highly-performant code without guidance.

There are some objective measures which can be pulled out of the code and automated (complexity measures, use of particular techniques / libs, etc.) These can be automated, and then LLMs can be trained to be decent at recognizing more subjective problems (e.g. naming, obviousness, etc.). There are a lot of good engineering practices which come down to doing the same thing as the usual thing which is in that space rather than doing something new. An engine that is good at detecting novelties seems intuitively like it would be helpful in recognizing good ideas (even given the problems of hallucinations so far seen). Extending the idea of the article to this aspect, the problem seems like it's one of prompting / training rather than a terminal blocker.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#380

Post by petee »

I've found them decent and mimicking existing code for boiler plate, or analysis (it feels neat when it 'catches' a race or timing issue) but writing code needs constant supervision and second guessing to the point I feel its more handy to have it show just comparisons of possible implementations, and you write the code with your new insight.

Learning a Lisp-y language, I do often find myself asking it for suggestions on how to write less imperative code, which seem to come out better than if conjured from a request alone. But again, thats feeding it examples

Post reply on HN