Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

Post reply on HN
Print view View original

121–130 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#121

Post by SkyBelow »

juujian wrote: post #55

I often run into LLMs writing "beginner code" that uses the most fundamental findings in really impractical ways. Trained on too many tutorials I assume. Usually, specifying the packages to use and asking for something less convoluted works really well. Problem is, how would you know if you have never learned to code without an LLM?

>Problem is, how would you know if you have never learned to code without an LLM?

The quick fix I use when needing to do something new is to ask the AI to list me different libraries and the pros and cons of using them. Then I quickly hop on google and check which have good documentation and examples so I know I have something to fall back on, and from there I ask the AI how to solve small simple version of my problem and explain what the library is doing. Only then do I ask it for a solution and see if it is reasonable or not.

It isn't perfect, but it saves enough time most times to more than make up for when it fails and I have to go back to old fashion RTFMing.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#122

Post by mackopes »

b112 wrote: post #82
ben_w wrote: post #65

Earlier quoted context omitted.

Even as someone with plenty of experience, this can still be a problem: I use them for stuff outside my domain, but where I can still debug the results. In my case, this means I use it for python and web frontend, where my professional experience has been iOS since 2010. ChatGPT has, for several generations, generally made stuff that works, but the libraries it gives me are often not the most appropriate, and are som…

I wish people would understand what a large language model is. There is no thinking. No comprehension. No decisions. Instead, think of your queries as super human friendly SQL. The database? Massive amounts of data boiled down to unique entries with probabilities. This is a simplistic, but accurate way to think of LLMs. So how much code is on the web for a particular problem solve? 10k blog entries, stackoverflow res…

> think of your queries as super human friendly SQL > The database? Massive amounts of data boiled down to unique entries with probabilities. This is a simplistic, but accurate way to think of LLMs.

I disagree that this is the accurate way to think about LLMs. LLMs still use a finite number of parameters to encode the training data. The amount of training data is massive in comparison to the number of parameters LLMs use, so they need to be somewhat capable of distilling that information into small pieces of knowledge they can then reuse to piece together the full answer.

But this being said, they are not capable of producing an answer outside of the training set distribution, and inherit all the biases of the training data as that's what they are trying to replicate.

> I guess my point is, when you use LLMs for tasks, you're getting whatever other humans have said. And I've seen some pretty poor code examples out there. Yup, exactly this.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#123

Post by arkh »

soulofmischief wrote: post #94
arkh wrote: post #77

Earlier quoted context omitted.

> The interface can do exactly what it says on the tin. In theory. Do some code maintenance and you'll soon find that many things don't do what it says on the tin. Hence the need for debug and maintenance. And then going through multiple levels of indirection to get to your bug will make you start hating some "good code".

Yes, that's what can means. It's still the developer's responsibility to correctly write and test code such that things do what they say on the tin. What's worse is trying to navigate an imperatively written 2000-line single-function, untestable module with undocumented, unabstracted routines found in ten other places in the codebase. This is something I've encountered plenty in my career, always written by people wh…

And on the other side you see a lot of single implementation interfaces; or 2 lines methods which call perfectly named methods 7 levels deep which could have been a 50 lines method easy to grok on a screen with zero scrolling.

So things are on a spectrum depending on the situation and what you want to accomplish => measuring code quality is not a simple thing.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#124

Post by bwilliams »

scosman wrote: post #118

By iterating it 5 times the author is using ~5x the compute. It’s kinda a strange chain of thought. Also: premature optimization is evil. I like the first iteration most. It’s not “beginner code”, it’s simple. Tell sonnet to optimize it IF benchmarks show it’s a pref problem. But a codebase full of code like this, even when unnecessary, would be a nightmare.

I had the same thought when reading the article too. I assumed (and hoped) it was for the sake of the article because there’s a stark difference between idiomatic code and performance focused code.

Living and working in a large code base that only focuses on “performance code” by default sounds very frustrating and time consuming.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#125

Post by martin_ »

I've observed given that LLM's inherently want to autocomplete, they're more inclined to keep complicating a solution than rewrite it because it was directionally bad. The most effective way i've found to combat this is to restart a session and prompt it such that it produces an efficient/optimal solution to the concrete problem... then give it the problematic code and ask it to refactor it accordingly

Re: Can LLMs write better code if you keep asking them to “write better code”?

#126

Post by tyingq »

juujian wrote: post #55

I often run into LLMs writing "beginner code" that uses the most fundamental findings in really impractical ways. Trained on too many tutorials I assume. Usually, specifying the packages to use and asking for something less convoluted works really well. Problem is, how would you know if you have never learned to code without an LLM?

Other imperfect things you can add to a prompt:

  - asking for fully type annotated python, rather than just python
  - specifically ask it for performance optimized code
  - specifically ask for code with exception handling
  - etc
Things that might lead it away from tutorial style code.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#127

Post by vitus »

Am I misinterpreting the prompt, or did the LLM misinterpret it from the get-go?

    Given a list of 1 million random integers between 1 and 100,000, find the difference between the smallest and the largest numbers whose digits sum up to 30.
That doesn't read to me as "generate a list of 1 million random integers, then find the difference ..." but rather, "write a function that takes a list of integers as input".

That said, my approach to "optimizing" this comes down to "generate the biggest valid number in the range (as many nines as will fit, followed by whatever digit remains, followed by all zeroes), generate the smallest valid number in the range (biggest number with its digits reversed), check that both exist in the list (which should happen With High Probability -- roughly 99.99% of the time), then return the right answer".

With that approach, the bottleneck in the LLM's interpretation is generating random numbers: the original random.randint approach takes almost 300ms, whereas just using a single np.random.randint() call takes about 6-7ms. If I extract the random number generation outside of the function, then my code runs in ~0.8ms.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#128

Post by SkyBelow »

b112 wrote: post #82
ben_w wrote: post #65

Earlier quoted context omitted.

Even as someone with plenty of experience, this can still be a problem: I use them for stuff outside my domain, but where I can still debug the results. In my case, this means I use it for python and web frontend, where my professional experience has been iOS since 2010. ChatGPT has, for several generations, generally made stuff that works, but the libraries it gives me are often not the most appropriate, and are som…

I wish people would understand what a large language model is. There is no thinking. No comprehension. No decisions. Instead, think of your queries as super human friendly SQL. The database? Massive amounts of data boiled down to unique entries with probabilities. This is a simplistic, but accurate way to think of LLMs. So how much code is on the web for a particular problem solve? 10k blog entries, stackoverflow res…

Every model for how to approach an LLM seems lacking to me. I would suggest anyone using AI heavily to take a weekend and make a simple one to do the handwriting digit recognition. Once you get a feel for basic neural network, then watch a good introduction to alexnet. Then you can think of an LLM as being the next step in the sequence.

>I guess my point is, when you use LLMs for tasks, you're getting whatever other humans have said.

This isn't correct. It embeds concepts that humans have discussed, but can combine them in ways that were never in the training set. There are issues with this, the more unique the combination of concepts, the more likely the output ends up being unrelated to what the user was wanting to see.

Post reply on HN