Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

Post reply on HN
Print view View original

61–70 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#61

Post by fragmede »

viraptor wrote: post #36
mrweasel wrote: post #21

Earlier quoted context omitted.

At that point isn't it starting to become easier to just write the code yourself? If I somehow have to formulate how I want a problem solved, then I've already done all the hard work myself. Having the LLM just do the typing of the code means that now not only did I have to solve the problem, I also get to do a code review.

There's no clear threshold with an universal answer. Sometimes prompting will be easier, sometimes writing things yourself. You'll have to add some debugging time to both sides in practice. Also, you can be opportunistic - you're going to write a commit anyway, right? A good commit message will be close to the prompt anyway, so why not start with that and see if you want to write your own or not? > I also get to do a…

why leave the commit message for the human to write? have the LLM start off and add relevant details it missed.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#62

Post by prettyStandard »

gweil wrote: post #59

has anyone tried saying "this will look good on your promo package"?

I'm not sure if you're joking or not, but I found I naturally encouraging remarks to the LLM saying

- You're doing better...

- Thanks that helps me...

And I just wonder if that actually has an improvement...

Re: Can LLMs write better code if you keep asking them to “write better code”?

#63

Post by netdevphoenix »

dash2 wrote: post #5

Something major missing from the LLM toolkit at the moment is that it can't actually run (and e.g. test or benchmark) its own code. Without that, the LLM is flying blind. I guess there are big security risks involved in making this happen. I wonder if anyone has figured out what kind of sandbox could safely be handed to a LLM.

Pretty sure this is done client-side by one of the big LLM companies. So it's virtually no risk for them

Re: Can LLMs write better code if you keep asking them to “write better code”?

#64

Post by GuB-42 »

mhmmmmmm wrote: post #22
AndrewOMartin wrote: post #10

Earlier quoted context omitted.

It'd be great if it could describe the performance of code in detail, but for now just adding a skill to detect if a bit of code has any infinite loops would be a quick and easy hack to be going on with.

Is reliably detecting if code has any infinite loops feasible? Sounds like the halting problem.

It is exactly the halting problem. Finding some infinite loops is possible, there are even some obvious cases, but finding "any" infinite loops is not. In fact, even the obvious cases are not if you take interrupts into account.

I think that's the joke. In a sci-fi story, that would make the computer explode.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#65

Post by ben_w »

juujian wrote: post #55

I often run into LLMs writing "beginner code" that uses the most fundamental findings in really impractical ways. Trained on too many tutorials I assume. Usually, specifying the packages to use and asking for something less convoluted works really well. Problem is, how would you know if you have never learned to code without an LLM?

Even as someone with plenty of experience, this can still be a problem: I use them for stuff outside my domain, but where I can still debug the results. In my case, this means I use it for python and web frontend, where my professional experience has been iOS since 2010.

ChatGPT has, for several generations, generally made stuff that works, but the libraries it gives me are often not the most appropriate, and are sometimes obsolete or no longer functional — and precisely because web and python are hobbies for me rather than my day job, it can take me a while to spot such mistakes.

Two other things I've noticed, related in an unfortunate way:

1) Because web and python not my day job, more often than not and with increasing frequency, I ultimately discover that when I disagree with ChatGPT, the AI was right and I was wrong.

2) These specific models often struggle when my response has been "don't use $thing or $approach"; unfortunately this seems to be equally applicable regardless of if the AI knew more than me or not, so it's not got predictive power for me.

(I also use custom instruction, you YMMV)

Re: Can LLMs write better code if you keep asking them to “write better code”?

#67

Post by whstl »

juujian wrote: post #55

I often run into LLMs writing "beginner code" that uses the most fundamental findings in really impractical ways. Trained on too many tutorials I assume. Usually, specifying the packages to use and asking for something less convoluted works really well. Problem is, how would you know if you have never learned to code without an LLM?

I actually find it super refreshing that they write "beginner" or "tutorial code".

Maybe because of experience: it's much simpler and easier to turn that into "senior code". After a few decades of experience I appreciate simplicity over the over-engineering mess that some mid-level developers tend to produce.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#68

Post by williamcotton »

mrweasel wrote: post #21
cranberryturkey wrote: post #3

its best to tell them how you want the code written.

At that point isn't it starting to become easier to just write the code yourself? If I somehow have to formulate how I want a problem solved, then I've already done all the hard work myself. Having the LLM just do the typing of the code means that now not only did I have to solve the problem, I also get to do a code review.

Spend your cognitive energy thinking about the higher level architecture, test cases and performance concerns rather than the minutia and you’ll find that you can get more work done with the less overall mental load.

This reduction in cognitive load is the real force multiplier.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#69

Post by whstl »

nuancebydefault wrote: post #46

My takeaway and also personal experience is that you get the best results is that you co-develop with the LLM. - write a simple prompt that explains in detail the wanted outcome. - look at the result, run it and ask it how it can improve. - tell it what to improve - tell it to make a benchmark and unit test - run it each time and see what is wrong or can be improved.

Same experience.

Also: If you're experienced at code reviews, you can get great results.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#70

Post by mixedbit »

dash2 wrote: post #5

Something major missing from the LLM toolkit at the moment is that it can't actually run (and e.g. test or benchmark) its own code. Without that, the LLM is flying blind. I guess there are big security risks involved in making this happen. I wonder if anyone has figured out what kind of sandbox could safely be handed to a LLM.

I have experimented with using LLM for improving unit test coverage of a project. If you provide the model with test execution results and updated test coverage information, which can be automated, the LLM can indeed fix bugs and add improvements to tests that it created. I found it has high success rate at creating working unit tests with good coverage. I just used Docker for isolating the LLM-generated code from the rest of my system.

You can find more details about this experiment in a blog post: https://mixedbit.org/blog/2024/12/16/improving_unit_test_cov...

Post reply on HN