Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

Post reply on HN
Print view View original

181–190 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#181

Post by asah »

ActivePattern wrote: post #155
gjm11 wrote: post #137

Earlier quoted context omitted.

There's another, arguably even simpler, optimization that makes me smile. (Because it's silly and arises only from the oddity of the task, and because it's such a huge performance gain.) You're picking 1,000,000 random numbers from 1 to 100,000. That means that any given number is much more likely to appear than not. In particular, it is very likely that the list contains both 3999 (which is the smallest number with…

This is actually a great example of an optimization that would be extremely difficult for an LLM to find. It requires a separate computation to find the smallest /largest numbers in the range with digits summing to 30. Hence, an LLM is unlikely to be able to generate them accurately on-the-fly.

You guys are picking on the problem statement. Here's a revised prompt, which also skips the silliness of single threading:

   Write __fully parallelized__ Python code to solve this problem: __Generate__ 1 million random integers between 1 and 10,000,000, find the difference between the smallest and the largest numbers whose digits sum up to 30.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#182

Post by lostmsu »

shahzaibmushtaq wrote: post #165

2 lessons to learn from this blog: > these LLMs won’t replace software engineers anytime soon, because it requires a strong engineering background to recognize what is actually a good idea, along with other constraints that are domain specific. > One issue with my experiments is that I’m benchmarking code improvement using Python, which isn’t the coding language developers consider when hyperoptimizing performance.

TBH I'm not sure how he arrived at "won’t replace software engineers anytime soon"

The LLM solved his task. With his "improved prompt" the code is good. The LLM in his setup was not given a chance to actually debug its code. It only took him 5 "improve this code" commands to get to the final optimized result, which means the whole thing was solved (LLM execution time) in under 1 minute.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#183

Post by danielbln »

SparkyMcUnicorn wrote: post #113
babyent wrote: post #81

Earlier quoted context omitted.

I used to really like Claude for code tasks but lately it has been a frustrating experience. I use it for writing UI components because I just don’t enjoy FE even though I have a lot of experience on it from back in the day. I tell it up front that I am using react-ts and mui. 80% of the time it will use tailwind classes which makes zero sense. It won’t use the sx prop and mui system. It is also outdated it seems. It…

To each their own, and everyone's experience seems to vary, but I have a hard time picturing people using Claude/ChatGPT web UIs for any serious developmen. It seems like so much time would he wasted recreating good context, copy/pasting, etc. We have tools like Aider (which has copy/paste mode if you don't have API access for some reason), Cline, CoPilot edit mode, and more. Things like having a conventions file and…

Before tue advent of proper IDE integrations and editors like Zed, copy pasting form the web UI was basically how things were done, and man was it daunting. As you say, having good, fine grained, repeatable and we'll integrated context management is paramount to efficient LLM based work.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#184

Post by james_marks »

lolinder wrote: post #104
jdietrich wrote: post #87

Earlier quoted context omitted.

>I often run into LLMs writing "beginner code" that uses the most fundamental findings in really impractical ways. Trained on too many tutorials I assume. In the absence of any other context, that's probably a sensible default behaviour. If someone is just asking "write me some code that does x", they're highly likely to be a beginner and they aren't going to be able to understand or reason about a more sophisticated…

In my experience the trouble with LLMs at the professional level is that they're almost as much work to prompt to get the right output as it would be to simply write the code. You have to provide context, ask nicely, come up with and remind it about edge cases, suggest which libraries to use, proofread the output, and correct it when it inevitably screws up anyway. I use Copilot for autocomplete regularly, and that's…

After 6 months of co-pilot autocomplete in my text editor feeling like an uninformed back seat driver with access to the wheel, I turned it off yesterday.

It’s night and day to what I get from Claude sonnet 3.5 in their UI, and even then only on mainstream languages.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#185

Post by ActivePattern »

asah wrote: post #181
ActivePattern wrote: post #155

Earlier quoted context omitted.

This is actually a great example of an optimization that would be extremely difficult for an LLM to find. It requires a separate computation to find the smallest /largest numbers in the range with digits summing to 30. Hence, an LLM is unlikely to be able to generate them accurately on-the-fly.

You guys are picking on the problem statement. Here's a revised prompt, which also skips the silliness of single threading: Write __fully parallelized__ Python code to solve this problem: __Generate__ 1 million random integers between 1 and 10,000,000, find the difference between the smallest and the largest numbers whose digits sum up to 30.

Correct, this optimization no longer works when you change the problem.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#186

Post by ADeerAppeared »

pbhjpbhj wrote: post #164
fhueller wrote: post #157

> how to completely uninstall and reinstall postgresql on a debian distribution without losing the data in the database. https://www.phind.com/search?cache=lrcs0vmo0wte5x6igp5i3607 Still seem to struggle on basic instructions, and even understanding what it itself is doing. sudo rm -rf /etc/postgresql sudo rm -rf /var/lib/postgresql sudo rm -rf /var/log/postgresql > This process removes all PostgreSQL components, cle…

Is the problem that the antonym is a substring within "without losing the data in the database"? I've seen problems with opposites for LLMs before. If you specify "retaining the data" or "keeping the data" does it get it right?

That's a red herring.

The problem is that these are fundamentally NOT reasoning systems. Even when contorted into "reasoning" models, these are just stochastic parrots guessing the next words in the hopes that it's the correct reasoning "step" in the context.

No approach is going to meaningfully work here. Fiddling with the prompt may get you better guesses, but they will always be guesses. Even without the antonym it's just a diceroll on whether the model will skip or add a step.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#187

Post by irthomasthomas »

This is great! I wish I could bring myself to blog, as I discovered this accidentally around March. I was experimenting with an agent that acted like a ghost in the machine and interacted via shell terminals. It would start every session by generating a greeting in ASCII art. On one occasion, I was shocked to see that the greeting was getting better each time it ran. When I looked into the logs, I saw that there was a mistake in my code which was causing it to always return an error message to the model, even when no error occurred. The model interpreted this as an instruction to try and improve its code.

Some more observations: New Sonnet is not universally better than Old Sonnet. I have done thousands of experiments in agentic workflows using both, and New Sonnet fails regularly at the same tasks Old Sonnet passes. For example, when asking it to update a file, Old Sonnet understands that updating a file requires first reading the file, whereas New Sonnet often overwrites the file with 'hallucinated' content.

When executing commands, Old Sonnet knows that it should wait for the execution output before responding, while New Sonnet hallucinates the command outputs.

Also, regarding temperature: 0 is not always more deterministic than temperature 1. If you regularly deal with code that includes calls to new LLMs, you will notice that, even at temperature 0, it often will 'correct' the model name to something it is more familiar with. If the subject of your prompt is newer than the model's knowledge cutoff date, then a higher temperature might be more accurate than a lower temperature.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#188

Post by qup »

emseetech wrote: post #112
babyent wrote: post #81

Earlier quoted context omitted.

I used to really like Claude for code tasks but lately it has been a frustrating experience. I use it for writing UI components because I just don’t enjoy FE even though I have a lot of experience on it from back in the day. I tell it up front that I am using react-ts and mui. 80% of the time it will use tailwind classes which makes zero sense. It won’t use the sx prop and mui system. It is also outdated it seems. It…

I've stopped using LLMs to write code entirely. Instead, I use Claude and Qwen as "brilliant idiots" for rubber ducking. I never copy and paste code it gives me, I use it to brainstorm and get me unstuck. I'm more comfortable using it this way.

People should try to switch to a more code-focused interface, like aider.

Copy and pasting code it gives you just means your workflow is totally borked, and it's no wonder you wouldn't want to try to let it generate code, because it's such a pain in your ass to try it, diff it, etc.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#189

Post by Vetch »

ActivePattern wrote: post #155
gjm11 wrote: post #137

Earlier quoted context omitted.

There's another, arguably even simpler, optimization that makes me smile. (Because it's silly and arises only from the oddity of the task, and because it's such a huge performance gain.) You're picking 1,000,000 random numbers from 1 to 100,000. That means that any given number is much more likely to appear than not. In particular, it is very likely that the list contains both 3999 (which is the smallest number with…

This is actually a great example of an optimization that would be extremely difficult for an LLM to find. It requires a separate computation to find the smallest /largest numbers in the range with digits summing to 30. Hence, an LLM is unlikely to be able to generate them accurately on-the-fly.

Excellent point. The hope is reasoning LLMs will make a difference for such problems. But it's also a great example of why the those who think being able to have the LLM iterate more will be crucial to reasoning are off base. There are many computations that a transformers (or humans for that matter) are not well equipped to represent internally, tool use during the reasoning process is unavoidable for all but the artificial or knowledge heavy problems.

Small examples, throwaway but involved calculations, prototypes, notes of what didn't work and what's promising are what's crucial for novel reasoning. It goes beyond just search or iterative refinement; there is no royal road to reasoning.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#190

Post by qup »

danielbln wrote: post #183
SparkyMcUnicorn wrote: post #113

Earlier quoted context omitted.

To each their own, and everyone's experience seems to vary, but I have a hard time picturing people using Claude/ChatGPT web UIs for any serious developmen. It seems like so much time would he wasted recreating good context, copy/pasting, etc. We have tools like Aider (which has copy/paste mode if you don't have API access for some reason), Cline, CoPilot edit mode, and more. Things like having a conventions file and…

Before tue advent of proper IDE integrations and editors like Zed, copy pasting form the web UI was basically how things were done, and man was it daunting. As you say, having good, fine grained, repeatable and we'll integrated context management is paramount to efficient LLM based work.

You gave me flashbacks of editing html, php, and perl in some shitty web editor within cPanel in the 1990s.

That was before, or maybe alongside, my Notepad++ / CuteFTP workflow.

Post reply on HN