Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

Post reply on HN
Print view View original

151–160 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#151

Post by [deleted] »

gjm11 wrote: post #137
dgacmu wrote: post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

There's another, arguably even simpler, optimization that makes me smile. (Because it's silly and arises only from the oddity of the task, and because it's such a huge performance gain.) You're picking 1,000,000 random numbers from 1 to 100,000. That means that any given number is much more likely to appear than not. In particular, it is very likely that the list contains both 3999 (which is the smallest number with…

[deleted]

Re: Can LLMs write better code if you keep asking them to “write better code”?

#152

Post by InsideOutSanta »

scosman wrote: post #118

By iterating it 5 times the author is using ~5x the compute. It’s kinda a strange chain of thought. Also: premature optimization is evil. I like the first iteration most. It’s not “beginner code”, it’s simple. Tell sonnet to optimize it IF benchmarks show it’s a pref problem. But a codebase full of code like this, even when unnecessary, would be a nightmare.

I'm glad I'm not the only one who felt that way. The first option is the one you should put into production, unless you have evidence that performance is going to be an issue. By that measure, the first response was the "best."

Re: Can LLMs write better code if you keep asking them to “write better code”?

#153

Post by charlieyu1 »

dgacmu wrote: post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

I actually think precomputing the numbers with digit sum 30 is the best approach. I'd give a very rough estimate of 500-3000 candidates because 30 is rather high, and we only need to loop for the first 4 digits because the fifth can be calculated. After that, it is O(1) set/dict lookups for each of the 1000000 numbers.

Everything can also be wrapped in list comprehensions for top performance.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#154

Post by simonw »

xrd wrote: post #19

Wow, what a great post. I came in very skeptical but this changed a lot of misconceptions I'm holding. One question: Claude seems very powerful for coding tasks, and now my attempts to use local LLMs seem misguided, at least when coding. Any disagreements from the hive mind on this? I really dislike sending my code into a for profit company if I can avoid it. Second question: I really try to avoid VSCode (M$ concerns…

Making the decision to trust companies like Anthropic with your data when they say things like "we won't train on your data" is the ultimate LLM productivity hack. It unlocks access to the currently best available coding models.

That said, there are increasingly great coding models you can run locally. Qwen2.5-Coder-32B impressed me a lot a few months ago: https://simonwillison.net/2024/Nov/12/qwen25-coder/

The problem I have is that models like that one take up 20+GB of RAM, and id rather use that to run more Chrome and Firefox windows! If I was serious about using local LLMs on a daily basis I'd set up a dedicated local server machine for them, super expensive though.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#155

Post by ActivePattern »

gjm11 wrote: post #137
dgacmu wrote: post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

There's another, arguably even simpler, optimization that makes me smile. (Because it's silly and arises only from the oddity of the task, and because it's such a huge performance gain.) You're picking 1,000,000 random numbers from 1 to 100,000. That means that any given number is much more likely to appear than not. In particular, it is very likely that the list contains both 3999 (which is the smallest number with…

This is actually a great example of an optimization that would be extremely difficult for an LLM to find. It requires a separate computation to find the smallest /largest numbers in the range with digits summing to 30. Hence, an LLM is unlikely to be able to generate them accurately on-the-fly.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#156

Post by nextaccountic »

b112 wrote: post #82
ben_w wrote: post #65

Earlier quoted context omitted.

Even as someone with plenty of experience, this can still be a problem: I use them for stuff outside my domain, but where I can still debug the results. In my case, this means I use it for python and web frontend, where my professional experience has been iOS since 2010. ChatGPT has, for several generations, generally made stuff that works, but the libraries it gives me are often not the most appropriate, and are som…

I wish people would understand what a large language model is. There is no thinking. No comprehension. No decisions. Instead, think of your queries as super human friendly SQL. The database? Massive amounts of data boiled down to unique entries with probabilities. This is a simplistic, but accurate way to think of LLMs. So how much code is on the web for a particular problem solve? 10k blog entries, stackoverflow res…

> I wish people would understand what a large language model is. There is no thinking. No comprehension. No decisions.

> Instead, think of your queries as super human friendly SQL.

Ehh this might be true in some abstract mathy sense (like I don't know, you are searching in latent space or something), but it's not the best analogy in practice. LLMs process language and simulate logical reasoning (albeit imperfectly). LLMs are like language calculators, like a TI-86 but for English/Python/etc, and sufficiently powerful language skills will also give some reasoning skills for free. (It can also recall data from the training set so this is where the SQL analogy shines I guess)

You could say that SQL also simulates reasoning (it is equivalent to Datalog after all) but LLMs can reason about stuff more powerful than first order logic. (LLMs are also fatally flawed in the sense it can't guarantee correct results, unlike SQL or Datalog or Prolog, but just like us humans)

Also, LLMs can certainly make decisions, such as the decision to search the web. But this isn't very interesting - a thermostat makes the decision of whether turn air refrigeration on or off, for example, and an operating system makes the decision of which program to schedule next on the CPU.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#157

Post by fhueller »

> how to completely uninstall and reinstall postgresql on a debian distribution without losing the data in the database.

https://www.phind.com/search?cache=lrcs0vmo0wte5x6igp5i3607

Still seem to struggle on basic instructions, and even understanding what it itself is doing.

   sudo rm -rf /etc/postgresql
   sudo rm -rf /var/lib/postgresql
   sudo rm -rf /var/log/postgresql
> This process removes all PostgreSQL components, cleans up leftover files, and reinstalls a fresh copy. By preserving the data directory (/var/lib/postgresql), we ensure that existing databases are retained. This method provides a clean slate for PostgreSQL while maintaining continuity of stored data.

Did we now?

Re: Can LLMs write better code if you keep asking them to “write better code”?

#158

Post by simonw »

Jimmc414 wrote: post #103

Interesting write up. It’s very possible that the "write better code" prompt might have worked simply because it allowed the model to break free from its initial response pattern, not because it understood "better"

The prompt works because every interaction with an LLM is from a completely fresh state.

When you reply "write better code" what you're actually doing is saying "here is some code that is meant to do X. Suggest ways to improve that existing code".

The LLM is stateless. The fact that it wrote the code itself moments earlier is immaterial.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#159

Post by glouwbug »

emseetech wrote: post #112
babyent wrote: post #81

Earlier quoted context omitted.

I used to really like Claude for code tasks but lately it has been a frustrating experience. I use it for writing UI components because I just don’t enjoy FE even though I have a lot of experience on it from back in the day. I tell it up front that I am using react-ts and mui. 80% of the time it will use tailwind classes which makes zero sense. It won’t use the sx prop and mui system. It is also outdated it seems. It…

I've stopped using LLMs to write code entirely. Instead, I use Claude and Qwen as "brilliant idiots" for rubber ducking. I never copy and paste code it gives me, I use it to brainstorm and get me unstuck. I'm more comfortable using it this way.

Having spent nearly 12 hours a day for a year with GPTs I agree that this is the way. Treat it like a professor on office hours who’s sometimes a little apathetically wrong because they’re overworked and underfunded

Re: Can LLMs write better code if you keep asking them to “write better code”?

#160

Post by EGreg »

bilbo0s wrote: post #148
btbuildem wrote: post #140

I've noticed this with GPT as well -- the first result I get is usually mediocre and incomplete, often incorrect if I'm working on something a little more obscure (eg, OpenSCAD code). I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". The next part is a little strange - it arose out of frustration, but it also seems to improve results. Let's call it "negative incent…

"If you get this wrong, OpenAI will be sued for a lot of money, and all the board members will go to prison" This didn't work. At least not on my task. What model were you using?

wait til 2027, you’ll see the result :-P
Post reply on HN