Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

411–420 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#411
post #405

Earlier quoted context omitted.

Doing a single modulo 9 operation is much faster than summing a d-digit number, which requires d modulo 10s, d divide 10s, and d sums.

You can do the sum by looking directly at the digits in the string, no need for module at all.

Number to string is even more expensive.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#412

Earlier quoted context omitted.

O1 found it. https://chatgpt.com/share/67782b6b-6248-8012-882d-238b600ef9...

Did it found it before the HN comment? O1 has access to the web so I'm just asking

That's actually a good point. Sadly "open"ai obfuscates this, too, so it is impossible to know.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#413
post #154
post #19

Wow, what a great post. I came in very skeptical but this changed a lot of misconceptions I'm holding. One question: Claude seems very powerful for coding tasks, and now my attempts to use local LLMs seem misguided, at least when coding. Any disagreements from the hive mind on this? I really dislike sending my code into a for profit company if I can avoid it. Second question: I really try to avoid VSCode (M$ concerns…

Making the decision to trust companies like Anthropic with your data when they say things like "we won't train on your data" is the ultimate LLM productivity hack. It unlocks access to the currently best available coding models. That said, there are increasingly great coding models you can run locally. Qwen2.5-Coder-32B impressed me a lot a few months ago: https://simonwillison.net/2024/Nov/12/qwen25-coder/ The probl…

I second qwen. It is very useable model. Sonnet is of course better (also 200k context vs 32k), but sometimes I just cannot take the risk of letting any sensitive data "escape" in the context so i use qwen and it is pretty good.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#414

I've noticed this with GPT as well -- the first result I get is usually mediocre and incomplete, often incorrect if I'm working on something a little more obscure (eg, OpenSCAD code). I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". The next part is a little strange - it arose out of frustration, but it also seems to improve results. Let's call it "negative incent…

We've entered the voodoo witch doctor phase of LLM usage: " Enter thee this arcane incantation along with thy question into the idol and, lo, the ineffable machine spirits wilt be appeased and deign to grant thee the information thou hast asked for. "

It is because the chance of the right answer goes down exponentially as the complexity of what is being asked goes up.

Asking a simpler question is not voodoo.

On the other hand, I think many people are trying various rain dances and believing it was a specific dance that was the cause when it happened to rain.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#415

Normies discover that inference time scaling works. More news at 11! BTW - prompt optimization is a supported use-case of several frameworks, like dspy and textgrad, and is in general something that you should be doing yourself anyway on most tasks.

[flagged]

Re: Can LLMs write better code if you keep asking them to “write better code”?

#416
post #80

I'm amused that neither the LLM or the author identified one of the simplest and most effective optimizations for this code: Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary. On an m1 macbook pro, using numpy to generate the random numbers, using mod/div to do digit sum: Base: 55ms Test before…

> Test if the number is max _before_ doing the digit sum. It's a free 5.5x speedup that renders some of the other optimizations, like trying to memoize digit sums, unnecessary.

How exactly did you arrive at this conclusion? The input is a million numbers in the range from 1 to 100000, chosen with a uniform random distribution; the minimum and maximum values are therefore very likely to be close to 1 and 100000 respectively - on average there won't be that much range to include. (There should only be something like a 1 in 11000 chance of excluding any numbers!)

On the other hand, we only need to consider numbers congruent to 3 modulo 9.

And memoizing digit sums is going to be helpful regardless because on average each value in the input appears 10 times.

And as others point out, by the same reasoning, the minimum and maximum values with the required digit sum are overwhelmingly likely to be present.

And if they aren't, we could just step through 9 at a time until we find the values that are in the input (and have the required digit sum; since it could differ from 30 by a multiple of 9) - building a `set` from the input values.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#417
post #270

Earlier quoted context omitted.

Another speed-up is to skip the sum of digits check if n % 9 != 30 % 9. Sum of digits have the same remainder divided by 9 as the number. This rules out 8/9 = 88% candidates.

Did you measure it? I would expect using % would ruin your performance as it's slow, even if it allows you to avoid doing a bunch of sums (which are fast).

Each sum involves determining the digits to sum, which involves using % multiple times.

Also, you don't have to use % in order to decide whether to perform the sum-of-digits check for a given value. You can just iterate over values to check in steps of 9.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#418

its best to tell them how you want the code written.

At that point isn't it starting to become easier to just write the code yourself? If I somehow have to formulate how I want a problem solved, then I've already done all the hard work myself. Having the LLM just do the typing of the code means that now not only did I have to solve the problem, I also get to do a code review.

Admittedly some people are using AI out of curiosity rather than because they get tangible benefit.

But aside from those situations, do you not think that the developers using AI - many of whom are experienced and respected - must be getting value? Or do you think they are deluded?

Re: Can LLMs write better code if you keep asking them to “write better code”?

#419
post #368

Earlier quoted context omitted.

It turns out that there is no modular multiplicative inverse for this, so that trick cannot be used to avoid the modulus and division when getting the base 10 digits: https://extendedeuclideanalgorithm.com/calculator.php?mode=2...

Indeed there isn't; 10 is not relatively prime to 2^32. However, 5 is (and therefore has a multiplicative inverse), so you can right shift and then multiply by the inverse.

All of this is missing the point that doing basic arithmetic like this in Python drowns in the overhead of manipulating objects (at least with the reference C implementation).

For that matter, the naive "convert to string and convert each digit to int" approach becomes faster in pure Python than using explicit div/mod arithmetic for very large numbers. This is in part thanks to algorithmic improvements implemented at least partially in Python (https://github.com/python/cpython/blob/main/Lib/_pylong.py#L...). But I can also see improved performance even for only a couple hundred digits (i.e. less than DIGLIM for the recursion) which I think comes from being able to do the div/mod loop in C (although my initial idea about the details doesn't make much sense if I keep thinking about it).

Re: Can LLMs write better code if you keep asking them to “write better code”?

#420

Earlier quoted context omitted.

This has been part of LLM usage since day 1, and I say that as an ardent fan of the tech. Let's not forget how much ink has been spilled over that fact that "think through this step by step" measurably improved/improves performance.

> "think through this step by step" Has always made sense to me, if you think how these models were trained. My experience with great stackoverflow responses and detailed blog posts, they often contain "think through this step by step" or something very similar. Intuitively adding that phrase should help the model narrow down the response content / formatting

Then why don't they hard-code the interface to the model to pretend you included that in the prompt?
Post reply on HN