Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

Post reply on HN
Print view View original

361–370 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#361

Post by IanCal »

withinboredom wrote: post #174
IanCal wrote: post #52

Earlier quoted context omitted.

It depends how you define reliably. The halting problem isn't so relevant in most development, and nothing stops you having a classifier that says "yes", "no" or "maybe". You can identify code that definitely finishes, and you can identify code that definitely doesn't. You can also identify some risky code that probably might. Under condition X, it would go into an infinite loop - even if you're not sure if condition…

The problem is that you can do this for specific functions/methods, but you cannot do this for a PROGRAM. All programs are "maybe", by definition. You want it to run until you tell it to stop, but you may never tell it to stop. Ergo, all programs have some sort of infinite loop in them somewhere, even if it is buried in your framework or language runtime.

I really don't think that's the definition of a program.

> You want it to run until you tell it to stop,

No? Many programs I don't want to run until I tell them to stop.

Even then, this reduces it to irrelevance.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#362

Post by afiodorov »

Dylan16807 wrote: post #357
afiodorov wrote: post #293

Earlier quoted context omitted.

I claim we can do O(1) complexity (minus precompute) in all cases, see another comment of mine. Curious if O1 will figure it out.

In that comment, you are generating your own random numbers and then optimizing away the actual generation. It can't take an input array. While clever, I think that strays too far from the initial prompt.

All I need is the proportion of the qualifying numbers to the input array to run the algorithm and the number of samples. Then we can sample min, max index of the qualifying array and return their difference without having to sample many times if we can derive the joined min max distribution conditional on the Bernoulli.

In other words the procedure can take any input array and qualifying criteria.

The joint distribution is relatively simple to derive. (This is related to the fact that min, max of continuous uniform on 0, 1 are Beta distributions.)

Re: Can LLMs write better code if you keep asking them to “write better code”?

#364

Post by the__alchemist »

Not ChatGPT in Kotlin/Android.

> You keep giving me code that calls nonexistant methods, and is deprecated, as shown in Android Studio. Please try again, using only valid code that is not deprecated.

Does not help. I use this example, since it seems good at all other sorts of programming problems I give it. It's miserable at Android for some reason, and asking it to do better doesn't work.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#365

Post by [deleted] »

minitech wrote: post #334
ryao wrote: post #295

Earlier quoted context omitted.

Iterating a precomputed list is a method of generating random numbers. It is used in the one time pad. Whether we iterate a precomputed list or use a pseudo random number generator, we can short circuit the random number generator using this trick. We cannot directly choose the output values, because then it would not be random.

They’re proposing choosing the output values randomly according to the distribution obtained by choosing input values uniformly at random for the original algorithm.

[deleted]

Re: Can LLMs write better code if you keep asking them to “write better code”?

#366

Post by _yb2s »

krackers wrote: post #355
_yb2s wrote: post #298

The headline question here alone gets at what is the biggest widespread misunderstanding of LLMs, which causes people to systematically doubt and underestimate their ability to exhibit real creativity and understanding based problem solving. At it's core an LLM is a sort of "situation specific simulation engine." You setup a scenario, and it then plays it out with it's own internal model of the situation, trained on…

Good perspective. Maybe it's because people are primed by sci-fi to treat this as a god-like oracle model. Note that even in the real-world simulations can give wrong results as we don't have perfect information, so we'll probably never have such an oracle model. But if you stick with the oracle framework, then it'd be better to model it as some sort of "fuzzy oracle" machine, right? I'm vaguely reminded of probabili…

I get that people really want an oracle, and are going to judge any AI system by how good it does at that - yes from sci-fi influenced expectations that expected AI to be rationally designed, and not inscrutable and alien like LLMs... but I think that will almost always be trying to fit a round peg into a square hole, and not using whatever we come up with very effectively. Surely, as LLMs have gotten better they have become more useful in that way so it is likely to continue getting better at pretending to be an oracle, even if never being very good at that compared to other things it can do.

Arguably, a (the?) key measure of intelligence is being able to accurately understand and model new phenomenon from a small amount of data, e.g. in a Bayesian sense. But in this case we are attempting to essentially evolve all of the structures of an intelligent system de novo from a stochastic optimization process- so is probably better compared to the entire history of evolution than to an individual human learning during their lifetime, although both analogies have big problems.

Overall, I think the training process will ultimately only be required to build a generally intelligent structure, and good inference from a small set of data or a totally new category of problem/phenomenon will happen entirely at the inference stage.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#367

Post by [deleted] »

cultureulterior wrote: post #265
ActivePattern wrote: post #155

Earlier quoted context omitted.

This is actually a great example of an optimization that would be extremely difficult for an LLM to find. It requires a separate computation to find the smallest /largest numbers in the range with digits summing to 30. Hence, an LLM is unlikely to be able to generate them accurately on-the-fly.

O1 found it. https://chatgpt.com/share/67782b6b-6248-8012-882d-238b600ef9...

[deleted]

Re: Can LLMs write better code if you keep asking them to “write better code”?

#368

Post by ryao »

ryao wrote: post #279
brabel wrote: post #270

Earlier quoted context omitted.

Did you measure it? I would expect using % would ruin your performance as it's slow, even if it allows you to avoid doing a bunch of sums (which are fast).

You can do this “without” using the modulus operation by storing the numbers in a boolean array. Start at 3999 and keep adding 9 to find the minimum. Then start at 99930 and keep subtracting 9 to find the maximum. You would need to check if the number is in the array and then if the number’s digits sum to 30. Note that the conversion of numbers to base 10 to check the digits typically involves doing division and modu…

It turns out that there is no modular multiplicative inverse for this, so that trick cannot be used to avoid the modulus and division when getting the base 10 digits:

https://extendedeuclideanalgorithm.com/calculator.php?mode=2...

Re: Can LLMs write better code if you keep asking them to “write better code”?

#369

Post by lblume »

Octoth0rpe wrote: post #219
ActivePattern wrote: post #155

Earlier quoted context omitted.

This is actually a great example of an optimization that would be extremely difficult for an LLM to find. It requires a separate computation to find the smallest /largest numbers in the range with digits summing to 30. Hence, an LLM is unlikely to be able to generate them accurately on-the-fly.

> This is actually a great example of an optimization that would be extremely difficult for an LLM to find It'll be somewhat more likely since the next gen training set includes your comment :) (disclaimer: I have no personal knowledge of ai companies scraping hacker news, but it wouldn't surprise me at all)

It would be very surprising if they would not scrape this site. The content is very high-quality in the general case and there are no giant barriers preventing entry (there even is a clean API!). One might even use us to fine-tune a coding assistant or the alike.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#370

Post by ryao »

minitech wrote: post #334
ryao wrote: post #295

Earlier quoted context omitted.

Iterating a precomputed list is a method of generating random numbers. It is used in the one time pad. Whether we iterate a precomputed list or use a pseudo random number generator, we can short circuit the random number generator using this trick. We cannot directly choose the output values, because then it would not be random.

They’re proposing choosing the output values randomly according to the distribution obtained by choosing input values uniformly at random for the original algorithm.

That removes the random element to this. The way that random numbers work is that it is possible (although unlikely) that the minimum and maximal values in the range will not be selected when generating the million random numbers. If you assume that they will always be selected and thus always return the same output, then your output will be wrong at least some of the time.
Post reply on HN