Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

291–300 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#291
post #205
post #181

Earlier quoted context omitted.

You guys are picking on the problem statement. Here's a revised prompt, which also skips the silliness of single threading: Write __fully parallelized__ Python code to solve this problem: __Generate__ 1 million random integers between 1 and 10,000,000, find the difference between the smallest and the largest numbers whose digits sum up to 30.

Whose digits sum up to 30, or the sum of whose digits equal 30? Btw, _whose_ digits are we talking about? — I just built a random program generator. After I finish optimizing, I'm gonna test it to see if works! — "If builders built houses the way programmers build programs, the first woodpecker to come along would destroy civilization" https://en.m.wikiquote.org/wiki/Gerald_Weinberg

> Btw, _whose_ digits are we talking about?

You seem to be under the impression that whose is not a form of which, which is incorrect.

whose:which::whose:who

Re: Can LLMs write better code if you keep asking them to “write better code”?

#292
post #281

Earlier quoted context omitted.

Would someone write a mathematical proof showing this is always true?

a = [int(x) for x in str(n)][::-1] assert n == sum(d*(10**i) for i, d in enumerate(a)) Now when you're operating mod 9, 10 == 1 % 9, thus 10**i == 1 % 9 Comes from the fact that (a*b) % 9 == (a % 9) * (b % 9) Now using (a+b) % 9 == (a % 9) + (b % 9) We get that that sum(a) and n are same mod 9.

Thank you for that.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#293

Earlier quoted context omitted.

For me O1 found this by telling it "There is a further significant optimization possible."

What if you keep telling it that "there is a further significant optimization possible"?

I claim we can do O(1) complexity (minus precompute) in all cases, see another comment of mine. Curious if O1 will figure it out.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#294
post #257

Earlier quoted context omitted.

Thanks for the tip, though I’m not sure how complexity theory will explain the impossibility of superhuman results. The main advantage ML methods have over humans is that they train much faster. Just like humans, they get better with more training. When they are good enough, they can be used to generate synthetic data, especially for cases like software optimization, when it is possible to verify the ground truth. A…

ML is better than biological neurons in some tasks, they are different contexts. Almost all the performance of say college tests are purely from the pre-training, pattern finding and detection. Transformers are limited to DLOGTIME-uniform TC0, they can't even do the Boolean circuit value problem. The ability to use the properties of BPP, does help. Understanding the power of, and limitations of iteration and improvin…

I read a book on recursively enumerable degrees once, which IIRC was a sort of introduction to complexity classes of various computable functions, but I never imagined it having practical use; so this post is eye-opening. I've been nattering about how the models are largely finding separating hyperplanes after non-linear transformations have been done, but this approach where the AI solving ability can't be more complex than the complexity class allows is an interesting one.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#295
post #276

Earlier quoted context omitted.

With this trick you can test while generating the random numbers and if you see both values, you can short circuit the generation of random numbers.

The input generation is outside the scope of this. Otherwise you could directly choose the output values with the apropriate distribution and just skip all the rest. (Arguably, this criticism applies to exchanging random.randint for a numpy equivalent as well, since that doesn't optimize the solution but only how quickly the question is being generated.)

Iterating a precomputed list is a method of generating random numbers. It is used in the one time pad. Whether we iterate a precomputed list or use a pseudo random number generator, we can short circuit the random number generator using this trick. We cannot directly choose the output values, because then it would not be random.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#296

I've noticed this with GPT as well -- the first result I get is usually mediocre and incomplete, often incorrect if I'm working on something a little more obscure (eg, OpenSCAD code). I've taken to asking it to "skip the mediocre nonsense and return the good solution on the first try". The next part is a little strange - it arose out of frustration, but it also seems to improve results. Let's call it "negative incent…

If they really care about the answer, they'll ask a second time sounds a lot like if your medical claims are real then you'll appeal.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#297
post #239

Earlier quoted context omitted.

My main concern with the simplification of memorization or near neighbor interpolation that is commonly assumed for LLMs is that these methods are ineffective at scale and unlikely to be used by decoder transformers in practice. That paper shows that the decoder transformer somehow came up with a better decision tree fitting algorithm for low data cases than any of the conventional or boosted tree solutions humans ty…

Littlestone and Warmuth make the connection to compression in1986, which was later shown to be equivalent to VC dimensionally or PAC learnablilty. Look into DBScan, OPTICs for far closer lenses on how clustering works in modern ML commercial ML, KNN not the only form of clustering. But it is still in-context, additional compression that depends on a decider function, or equivalently a composition linearized set shatt…

I am very familiar with these and other clustering methods in modern ML, and have been involved in inventing and publishing some such methods myself in various scientific contexts. The paper I cited above only used 3 nearest neighbors as one baseline IIRC; that is why I mentioned KNN. However, even boosted trees failed to reduce the loss as much as the algorithm learned from the data by the decoder transformer.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#298
The headline question here alone gets at what is the biggest widespread misunderstanding of LLMs, which causes people to systematically doubt and underestimate their ability to exhibit real creativity and understanding based problem solving.

At it's core an LLM is a sort of "situation specific simulation engine." You setup a scenario, and it then plays it out with it's own internal model of the situation, trained on predicting text in a huge variety of situations. This includes accurate real world models of, e.g. physical systems and processes, that are not going to be accessed or used by all prompts, that don't correctly instruct it to do so.

At its core increasingly accurate prediction of text, that is accurately describing a time series of real world phenomena, requires an increasingly accurate and general model of the real world. There is no sense in which there is a simpler way to accurately predict text that represents real world phenomena in cross validation, without actually understanding and modeling the underlying processes generating those outcomes represented in the text.

Much of the training text is real humans talking about things they don't understand deeply, and saying things that are wrong or misleading. The model will fundamentally simulate these type of situations it was trained to simulate reliably, which includes frequently (for lack of a better word) answering things "wrong" or "badly" "on purpose" - even when it actually contains an accurate heuristic model of the underlying process, it will still, faithfully according to the training data, often report something else instead.

This can largely be mitigated with more careful and specific prompting of what exactly you are asking it to simulate. If you don't specify, there will be a high frequency of accurately simulating uninformed idiots, as occur in much of the text on the internet.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#299
post #298

The headline question here alone gets at what is the biggest widespread misunderstanding of LLMs, which causes people to systematically doubt and underestimate their ability to exhibit real creativity and understanding based problem solving. At it's core an LLM is a sort of "situation specific simulation engine." You setup a scenario, and it then plays it out with it's own internal model of the situation, trained on…

Garbage in garbage out.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#300
post #257

Earlier quoted context omitted.

Thanks for the tip, though I’m not sure how complexity theory will explain the impossibility of superhuman results. The main advantage ML methods have over humans is that they train much faster. Just like humans, they get better with more training. When they are good enough, they can be used to generate synthetic data, especially for cases like software optimization, when it is possible to verify the ground truth. A…

ML is better than biological neurons in some tasks, they are different contexts. Almost all the performance of say college tests are purely from the pre-training, pattern finding and detection. Transformers are limited to DLOGTIME-uniform TC0, they can't even do the Boolean circuit value problem. The ability to use the properties of BPP, does help. Understanding the power of, and limitations of iteration and improvin…

The discussion cannot go deeper than the current level, unfortunately. One thing to not forget when thinking about decoder transformer models is that there is no limitation to having parts of the output / input stream be calculated by other circuits if it helps the cause. Eg send a token to use a calculator, compute and fill the answer; send a token to compile and run a code and fill the stream with the results. The complexity class of the main circuit might not need be much more complicated than the 200-level deep typical architectures of today as long as they can have access to memory and tools. You can call this system something else if you prefer (decoder-transformer-plus-computer), but that is what people interact with in ChatGPT, so not sure I agree that complexity theory limits the superhuman ability. Humans are not good with complexity.
Post reply on HN