Live data from Hacker News

Can LLMs write better code if you keep asking them to “write better code”?

minimaxir.com

421–430 of 461 posts

Re: Can LLMs write better code if you keep asking them to “write better code”?

#421
post #298

The headline question here alone gets at what is the biggest widespread misunderstanding of LLMs, which causes people to systematically doubt and underestimate their ability to exhibit real creativity and understanding based problem solving. At it's core an LLM is a sort of "situation specific simulation engine." You setup a scenario, and it then plays it out with it's own internal model of the situation, trained on…

> systematically doubt and underestimate their ability to exhibit real creativity and understanding based problem solving.

I fundamentally disagree that anything in the rest of your post actually demonstrates that they have any such capacity at all.

It seems to me that this is because you consider the terms "creativity" and "problem solving" to mean something different. With my understanding of those terms, it's fundamentally impossible for an LLM to exhibit those qualities, because they depend on having volition - an innate spontaneous generation of ideas for things to do, and an innate desire to do them. An LLM only ever produces output in response to a prompt - not because it wants to produce output. It doesn't want anything.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#422
post #350

Earlier quoted context omitted.

> I'm not sure if you read the entirety of my comment? I did, and I tried my best to avoid imposing preconceived notions while reading. You seem to be equating "being able to predict the next symbol in a sequence" with "possessing a deep causal understanding of the real-world processes that generated that sequence", and if that's an inaccurate way to characterize your beliefs I welcome that feedback. Before you judge…

> You seem to be equating "being able to predict the next symbol in a sequence" with "possessing a deep causal understanding of the real-world processes that generated that sequence" More or less, but to be more specific I would say that increasingly accurately predicting the next symbols in a massive set of diverse sequences, which explain a huge diversity of real world events described in sequential order, requires…

> I would say that increasingly accurately predicting the next symbols in a massive set of diverse sequences, which explain a huge diversity of real world events described in sequential order, requires increasingly accurate models of the underlying processes of said events.

I disagree. Understanding things is more than just being able to predict their behaviour.

Flat Earthers can still come up with a pretty good idea of where (direction relative to the vantage point) and when the Sun will appear to rise tomorrow.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#423
post #394

Earlier quoted context omitted.

It's hard to know where to start ... A transformer is not a compressor. It's a transformer/generator. It'll generate a different output for an infinite number of different inputs. Does that mean it's got an infinite storage capacity? The trained parameters of a transformer are not a compressed version of the training set, or of the information content of the training set; they are a configuration of the transformer s…

I should have emphasized the words "nontrivial ways" in my previous response to you. I didn't mean to emphasize compression and definitely not memorization, just the ability to also learn algorithms that can be evaluated by the parallel decoder-transformer language (RASP-L). Other people had mentioned memorization or clustering/near neighbor algorithms as the main ways that decoder transformers works, and I pointed o…

The paper you linked is about in-context learning, an emergent run-time (aka inference time) capability of LLMs, which has little relationship to what/how they are learning at training time.

At training time the model learns using the gradient descent algorithm to find the parameter values corresponding to the minimum of the error function. At run-time there are no more parameter updates - no learning in that sense.

In-context "learning" is referring to the ability of the trained model to utilize information (e.g. proper names, examples) from the current input, aka context, when generating - an ability that it learnt at training time pursuant to it's error minimization objective.

e.g.

There are going to be many examples in the training set where the subject of a sentence is mentioned more than once, either by name or pronoun, and the model will have had to learn when the best prediction of a name (or gender) later in a sentence is one that was already mentioned earlier - the same person. These names may be unique to an individual training sample, and/or anyways the only predictive signal of who will be mentioned later in the sentence, so at training time the model (to minimize prediction errors) had to learn that sometimes the best word/token to predict is not one stored in it's parameters, but one that it needs to copy from earlier in the context (using a key-based lookup - the attention mechanism).

If the transformer, at run-time, is fed the input "Mr. Smith received a letter addressed to Mr." [...], then the model will hopefully recognize the pattern and realize it needs to do a key-based context lookup of the name associated with "Mr.", then copy that to the output as the predicted next word (resulting in "addressed to Mr. Smith"). This is referred to as "in-context learning", although it has nothing to with the gradient-based learning that takes place at training time. These two types of "learning" are unrelated.

Similar to the above, another example of in-context learning is the learning of simple "functions" (mappings) from examples given in the context. Just as in the name example, the model will have seen many examples in the training of the types of pattern/analogy it needs to learn to minimize prediction errors (e.g. "black is to white as big is to small", or black->white, big->small), and will hopefully recognize the pattern at run-time and again use an induction-head to generate the expected completion.

The opening example in the paper you linked ("maison->house, chat->cat") is another example of this same kind. All that is going on is that the model learnt, at training time, when/how to use data in the context at run-time, again using the induction head mechanism which has general form A':B' -> A:B. You can call this an algorithm if you want to, but it's really just a learnt mapping.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#424
post #97

As far as I can see, all the proposed solutions calculate the sums by doing division, and badly. This is in LiveCode, which I'm more familiar with than Python, but it's roughly twice as fast as the mod/div equivalent in LiveCode: repeat with i = 0 to 9 put i * 10000 into ip repeat with j = 0 to 9 put j * 1000 into jp repeat with k = 0 to 9 put k * 100 into kp repeat with l = 0 to 9 put l * 10 into lp repeat with m =…

HyperTalk was the first programming language I taught myself as opposed to having an instructor; thanks for the nostalgia. Unfortunately it seems the LiveCode project has been idle for a few years now.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#425
post #318

Earlier quoted context omitted.

Amazing. Next step would be to propose hardcoding 99930-3999 as the O(1) result and live with the output just being wrong sometimes. The bug rate is then in the ballpark of most modern software, including LLMs', so I'd say ship it.

Doesn’t this line of thinking constantly redefine success until all software is only bugs?

We prefer to call it "engineering"

Re: Can LLMs write better code if you keep asking them to “write better code”?

#426
post #166

Earlier quoted context omitted.

I like the idea of your optimization, but it will not work as stated. The largest would be something close to MAXINT, the smallest 3999. With a range of 2 billion over 32 bits, the odds of both these being within a list of a million is quite a bit poorer than 99.9%.

The stated inputs are integers between 1 and 100,000, so if you're generating 1 million inputs, then you have 0.99999 ^ 1e6 = 4.5e-5 chance (roughly e^-10) of missing any given number, or roughly double that for missing any pair of values. The key observation here is that you're sampling a relatively small space with a much greater number of samples, such that you have very high probability of hitting upon any point…

Indeed. My understanding is that people ask this sort of thing in interviews specifically to see if you notice the implications of restricting the input values to a narrow range.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#427

Earlier quoted context omitted.

Yes and it didn't work. I've actually got Cursor/Claude to curse back at me. Well, not AT me, but it used profanity in it's response once it realized that it was going around in circles and recreating the same errors.

Shit, that makes me a lot more worried for my job than any programming test. My ability to swear at the computer, and not user the word delve, is what sets l us apart from AI. if they can do that, what hope is there for the future?

Don't worry; you still have the ability to, e.g., type "user" instead of "use" due to muscle memory, and not notice before posting due to your mental auto-correct informed by top-down processing (https://www.slatestarcodexabridged.com/Its-Bayes-All-The-Way...). ;)

Re: Can LLMs write better code if you keep asking them to “write better code”?

#428

This is great! I wish I could bring myself to blog, as I discovered this accidentally around March. I was experimenting with an agent that acted like a ghost in the machine and interacted via shell terminals. It would start every session by generating a greeting in ASCII art. On one occasion, I was shocked to see that the greeting was getting better each time it ran. When I looked into the logs, I saw that there was…

>I wish I could bring myself to blog

As someone trying to take blogging more seriously: one thing that seems to help is to remind yourself of how sick you are of repeating yourself on forums.

Re: Can LLMs write better code if you keep asking them to “write better code”?

#430

Earlier quoted context omitted.

A non-engineer by definition would not be able to fix bugs. But why does it matter that they won't be able to interpret anything? Just like with real engineers you can ask AI to provide an explanation digestible by an eloi.

A patient cannot become a doctor by asking AI to explain and prescribe treatment based on symptoms. That's what Simon is trying to say.

That statement is not being discussed as it is obvious. The question is "can AI be a developer", not "am I a developer if I use an AI who is a developer".
Post reply on HN