Earlier quoted context omitted.
You're anthropomorphizing. LLMs can't lie nor can they tell the truth. These concepts just don't apply to them. They also cannot tell you what they were "thinking" when they wrote a piece of code. If you "ask" them what they were thinking, you just get a plausible response, not the "intention" that may or may not have existed in some abstract form in some layer when the system selected tokens*. That information is go…
I’m not anthropomorphizing. I’ve been in many situation where the AI wrote some code some way and I had to ask why, it told me why and then we moved on to better solutions as needed. Better if it just wrote the code and its reasoning was still in context, but even if it’s not, it can usually reverse engineer what it wrote well enough. Then it’s a conversation about whether there is a better clearer way to do it, the…
> where the AI wrote some code some way and I had to ask why, it told me why
I just explained that it cannot tell you why. It's simply not how they work. You might as well tell me that it cooked you dinner and did your laundry.
> the code improves.
We can agree on this. The iterative process works. The understanding of it is incorrect. If someone's understanding of a hammer superficially is "tool that drives pointy things into wood", they'll inevitably try to hammer a screw at some point - which might even work, badly.
> It sounds like you either have access to bad models or you are just imagining what it’s like to use an LLM in this way
Quoting this is really enough. You may imagine me sighing.
> Also, pretending that the LLM is still just token predicting
Strawman.
Overall your comment is dancing around engaging with what is being said, so I will not waste my time here.