If I have learnt one thing working in software engineering, specifically on AI-enabled products empowering junior engineers, and using Copilot professionally, it’s that you need even more experience to detect the subtleties in the lack of the models understanding of your domain, your specific intent. If you don’t know exactly what you’re after, and use the LLM as a sparring partner to bounce your ideas off, you’re in…
Therein lies the mistake. Too many people assume ChatGPT (and similar LLMs) are capable of reasoning. It's not. It is simply just giving you what is likely the 'correct' answer based on some sort of pattern.
It doesn't know what's wrong, so it's not aware it's giving you an inappropriate answer.