I like how this article was itself clearly written with the help of an LLM. (You can particularly tell from the "Conclusions" section. The formatting, where each list item starts with a few-word bolded summary, is already a strong hint, but the real issue is the repetitiveness of the list items. For bonus points there's a "not X, but Y", as well as a dash, albeit not an em dash.)
Case study: Creative math – How AI fakes proofs
41–50 of 107 posts
Re: Case study: Creative math – How AI fakes proofs
#42Re: Case study: Creative math – How AI fakes proofs
#43The simpler and I think correct conclusion is that the LLM simply does not reason in our sense of the word. It mimics the reasoning pattern and try to get it right but could not.
Re: Case study: Creative math – How AI fakes proofs
#44I like how this article was itself clearly written with the help of an LLM. (You can particularly tell from the "Conclusions" section. The formatting, where each list item starts with a few-word bolded summary, is already a strong hint, but the real issue is the repetitiveness of the list items. For bonus points there's a "not X, but Y", as well as a dash, albeit not an em dash.)
Re: Case study: Creative math – How AI fakes proofs
#45I like how this article was itself clearly written with the help of an LLM. (You can particularly tell from the "Conclusions" section. The formatting, where each list item starts with a few-word bolded summary, is already a strong hint, but the real issue is the repetitiveness of the list items. For bonus points there's a "not X, but Y", as well as a dash, albeit not an em dash.)
Not only that, it even looks like the fabrication example is generated by AI, as the entire question seem too "fabricated". Also gemini web app queries the tool and returns correct answer, so don't know which gemini the author is talking about.
Re: Case study: Creative math – How AI fakes proofs
#46Re: Case study: Creative math – How AI fakes proofs
#47that's not a proof
Re: Case study: Creative math – How AI fakes proofs
#48I've found a funny and simple technique for this. Just write "what the F$CK" and it will often seem to unstick from repetitiveness or refusals(i cant do that).
Actually just writing the word F#ck often will do it. Works on coding too.
Re: Case study: Creative math – How AI fakes proofs
#49This is the classic 'plausible hallucination' problem. In my own testing with coding agents, we see this constantly—LLMs will invent a method that sounds correct but doesn't exist in the library. The only fix is tight verification loops. You can't trust the generative step without a deterministic compilation/execution step immediately following it. The model needs to be punished/corrected by the environment, not just…
Re: Case study: Creative math – How AI fakes proofs
#50You don’t need a test to know this we already know there’s heavy reinforcement training done on these models so it optimizes for passing the training. Passing the training means convincing the person rating the answers and that the answer is good. The keyword is convince. So it just needs to convince people that’s it’s right. It is optimizing for convincing people. Out of all answers that can convince people some can…
Yet people often forget this. We don't have mathematical models of truth, beauty, or many abstract things. Thus we proxy it with "I know it when I see it." It's a good proxy for lack of anything better but it also creates a known danger: the model optimizes deception. The proxy helps it optimize the answers we want but if we're not incredibly careful they also optimize deception. This makes them frustrating and poten…