Live data from Hacker News

Case study: Creative math – How AI fakes proofs

tomaszmachnik.pl

41–50 of 107 posts

Re: Case study: Creative math – How AI fakes proofs

#41
post #40

I like how this article was itself clearly written with the help of an LLM. (You can particularly tell from the "Conclusions" section. The formatting, where each list item starts with a few-word bolded summary, is already a strong hint, but the real issue is the repetitiveness of the list items. For bonus points there's a "not X, but Y", as well as a dash, albeit not an em dash.)

Not only that, it even looks like the fabrication example is generated by AI, as the entire question seem too "fabricated". Also gemini web app queries the tool and returns correct answer, so don't know which gemini the author is talking about.

Re: Case study: Creative math – How AI fakes proofs

#44
post #40

I like how this article was itself clearly written with the help of an LLM. (You can particularly tell from the "Conclusions" section. The formatting, where each list item starts with a few-word bolded summary, is already a strong hint, but the real issue is the repetitiveness of the list items. For bonus points there's a "not X, but Y", as well as a dash, albeit not an em dash.)

“This is key!”

Re: Case study: Creative math – How AI fakes proofs

#45
post #40

I like how this article was itself clearly written with the help of an LLM. (You can particularly tell from the "Conclusions" section. The formatting, where each list item starts with a few-word bolded summary, is already a strong hint, but the real issue is the repetitiveness of the list items. For bonus points there's a "not X, but Y", as well as a dash, albeit not an em dash.)

Not only that, it even looks like the fabrication example is generated by AI, as the entire question seem too "fabricated". Also gemini web app queries the tool and returns correct answer, so don't know which gemini the author is talking about.

Probably gemini on aistudio.google.com, you can configure if it is allowed to access code execution / web search / others

Re: Case study: Creative math – How AI fakes proofs

#46
What's interesting about this is that a human would hypothetically produce a similar error, but in practice would reject the question as beyond their means. I'd assume something about supervised learning makes the models overestimate their abilities. It probably learns that “good” responses attempt to answer the question rather than giving up.

Re: Case study: Creative math – How AI fakes proofs

#48
>STEP 2: The Shock (Reality Check)

I've found a funny and simple technique for this. Just write "what the F$CK" and it will often seem to unstick from repetitiveness or refusals(i cant do that).

Actually just writing the word F#ck often will do it. Works on coding too.

Re: Case study: Creative math – How AI fakes proofs

#49

This is the classic 'plausible hallucination' problem. In my own testing with coding agents, we see this constantly—LLMs will invent a method that sounds correct but doesn't exist in the library. The only fix is tight verification loops. You can't trust the generative step without a deterministic compilation/execution step immediately following it. The model needs to be punished/corrected by the environment, not just…

Yes, and better still the AI will fix its mistakes if it has access to verification tools directly. You can also have it write and execute tests, and then on failure, decide if the code it wrote or the tests it wrote are wrong, snd while there is a chance of confirmation bias, it often works well enough

Re: Case study: Creative math – How AI fakes proofs

#50

You don’t need a test to know this we already know there’s heavy reinforcement training done on these models so it optimizes for passing the training. Passing the training means convincing the person rating the answers and that the answer is good. The keyword is convince. So it just needs to convince people that’s it’s right. It is optimizing for convincing people. Out of all answers that can convince people some can…

Yet people often forget this. We don't have mathematical models of truth, beauty, or many abstract things. Thus we proxy it with "I know it when I see it." It's a good proxy for lack of anything better but it also creates a known danger: the model optimizes deception. The proxy helps it optimize the answers we want but if we're not incredibly careful they also optimize deception. This makes them frustrating and poten…

No the question is, how do you train the system so it doesn't deceive you?
Post reply on HN