Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Case study: Creative math – How AI fakes proofs

tomaszmachnik.pl

Post reply on HN
Print view View original

71–80 of 107 posts

Re: Case study: Creative math – How AI fakes proofs

#71

Post by croisillon »

RestartKernel wrote: post #64
simonw wrote: post #19

Somewhat ironic that the author calls out model mistakes and then presents https://tomaszmachnik.pl/gemini-fix-en.html - a technique they claim reduces hallucinations which looks wildly superstitious to me. It involves spinning a whole yarn to the model about how it was trained to compete against other models but now it's won so it's safe for it to admit when it doesn't know something. I call this a superstition beca…

Is there a term for "LLM psychology" like this? If so, it seems closer to a soft science than anything definitive.

vibe massaging?

Re: Case study: Creative math – How AI fakes proofs

#72

Post by musculus »

comex wrote: post #40

I like how this article was itself clearly written with the help of an LLM. (You can particularly tell from the "Conclusions" section. The formatting, where each list item starts with a few-word bolded summary, is already a strong hint, but the real issue is the repetitiveness of the list items. For bonus points there's a "not X, but Y", as well as a dash, albeit not an em dash.)

Good catch. You are absolutely right.

My native language is Polish. I conducted the original research and discovered the 'square root proof fabrication' during sessions in Polish. I then reproduced the effect in a clean session for this case study.

Since my written English is not fluent enough for a technical essay, I used Gemini as a translator and editor to structure my findings. I am aware of the irony of using an LLM to complain about LLM hallucinations, but it was the most efficient way to share these findings with an international audience.

Re: Case study: Creative math – How AI fakes proofs

#74

Post by lancebeet »

godelski wrote: post #20

I thought it funny a few weeks ago Karpathy shared a sample od NanoBannana solving some physics problems but despite getting the right output it isn't get the right answers. I think it's quite illustrative of the problem even with coding LLMs. Code and math proofs aren't so different, what matters is the steps to generate the output. All that matters far more than the actual output. The output is meaningless if the s…

>Maybe what surprised me most is that the mistakes NanoBananna made are simple enough that I'm absolutely positive Karpathy could have caught them. Even if his physics is very rusty. I'm often left wondering if people really are true believers and becoming blind to the mistakes or if they don't care.

I've seen this interesting phenomenon many times. I think it's a kind of subconscious bias. I call it "GeLLMann amnesia".

Re: Case study: Creative math – How AI fakes proofs

#75

Post by oasisaimlessly »

mlpoknbji wrote: post #34
tombert wrote: post #23

I remember when ChatGPT first came out, I asked it for a proof for Fermat's Last Theorem, which it happily gave me. It was fascinating, because it was doing a lot of understandable mistakes that 7th graders make. For example, I don't remember the surrounding context but it decided that you could break `sqrt(x^2 + y^2)` into `sqrt(x^2) + sqrt(y^2) => x + y`. It's interesting because it was one of those "ASSUME FALSE"…

My favorite early chatgpt math problem was "prove there exists infinitely many even primes" . Easy! Take a finite set of even primes, multiply them and add one to get a number with a new even prime factor. Of course, it's gotten a bit better than this.

IIRC, that is actually the standard proof that there are infinitely many primes[1] or maybe this variation on it[2].

[1]: https://en.wikipedia.org/wiki/Euclid%27s_theorem#Euclid's_pr...

[2]: https://en.wikipedia.org/wiki/Euclid%27s_theorem#Proof_using...

Re: Case study: Creative math – How AI fakes proofs

#76

Post by calhoun137 »

simonw wrote: post #19

Somewhat ironic that the author calls out model mistakes and then presents https://tomaszmachnik.pl/gemini-fix-en.html - a technique they claim reduces hallucinations which looks wildly superstitious to me. It involves spinning a whole yarn to the model about how it was trained to compete against other models but now it's won so it's safe for it to admit when it doesn't know something. I call this a superstition beca…

> Does replacing that lengthy text with "if you aren't sure of the answer say you don't know" have the same exact effect?

i believe it makes a substantial difference. the reason is that a short query contains a small number of tokens, whereas a large “wall of text” contains a very large number of tokens.

I strongly suspect that a large wall of text implicitly activates the models persona behavior along the lines of the single sentence “if you aren't sure of the answer say you don't know” but the lengthy argument version of that is a form of in-context learning that more effectively constrains the models output because you used more tokens.

Re: Case study: Creative math – How AI fakes proofs

#77

Post by calhoun137 »

My experience leads to the same conclusion that the models are very good at math reasoning, but you have to really know what you are doing and be aware of the blatant lies that result from poorly phrased queries.

I recently prompted Gemini Deep Research to “solve the Riemann Hypothesis” using a specific strategy and it just lied and fabricated the result of a theorem in its output, which otherwise looked very professional.

Re: Case study: Creative math – How AI fakes proofs

#78

Post by te7447 »

vrighter wrote: post #69
v_CodeSentinal wrote: post #6

This is the classic 'plausible hallucination' problem. In my own testing with coding agents, we see this constantly—LLMs will invent a method that sounds correct but doesn't exist in the library. The only fix is tight verification loops. You can't trust the generative step without a deterministic compilation/execution step immediately following it. The model needs to be punished/corrected by the environment, not just…

So you want the program to always halt at some point. How would you write a deterministic test for it?

I imagine you would use something that errs on the side of safety - e.g. insist on total functional programming and use something like Idris' totality checker.

Re: Case study: Creative math – How AI fakes proofs

#79

Post by mlpoknbji »

oasisaimlessly wrote: post #75
mlpoknbji wrote: post #34

Earlier quoted context omitted.

My favorite early chatgpt math problem was "prove there exists infinitely many even primes" . Easy! Take a finite set of even primes, multiply them and add one to get a number with a new even prime factor. Of course, it's gotten a bit better than this.

IIRC, that is actually the standard proof that there are infinitely many primes[1] or maybe this variation on it[2]. [1]: https://en.wikipedia.org/wiki/Euclid%27s_theorem#Euclid's_pr... [2]: https://en.wikipedia.org/wiki/Euclid%27s_theorem#Proof_using...

Yes this is the standard proof of infinitely many primes but note that my prompt asked for infinitely many even primes. The point is that GPT would take the correct proof and insert "even" at sensible places to get something that looks like a proof but is totally wrong.

Of course it's much better now, but with more pressure to prove something hard the models still just insert nonsense steps.

Re: Case study: Creative math – How AI fakes proofs

#80

Post by PlatoIsADisease »

simonw wrote: post #19

Somewhat ironic that the author calls out model mistakes and then presents https://tomaszmachnik.pl/gemini-fix-en.html - a technique they claim reduces hallucinations which looks wildly superstitious to me. It involves spinning a whole yarn to the model about how it was trained to compete against other models but now it's won so it's safe for it to admit when it doesn't know something. I call this a superstition beca…

Wow that link was absurdly bad.

Reading that makes me unbelievably happy I played with GPT3 and learned how/when LLMs fail.

Telling it not to hallucinate is a serious misunderstanding of LLMs. At most in 2026, you are telling thinking/COT to double check.

Post reply on HN