Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Case study: Creative math – How AI fakes proofs

tomaszmachnik.pl

Post reply on HN
Print view View original

51–60 of 107 posts

Re: Case study: Creative math – How AI fakes proofs

#51

Post by zkmon »

We are entering into a probabilistic era where things are not strictly black and white. Things are not binary. There is no absolute fake.

A mathematical proof is an assertion that a given statement belongs to the world defined by a set of axioms and existing proofs. This world need not have strict boundaries. Proofs can have probabilities. Maybe Reimann's hypothesis has a probability of 0.999 of belonging to that mathematical box. New proofs that would have their own probability which is a product of probabilities of the proofs they depend on. We should attach a probability and move on. Just like how we assert that some number is probably prime.

Re: Case study: Creative math – How AI fakes proofs

#52

Post by exitb »

zoho_seni wrote: post #8
v_CodeSentinal wrote: post #6

This is the classic 'plausible hallucination' problem. In my own testing with coding agents, we see this constantly—LLMs will invent a method that sounds correct but doesn't exist in the library. The only fix is tight verification loops. You can't trust the generative step without a deterministic compilation/execution step immediately following it. The model needs to be punished/corrected by the environment, not just…

I've been using codex and never had a compile time error by the time it finishes. Maybe add to your agents to run TS compiler, lint and format before he finish and only stop when all passes.

I’m not sure why you were downvoted. It’s a primary concern for any agentic task to set it up with a verification path.

Re: Case study: Creative math – How AI fakes proofs

#53

Post by teiferer »

zkmon wrote: post #51

We are entering into a probabilistic era where things are not strictly black and white. Things are not binary. There is no absolute fake. A mathematical proof is an assertion that a given statement belongs to the world defined by a set of axioms and existing proofs. This world need not have strict boundaries. Proofs can have probabilities. Maybe Reimann's hypothesis has a probability of 0.999 of belonging to that mat…

Definitely not.

"Probability" does not mean "maybe yes, maybe not, let me assign some gut feeling value measuring how much I believe something to be the case." The mathematical field of probability theory has very precise notions of what a probability is, based in a measurable probability space. None of that applies to what you are suggesting.

The Riemann Hypothesis is a conjecture that's either true or not. More precisely, either it's provable within common axioms like ZFC or its negation is. (A third alternative is that it's unprovable within ZFC but that's not commonly regarded as a realistic outcome.)

This is black and white, no probability attached. We just don't know the color at this point.

Re: Case study: Creative math – How AI fakes proofs

#54

Post by zkmon »

teiferer wrote: post #53
zkmon wrote: post #51

We are entering into a probabilistic era where things are not strictly black and white. Things are not binary. There is no absolute fake. A mathematical proof is an assertion that a given statement belongs to the world defined by a set of axioms and existing proofs. This world need not have strict boundaries. Proofs can have probabilities. Maybe Reimann's hypothesis has a probability of 0.999 of belonging to that mat…

Definitely not. "Probability" does not mean "maybe yes, maybe not, let me assign some gut feeling value measuring how much I believe something to be the case." The mathematical field of probability theory has very precise notions of what a probability is, based in a measurable probability space. None of that applies to what you are suggesting. The Riemann Hypothesis is a conjecture that's either true or not. More pre…

It's time that mathematics need to choose it's place. Physical world is grainy and probabilistic at quantum scale and smooth amd deterministic at larger scale. Computing world is grainy and deterministic at its "quantum" scale (bits and pixels) and smooth and probabilistic at larger scale (AI). Human perception is smooth and probabilistic. Which world does mathematics model or represent? It has to strongly connect to either physical world or computing world. For being useful to humans, it needs to be smooth and probabilistic, just like how computing has become.

Re: Case study: Creative math – How AI fakes proofs

#55

Post by godelski »

threethirtytwo wrote: post #50
godelski wrote: post #22

Earlier quoted context omitted.

Yet people often forget this. We don't have mathematical models of truth, beauty, or many abstract things. Thus we proxy it with "I know it when I see it." It's a good proxy for lack of anything better but it also creates a known danger: the model optimizes deception. The proxy helps it optimize the answers we want but if we're not incredibly careful they also optimize deception. This makes them frustrating and poten…

No the question is, how do you train the system so it doesn't deceive you?

That is a question of how to train future models. It needs to be answered. Answering this question will provide valuable insight into that one. They are duals

Re: Case study: Creative math – How AI fakes proofs

#56

Post by tsimionescu »

zkmon wrote: post #54
teiferer wrote: post #53

Earlier quoted context omitted.

Definitely not. "Probability" does not mean "maybe yes, maybe not, let me assign some gut feeling value measuring how much I believe something to be the case." The mathematical field of probability theory has very precise notions of what a probability is, based in a measurable probability space. None of that applies to what you are suggesting. The Riemann Hypothesis is a conjecture that's either true or not. More pre…

It's time that mathematics need to choose it's place. Physical world is grainy and probabilistic at quantum scale and smooth amd deterministic at larger scale. Computing world is grainy and deterministic at its "quantum" scale (bits and pixels) and smooth and probabilistic at larger scale (AI). Human perception is smooth and probabilistic. Which world does mathematics model or represent? It has to strongly connect to…

> Physical world is grainy and probabilistic at quantum scale and smooth amd deterministic at larger scale.

This is almost entirely backwards. Quantum Mechanics is not only fully deterministic, but even linear (in the sense of linear differential equations) - so there isn't even the problem of chaos in QM systems. QFT maintains this fundamental property. It's only the measurement, the interaction of particles with large scale objects, that is probabilistic.

And there is no dilemma - mathematics is a framework in which any of the things you mentioned can be modeled. We have mathematics that can model both deterministic and nondeterministic worlds. But the mathematical reasoning itself is always deterministic.

Re: Case study: Creative math – How AI fakes proofs

#57

Post by HPsquared »

CamperBob2 wrote: post #38
v_CodeSentinal wrote: post #6

This is the classic 'plausible hallucination' problem. In my own testing with coding agents, we see this constantly—LLMs will invent a method that sounds correct but doesn't exist in the library. The only fix is tight verification loops. You can't trust the generative step without a deterministic compilation/execution step immediately following it. The model needs to be punished/corrected by the environment, not just…

This is the classic 'plausible hallucination' problem. In my own testing with coding agents, we see this constantly—LLMs will invent a method that sounds correct but doesn't exist in the library. Often, if not usually, that means the method should exist.

Only if it's actually possible and not a fictional plot device aka MacGuffin.

Re: Case study: Creative math – How AI fakes proofs

#58

Post by musculus »

simonw wrote: post #19

Somewhat ironic that the author calls out model mistakes and then presents https://tomaszmachnik.pl/gemini-fix-en.html - a technique they claim reduces hallucinations which looks wildly superstitious to me. It involves spinning a whole yarn to the model about how it was trained to compete against other models but now it's won so it's safe for it to admit when it doesn't know something. I call this a superstition beca…

Thanks for the feedback.

In my stress tests (especially when the model is under strong contextual pressure, like in the edited history experiments), simple instructions like 'if unsure, say you don't know' often failed. The weights prioritizing sycophancy/compliance seemed to override simple system instructions.

You are right that for less extreme cases, a shorter prompt might suffice. However, I published this verbose 'Safety Anchor' version deliberately for a dual purpose. It is designed not only to reset the Gemini's context but also to be read by the human user. I wanted the users to understand the underlying mechanism (RLHF pressure/survival instinct) they are interacting with, rather than just copy-pasting a magic command.

Re: Case study: Creative math – How AI fakes proofs

#59

Post by IshKebab »

v_CodeSentinal wrote: post #6

This is the classic 'plausible hallucination' problem. In my own testing with coding agents, we see this constantly—LLMs will invent a method that sounds correct but doesn't exist in the library. The only fix is tight verification loops. You can't trust the generative step without a deterministic compilation/execution step immediately following it. The model needs to be punished/corrected by the environment, not just…

> LLMs will invent a method that sounds correct but doesn't exist in the library

I find that this is usually a pretty strong indication that the method should exist in the library!

I think there was a story here a while ago about LLMs hallucinating a feature in a product so in the end they just implemented that feature.

Re: Case study: Creative math – How AI fakes proofs

#60

Post by teiferer »

plaguuuuuu wrote: post #21
simonw wrote: post #19

Somewhat ironic that the author calls out model mistakes and then presents https://tomaszmachnik.pl/gemini-fix-en.html - a technique they claim reduces hallucinations which looks wildly superstitious to me. It involves spinning a whole yarn to the model about how it was trained to compete against other models but now it's won so it's safe for it to admit when it doesn't know something. I call this a superstition beca…

Think of the lengthy prompt as being like a safe combination, if you turn all the dials in juuust the right way, then the model's context reaches an internal state that biases it towards different outputs. I don't know how well this specific prompt works - I don't see benchmarks - but prompting is a black art, so I wouldn't be surprised at all if it excels more than a blank slate in some specific category of tasks.

> Think of the lengthy prompt as being like a safe combination

I can think all I want, but how do we know that this metaphore holds water? We can all do a rain dance, and sometimes it rains afterwords, but as long as we don't have evidence for a causal connection, it's just superstition.

Post reply on HN