More questions about whether researchers can trust OpenAI with unpublished math • Hacker News Board

Re: More questions about whether researchers can trust OpenAI with unpublished math

#421

Post by jameslars » Thu, Sep 10, 2026, 6:50 PM UTC

nezi wrote: post #412

I think it's a useful analogy to compare OpenAI to a human collaborator. These researchers willingly collaborated with an OpenAI model, giving it ideas, and OpenAI provided useful replies. Then, OpenAI goes ahead and publishes work along the lines of this collaboration, without attributing the researchers. If OpenAI was in fact a human researcher, this would be highly unethical. Now, OpenAI is claiming that the model…

> Provenance is hard to track - you would hope OpenAI has very good tools for this, but a full data trail of all inputs is difficult to trace through.

What would OpenAIs incentive for this be? They've gotten away with scraping everything and getting it ruled fair use. It seems like willful ignorance is an affirmative defense today. Why would they want to have some sort of audit trail that could prove otherwise?

Re: More questions about whether researchers can trust OpenAI with unpublished math

#422

Post by contubernio » Thu, Sep 10, 2026, 6:51 PM UTC

Razengan wrote: post #387

ozgung wrote: post #293

Earlier quoted context omitted.

If your rumor is true, what we are witnessing is a giant paradigm shift rather than individual incidents. Mathematicians were the first victims of super-intelligence. Of course it’s not an endless source. They had to burn millions of dollars to solve a single problem.

> were the first victims Spinning it negatively like that doesn't do anybody good. Were mathematicians the "victims" of calculators? of Matlab? Were writers the ""vIcTiMs"" of word processors?? (apparently yes, according to old TV shows about computers during the 1980s, that you can see on YouTube) > "tHiS iS nOt ThE sAmE" — Everyone every time. No, just look it up. Look into old magazines and TV shows or newspaper a…

What you say is true but ... This is qualitatively different than calculators or computers.

I'm a professional mathematician and all the better mathematicians I know are in crisis mode. Most of us hadn't taken this sufficiently seriously and don't know how to use these models effectively but we play with them and immediately see that the entire way we've worked all our professional lives has to change. We worry less about ourselves than about the younger folks. I've got good ideas ai still doesn't know about ... Younger folks may not get the chance.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#423

Post by jjwiseman » Thu, Sep 10, 2026, 6:56 PM UTC

nezi wrote: post #412

I think it's a useful analogy to compare OpenAI to a human collaborator. These researchers willingly collaborated with an OpenAI model, giving it ideas, and OpenAI provided useful replies. Then, OpenAI goes ahead and publishes work along the lines of this collaboration, without attributing the researchers. If OpenAI was in fact a human researcher, this would be highly unethical. Now, OpenAI is claiming that the model…

First, OpenAI is not claiming that the model wasn't trained on those sessions. What they've said is “We (the researchers and the agents) did not see any of their work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem.” and “We did not use their prompts or proofs to prompt our models or direct our agents.” and “While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.”

They also said “Our effort began on September 1st after hearing a rumor which we later realized was related to Levent Alpöge … and Tristan Buckmaster….” They say the rumor was that two Millennium Prize problems had been resolved, and that this prompted them to launch "an effort to evaluate it on all open Millennium Prize problems and a few other high-impact problems."

It's not obvious to me that's an unethical thing to do, if it happened as they described.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#424

Post by cjf101 » Thu, Sep 10, 2026, 6:56 PM UTC

nezi wrote: post #412

I think it's a useful analogy to compare OpenAI to a human collaborator. These researchers willingly collaborated with an OpenAI model, giving it ideas, and OpenAI provided useful replies. Then, OpenAI goes ahead and publishes work along the lines of this collaboration, without attributing the researchers. If OpenAI was in fact a human researcher, this would be highly unethical. Now, OpenAI is claiming that the model…

If the model was trained proper to the conversation with the researcher took place, there'd be no question of tainting the results. But if any amount of training on the model took place afterward, then yes, everything is thrown into doubt (a core problem with considering anything "original" from a model because of how >a % of everything ever written has been used a corpus for the training).

Re: More questions about whether researchers can trust OpenAI with unpublished math

#425

Post by olalonde » Thu, Sep 10, 2026, 6:58 PM UTC

Planktonne wrote: post #259

olalonde wrote: post #109

Earlier quoted context omitted.

Example?

You can read their Wikipedia page [1]. [1] https://en.wikipedia.org/wiki/OpenAI#Governance_and_legal_is...

These examples aren't really similar. None of those situations involve harming and lying to their own customers.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#426

Post by palmotea » Thu, Sep 10, 2026, 6:58 PM UTC

kzz102 wrote: post #360

sashank_1509 wrote: post #266

Both things can be true: 1. OpenAI when using your chats in pretraining is improving its model’s intuition. The model parameter size is massive, and while the data is OOM larger it is plausible that model remembers stuff about chats that improves its latent representation. 2. During RL on verifiable math and massive compute, the model discovers techniques and connections to solve math problems that are superhuman and…

On your second point: there is a more plausible explanation which David Bessis calls the "overhang". The short version is that there is a large amount of relatively low hanging fruits in mathematics, because no human has broad enough knowledge and enough time to try them all. AI is not constraint by that, and therefore can systematically pluck all those low hanging fruits. Quote: "The Overhang consists of the unreali…

> Quote: "The Overhang consists of the unrealized capital gains of past mathematical creativity, the latent value from connecting the dots in the existing corpus. It is a dividend of canonization. Mathematician X states problem A, mathematician Y crafts concept B, then mathematician Z notices that B trivially solves A and “captures” the social reward. But in the process of capturing the reward, Z usually introduces new concepts and new open problems, reinjecting latent value into the Overhang.

That overhang seems like a precious resource for AI companies. They can exploit that overhang to inflate the impression of AI's capabilities, and hopefully that exploitation will discourage the next generation of mathematicians from pursuing math. If they play their cards right, OpenAI and Anthropic can dominate the field even if they ultimately can't replicate the creativity of human mathematicians, because they'll have driven their competition out.

What we should be trying to achieve is a ladder-breaking maneuver: knock out the lower rungs so no person can reasonably climb to the top-reaches of mathematical skill anymore. That may ultimately result in stagnation, but it's what's best for AI, so it's what should be done now.

We need to do everything we can to create the greatest-possible dependence on AI tools.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#427

Post by lowbloodsugar » Thu, Sep 10, 2026, 6:59 PM UTC

SpicyLemonZest wrote: post #243

lowbloodsugar wrote: post #239

Earlier quoted context omitted.

That’s … Googles entire reason for making these “you don’t pay with money” tools. Did you not understand that?

What? I don't understand how you even came up with this idea, much less consider it so obvious to condescend about it. Do you have even a single example of a research project that got scooped because the Google Docs team forwarded their private documents to someone?

Lol. That's not what I said nor what TFA is claiming. The claim is that private data is used for training.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#428

Post by Lerc » Thu, Sep 10, 2026, 7:01 PM UTC

freejazz wrote: post #383

cbarrick wrote: post #135

Earlier quoted context omitted.

I think people are focusing on the training data issue too much. If the data was contaminated, I can still blame that on negligence. But, at least with the Navier-Stokes solution, it's clear [^1] that they learned that Alpöge and Buckmaster were getting close to a solution and learned of the general approach they were taking. Only after learning the secret to cracking the problem did they send the first prompt. What…

> but I can believe it to be accidental What accident is it when the system is designed to function that way?

Their claim is that training on their solution is "unlikely but possible".

Consider this scenario.

Has a google crawler read my new novel, which I may or may not have posted on my blog, page by page, as I wrote it?

Can you, without knowledge of what I have actually done, claim that the google crawler has not seen the novel?

Without any evidence that I have posted the novel online, it might be tempting to say that the crawler has not seen the novel, but what if I were in an adversarial position against Google on this topic and were challenging them to make that claim. You would wonder if I were hoping Google to overreach by making a definitive claim without taking into account some action that they had no knowledge of. It becomes difficult to use the scientific expression "There is no evidence for this" when there is an accusation of malfeasance because it can be so easily be conflated as "You can't prove we did it". It seems like the best you could say would be 'Unlikely, but possible'

Re: More questions about whether researchers can trust OpenAI with unpublished math

#429

Post by ImPostingOnHN » Thu, Sep 10, 2026, 7:05 PM UTC

YeGoblynQueenne wrote: post #393

ImPostingOnHN wrote: post #238

Earlier quoted context omitted.

They used words to mean what the words mean. What specific issue do you take with that? "prompt", as in prompting an AI, has the same definition as "prompt", as in prompting a person. They mean the same thing, that's why the term was applied to AI after already applying people.

That's a jingle fallacy. *Jingle-jangle fallacies are erroneous assumptions that either two different things are the same because they bear the same name (jingle fallacy); or two identical or almost identical things are different because they are labeled differently (jangle fallacy).[1][2][3] The term was coined by Truman Lee Kelley in his 1927 book Interpretation of educational measurements.[4] In research, a jangle…

You are simply incorrect. It is not a fallacy of that type, or any other type, because the words do, in fact, mean the same thing, as multiple people have pointed out here. Whether referring to chatbots or people, "prompt" means "to move to action".

If you have some reliable source supporting your unilateral claims that "prompt" does not mean this, please share. Otherwise, the consensus seems to be contrary to your claims.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#430

Post by ModernMech » Thu, Sep 10, 2026, 7:08 PM UTC

kzz102 wrote: post #360

sashank_1509 wrote: post #266

Both things can be true: 1. OpenAI when using your chats in pretraining is improving its model’s intuition. The model parameter size is massive, and while the data is OOM larger it is plausible that model remembers stuff about chats that improves its latent representation. 2. During RL on verifiable math and massive compute, the model discovers techniques and connections to solve math problems that are superhuman and…

On your second point: there is a more plausible explanation which David Bessis calls the "overhang". The short version is that there is a large amount of relatively low hanging fruits in mathematics, because no human has broad enough knowledge and enough time to try them all. AI is not constraint by that, and therefore can systematically pluck all those low hanging fruits. Quote: "The Overhang consists of the unreali…

> no human has broad enough knowledge and enough time to try them all.

The other part is, humans don’t really want to fund other humans doing this.

Very few want to be a math major; and of those that do, fewer complete a grad degree; and for those that do get grad degrees, there’s scant few research jobs; and for those who do get jobs there’s hardly any research funding to go around.

There does seem to be unlimited money for ai researchers to use ai to solve these problems though.

We’ve turned education into job training, so because there’s no jobs in solving math problems, few aspire to do it. If there were more opportunities for people, more people would do it, and more low hanging fruit would be plucked.