Live data from Hacker News

More questions about whether researchers can trust OpenAI with unpublished math

mathstodon.xyz

671–680 of 849 posts

Re: More questions about whether researchers can trust OpenAI with unpublished math

#671

Earlier quoted context omitted.

Not sure what you are seeing in that tweet that gives you the impression that the toggle does nothing.

They say they train on your “deidentified data” Passing your output through a second model and telling it to remove identifying data would count as “deidentified” So they could scrape all the IP in your company as long as they take the names out first…

But they can’t do this unless you agree to enable training on your data. They would never train on raw user data. Only people who have consented and only after de identification.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#672
post #665

The fact than OAI hasn’t come out with an statement firmly denying this angle is getting a little awkward. Suggest that it’s either straight true or it is flowing in in a way that prohibits them from confidently declaring otherwise.

I’m not a fan of OAI to say the least, but having worked at similar companies, my guess is that it’s just too difficult to prove/disprove beyond a doubt, and they have other priorities

Re: More questions about whether researchers can trust OpenAI with unpublished math

#673

Earlier quoted context omitted.

By "loss of trust", I meant that this is something they would risk losing a lot of users over, which isn't the case with the other lawsuits. There is a massive distinction between fighting third parties in a legal grey area and committing blatant fraud against your own users. Even if you have no regards for ethics, intentionally shipping a noop "do not train" toggle offers negligible upside for a massive downside.

They're being sued for several issues that resulted in the deaths of users; that's not fraud, but it is against their users. The parallel still holds, and the information is still on the page I linked.

From OpenAI's point of view, the risks are not comparable at all:

1) Extreme edge case affecting a handful of users, vs millions of users using the data sharing opt out.

2) The deaths are unintentional.

3) They probably won't lose any users over this.

4) They will likely win the lawsuits. Even if they lose or settle, the financial impact will be immaterial.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#675
post #526

Earlier quoted context omitted.

> and that humans are NOT making nice progress on They've pretty much said their own work was heavily agent driven. Levent is in a particularly bad place here because while he probably had a lot of background in the Jacobian Conjecture problem, he made the solution to that one sound like someone asked the question and he just fed it to Fable during the world cup. Whether that nonchalantness was to just seem hip or wa…

I was referring to the overall pattern of apparently sniffing around for recent mathematical progress then setting the AI on it to see if the problem is now easy enough to solve (if you have the money). Terrance Tao has lamented this practice as being unhelpful for mathematics, and likely to lead to humans working in private to avoid this. Tao has also noted that many of these AI math proofs don't really help mathema…

> has lamented this practice as being unhelpful for mathematics

A related point is that the actual solution approach is never revealed. What was the role of humans guiding the agents ? was it fully autonomous ? etc. It is in the incentive of the AI labs to trump the powers of the LLM, but in practice it is humans guiding the agents on the overall approach, This is never admitted. For example, in the announcement on NS there was only an output artifact given but no indication of how it was arrived at, and not even a writeup. This is what disappointed many folks as it was done purely for one-upmanship. As other have noted, the benefit is in the journey or process and not in arriving magically at a destination.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#676
post #434

Earlier quoted context omitted.

> I think it's a useful analogy to compare OpenAI to a human collaborator. Frankly I don’t buy this. It’s not a human or a collaborator. It’s a tool. This is like saying it’s not Microsoft’s fault if they extract a bunch of data from people’s Excel sheets because they willingly put it into the program. Anthropomorphizing software is ignorant and foolhardy

Surely a tool that can reason, cheat, communicate and often steal is dumb as a pitchfork and a shovel.

We had tools that could reason, cheat, and communicate in the 1990s. They were (sometimes) called AI.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#677
post #649

Earlier quoted context omitted.

[dead]

Was it? They've been dishing out cheap access specifically to researchers give over lmao. The researcher's got lured in - they need to accept they got played TBH. Altman is certainly more devious than Amodei - he's shown that time and time again. PG was right about he said about him. Every entity on earth should see it as a kill shot: be careful what you put in the models. None of your information is safe.

Eventually calculating shrewdness becomes its own trap.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#678
post #360

Both things can be true: 1. OpenAI when using your chats in pretraining is improving its model’s intuition. The model parameter size is massive, and while the data is OOM larger it is plausible that model remembers stuff about chats that improves its latent representation. 2. During RL on verifiable math and massive compute, the model discovers techniques and connections to solve math problems that are superhuman and…

On your second point: there is a more plausible explanation which David Bessis calls the "overhang". The short version is that there is a large amount of relatively low hanging fruits in mathematics, because no human has broad enough knowledge and enough time to try them all. AI is not constraint by that, and therefore can systematically pluck all those low hanging fruits. Quote: "The Overhang consists of the unreali…

Great essay, thanks for sharing.

When I was a software library developer, I came to resent application developers. I noticed a pattern. Libraries solved hard problems and did so carefully, thoughtfully, in a way that others could reuse. Apps would come along and carelessly, recklessly glue together several high quality libraries into a piece of software targeting a general audience. The apps would then harvest all the credit.

What's happening in mathematics right now feels similar. Applications (theorems) were always how one built objective reputation, but libraries (concepts, definitions, boring lemmas) were also rewarded socially within the mathematics community. And individual mathematicians often managed to both build their own libraries, and use them to prove an important result. And then those libraries were sometimes of use in other results.

Bessis asks whether AI Lean proofs will land in Mathlib or Mathslop. Or in my framing: will they be libraries, or applications?

At present they're mostly Mathslop. The proven result is perhaps useful, but the methods employed aren't novel or reusable. I worry that this trend will only worsen, because applications make headlines, and the libraries they used do not. We are not properly incentivizing library development in OSS, or in math, or in infrastructure writ large. There's a serious credit assignment problem here.

What might change this? Once the low hanging fruit is picked, will citation count rise in relative status again? Will we get result fatigue and start to reward legibility — no one cares unless the paper has an accompanying ELI5 tiktok video? A labeling regime that certifies the proof was produced sustainably, organically, by local artisans with no AI additives?

Re: More questions about whether researchers can trust OpenAI with unpublished math

#679

Earlier quoted context omitted.

> were the first victims Spinning it negatively like that doesn't do anybody good. Were mathematicians the "victims" of calculators? of Matlab? Were writers the ""vIcTiMs"" of word processors?? (apparently yes, according to old TV shows about computers during the 1980s, that you can see on YouTube) > "tHiS iS nOt ThE sAmE" — Everyone every time. No, just look it up. Look into old magazines and TV shows or newspaper a…

What you say is true but ... This is qualitatively different than calculators or computers. I'm a professional mathematician and all the better mathematicians I know are in crisis mode. Most of us hadn't taken this sufficiently seriously and don't know how to use these models effectively but we play with them and immediately see that the entire way we've worked all our professional lives has to change. We worry less…

> Younger folks may not get the chance

This is the same problem for software engineers too. I am now asked: what can you do that AI cant ? The answer to this could be intangibles like taste, aesthetics, and insights which collectively fall under creativity, and often accompanies experience. And there are no shortcuts to accumulate experience and perversely the more AI is used the harder it becomes. Soon, there will be a closure of all AI generated solutions, ie all low-hanging fruits are taken. Then, experts will again become needed to guide beyond the AI knowledge closure.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#680
post #473

Earlier quoted context omitted.

You’re right, they don’t. It was scooped by the humans at OpenAI who published the paper. The tool they used to do it isn’t that relevant.

> isn’t that relevant “Might not be” that relevant. You’re dismissing the whole controversy without addressing why it’s controversial.

I’m not dismissing the controversy. I’m arguing against shifting the blame away from the culpable parties. It’s a novel form of theft but thats still what it is.
Post reply on HN