Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

More questions about whether researchers can trust OpenAI with unpublished math

mathstodon.xyz

Post reply on HN
Print view View original

491–500 of 844 posts

Re: More questions about whether researchers can trust OpenAI with unpublished math

#491

Post by pixl97 »

ForHackernews wrote: post #450
pixl97 wrote: post #311

Earlier quoted context omitted.

>They had to burn millions of dollars to solve a single problem I'd like to adjust that to "They had to burn a lot of energy (create a lot of entropy) to solve a single problem. As we go into the super-intelligence age the current paradigm of money as humans understand it may break at some point. For example to a paperclip-maximizer money at best is a short term instrumental goal, hard power of matter conversion mach…

I'd wager a fair chunk of my money that money breaks OpenAI before OpenAI breaks money.

OpenAI != AI.

If you were in 1999 you'd be saying pets.com = internet.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#492

Post by nsagent »

tedsanders wrote: post #487
jjwiseman wrote: post #423

Earlier quoted context omitted.

First, OpenAI is not claiming that the model wasn't trained on those sessions. What they've said is “We (the researchers and the agents) did not see any of their work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem.” and “We did not use their prompts or proofs to prompt our models or direct our agents.” and “While unlikely, we cannot…

We were also curious and we looked further into this. We've determined it was impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training. This goes beyond what we said earlier, when we were less sure. If prompts were submitted earlier than that and training was not opted out, there may be a chance they made their way into our training pipeline i…

Can you speak to why in both cases, the problems OpenAI's models solved used the same techniques the mathematicians were exploring, which also happened to be niche approaches to the problem. As an NLP researcher myself, I find that coincidence highly suspect unless the models focused most of their attempts on the predominant approaches (they are trained for MLE after all).

Re: More questions about whether researchers can trust OpenAI with unpublished math

#493

Post by square_usual »

abathologist wrote: post #467
lnrd wrote: post #398

Earlier quoted context omitted.

> They intentionally threw $15 million in compute at the problem what? really?

Yes. Maybe much more: > Such intensive use of AI doesn't come cheap. In a post on X, LisanBench, an LLM benchmark evaluator, estimated that the output tokens alone would cost about $6.5 million at OpenAI's average consumer price. Including the far larger volume of input tokens, the post estimated the total could reach $10 million to $40 million. https://www.businessinsider.com/openai-math-problem-solved-t...

That's their API pricing. There's no way they actually paid $15M in compute. I'd say much more likely it's in the order of $1M.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#494

Post by bitexploder »

aaronharnly wrote: post #461
bitexploder wrote: post #408

Earlier quoted context omitted.

That is a better idea. Ingesting your corpus with a lot of traces that have semantic patterns. Semantic steganography that suffixes well to real math and science (and any) topics. heh.

"Semantic steganography" is my new favorite search term – thank you for this rabbit hole.

Hah, np, stego in general is really cool :)

Re: More questions about whether researchers can trust OpenAI with unpublished math

#495

Post by ForHackernews »

pixl97 wrote: post #491
ForHackernews wrote: post #450

Earlier quoted context omitted.

I'd wager a fair chunk of my money that money breaks OpenAI before OpenAI breaks money.

OpenAI != AI. If you were in 1999 you'd be saying pets.com = internet.

yeah yeah yeah. I agree that AI is and will be a very useful tool, it's just not going to be worth $30T like OpenAI/Anthropic are pretending.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#496

Post by btilly »

nautilus12 wrote: post #356
btilly wrote: post #309

Earlier quoted context omitted.

Yes. See https://www.anthropic.com/research/small-samples-poison?from... . 250 documents ingested from somewhere is enough to become part of the knowledge of a model of arbitrarily large size. I would expect that a good idea that fits in a framework that is already being ingested would be more easily taken up than some random thing unassociated with anything else. Could that go down to a single transcript? If the mod…

Thats not what they are asking. This paper is discussing documents in the training dataset poisoning the LLM for malicious behavior. This person are asking if anyone has deliberately put something in a private chat (presumably with retrain on my data turned off), to see if they can get it to leak across sessions from distinct users. I am positive this happens but I have not seen the proof. I also want to know the ans…

Did you miss my last paragraph?

I presented the research that I knew was somewhat relevant. Then made it clear that that wasn't what was being asked, and why my expectation is what it is.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#497

Post by Enginerrrd »

techblueberry wrote: post #156
Robotbeat wrote: post #152

Earlier quoted context omitted.

Neither? Competitive academic researchers are susceptible to exaggeration and self-aggrandizing, and CEOs are that and also mostly psychopaths. I tend to think there isn’t systematic spying on researchers looking for breakthroughs. A lot of people are looking for the same things using similar approaches.

I mean the accusation is that they were using private ChatGPT conversations. Given the extent to of the gold rush and the long history of Silicon Valley stealing ideas, and arguing it’s not immoral, It almost seems like your making the exceptional claim that this is the one time where Silicon Valley didn’t use information that was at their disposal. Sam Altman might himself be offended you would presume he’s not ambi…

> Given the extent to of the gold rush and the long history of Silicon Valley stealing ideas, and arguing it’s not immoral

Not just Silicon Valley but also OpenAI, specifically.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#498

Post by HarHarVeryFunny »

famouswaffles wrote: post #371
HarHarVeryFunny wrote: post #303

Earlier quoted context omitted.

OpenAI said they sicced this agent army on Navier-Stokes on Sept 1st, while only a couple of days earlier OpenAI's Noam Brown happened to reply to a tweet saying that they had already tried to solve all the Millennium Prize problems and failed... So, it seems either the previous attempt didn't have the training to succeed, or was just not given the compute to do so. Once OpenAI heard that Navier-Stokes was solved, th…

OpenAI have come out and said: >The Wednesday evening statement from OpenAI was more emphatic: “We can say categorically that it is impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training.” >The statement added, “After investigating, we can say with full confidence that no user inputs past July 3rd could have influenced this system in any way…

OK, good to know (if they can be trusted - Altman clearly is a liar), but it doesn't really change the big picture much.

1) OpenAI by their own admission, only re-tackled Navier-Stokes because they heard it had already been solved (but not yet published). This isn't advancing science or helping the mathematical community, this is just being a dick.

2) OpenAI, specifically Sebastien Brubeck, then threaten to "not be nice" and "ruin the career" of one of the mathematicians whose work they had succeeded in duplicating, unless he agreed (which he refused to do) that his collaborator, an Anthropic employee, was not named. This is not only against mathematical norms of credit assignment, it is also being a pathetic human being.

OpenAI would have you believe this result shows how powerful their mystery better-than-Astra model is, but the reality here is that this model needed 10,000 agents, $20M of compute, and the assistance of a whole team of people at OpenAI, to replicate (then exceed) the work that just took two people, with some academic grants as an AI spending budget to achieve (a few $100K - listed below).

https://cims.nyu.edu/~tristanb/

I'd say advantage humans this time. Better luck next time OpenAI - and if you don't want unfavorable comparisons then maybe choose to work on problems that have not been solved yet, and that humans are NOT making nice progress on.

Re: More questions about whether researchers can trust OpenAI with unpublished math

#499

Post by magicalist »

ozgung wrote: post #486
raincole wrote: post #454

Earlier quoted context omitted.

The irony is that OpenAI got into this trouble only because they tried to play "nice". They told Buckmaster that he could publish the final result as the author as long as he removed Alpöge from the author list. They wanted to give Buckmaster a chance to be the one solved N-S problem. While this behavior is highly questionable, if OpenAI just published the final result without notifying Buckmaster first and simply ci…

No. 1. Buckmaster contacted OpenAI first. Not the other way. 2. Giving the $1M bounty to a human mathematician for the effort and giving him credit would be excellent PR. They had already burned much more than $1M for the generation. Adding him as author also costs nothing. Purely pragmatical. 3. “As long as he removed Alpöge” part itself is against academic honesty by all means. 4. Buckmaster rejected fame and $1M o…

There are some mixed up things in your post, maybe double check next time, especially before quoting anyone, as you really undermine your point even if you're directionally right.

> Buckmaster rejected fame and $1M only because doing (3) would be wrong

I doubt Buckmaster would have accepted the offer to "write a paper presenting the Navier-Stokes result, acknowledging that an internal OpenAI model had resolved it" even if removing Alpöge from authorship wasn't a requirement. He clearly wanted nothing to do with OpenAI's actions here.

edit: I don't know if people think I'm disagreeing here, I'm certainly not, I'm just pointing out that playing the game of telephone with easily verifiable quotes is lazy and bad. For example, "end [your] career" was "ruin your career", and it was phrased as the much more "it would be a shame if something happened to you" like "Why would you ruin your career?" when Buckmaster said he would go public with this conversation: https://cims.nyu.edu/~tristanb/statement.pdf

Re: More questions about whether researchers can trust OpenAI with unpublished math

#500

Post by atleastoptimal »

robotpepi wrote: post #364
atleastoptimal wrote: post #343

Most scientific breakthroughs are simply a continuation of previous work. I feel that these suspicions of mathematicians "seeding" the models' with intuition on how to solve these problems massively overestimates how much their prompts helped the models, and underestimated how much work the models did. Why? We are scared of AI being smarter than us, the "human helped the AI" narrative is more psychologically comforti…

> We are scared of AI being smarter than us, the "human helped the AI" narrative is more psychologically comforting. We're scared of big tech companies concentrating ridiculous amounts of power, destroying the communities that support and guide scientific research, without even thinking about the dangers and possible consequences, because a PR stunt is more important in the short term.

If this were true, it should be stated more clearly, than most of the criticism which seems to aim to minimize the capabilities of these models.

Way more often I see

>AI is a scam and steals human insight and doesn't produce anything original

vs

>AI is too capable/powerful and will concentrate power even more than it does already due to its capabilities

The latter is rarer because it requires admitting that AI is useful and inventive

Post reply on HN