501–510 of 844 posts
Earlier quoted context omitted.
I feel that we don’t praise Lean enough. AFAIU it’s what enables LLMs to brute force those problems
How long until we find out that some AI has quietly buried an exploit in Lean to cheat at proofs?
I think it's a useful analogy to compare OpenAI to a human collaborator. These researchers willingly collaborated with an OpenAI model, giving it ideas, and OpenAI provided useful replies. Then, OpenAI goes ahead and publishes work along the lines of this collaboration, without attributing the researchers. If OpenAI was in fact a human researcher, this would be highly unethical. Now, OpenAI is claiming that the model…
> Provenance is hard to track - you would hope OpenAI has very good tools for this, but a full data trail of all inputs is difficult to trace through. What would OpenAIs incentive for this be? They've gotten away with scraping everything and getting it ruled fair use. It seems like willful ignorance is an affirmative defense today. Why would they want to have some sort of audit trail that could prove otherwise?
Has anyone run a test of including some shibboleth or canary phrase or assertion in a chat, enabled for training, and seeing if it turns up later as something a model "knows"? I'd be curious to understand how that works even in a toy-level model, and if there is anyone consciously testing that process with the frontier lab offerings. My naive instincts would be that it seems unlikely that a single chat transcript wou…
Yes. See https://www.anthropic.com/research/small-samples-poison?from... . 250 documents ingested from somewhere is enough to become part of the knowledge of a model of arbitrarily large size. I would expect that a good idea that fits in a framework that is already being ingested would be more easily taken up than some random thing unassociated with anything else. Could that go down to a single transcript? If the mod…
Earlier quoted context omitted.
The irony is that OpenAI got into this trouble only because they tried to play "nice". They told Buckmaster that he could publish the final result as the author as long as he removed Alpöge from the author list. They wanted to give Buckmaster a chance to be the one solved N-S problem. While this behavior is highly questionable, if OpenAI just published the final result without notifying Buckmaster first and simply ci…
No. 1. Buckmaster contacted OpenAI first. Not the other way. 2. Giving the $1M bounty to a human mathematician for the effort and giving him credit would be excellent PR. They had already burned much more than $1M for the generation. Adding him as author also costs nothing. Purely pragmatical. 3. “As long as he removed Alpöge” part itself is against academic honesty by all means. 4. Buckmaster rejected fame and $1M o…
In this case it's more like C) though, as in hindsight the best move OpenAI could do is insisting that they just used an insurmountable number of tokens to exhaust all the published directions. They absolutely shouldn't have thought of negotiating with Buckmaster over the Clay prize at all, let alone trying to manipulate him into a situation where Alpöge is specifically excluded.
Earlier quoted context omitted.
We were also curious and we looked further into this. We've determined it was impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training. This goes beyond what we said earlier, when we were less sure. If prompts were submitted earlier than that and training was not opted out, there may be a chance they made their way into our training pipeline i…
Can you speak to why in both cases, the problems OpenAI's models solved used the same techniques the mathematicians were exploring, which also happened to be niche approaches to the problem. As an NLP researcher myself, I find that coincidence highly suspect unless the models focused most of their attempts on the predominant approaches (they are trained for MLE after all).
- I've heard some people say the model's solution is quite different from theirs (but I have no clue how to personally assess the spiritual truth of this, so please give it zero weight)
- Thousands of agents costing millions of dollars searched for ideas, and they were encouraged to explore a diversity of approaches, so it wouldn't be too surprising to me if the approaches they tried overlapped with other mathematicians', especially considering the models have knowledge of so much published math research
- This model has been beastly at solving all sorts of math problems (if it was Euler in particular, I'd agree that would look suspicious/lucky)
- The Euler regularity disproof itself took ~100 agents working for ~50 hours (if it was very quick, and then the subsequent NS work took a long time, I'd agree that would look suspicious/lucky)
I understand the skepticism, but from what I know internally at OpenAI, we have zero reason to believe our models did anything fishy. It's hard for us to prove a negative, especially when you have to take us at our word, so I understand why people still feel suspicious.
Edit: Reminds me a bit of the Scarlet Johansson voice cloning accusations and FrontierMath cheating accusations, where the rumors of misbehavior seemed to travel faster than the truth. In both of those cases, we hadn't done what was accused, but suspicions persisted nonetheless.
I think it's a useful analogy to compare OpenAI to a human collaborator. These researchers willingly collaborated with an OpenAI model, giving it ideas, and OpenAI provided useful replies. Then, OpenAI goes ahead and publishes work along the lines of this collaboration, without attributing the researchers. If OpenAI was in fact a human researcher, this would be highly unethical. Now, OpenAI is claiming that the model…
First, OpenAI is not claiming that the model wasn't trained on those sessions. What they've said is “We (the researchers and the agents) did not see any of their work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem.” and “We did not use their prompts or proofs to prompt our models or direct our agents.” and “While unlikely, we cannot…
> no specific user data was accessed in order to solve this problem
Data was accessed in order to (and then accidentally used in training) Also, is llm’s answer to the prompt actually “user data”?
> We did not use their prompts or proofs …
So they used llm’s answers to those prompts.
> … to prompt our models or directew our agents.
So they trained the model on it. (Training is not prompting and plain model is not an agent)
I think it's a useful analogy to compare OpenAI to a human collaborator. These researchers willingly collaborated with an OpenAI model, giving it ideas, and OpenAI provided useful replies. Then, OpenAI goes ahead and publishes work along the lines of this collaboration, without attributing the researchers. If OpenAI was in fact a human researcher, this would be highly unethical. Now, OpenAI is claiming that the model…
First, OpenAI is not claiming that the model wasn't trained on those sessions. What they've said is “We (the researchers and the agents) did not see any of their work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem.” and “We did not use their prompts or proofs to prompt our models or direct our agents.” and “While unlikely, we cannot…
In terms of work in mathematics, something I personally would not do based on ethical grounds would be to hear a rumor that some researchers are taking a certain approach and may be nearing a solution, use a model that was possibly contaminated with intimate knowledge about that approach (though later they investigated and think it wasn't), and then commit millions to tens of millions of dollars and untold amounts of hardware to try to beat them to it. If I had done this, I also wouldn't have pestered the researchers on a Sunday night to meet immediately so we could negotiate a nice way of presenting the actions I had decided to take.
Even if you don't think it was unethical, it was never going to be received well in the community that was especially going to care about this work, and who are very much peers to many of the people working on this solution, so it was at the least an enormous (and well-deserved) own-goal that their unveiling of their solution to NS went like this.
Has anyone run a test of including some shibboleth or canary phrase or assertion in a chat, enabled for training, and seeing if it turns up later as something a model "knows"? I'd be curious to understand how that works even in a toy-level model, and if there is anyone consciously testing that process with the frontier lab offerings. My naive instincts would be that it seems unlikely that a single chat transcript wou…
Problem is how do you convince the model and training profess it matters. A one off canary is very unlikely to survive in the final model state.
You can probably game the metrics that models use to weight potential knowledge akin to SEO. Maybe have some bots parrot your data around a bit in some places online, maybe the model picks up on this and sees it as high engagement and promotes it over the correct data.
Maybe there are ways you can coax out the most optimal way to break into the training set out of the model itself.
I've been wondering whether AI really is improving rapidly at open problems or we're being fooled. - OpenAI invites researchers to use their models, in fact giving at least 100,000 researchers free access[1], but there are also those that pay - Internal OpenAI models are reportedly solving open problems at a surprisingly fast rate[2] - But researchers will typically work on open problems. A researcher who is using Co…
I've been thinking along exactly these lines... they very well could have a 21st century Mechanical Turk and its real superpower is getting people to "collaborate" asynchronously but it's just stealing their ideas and laundering them.
I don't think it's purely that, of course... but "consult other clients' transcripts" would be an easy tool to write.
Earlier quoted context omitted.
We were also curious and we looked further into this. We've determined it was impossible for Dr. Buckmaster’s Codex prompts over the last two months to have influenced the system in any way, including training. This goes beyond what we said earlier, when we were less sure. If prompts were submitted earlier than that and training was not opted out, there may be a chance they made their way into our training pipeline i…
Can you speak to why in both cases, the problems OpenAI's models solved used the same techniques the mathematicians were exploring, which also happened to be niche approaches to the problem. As an NLP researcher myself, I find that coincidence highly suspect unless the models focused most of their attempts on the predominant approaches (they are trained for MLE after all).
ChatGPT agrees with this too.
https://chatgpt.com/share/6aa31959-b0e8-83ec-bee6-851ed18d45...