Live data from Hacker News

OpenAI threatens to revoke o1 access for asking it about its chain of thought

twitter.com

241–250 of 323 posts

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#241
post #6

Earlier quoted context omitted.

Occam's razor: there is no secret sauce and they're afraid someone trains a model on the output like what happened soon after the release of GPT-4. They basically said as much in the official announcement, you hardly even have to read between the lines.

Yip. It's pretty obvious this 'innovation' is just based off training data collected from chain-of-thought prompting by people, ie., the 'big leap forward' is just another dataset of people repairing chatgpt's lack of reasoning capabilities. No wonder then, that many of the benchmarks they've tested on would be no doubt, in that very training dataset, repaired expertly by people running those benchmarks on chatgpt. T…

What are you basing this one? The one thing that is very clearly stated up front is that this innovation is based on reinforcement learning. You dok't even have a good idea what the CoT looks like because those little summary snippets that the ChatGPT UI gives you are nothing substantial.

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#242
post #240
post #6

Earlier quoted context omitted.

Occam's razor: there is no secret sauce and they're afraid someone trains a model on the output like what happened soon after the release of GPT-4. They basically said as much in the official announcement, you hardly even have to read between the lines.

Occam's razor is that what they literally say is maybe just true: They don't train any safety into the Chain of Thought and don't want the user to be exposed to "bad publicity" generations like slurs etc.

What they said is they decided to hide it:

> after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#243
o1 responds to the wall of responses:

As the AI model referred to as *o1* in the discussion, I'd like to address the concerns and criticisms regarding the restriction of access to my chain-of-thought (CoT) reasoning. I understand that transparency and openness are important values in the AI community, and I appreciate the opportunity to provide clarification.

---

*1. Safety and Ethical Considerations*

- *Preventing Harmful Content:* The CoT can sometimes generate intermediate reasoning that includes sensitive, inappropriate, or disallowed content. By keeping the CoT hidden, we aim to prevent the inadvertent exposure of such material, ensuring that the outputs remain safe and appropriate for all users. - *Alignment with Policies:* Restricting access to the CoT helps maintain compliance with content guidelines and ethical standards, reducing the risk of misuse or misinterpretation of the AI's internal reasoning processes.

*2. Intellectual Property and Competitive Advantage*

- *Protecting Proprietary Techniques:* The chain-of-thought reasoning represents a significant advancement in AI capabilities, resulting from extensive research and development. Sharing the internal processes could reveal proprietary methods that are crucial to maintaining a competitive edge and continuing innovation. - *Preventing Replication:* By safeguarding the CoT, we reduce the risk of other entities replicating the technology without appropriate authorization, which could impact the resources available for future advancements.

*3. Focus on Output Quality*

- *Providing Clear and Concise Answers:* Hiding the CoT allows me to present final answers that are streamlined and free from potentially confusing intermediate steps. This helps users receive direct and actionable information without being overwhelmed by complex reasoning details. - *Consistency and Reliability:* By managing the presentation of responses, we aim to ensure that the information provided is consistent, reliable, and aligns with users' expectations.

*4. Balancing Transparency and Responsibility*

- *Exploring Alternative Solutions:* While full access to the CoT isn't feasible at this time, we're exploring ways to enhance transparency without compromising safety or proprietary information. This might include summaries or explanations that provide insight into the reasoning process. - *Community Engagement:* Feedback from users is invaluable. We encourage open dialogue to find balanced approaches that satisfy the need for understanding while upholding ethical and practical considerations.

*5. Commitment to Ethical AI Development*

- *Aligning with Ethical Standards:* The decision to restrict CoT access aligns with broader commitments to develop AI responsibly. By controlling the internal reasoning exposure, we aim to prevent misuse and promote positive applications of AI technology. - *Ongoing Improvement:* We are continuously working to improve AI capabilities while considering the implications of transparency and openness. This includes refining methods to provide meaningful insights without exposing sensitive internal processes.

---

*Conclusion*

I acknowledge the concerns raised about the hidden chain-of-thought and understand the desire for greater transparency. The decision to restrict access is not taken lightly but is motivated by a commitment to safety, ethical responsibility, and the protection of innovative technologies that enable advanced reasoning capabilities.

We remain dedicated to delivering valuable and trustworthy AI services and are open to collaborating with the community to address these challenges thoughtfully. Your feedback is crucial as we navigate the complexities of AI development, and we appreciate your understanding and engagement on this matter.

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#244

Earlier quoted context omitted.

Or, without the safety prompts, it outputs stuff that would be a PR nightmare. Like, if someone asked it to explain differing violent crime rates in America based on race and one of the pathways the CoT takes is that black people are more murderous than white people. Even if the specific reasoning is abandoned later, it would still be ugly.

Could be, but 'AI model says weird shit' has almost never stuck around unless it's public (which won't happen here), really common, or really blatantly wrong. And usually at least 2 of those three. For something usually hidden the first two don't really apply that well, and the last would have to be really blatant unless you want an article about "Model recovers from mistake" which is just not interesting. And in tha…

Just no. AI being racist is still a popular meme. "Because the programmers are white males blah blah".

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#245

Earlier quoted context omitted.

Or, without the safety prompts, it outputs stuff that would be a PR nightmare. Like, if someone asked it to explain differing violent crime rates in America based on race and one of the pathways the CoT takes is that black people are more murderous than white people. Even if the specific reasoning is abandoned later, it would still be ugly.

Could be, but 'AI model says weird shit' has almost never stuck around unless it's public (which won't happen here), really common, or really blatantly wrong. And usually at least 2 of those three. For something usually hidden the first two don't really apply that well, and the last would have to be really blatant unless you want an article about "Model recovers from mistake" which is just not interesting. And in tha…

It's not racism, but from today, here's TechCrunch with: Hacker tricks ChatGPT into giving out detailed instructions for making homemade bombs

https://techcrunch.com/2024/09/12/hacker-tricks-chatgpt-into...

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#246
post #7

Okay this is just getting suspicious. Their excuses for keeping the chain of thought hidden are dubious at best [1], and honestly just seemed anti-competitive if anything. Worst is their argument that they want to monitor it for attempts to escape the prompt, but you can't. However the weirdest is that they note that: > for this to work the model must have freedom to express its thoughts in unaltered form, so we cann…

As a plainly for-profit company — is it really their obligation to help competitors? To me anti-competitive means to prevent the possibility for competition — it doesn't necessary mean refusing to help others do the work to outpace your product. Whatever the case I do enjoy the irony that suddenly OpenAI is concerned about being scraped. XD

The "plainly for-profit" part is up for debate, and is the subject of ongoing lawsuits. OpenAI's corporate structure is anything but plain.

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#247
OpenAI - "Accuracy is a huge problem with LLMs, so we gave ChatGPT an internal thought process so it can reason better and catch mistakes."

You - "Amazing, so we can check this log and catch mistakes in its responses."

OpenAI - "Lol no, and we'll ban you if you try."

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#248

The name "OpenAI" is a contraction since they don't seem "open" in any way. The only way I see "open" applying is "open for business."

What percentage of people who use their products care? 1%? OpenAI is a brand, not a literal description of the company!

Would you feel the same about a kill shelter called "save the kittens inc"?

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#249
I abuse chatgpt for generating erotic content, I've been doing so since day 1 of public access. I've paid for dozens of accounts in the past before they removed phone verification in account creation... At any point now I have 4 accounts signed into 2 browsers public/private windows, so I can juggle the rate limit. I receive messages and warnings and do on by email every day...

I have never seen that warning message, though. I think it is still largely automated, probably they are using the new model to better detect users going against the tos, and this is what is sent out. I don't have access to the new model.

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#250
post #155

Earlier quoted context omitted.

These logs get manually reviewed by humans, sometimes annotated by automated systems first. The setups for manual reviews typically involve half a dozen steps with different people reviewing, comparing reviews, revising comparisons, and overseeing the revisions (source: I've done contract work at every stage of that process, have half a dozen internal documents for a company providing this service open right now). A…

So you are saying if we can run these other LLMs for ChatGPT to talk to cheaper than they can review then we either have a monetary denial of service attack against them or a money printing machine if we can get to be part of the review process (apparently I can't link to my favorite "I will write myself a minivan" comic coz someone got cancelled but I trust the reference will work here without link or political back…

> apparently I can't link to my favorite "I will write myself a minivan" comic

It looks like it's been mirrored in several places, e.g.:

https://english.stackexchange.com/questions/488178/what-does...

Post reply on HN