Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI threatens to revoke o1 access for asking it about its chain of thought

twitter.com

Post reply on HN
Print view View original

141–150 of 323 posts

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#141

Post by icpmacdo »

DeepYogurt wrote: post #13

Would be funny if there was a human in the loop that they're trying to hide

Scaling The Turk to OpenAI scale would be as impressive as agi

"The Turk was not a real machine, but a mechanical illusion. There was a person inside the machine working the controls. With a skilled chess player hidden inside the box, the Turk won most of the games. It played and won games against many people including Napoleon Bonaparte and Benjamin Franklin"

https://simple.wikipedia.org/wiki/The_Turk#:~:text=The%20Tur....

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#142

Post by wg0 »

CoT again is result of computing probabilities on tokens which happen to be reasoning steps. So those are subject to the same limitations as LLMs themselves.

And OpenAI knows this because exactly CoT output is the dataset that's needed to train another model.

The general euphoria around this advancement is misplaced.

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#143

Post by quantified »

bugglebeetle wrote: post #43
fallingknife wrote: post #26

Earlier quoted context omitted.

This is correct. Most people hear about AI from two sources, AI companies and journalists. Both have an incentive to make it sound more powerful than it is. On the other hand this thing got 83% on a test I got 47% on...

This thing also hallucinated a test directly into a function when I asked it to use a different data structure, which is not something I ever recall doing during all my years of tests and schooling.

Must have been quite the hangover to prevent your recalling this.

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#144

Post by quantified »

jasonlfunk wrote: post #129
IncreasePosts wrote: post #33

Earlier quoted context omitted.

Or, without the safety prompts, it outputs stuff that would be a PR nightmare. Like, if someone asked it to explain differing violent crime rates in America based on race and one of the pathways the CoT takes is that black people are more murderous than white people. Even if the specific reasoning is abandoned later, it would still be ugly.

This is 100% a factor. The internet has some pretty dark and nasty corners; therefore so does the model. Seeing it unfiltered would be a PR nightmare for OpenAI.

I trust that Grok won't be limited by avoiding the dark and nasty corners.

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#145

Post by [deleted] »

contravariant wrote: post #4

Okay this is just getting suspicious. Their excuses for keeping the chain of thought hidden are dubious at best [1], and honestly just seemed anti-competitive if anything. Worst is their argument that they want to monitor it for attempts to escape the prompt, but you can't. However the weirdest is that they note that: > for this to work the model must have freedom to express its thoughts in unaltered form, so we cann…

[deleted]

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#146

Post by mianos »

dontlikeyoueith wrote: post #42
fallingknife wrote: post #26

Earlier quoted context omitted.

This is correct. Most people hear about AI from two sources, AI companies and journalists. Both have an incentive to make it sound more powerful than it is. On the other hand this thing got 83% on a test I got 47% on...

> On the other hand this thing got 83% on a test I got 47% on Easy to do when it can memorize the answers in its training data and didn't get drunk while reviewing the textbook (that last part might just be me).

The Olympiad questions are puzzles, so you can't memorise the answers. To do well you need to both remember the foundations and exercise reasoning. They are written to be slightly novel to test this and not the same every year.

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#147

Post by a2128 »

If OpenAI really cares about AI safety, they should be all about humans double-checking the thought process and making sure it hasn't made a logical error that completely invalidates the result. Instead, they're making the conscious decision to close off the AI thinking process, and they're being as strict about keeping it secret as information about how to build a bomb.

This feels like an absolute nightmare scenario for AI transparency and it feels ironic coming from a company pushing for AI safety regulation (that happens to mainly harm or kill open source AI)

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#148

Post by vidarh »

COAGULOPATH wrote: post #92
DiscourseFan wrote: post #41

Earlier quoted context omitted.

It seems like the best AI models are increasingly just combinations of writings of various people thrown together. Like they hired a few hundred professors, journalists and writers to work with the model and create material for it, so you just get various combinations of their contributions. It's very telling that this model, for instance, is extraordinarily good at STEM related queries, but much worse (and worse eve…

>but much worse (and worse even in comparison to GPT4) than English composition O1 is supposed to be a reasoning model, so I don't think judging it by its English composition abilities is quite fair. When they release a true next-gen successor to GPT-4 (Orion, or whatever), we may see improvements. Everyone complains about the "ChatGPTese" writing style, and surely they'll fix that eventually. >Like they hired a few…

The bulk in terms of the number of tokens may well be synthetic data, but I personally know of at least 3 companies, 2 of whom I've done work for, that have people doing substantial amounts of bespoke writing under rather heavy NDAs. I've personally done a substantial amount of bespoke writing for training data for one provider, at good tech contractor fees (though I know I'm one of the highest-paid people for that company and the span of rates is a factor of multiple times even for a company with no exposure to third world contractors).

That said, the speculation you just "get various combinations" of those contributions is nonsense, and it's also by no means only STEM data.

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#149

Post by rmbyrro »

kgeist wrote: post #132
hiddencost wrote: post #107

Earlier quoted context omitted.

They can charge whatever they want.

In my country, it's illegal to charge different people differently if there's no explicitly signed agreement where the both sides agree to it. Without an agreement, there must be a reasonable and verifiable justification for a change in the price. I think suddenly charging you $100 more (compared to other consumers) without explaining how you calculated it is somewhat illegal here.

There's no change in price. They charge the same amount per token from everyone. You pay more if you use more tokens. If some tokens are hidden, used internally to generate the final 'public' tokens is just a matter of technical implementation and business choice. If you're not happy, don't use the service.

Re: OpenAI threatens to revoke o1 access for asking it about its chain of thought

#150

Post by vidarh »

mattkrause wrote: post #70
GaggiX wrote: post #47

Earlier quoted context omitted.

Do you have a source about OpenAI hiring a few hundred professors, journalists and writers? Because I honestly doubt.

A few recruiters have contacted me (a scientist) about doing RLHF and annotation on biomedical tasks. I don’t know if the eventual client was OpenAI or some other LLM provider but they seemed to have money to burn.

I fill in gaps in my contracting with one of these providers, and I know who the ultimate client is, and if you were to list 4-5 options they'd be in there. I've also done work for another company doing work in this space that had at least 4-5 different clients in that space that I can't be sure about. So, yes, while I can't confirm if OpenAI does this, I know one of the big players do, and it's likely most of the other clients are among the top ones...
Post reply on HN