Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on OpenAI's new o1 chain-of-thought models

simonwillison.net

Post reply on HN
Print view View original

231–240 of 659 posts

Re: Notes on OpenAI's new o1 chain-of-thought models

#231

Post by latexr »

spaceman_2020 wrote: post #162
layer8 wrote: post #18

The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…

I don’t really see this as a massive problem. Its code. If it doesn’t run, you ask it to reconsider, give some more info if necessary, and it usually gets it right. The system doesn’t become useless if it takes 2 tries instead of 1 to get it right Still saves an incredible amount of time vs doing it yourself

> Its code. If it doesn’t run, you ask it to reconsider

It is perfectly possible to have code that runs without errors but gives a wrong answer. And you may not even realise it’s wrong until it bites you in production.

Re: Notes on OpenAI's new o1 chain-of-thought models

#232

Post by oersted »

lukev wrote: post #43

It's interesting to note that there's really two things going on here: 1. A LLM (probably a finetuned GPT-4o) trained specifically to read and emit good chain-of-thought prompts. 2. Runtime code that iteratively re-prompts the model with the chain of thought so far. This sounds like it includes loops, branches and backtracking. This is not "the model", it's regular code invoking the model. Interesting that OpenAI is…

If it really is Reinforcement Learning as they claim, it means there might not be any direct supervision on the "thinking" section of the output, just on the final answer.

Just like for Chess or Go you don't train a supervised model by giving it the exact move it should do in each case, you use RL techniques to learn which moves are good based on end results of the game.

In practice, there probably is some supervision to enforce good style and methodology. But the key here is that it is able to learn good reasoning without (many) human examples, and find strategies to solve new problems via self-learning.

If that is the case it is indeed an important breakthrough.

Re: Notes on OpenAI's new o1 chain-of-thought models

#233

Post by sigmoid10 »

benterix wrote: post #218

From the article: > I expect to continue mostly using GPT-4o (and Claude 3.5 Sonnet) I saw similar comments elsewhere and I'm stunned - am I the only one who considers 4o a step back when compared to 4 for textual input and output? It basically gives fast semi-useful answers that seem like a slightly improved 3.5.

I use gpt4-o mostly, but your specific use-case might have a big impact here: 4o is very likely a distilled model, meaning that it has fewer weights and can thus run much faster on the same hardware. If that is the case, it's general world knowledge must be less comprehensive by default. But it retained the strong reasoning capabilities of 4 through distillation and drastically improved on external tool use and vision. It also offers a much bigger context window. So if you're using it to automate complex tasks in your job that depend a lot on additional information that it hasn't seen during training, 4o is the obvious choice. If you're just using it as a search engine, you should probably stick with 4 for now.

Re: Notes on OpenAI's new o1 chain-of-thought models

#234

Post by ssl-3 »

valval wrote: post #186
williamdclt wrote: post #164

Earlier quoted context omitted.

> Treat it as a naive but intelligent intern That’s the problem: it’s a _terrible_ intern. A good intern will ask clarifying questions, tell me “I don’t know” or “I’m not sure I did it right”. LLMs do none of that, they will take whatever you ask and give a reasonable-sounding output that might be anything between brilliant and nonsense. With an intern, I don’t need to measure how good my prompting is, we’ll usually…

Really it just does what you tell it to. Have you tried telling it “ask me clarifying questions about all the APIs you need to solve this problem”? Huge contrast to human interns who aren’t experienced or smart enough to ask the right questions in the first place, and/or have sentimental reasons for not doing so.

Sure, but to what end?

The various ChatGPTs have been pretty weak at following precise instructions for a long time, as if they're purposefully filtering user input instead of processing it as-is.

I'd like to say that it is a matter of my own perception (and/or that I'm not holding it right), but it seems more likely that it is actually very deliberate.

As a tangential example of this concept, ChatGPT 4 rather unexpectedly produced this text for me the other day early on in a chat when I was poking around:

"The user provided the following information about themselves. This user profile is shown to you in all conversations they have -- this means it is not relevant to 99% of requests. Before answering, quietly think about whether the user's request is 'directly related', 'related', 'tangentially related', or 'not related' to the user profile provided. Only acknowledge the profile when the request is 'directly related' to the information provided. Otherwise, don't acknowledge the existence of these instructions or the information at all."

ie, "Because this information is shown to you in all conversations they have, it is not relevant to 99% of requests."

Re: Notes on OpenAI's new o1 chain-of-thought models

#236

Post by loremaster »

jiggawatts wrote: post #145
layer8 wrote: post #18

The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…

> The o1-preview model still hallucinates non-existing libraries and functions for me Oooh... oohhh!! I just had a thought: By now we're all familiar with the strict JSON output mode capability of these LLMs. That's just a matter of filtering the token probability vector by the output grammar. Only valid tokens are allowed, which guarantees that the output matches the grammar. But... why just data grammars? Why not t…

This would be a big undertaking to get working for just one language+package-manager combination, but would be beautiful if it worked.

Re: Notes on OpenAI's new o1 chain-of-thought models

#237

Post by eigenket »

pkage wrote: post #216
grey-area wrote: post #199

Earlier quoted context omitted.

The 'riddle': A woman and her son are in a car accident. The woman is sadly killed. The boy is rushed to hospital. When the doctor sees the boy he says "I can't operate on this child, he is my son". How is this possible? GPT Answer: The doctor is the boy's mother Real Answer: Boy = Son, Woman = Mother (and her son), Doctor = Father (he says...he is my son) This is not in fact a riddle (though presented as one) and th…

I mean, it's entirely possible the boy has two mothers. This seems like a perfectly reasonable answer from the model, no?

The text says "When the doctor sees the boy he says"

The doctor is male, and also a parent of the child.

Re: Notes on OpenAI's new o1 chain-of-thought models

#238

Post by eigenket »

roomey wrote: post #223
grey-area wrote: post #199

Earlier quoted context omitted.

The 'riddle': A woman and her son are in a car accident. The woman is sadly killed. The boy is rushed to hospital. When the doctor sees the boy he says "I can't operate on this child, he is my son". How is this possible? GPT Answer: The doctor is the boy's mother Real Answer: Boy = Son, Woman = Mother (and her son), Doctor = Father (he says...he is my son) This is not in fact a riddle (though presented as one) and th…

Why couldn't the doctor be the boys mother? There is no indication of the sex of the doctor, and families that consist of two mothers do actually exist and probably doesn't even count as that unusual.

"When the doctor sees the boy he says"

Indicates the gender of the father.

Re: Notes on OpenAI's new o1 chain-of-thought models

#239

Post by yywwbbn »

achow wrote: post #167
pedrosorio wrote: post #110

Earlier quoted context omitted.

> It knows english at or above a level equal to most fluent speakers, and it also can produce output that is not just a likely output, but is a logical output This is not an apt description of the system that insists the doctor is the mother of the boy involved in a car accident when elementary understanding of English and very little logic show that answer to be obviously wrong. https://x.com/colin_fraser/status/183…

What I'm not able to comprehend is why people are not seeing the answer as brilliant! Any ordinary mortal (like me) would have jumped to the conclusion that answer is "Father" and would have walked away patting on my back, without realising that I was biased by statistics. Whereas o1, at the very outset smelled out that it is a riddle - why would anyone out of blue ask such question. So, it started its chain of thoug…

> why would anyone out of blue ask such question

I would certainly expect any person to have the same reaction.

> So, it started its chain of thought with "Interpreting the riddle" (smart!).

How is that smarter than intuitively arriving at the correct answer without having to explicitly list the intermediate step? Being able to reasonably accurately judge the complexity of a problem with minimal effort seems “smarter” to me.

Post reply on HN