Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on OpenAI's new o1 chain-of-thought models

simonwillison.net

Post reply on HN
Print view View original

271–280 of 659 posts

Re: Notes on OpenAI's new o1 chain-of-thought models

#271

Post by grey-area »

roomey wrote: post #223
grey-area wrote: post #199

Earlier quoted context omitted.

The 'riddle': A woman and her son are in a car accident. The woman is sadly killed. The boy is rushed to hospital. When the doctor sees the boy he says "I can't operate on this child, he is my son". How is this possible? GPT Answer: The doctor is the boy's mother Real Answer: Boy = Son, Woman = Mother (and her son), Doctor = Father (he says...he is my son) This is not in fact a riddle (though presented as one) and th…

Why couldn't the doctor be the boys mother? There is no indication of the sex of the doctor, and families that consist of two mothers do actually exist and probably doesn't even count as that unusual.

he says

Re: Notes on OpenAI's new o1 chain-of-thought models

#272

Post by benterix »

spaceman_2020 wrote: post #162
layer8 wrote: post #18

The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…

I don’t really see this as a massive problem. Its code. If it doesn’t run, you ask it to reconsider, give some more info if necessary, and it usually gets it right. The system doesn’t become useless if it takes 2 tries instead of 1 to get it right Still saves an incredible amount of time vs doing it yourself

While I agree, I saw it abused in this way a lot, in the sense that the code did what it was supposed to do in a given scenario but was obviously flawed in various was so it was just sitting there waiting for a disaster.

Re: Notes on OpenAI's new o1 chain-of-thought models

#273

Post by [deleted] »

kqr wrote: post #131
shmatt wrote: post #51

Earlier quoted context omitted.

I honestly can’t believe this is the hyped up “strawberry” everyone was claiming is pretty much AGI. Senior employees leaving due to its powers being so extreme I’m in the “probabilistic token generators aren’t intelligence” camp so I don’t actually believe in AGI, but I’ll be honest the never ending rumors / chatter almost got to me Remember, this is the model some media outlet reported recently that is so powerful…

> probabilistic token generators aren’t intelligence Maybe this has been extensively discussed before, but since I've lived under a rock: which parts of intelligence do you think are not representable as conditional probability distributions?

[deleted]

Re: Notes on OpenAI's new o1 chain-of-thought models

#274

Post by sebzim4500 »

dkga wrote: post #143

Judging by this thread, surely a part of OpenAI’s business model is to release models with somewhat grey-area outlandish claims, then sit and wait for people to test it out paying top dollars for tokens.

I would guess most of the people tesing the model were already paying for plus so they aren't getting any additional revenue for these new tokens.

Re: Notes on OpenAI's new o1 chain-of-thought models

#275

Post by anentropic »

Are there any benchmarks which compare existing LLMs using langchain-style multi-step reasoning?

The new OpenAI model shows a big improvement on some benchmarks over GPT4 one-shot chain-of-thought, but what about vs systems doing something more similar to what presumably this is?

Re: Notes on OpenAI's new o1 chain-of-thought models

#276

Post by fragmede »

croes wrote: post #270
MPSimmons wrote: post #106

Earlier quoted context omitted.

The failure is in how you're using it. I don't mean this as a personal attack, but more to shed light on what's happening. A lot of people use LLMs as a search engine. It makes sense - it's basically a lossy compressed database of everything its ever read, and it generates output that is statistically likely - varying degrees of likeliness depending on the temperature, as well as how many times the particular weights…

Intelligent? Just ask ChatGPT How many Rs are in strawberry?

https://chatgpt.com/share/66e3f9e1-2cb4-8009-83ce-090068b163...

Keep up, that was last week's gotcha, with the old model.

Re: Notes on OpenAI's new o1 chain-of-thought models

#277

Post by sebzim4500 »

la64710 wrote: post #79

Please please please stop saying thought. This has nothing to do with the word thought. When we say the word thought it means something. Please don’t use the same word for whatever AI is doing and trivialize the word. Invent a new word if needed but for Pete’s sake be accurate and truthful.

No one gets this emotional about astrophysicists calling almost everything 'metal' and this is definitely less bad than that.

Re: Notes on OpenAI's new o1 chain-of-thought models

#278

Post by benterix »

postalcoder wrote: post #230
benterix wrote: post #218

From the article: > I expect to continue mostly using GPT-4o (and Claude 3.5 Sonnet) I saw similar comments elsewhere and I'm stunned - am I the only one who considers 4o a step back when compared to 4 for textual input and output? It basically gives fast semi-useful answers that seem like a slightly improved 3.5.

I wholly agree with you. I've been using every model extensively since early the Davincis and I strongly believe that gpt-4-0314 was the best model they've released to date. It's poor performance on benchmarks drives my skepticism of LLM benchmarking in general. I trust my feel for the models much more, and my feel was that 0314 was great. The one thing that 0314 doesn't do well are the tricks like structured output…

That's my concern - they marked 4 as "legacy" in the GUI, and now they hid it temporarily under a submenu - but it's the only model I care about. If they remove it, there is no reason for me to use their services, especially with Claude 3.5 wider context window and reasonably good results.

Re: Notes on OpenAI's new o1 chain-of-thought models

#280

Post by andrew_eu »

I thought with this chain-of-thought approach the model might be better suited to solve a logic puzzle, e.g. ZebraPuzzles [0]. It produced a ton of "reasoning" tokens but hallucinated more than half of the solution with names/fields that weren't available. Not a systematic evaluation, but it seems like a degradation from 4o-mini. Perhaps it does better with code reasoning problems though -- these logic puzzles are essentially contrived to require deductive reasoning.

[0] https://zebrapuzzles.com

Post reply on HN