[dead]
Notes on OpenAI's new o1 chain-of-thought models
211–220 of 659 posts
Re: Notes on OpenAI's new o1 chain-of-thought models
#212The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…
Yes, this only helps multi-step reasoning. The model still has problems with general knowledge and deep facts. There's no way you can "reason" a correct answer to "list the tracklisting of some obscure 1991 demo by a band not on Wikipedia." You either know or you don't. I usually test new models with questions like "what are the levels in [semi-famous PC game from the 90s]?" The release version of GPT-4 could get abo…
o1-mini is a small model (knows a lot less about the world) and is tuned for reasoning through symbolic problems (maths, programming, chemistry etc.).
You're using a calculator as a search engine.
Re: Notes on OpenAI's new o1 chain-of-thought models
#213Earlier quoted context omitted.
Really it just does what you tell it to. Have you tried telling it “ask me clarifying questions about all the APIs you need to solve this problem”? Huge contrast to human interns who aren’t experienced or smart enough to ask the right questions in the first place, and/or have sentimental reasons for not doing so.
I have to say, having to tell it to ask me clarifying questions DOES make it really look smart!
imagine if you make it keep going without having to reprompt it
Re: Notes on OpenAI's new o1 chain-of-thought models
#214Earlier quoted context omitted.
> We’re no closer to “AI” today than we were 20 years ago. 20 years ago we had barely figured out how to create superhuman agents to play chess. We have now created a new algorithm to solve Go, which is a much harder game. We then created an algorithm (alpha zero) to teach itself to play any game, and which became the best chess player in the world in hours. We next created a superhuman poker agent. Poker is even mor…
You can point a tool at a solution and certainly get results. Doesn’t mean it’s intelligent.
At what point are we better described as tools?
Re: Notes on OpenAI's new o1 chain-of-thought models
#215Earlier quoted context omitted.
Keep in mind that the system always chooses randomly so there is always a possibility it commits to the wrong output. I don't know why openAi won't allow determinism but it doesn't, even with temperature set to zero
Would picking deterministically help through? Then in some cases it’s always 100% wrong
Yes, it is better if for example using it via an API to classify. Deterministic behavior makes it a lot easier to debug the prompt.
Re: Notes on OpenAI's new o1 chain-of-thought models
#216Earlier quoted context omitted.
What I'm not able to comprehend is why people are not seeing the answer as brilliant! Any ordinary mortal (like me) would have jumped to the conclusion that answer is "Father" and would have walked away patting on my back, without realising that I was biased by statistics. Whereas o1, at the very outset smelled out that it is a riddle - why would anyone out of blue ask such question. So, it started its chain of thoug…
The 'riddle': A woman and her son are in a car accident. The woman is sadly killed. The boy is rushed to hospital. When the doctor sees the boy he says "I can't operate on this child, he is my son". How is this possible? GPT Answer: The doctor is the boy's mother Real Answer: Boy = Son, Woman = Mother (and her son), Doctor = Father (he says...he is my son) This is not in fact a riddle (though presented as one) and th…
I mean, it's entirely possible the boy has two mothers. This seems like a perfectly reasonable answer from the model, no?
Re: Notes on OpenAI's new o1 chain-of-thought models
#217the censors need to know what they are censoring. Now if they are going to sell to the censors, presumably the censors will pay for seeing the full reasoning capability. hopefully the reasoning demonstrates the counterproductiveness of hiding the reasoning in the first place.
Yes, it's a sad world where authoritarianism will be supported and enforced by sophisticated technical solutions for mass surveillance and mass censorship.
Re: Notes on OpenAI's new o1 chain-of-thought models
#218From the article:
> I expect to continue mostly using GPT-4o (and Claude 3.5 Sonnet)
I saw similar comments elsewhere and I'm stunned - am I the only one who considers 4o a step back when compared to 4 for textual input and output? It basically gives fast semi-useful answers that seem like a slightly improved 3.5.
Re: Notes on OpenAI's new o1 chain-of-thought models
#219The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…
o1-preview != o1. In public coding AI comparison tests, results showed 4o scoring around 35%, o1-preview scoring ~50% and o1 scoring ~85%. o1 is not yet released, but has been run through many comparison tests with public results posted.
Don't forget about o1-mini. It seems better than o1-preview for problems that fit it (don't require so much real world knowledge).
Re: Notes on OpenAI's new o1 chain-of-thought models
#220Earlier quoted context omitted.
And how much data can you give it? I'm not up to date with these things because I haven't found them useful. But with what you said, and previous limitations in how much data they can retain essentially makes them pretty darn useless for that task. Great learning tool on common subjects you don't know, such as learning a new programming-language. Also great for inspiration etc. But that's pretty much it? Don't get me…
>And how much data can you give it? Well, theoretically you can give it up to the context size minus 4k tokens, because the maximum it can output is 4k. In practice, though, its ability to effectively recall information in the prompt drops off. Some people have studied this a bit - here's one such person: https://gritdaily.com/impact-prompt-length-llm-performance/
You should be able to provide more data than that in the input if the output doesn't use the full 4k tokens. So limit is context_size minus expected length of output.