Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on OpenAI's new o1 chain-of-thought models

simonwillison.net

Post reply on HN
Print view View original

431–440 of 659 posts

Re: Notes on OpenAI's new o1 chain-of-thought models

#431

Post by accountnum »

geysersam wrote: post #414
accountnum wrote: post #333

Earlier quoted context omitted.

It hasn't read that riddle because it is a modified version. The model would in fact solve this trivially if it _didn't_ see the original in its training. That's the entire trick.

Sure but the parent was praising the model for recognizing that it was a riddle in the first place: > Whereas o1, at the very outset smelled out that it is a riddle That doesn't seem very impressive since it's (an adaptation of) a famous riddle The fact that it also gets it wrong after reasoning about it for a long time doesn't make it better of course

Recognizing that it is a riddle isn't impressive, true. But the duration of its reasoning is irrelevant, since the riddle works on misdirection. As I keep saying here, give someone uninitiated the 7 wives with 7 bags going (or not) to St Ives riddle and you'll see them reasoning for quite some time before they give you a wrong answer.

If you are tricked about the nature of the problem at the outset, then all reasoning does is drive you further in the wrong direction, making you solve the wrong problem.

Re: Notes on OpenAI's new o1 chain-of-thought models

#432

Post by noisy_boy »

williamdclt wrote: post #164
MPSimmons wrote: post #106

Earlier quoted context omitted.

The failure is in how you're using it. I don't mean this as a personal attack, but more to shed light on what's happening. A lot of people use LLMs as a search engine. It makes sense - it's basically a lossy compressed database of everything its ever read, and it generates output that is statistically likely - varying degrees of likeliness depending on the temperature, as well as how many times the particular weights…

> Treat it as a naive but intelligent intern That’s the problem: it’s a _terrible_ intern. A good intern will ask clarifying questions, tell me “I don’t know” or “I’m not sure I did it right”. LLMs do none of that, they will take whatever you ask and give a reasonable-sounding output that might be anything between brilliant and nonsense. With an intern, I don’t need to measure how good my prompting is, we’ll usually…

I feel like it almost always starts well, given the full picture, but then for non-trivial stuff, gets stuck towards the end. The longer the conversation goes, the more wheel-spinning occurs and before you know it, you have spent an hour chasing that last-mile-connectivity.

For complex questions, I now only use it to get the broad picture and once the output is good enough to be a foundation, I build the rest of it myself. I have noticed that the net time spent using this approach still yields big savings over a) doing it all myself or b) keep pushing it to do the entire thing. I guess 80/20 etc.

Re: Notes on OpenAI's new o1 chain-of-thought models

#433

Post by falcor84 »

AbstractH24 wrote: post #370
falcor84 wrote: post #334

Earlier quoted context omitted.

I wonder what would our world look like if these two expectations that you seem to be taking for granted were applied to our politicians.

Are you suggesting people are satisfied with our politicians and aspire for other things to be just as good as them? What if we applied those two expectations to building construction? What if we didn’t?

I think it's always good to aspire for more, but we shouldn't be expecting perfect results in novel areas of technology.

Taking up your construction metaphor, LLMs are now where construction was perhaps 3000 years ago; buildings weren't that sturdy, but even if the roofs leaked a bit, I'm sure it beat sleeping outside on a rainy night. We need to continue iterating.

Re: Notes on OpenAI's new o1 chain-of-thought models

#434

Post by roywiggins »

lucubratory wrote: post #182
hmottestad wrote: post #161

Earlier quoted context omitted.

Reminds me of a trick question about Schrödinger's cat. “I’ve put a dead cat in a box with a poison and an isotope that will trigger the poison at a random point in time. Right now, is the cat dead or alive?” The answer is that the cat is dead, because it was dead to begin with. Understanding this doesn’t mean that you are good at deductive reasoning. It just means that I didn’t manage to trick you. Same goes for an…

Yeah, I think what a lot of people miss about these sort of gotchas are that most of them were invented explicitly to gotcha humans, who regularly get got by them. This is not a failure mode unique to LLMs.

One that trips up LLMs in ways that wouldn't trip up humans is the chicken, fox and grain puzzle but with just the chicken. They tend to insist that the chicken be taken across the river, then back, then across again, for no reason other than the solution to the classic puzzle requires several crossings. No human would do that, by the time you've had the chicken across then even the most unobservant human would realize this isn't really a puzzle and would stop. When you ask it to justify each step you get increasingly incoherent answers.

Has anyone tried this on o1?

Re: Notes on OpenAI's new o1 chain-of-thought models

#435

Post by ChicagoDave »

SubiculumCode wrote: post #214
ChicagoDave wrote: post #201

Earlier quoted context omitted.

You can point a tool at a solution and certainly get results. Doesn’t mean it’s intelligent.

At what point are we better described as tools?

Humans can be a lot of things.

AI can only do what it knows and what it’s been programmed to do.

Re: Notes on OpenAI's new o1 chain-of-thought models

#436

Post by roywiggins »

anon291 wrote: post #111
pedrosorio wrote: post #110

Earlier quoted context omitted.

> It knows english at or above a level equal to most fluent speakers, and it also can produce output that is not just a likely output, but is a logical output This is not an apt description of the system that insists the doctor is the mother of the boy involved in a car accident when elementary understanding of English and very little logic show that answer to be obviously wrong. https://x.com/colin_fraser/status/183…

Keep in mind that the system always chooses randomly so there is always a possibility it commits to the wrong output. I don't know why openAi won't allow determinism but it doesn't, even with temperature set to zero

Determinism only helps if you always ask the question with exactly the same words. There's no guarantee a slightly rephrased version will give the same answer, so a certain amount of unpredictability is unavoidable anyway. With a deterministic LLM you might find one phrasing that always gets it right and a dozen basically indistinguishable ones that always get it wrong.

Re: Notes on OpenAI's new o1 chain-of-thought models

#437

Post by iinnPP »

ksplicer wrote: post #425
bambax wrote: post #281

Near the end, the quote from OpenAI researcher Jason Wei seems damning to me: > Results on AIME and GPQA are really strong, but that doesn’t necessarily translate to something that a user can feel. Even as someone working in science, it’s not easy to find the slice of prompts where GPT-4o fails, o1 does well, and I can grade the answer. But when you do find such prompts, o1 feels totally magical. We all need to find…

This is something we've been grappeling with on my team. Many of the researchers in the org want to try all these reasoning techniques to increase performance, and my team keeps pushing back that we don't actually need that extra performance- we just want to decrease latency and cost.

So make the requirement using a cheaper and lower latency model and try to increase the performance to a satisfactory level. Assuming that you are not already using the cheapest/lowest latency model.

Re: Notes on OpenAI's new o1 chain-of-thought models

#438

Post by arb_ »

flanked-evergl wrote: post #321
MPSimmons wrote: post #106

Earlier quoted context omitted.

The failure is in how you're using it. I don't mean this as a personal attack, but more to shed light on what's happening. A lot of people use LLMs as a search engine. It makes sense - it's basically a lossy compressed database of everything its ever read, and it generates output that is statistically likely - varying degrees of likeliness depending on the temperature, as well as how many times the particular weights…

> The failure is in how you're using it. People, for the most part, know what they know and don't know. I am not uncertain that the distance between the earth and the sun varies, but I'm certain that I don't know the distance from the earth to the sun, at least not with better precision than about a light week. This is going to have to be fixed somehow to progress past where we are now with LLMs. Maybe expecting an L…

Empirically, they have reduced hallucinations. Where do OpenAI / Anthropic claim that their models won't hallucinate?

Re: Notes on OpenAI's new o1 chain-of-thought models

#439

Post by HarHarVeryFunny »

layer8 wrote: post #18

The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…

To the extent we've now got the output of the underlying model wrapped in an agent that can evaluate that output, I'd expect it to be able to detect it's own hallucinations some of the time and therefore provide an alternate answer.

It's like when an LLM gives you a wrong answer and all it takes is "are you sure?" to get it to generate a different answer.

Of course the underlying problem of the model not knowing what it knows or doesn't know persists, so giving it the ability to reflect on what it just blurted out isn't always going to help. It seems the next step is for them to integrate RAG and tool use into this agentic wrapper, which may help in some cases.

Re: Notes on OpenAI's new o1 chain-of-thought models

#440

Post by arb_ »

aubanel wrote: post #296
bambax wrote: post #281

Near the end, the quote from OpenAI researcher Jason Wei seems damning to me: > Results on AIME and GPQA are really strong, but that doesn’t necessarily translate to something that a user can feel. Even as someone working in science, it’s not easy to find the slice of prompts where GPT-4o fails, o1 does well, and I can grade the answer. But when you do find such prompts, o1 feels totally magical. We all need to find…

He's speaking about his objective to make ever stronger LLMs: so for this his secondary objective is to measure their real performance. The human preference is not that good of a proxy measurement: for instance, it can be gamed by making the model more assertive, causing the human error-spotting ability to decrease a lot [0]. So what he's really saying is that non-rigorous human vibe checks (like those LMSys Chatbot…

As models improve, human preference will become worse as a proxy measurement (e.g. as model capabilities surpass the human's ability to judge correctness at a glance). This can be due to more raw capability - or more persuasion / charisma.
Post reply on HN