Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Notes on OpenAI's new o1 chain-of-thought models

simonwillison.net

Post reply on HN
Print view View original

31–40 of 659 posts

Re: Notes on OpenAI's new o1 chain-of-thought models

#31

Post by vrighter »

bamboozled wrote: post #9
ldjkfkdsjnv wrote: post #5

The theory is that this solves the data shortage problem, they can generate a ton of chain of reasoning data from what we already have. True iterative improvement, like out of a science fiction novel These models are going to get embedded deeply into IDE's, like cursor has, and essentially end software development as we know it. A properly written requirements spec, and an engineer, can do the work of 5. Software eng…

Which will mean...there is going to be a lot more software?

a lot more broken software. Companies release broken software intentionally just to be quick to market. Now can you imagine the same, but the "engineers" literally cannot make the product better even if they wanted to. They never learned to code properly. So they can't tell whether the code is good.

Re: Notes on OpenAI's new o1 chain-of-thought models

#32

Post by knicholes »

ldjkfkdsjnv wrote: post #5

The theory is that this solves the data shortage problem, they can generate a ton of chain of reasoning data from what we already have. True iterative improvement, like out of a science fiction novel These models are going to get embedded deeply into IDE's, like cursor has, and essentially end software development as we know it. A properly written requirements spec, and an engineer, can do the work of 5. Software eng…

I dunno man. I just spent a couple hours trying to get it to write functioning code to read from my RTSP stream, detect if my kid is playing piano, and send the result to HomeAssistant. It did not succeed.

Re: Notes on OpenAI's new o1 chain-of-thought models

#34

Post by [deleted] »

jumploops wrote: post #3

> the idea that I can run a complex prompt and have key details of how that prompt was evaluated hidden from me feels like a big step backwards. As a developer, this is highly concerning, as it makes it much harder to debug where/how the “reasoning” went wrong. The pricing is also silly, because I’m paying for tokens I can’t see. As a user, I don’t really care. LLMs are already magic boxes and I usually only care abo…

[deleted]

Re: Notes on OpenAI's new o1 chain-of-thought models

#35

Post by 7thpower »

ldjkfkdsjnv wrote: post #5

The theory is that this solves the data shortage problem, they can generate a ton of chain of reasoning data from what we already have. True iterative improvement, like out of a science fiction novel These models are going to get embedded deeply into IDE's, like cursor has, and essentially end software development as we know it. A properly written requirements spec, and an engineer, can do the work of 5. Software eng…

I think SaaS startups with a Harvard founder and 5 million are going to crush it in the world you describe. The marginal cost of building decreases, but brands, trust, and reach do not follow the same scaling laws.

Access to capital and pedigree are still going to be a big plus.

Re: Notes on OpenAI's new o1 chain-of-thought models

#36

Post by colordrops »

mrtesthah wrote: post #25
layer8 wrote: post #18

The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…

It begs the question of whether we can supply a function to be called (e.g., one that compiles and runs code) to evaluate intermediate CoT results

The answer is yes if you are willing to code it. OpenAI supports tool calls. Even if it didn't you could just make multiple calls to their API and submit the result of the code execution yourself.

Re: Notes on OpenAI's new o1 chain-of-thought models

#37

Post by OutOfHere »

jumploops wrote: post #3

> the idea that I can run a complex prompt and have key details of how that prompt was evaluated hidden from me feels like a big step backwards. As a developer, this is highly concerning, as it makes it much harder to debug where/how the “reasoning” went wrong. The pricing is also silly, because I’m paying for tokens I can’t see. As a user, I don’t really care. LLMs are already magic boxes and I usually only care abo…

> As a user, I don’t really care.

Tell me: Just how is it fair for a user to pay for the reasoning tokens without actually seeing them? If they are not shared, the service can bill you anything they want for them!

Re: Notes on OpenAI's new o1 chain-of-thought models

#38

Post by mensetmanusman »

layer8 wrote: post #23
tptacek wrote: post #20

Earlier quoted context omitted.

I've had the opposite experience with some coding samples. After reading Nick Carlini's post, I've gotten into the habit of powering through coding problems with GPT (where previously I'd just laugh and immediately give up) by just presenting it the errors in its code and asking it to fix them. o1 seems to be effectively screening for some of those errors (I assume it's just some, but I've noticed that the o1 things…

My experience is likely colored by the fact that I tend to turn to LLMs for problems I have trouble solving by myself. I typically don't use them for the low-hanging fruits. That's the frustrating thing. LLMs don't materially reduce the set of problems where I'm running against a wall or have trouble finding information.

LLMs are not for expanding the sphere of human knowledge, but for speeding up auto-correct of higher order processing to help you more quickly reach the shell of the sphere and make progress with your own mind :)

Re: Notes on OpenAI's new o1 chain-of-thought models

#39

Post by cma »

colordrops wrote: post #36
mrtesthah wrote: post #25

Earlier quoted context omitted.

It begs the question of whether we can supply a function to be called (e.g., one that compiles and runs code) to evaluate intermediate CoT results

The answer is yes if you are willing to code it. OpenAI supports tool calls. Even if it didn't you could just make multiple calls to their API and submit the result of the code execution yourself.

The intermediate CoT results aren't in the API.

Re: Notes on OpenAI's new o1 chain-of-thought models

#40

Post by CSSer »

w4 wrote: post #29
OutOfHere wrote: post #19

Earlier quoted context omitted.

Don't you still have to explain your requirement really well to it, in a lot of detail? In a terse language like Python, I might as well just write the code. In a verbose language like Java, perhaps there is more of a value in detailing the requirement.

It depends on what you're doing. If you're writing something specific to your particular problem, or thinking through how to structure your data, or even working on something tough to describe in words like UI design, it probably is easier to just code it yourself in most high-level languages. On the other hand, if you're just trying to get a framework or library to do something and you don't want to spend a bunch of…

An abstraction machete. Heh.
Post reply on HN