the censors need to know what they are censoring. Now if they are going to sell to the censors, presumably the censors will pay for seeing the full reasoning capability. hopefully the reasoning demonstrates the counterproductiveness of hiding the reasoning in the first place.
Notes on OpenAI's new o1 chain-of-thought models
21–30 of 659 posts
Re: Notes on OpenAI's new o1 chain-of-thought models
#22Earlier quoted context omitted.
> A properly written requirements spec, and an engineer, can do the work of 5. I do not think this will scale. GPT o1 is presumably good for bootstrapping a project using tools that the engineer is not familiar with. The model will struggle to update a sizable codebase, however, with dependencies between the files. Secondly, no matter the size of the codebase and no matter the model used, the engineer still has to re…
I respectfully, but completely disagree. Right now with sonnet 3.5 + cursor ide, I'm not writing that much of my own code at my FAANG job. I am generating a ton, passing in documentation from internal libraries, iterating on the result. Most of the time, I just accept its changes. This is going to rapidly happen. All we need are a few more model releases, not even a step function improvement
I use the same workflow. It’s taking a while for me to learn to sense when it’s getting off track and I need to start a new chat session in general it’s pretty amazing if given very clear guidance at the right moments.
Re: Notes on OpenAI's new o1 chain-of-thought models
#23The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…
I've had the opposite experience with some coding samples. After reading Nick Carlini's post, I've gotten into the habit of powering through coding problems with GPT (where previously I'd just laugh and immediately give up) by just presenting it the errors in its code and asking it to fix them. o1 seems to be effectively screening for some of those errors (I assume it's just some, but I've noticed that the o1 things…
My experience is likely colored by the fact that I tend to turn to LLMs for problems I have trouble solving by myself. I typically don't use them for the low-hanging fruits.
That's the frustrating thing. LLMs don't materially reduce the set of problems where I'm running against a wall or have trouble finding information.
Re: Notes on OpenAI's new o1 chain-of-thought models
#24> No system prompt support—the models use the existing chat completion API but you can only send user and assistant messages.
> No streaming support, tool usage, batch calls or image inputs either.
I think it's worth adding a note explaining that many of these limitations are due to the beta status of the API. max_tokens is the only parameter I've seen deprecated in the API docs.
From https://platform.openai.com/docs/guides/reasoning
> We will be adding support for some of these parameters in the coming weeks as we move out of beta. Features like multimodality and tool usage will be included in future models of the o1 series.
Re: Notes on OpenAI's new o1 chain-of-thought models
#25The o1-preview model still hallucinates non-existing libraries and functions for me, and is quickly wrong about facts that aren't well-represented on the web. It's the usual string of "You're absolutely correct, and I apologize for the oversight in my previous response. [Let me make another guess.]" While the reasoning may have been improved, this doesn't solve the problem of the model having no way to assess if what…
It begs the question of whether we can supply a function to be called (e.g., one that compiles and runs code) to evaluate intermediate CoT results
Re: Notes on OpenAI's new o1 chain-of-thought models
#26Earlier quoted context omitted.
I respectfully, but completely disagree. Right now with sonnet 3.5 + cursor ide, I'm not writing that much of my own code at my FAANG job. I am generating a ton, passing in documentation from internal libraries, iterating on the result. Most of the time, I just accept its changes. This is going to rapidly happen. All we need are a few more model releases, not even a step function improvement
Don't you still have to explain your requirement really well to it, in a lot of detail? In a terse language like Python, I might as well just write the code. In a verbose language like Java, perhaps there is more of a value in detailing the requirement.
Not really, most of the changes are straightforward. Also, alot of the time it writes better syntax than i would. Sometimes I write a bunch of psuedo code and hsave it fill in the detials, then write the tests
Re: Notes on OpenAI's new o1 chain-of-thought models
#27It kind of seems like they just wrote a generalized DSPy program. Can anyone confirm?
This has been a very incremental year for OpenAI. If this is what it seems like, then I’ve got to believe they’re stalling for time.
Re: Notes on OpenAI's new o1 chain-of-thought models
#28Earlier quoted context omitted.
Don't you still have to explain your requirement really well to it, in a lot of detail? In a terse language like Python, I might as well just write the code. In a verbose language like Java, perhaps there is more of a value in detailing the requirement.
Not really, most of the changes are straightforward. Also, alot of the time it writes better syntax than i would. Sometimes I write a bunch of psuedo code and hsave it fill in the detials, then write the tests
> Not really
How on earth are you conveying your intent to the model? Or is your intent so CRUDdy that it doesn't need to be conveyed?
Re: Notes on OpenAI's new o1 chain-of-thought models
#29Earlier quoted context omitted.
I respectfully, but completely disagree. Right now with sonnet 3.5 + cursor ide, I'm not writing that much of my own code at my FAANG job. I am generating a ton, passing in documentation from internal libraries, iterating on the result. Most of the time, I just accept its changes. This is going to rapidly happen. All we need are a few more model releases, not even a step function improvement
Don't you still have to explain your requirement really well to it, in a lot of detail? In a terse language like Python, I might as well just write the code. In a verbose language like Java, perhaps there is more of a value in detailing the requirement.
It depends on what you're doing.
If you're writing something specific to your particular problem, or thinking through how to structure your data, or even working on something tough to describe in words like UI design, it probably is easier to just code it yourself in most high-level languages. On the other hand, if you're just trying to get a framework or library to do something and you don't want to spend a bunch of time reading the docs to remember the special incantations needed to just make it do the thing you already know it can do, the AI speeds things up considerably.
Re: Notes on OpenAI's new o1 chain-of-thought models
#30The theory is that this solves the data shortage problem, they can generate a ton of chain of reasoning data from what we already have. True iterative improvement, like out of a science fiction novel These models are going to get embedded deeply into IDE's, like cursor has, and essentially end software development as we know it. A properly written requirements spec, and an engineer, can do the work of 5. Software eng…
a properly written requirements spec is something that doesn't exist in the vast majority of cases.