5 hours in 500 odd comments. Definitely feels like this has less wow factor than previous OAI releases.
OpenAI O3-Mini
561–570 of 944 posts
Re: OpenAI O3-Mini
#562This is.... underwhelming.
Re: OpenAI O3-Mini
#563Well, o3-mini-high just successfully found the root cause of a seg fault that o1 missed: mistakenly using _mm512_store_si512 for an unaligned store that should have been _mm512_storeu_si512.
why is this impressive at all? It effectively amounts to correcting a typo.
Re: OpenAI O3-Mini
#564Earlier quoted context omitted.
Yeah. I immediately thought: I wonder if that 56% is in one or two categories and the rest are worse?
44% of the people prefers the existing model ?
Each question falls into a different category (ie math, coding, story writing etc). Typically models are better at some categories and worse at others. Saying "56% of people preferred responses from o3-mini" makes me wonder if those 56 are only from certain categories and the model isn't uniformly 56% preferred.
Re: OpenAI O3-Mini
#565I've been using cursor since it launched, sticking almost exclusively to claude-3.5-sonnet because it is incredibly consistent, and rarely loses the plot. As subsequent models have been released, most of which claim to be better at coding, I've switched cursor to it to give them a try. o1, o1-pro, deepseek-r1, and the now o3-mini. All of these models suffer from the exact same "adhd." As an example, in a NextJS app,…
My experience with cursor and sonnet is that it is relatively good at first tries, but completely misses the plot during corrections. "My attempt at solving the problem contains a test that fails? No problem, let me mock the function I'm testing, so that, rather than actually run, it returns the expected value!" It keeps doing that kind of shenanigans, applying modifications that solve the newly appearing problem whi…
Can't you select Chatgpt as the model in cursor?
Re: OpenAI O3-Mini
#566Earlier quoted context omitted.
My experience with cursor and sonnet is that it is relatively good at first tries, but completely misses the plot during corrections. "My attempt at solving the problem contains a test that fails? No problem, let me mock the function I'm testing, so that, rather than actually run, it returns the expected value!" It keeps doing that kind of shenanigans, applying modifications that solve the newly appearing problem whi…
Can't you select Chatgpt as the model in cursor?
Yes, but for some reason it seems to perform worse there.
Perhaps whatever algorithms Cursor uses to prepare the context it feeds the model are a good fit for Claude but not so much for the others (?). It's a random guess, but whatever the reason, there's a weird worsening of performance vs pure chat.
Re: OpenAI O3-Mini
#567O3-mini solved this prompt. DeepSeek R1 had a mental breakdown. The prompt: “Bob is facing forward. To his left is Ann, to his right is Cathy. Ann and Cathy are facing backwards. Who is on Ann’s left?”
That's a fun, simple test! I tried a few models, and mistral-nemo gets it every time, even when run locally without any system prompt! https://build.nvidia.com/nv-mistralai/mistral-nemo-12b-instr...
Re: OpenAI O3-Mini
#568Earlier quoted context omitted.
I normally use aider by just typing in what I want and it magically does it. How do I use o1 or R1 to play the role of the "architect"?
You can start it with something like: aider --architect --model o1 --editor-model sonnet Then you'll be in "architect" mode, which first prompts o1 to design the solution, then you can accept it and allow sonnet to actually create the diffs. Most of the time your way works well—I use sonnet alone 90% of the time, but the architect mode is really great at getting it unstuck when it can't seem to implement what I want…
I really want to see how apps created this way scale to large codebases. I’m very skeptical they don’t turn into spaghetti messes.
Coding is basically just about the most precise way to encapsulate a problem as a solution possible. Taking a loose English description and expanding it into piles of code is always going to be pretty leaky no matter how much these models spit out working code.
In my experience you have to pay a lot of attention to every single line these things write because they’ll often change stuff or more often make wrong assumptions that you didn’t articulate. And in my experience they never ask you questions unless you specifically prompt them to (and keep reminding them to), which means they are doing a hell of a lot of design and implementation that unless carefully looked over will ultimately be wrong.
It really reminds me a bit of when Ruby on Rails came out and the blogosphere was full of gushing “I’ve never been more productive in my life” posts. And then you find out they were basically writing a TODO app and their previous development experience was doing enterprise Java for some massive non-tech company. Of course RoR will be a breath of fresh air for those people.
Don’t get me wrong I use cursor as my daily driver but I am starting to find the limits for what these things can do. And the idea of having two of these LLM’s taking some paragraph long feature description and somehow chatting with each other to create a scalable bit of code that fits into a large or growing codebase… well I find that kind of impossible. Sure the code compiles and conforms to whatever best practices are out there but there will be absolutely no constancy across the app—especially at the UX level. These things simply cannot hold that kind of complexity in their head and even if they could part of a developers job is to translate loose English into code. And there is much, much, much, much more to that than simply writing code.
Re: OpenAI O3-Mini
#569> Testers preferred o3-mini's responses to o1-mini 56% of the time I hope by this they don't mean me, when I'm asked 'which of these two responses do you prefer'. They're both 2,000 words, and I asked a question because I have something to do. I'm not reading them both ; I'm usually just selecting the one that answered first. That prompt is pointless. Perhaps as evidenced by the essentially 50% response rate: it's a…
It seems like the the first response must get chosen a majority of the time just to account for friction
Re: OpenAI O3-Mini
#570Earlier quoted context omitted.
The -mini postfix makes perfect sense, probably even clearer than the old "turbo" wording. Naturally, the latest small model may be better than larger older models... but not always and not necessarily in everything. What you'd expect from a -mini model is exactly what is delivered. The non-reasoning line was also pretty straightforward. Newer base models get a larger prefix number and some postfixes like 'o' were ad…
> I wonder if we'll end up with both a 4o and o4... The perplexing thing is that someone has to have said that, right? It has to have been brought up in some meeting when they were brainstorming names that if you have 4o and o1 with the intention of incrementing o1 you'll eventually end up with an o4. Where they really went off the rails was not just bailing when they realized they couldn't use o2. In that moment the…
Why not let ChatGPT decide the naming? Surely it will be replacing humans at this task any day now?