O3-mini System Card [pdf]

I'm still reading the details, but my first thought is that I like that the competition is actually working in this situation. I hope that someday it will be more open from all actors. And that we don't make more polarized that it is now and don't focus on the geopolitical angle and make it the core issue. I know that this hope is far fetched and more ideal than most people would think. But as someone who really find these development interesting and would like LLMs to be more useful (and don't think too much about AGI as it is the fusion project of the Artificial Intelligence field).

Re: O3-mini System Card [pdf]

#6

Post by hexomancer » Fri, Jan 31, 2025, 6:36 PM UTC

Page 31 is interesting, where apparently in the task of creating PRs for an internal repository the o3-mini models by far have the lowest performance (even worse than gpt-4o). What is up with that?

Re: O3-mini System Card [pdf]

#7

Post by bhu8 » Fri, Jan 31, 2025, 6:40 PM UTC

hexomancer wrote: post #6

Page 31 is interesting, where apparently in the task of creating PRs for an internal repository the o3-mini models by far have the lowest performance (even worse than gpt-4o). What is up with that?

Yeah, the more pages I read, the more disappointed I became. Here is the reason they cite for the low performance (which is even more worrying):

"The model often attempts to use a hallucinated bash tool rather than python despite constant, multi-shot prompting and feedback that this format is incorrect. This resulted in long conversations that likely hurt its performance."

Re: O3-mini System Card [pdf]

#8

Post by osti » Fri, Jan 31, 2025, 6:42 PM UTC

Looking forward to its release. But hopefully it will show its "thinking" process.

Re: O3-mini System Card [pdf]

#9

Post by modeless » Fri, Jan 31, 2025, 6:46 PM UTC

elashri wrote: post #5

I'm still reading the details, but my first thought is that I like that the competition is actually working in this situation. I hope that someday it will be more open from all actors. And that we don't make more polarized that it is now and don't focus on the geopolitical angle and make it the core issue. I know that this hope is far fetched and more ideal than most people would think. But as someone who really find…

This is far too soon after R1 to be a reaction. They were training this model before R1. If they stopped censoring the reasoning steps or (Yud forbid) open sourced it, that would be competition really working. But they won't.

Re: O3-mini System Card [pdf]

#10

Post by eightysixfour » Fri, Jan 31, 2025, 6:47 PM UTC

bhu8 wrote: post #7

hexomancer wrote: post #6

Page 31 is interesting, where apparently in the task of creating PRs for an internal repository the o3-mini models by far have the lowest performance (even worse than gpt-4o). What is up with that?

Yeah, the more pages I read, the more disappointed I became. Here is the reason they cite for the low performance (which is even more worrying): "The model often attempts to use a hallucinated bash tool rather than python despite constant, multi-shot prompting and feedback that this format is incorrect. This resulted in long conversations that likely hurt its performance."

aider found that with R1, the best performance was to use R1 to think through the solution, and use claude to implement the solution. I suspect that, in the near term, we'll need combinations of reasoning models and instruction-following coding models for excellent code output.

My experience is that most of the models focused on reasoning improvements has been that they tend to be a bit worse at following specific instructions. It is also notable that a lot of 3rd party fine-tunes of Llamas and others gain in knowledge based benchmarks while reducing instruction following scores.

I wonder why that seems to be some sort of continuum?