Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o3-pro

help.openai.com

Post reply on HN
Print view View original

151–160 of 209 posts

Re: OpenAI o3-pro

#151

Post by avereveard »

DanMcInerney wrote: post #31

I'm really hoping GPT5 is a larger jump in metrics than the last several releases we've seen like Claude3.5 - Claude4 or o3-mini-high to o3-pro. Although I will preface that with the fact I've been building agents for about a year now and despite the benchmarks only showing slight improvement, I have seen that each new generation feels actively better at exactly the same tasks I gave the previous generation. It would…

There's a new set of metrics that capture advances better than MMLU or it's pro version but nothing yet as standardized and specifically very few have a hidden set of tests to keep advancements from been from directional fine tuning.

Re: OpenAI o3-pro

#152

Post by torginus »

simonw wrote: post #36
chad1n wrote: post #28

The guys in the other thread who said that OpenAI might have quantized o3 and that's how they reduced the price might be right. This o3-pro might be the actual o3-preview from the beginning and the o3 might be just a quantized version. I wish someone benchmarks all of these models to check for drops in quality.

That's definitely not the case here. The new o3-pro is slow - it took two minutes just to draw me an SVG of a pelican riding a bicycle. o3-preview was much faster than that. https://simonwillison.net/2025/Jun/10/o3-pro/

This made me think of the 'draw a bike experiment', where people were asked to draw a bike from memory, and were suprisingly bad at recreating how the parts fit together in a sensible manner:

https://road.cc/content/blog/90885-science-cycology-can-you-...

ChatGPT seems to perform better than most, but with notable missing elements (where's the chain or the handlebars?). I'm not sure if those are due to a lack of understanding, or artistic liberties taken by the model?

Re: OpenAI o3-pro

#153

Post by iLoveOncall »

eru wrote: post #148
iLoveOncall wrote: post #138

Earlier quoted context omitted.

You can write projects with LLMs thanks to tools that can analyze your local project's context, which didn't exist a year ago. You could use Cursor, Windsurf, Q CLI, Claude Code, whatever else with Claude 3 or even an older model and you'd still get usable results. It's not the models which have enabled "vibe coding", it's the tools. An additional proof of that is that the new models focus more and more on coding in…

You need a certain quality of model to make 'vibe coding' work. For example, I think even with the best tooling in the world, you'd be hard pressed to make GPT 2 useful for vibe coding.

I'm not claiming otherwise. I'm just saying that people say "look what we can do with the new models" when they're completely ignoring the fact that the tooling has improved a hundred fold (or rather, there was no tooling at all and now there is).

Re: OpenAI o3-pro

#154

Post by eru »

iLoveOncall wrote: post #153
eru wrote: post #148

Earlier quoted context omitted.

You need a certain quality of model to make 'vibe coding' work. For example, I think even with the best tooling in the world, you'd be hard pressed to make GPT 2 useful for vibe coding.

I'm not claiming otherwise. I'm just saying that people say "look what we can do with the new models" when they're completely ignoring the fact that the tooling has improved a hundred fold (or rather, there was no tooling at all and now there is).

OK, no objections from me there.

Re: OpenAI o3-pro

#155

Post by saberience »

nipah wrote: post #102
saberience wrote: post #66

Earlier quoted context omitted.

I’m not sure the arcagi are interesting benchmarks, for one they are image based and for two most people I show them too have issues understanding them, and in fact I had issues understanding them. Given the models don’t even see the versions we get to see it doesn’t surprise me they have issues we these. It’s not hard to make benchmarks that are so hard that humans and Lims can’t do.

"most people I show them too have issues understanding them, and in fact I had issues understanding them" ??? those benchmarks are so extremely simple they have basically 100% human approval rates, unless you are saying "I could not grasp it immediately but later I was able to after understanding the point" I think you and your friends should see a neurologist. And I'm not mocking you, I mean seriously, those are tas…

lol 100% approval rates? No they don’t.

Also mammals? What mammals could even understand we were giving it a test?

Have you seen them or shown them to average people? I’m sure the people who write them understand them but if you show these problems to average people in the street they are completely clueless.

This is a classic case of some phd ai guys making a benchmark and not really considering what average people are capable of.

Look, these insanely capable ai systems can’t do these problems but the boys in the lab can do them, what a good benchmark.

Re: OpenAI o3-pro

#156

Post by varjag »

eru wrote: post #147

I'm trying out o3-pro now with some algorithmic questions. It seems to be doing alright, but it's taking an awfully long time (as expected) and the UIs seem to time out a lot, especially the Android app and the MacOS desktop app. The web interface seems the least flaky, but that's not saying much.

I had my own programming question for a while, which all models from all vendors been robustly failing so far. It is a known problem with surprisingly few published implementations as it had never been a part of leetcode, Euler or typical homework assignments. Yesterday o3-pro cleared it, using a more obscure algorithm I never even heard of.

Re: OpenAI o3-pro

#157

Post by varjag »

petesergeant wrote: post #113
manmal wrote: post #3

The benchmarks don’t look _that_ much better than o3. Does that mean Pro models are just incrementally better than base models, or are we approaching the higher end of a sigmoid function, with performance gains leveling off?

I am starting to feel like hallucination is a fundamentally unsolvable problem with the current architecture, and is going to keep squeezing the benchmarks until something changes. At this point I don't need smarter general models for my work, I need models that don't hallucinate, that are faster/cheaper, and that have better taste in specific domains. I think that's where we're going to see improvements moving forwa…

Hallucination rate from o3 onward appear to be very low, to the point I rarely have to check.

Re: OpenAI o3-pro

#158

Post by saberience »

HDThoreaun wrote: post #97
saberience wrote: post #66

Earlier quoted context omitted.

I’m not sure the arcagi are interesting benchmarks, for one they are image based and for two most people I show them too have issues understanding them, and in fact I had issues understanding them. Given the models don’t even see the versions we get to see it doesn’t surprise me they have issues we these. It’s not hard to make benchmarks that are so hard that humans and Lims can’t do.

arc agi is the closest any widely used benchmark is coming to an IQ test, its straight logic/reasoning. Looking at the problem set its hard for me to choose a better benchmark for "when this is better than humans we have agi"

There are humans who cannot do arc agi though so how does an LLM not doing it mean that LLMs don’t have general intelligence?

LLMs have obviously reached the point where they are smarter than almost every person alive, better at maths, physics, biology, English, foreign languages, etc.

But because they can’t solve this honestly weird visual/spatial reasoning test they aren’t intelligent?

That must mean most humans on this planet aren’t generally intelligent too.

Re: OpenAI o3-pro

#159

Post by viraptor »

nipah wrote: post #102
saberience wrote: post #66

Earlier quoted context omitted.

I’m not sure the arcagi are interesting benchmarks, for one they are image based and for two most people I show them too have issues understanding them, and in fact I had issues understanding them. Given the models don’t even see the versions we get to see it doesn’t surprise me they have issues we these. It’s not hard to make benchmarks that are so hard that humans and Lims can’t do.

"most people I show them too have issues understanding them, and in fact I had issues understanding them" ??? those benchmarks are so extremely simple they have basically 100% human approval rates, unless you are saying "I could not grasp it immediately but later I was able to after understanding the point" I think you and your friends should see a neurologist. And I'm not mocking you, I mean seriously, those are tas…

> so extremely simple they have basically 100% human approval rates

Are you thinking of a different set? Arc-agi-2 has average 60% success for a single person and questions require only 2 out of 9 correct answers to be accepted. https://docs.google.com/presentation/d/1hQrGh5YI6MK3PalQYSQs...

> and even some other mammals to do.

No, that's not the case.

Re: OpenAI o3-pro

#160

Post by yorwba »

achierius wrote: post #137
clbrmbr wrote: post #128

Earlier quoted context omitted.

You may be above average intelligence. Those challenges are like classic IQ tests and I bet have a significant distribution among humans.

No, they've done testing against samples from the general population.

The ARC-AGI-2 paper https://arxiv.org/pdf/2505.11831#figure.4 uses a non-representative sample, success rate differs widely across participants and "final ARC-AGI-2 test pairs were solved, on average, by 75% of people who attempted them. The average test-taker solved 66% of tasks they attempted. 100% of ARC-AGI-2 tasks were solved by at least two people (many were solved by more) in two attempts or less."

Certainly those non-representative humans are much better than current models, but they're also far from scoring 100%.

Post reply on HN