Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o3-pro

help.openai.com

Post reply on HN
Print view View original

121–130 of 209 posts

Re: OpenAI o3-pro

#121

Post by lhl »

manmal wrote: post #99
lhl wrote: post #94

Earlier quoted context omitted.

I've been using o3 extensively since release (and a lot of Deep Research). I also use a lot of Claude and Gemini 2.5 Pro (most of the times, for code I'll let all of them go at it and iterate on my fav results). So far I've only used o3-pro a bit today, and it's a bit too heavy to use interactively (fire it off, revisit in 10-15 minutes), but it seems to generate much cleaner/more well organized code and answers. I f…

Thanks for your input, very appreciated. Just in case you didn’t mean Claude Code, it’s really good in my experience and mostly stable. If something fails, it just retries and I don’t notice it much. Its autonomous discovery and tool use is really good and I‘m relying more and more on it.

For the Claude issues, I'm referring to the claude.ai frontend. While I use some Codex, Aider, and other agentic tools, I found Claude Code to be not to my taste - for my uses it tended burn a lot of tokens and gave relatively mediocre results, but I know it works well for others, so YMMV.

Re: OpenAI o3-pro

#122

Post by lhl »

petesergeant wrote: post #114
lhl wrote: post #94

Earlier quoted context omitted.

I've been using o3 extensively since release (and a lot of Deep Research). I also use a lot of Claude and Gemini 2.5 Pro (most of the times, for code I'll let all of them go at it and iterate on my fav results). So far I've only used o3-pro a bit today, and it's a bit too heavy to use interactively (fire it off, revisit in 10-15 minutes), but it seems to generate much cleaner/more well organized code and answers. I f…

I wonder if we'll start to see artisanal benchmarks. You -- and I -- have preferred models for certain tasks. There's a world in which we start to see how things score on the "simonw chattiness index", and come to rely on smaller more specific benchmarks I think

Yeah, I think personalized evals will definitely be a thing. Besides reviewing way too much Arena, WildChat and having now seen lots of live traces firsthand, there's a wide range of LLM usage (and preferences), which really don't match my own tastes or requirements, lol.

For the past year or two, I've had my own personal 25 question vibe-check I've used on new models to kick the tires, but I think the future is something both a little more rigorous and a little more automated (something like LLM Jury w/ an UltraFeedback criteria based off of your own real world exchanges and then BTL ranked)? A future project...

Re: OpenAI o3-pro

#123

Post by OccamsMirror »

petesergeant wrote: post #113
manmal wrote: post #3

The benchmarks don’t look _that_ much better than o3. Does that mean Pro models are just incrementally better than base models, or are we approaching the higher end of a sigmoid function, with performance gains leveling off?

I am starting to feel like hallucination is a fundamentally unsolvable problem with the current architecture, and is going to keep squeezing the benchmarks until something changes. At this point I don't need smarter general models for my work, I need models that don't hallucinate, that are faster/cheaper, and that have better taste in specific domains. I think that's where we're going to see improvements moving forwa…

If you could actually teach these models things, not just in the current context, but as temporal learning, then that would alleviate a lot of the issues of hallucination. I imagine being able to say "that method doesn't exist, don't recommend it again" and then give it the documentation and it would absorb that information permanently, that would fundamentally change how we interact with these models. But can that work for models hosted for everyone to use at once?

Re: OpenAI o3-pro

#124

Post by baq »

mark_l_watson wrote: post #87

I am still not willing to upgrade to a Pro account. I pay $20 a month for both Gemini and ChatGPT, and for what I need this is currently enough. I have dreamed of having powerful AI ever since I read Bertram Raphael's great book Mind Inside Matter around 1978, getting hooked on AI research and sometimes practical applications for my life since then. I can easily afford $200 for a Pro account but I get this nagging fe…

LLMs would be transformative technology if all progress stopped today if only for their NLP capabilities, but the recent models obviously do so much more than that. Winter isn’t coming in that regard; what might happen if models won’t get smarter from here is a race to the bottom in token prices, which would still be not bad at all for token buyers.

Re: OpenAI o3-pro

#125

Post by hapticmonkey »

chad1n wrote: post #28

The guys in the other thread who said that OpenAI might have quantized o3 and that's how they reduced the price might be right. This o3-pro might be the actual o3-preview from the beginning and the o3 might be just a quantized version. I wish someone benchmarks all of these models to check for drops in quality.

What a great future we are building. If AI is supposed to run everything, everywhere....then there will be 2, maybe 3, AI companies. And nobody outside those companies knows how they work.

Re: OpenAI o3-pro

#126

Post by dmos62 »

nickysielicki wrote: post #14
mmsc wrote: post #2

I understand that things are moving fast and all, but surely the.. 8? models which are currently available is a bit .. overwhelming for users that just want to get answers to their questions of life? What's the end goal with having so many models available?

I just can’t believe nobody at the company has enough courage to tell their leadership that their naming scheme is completely stupid and insane. Four is greater than three, and so four should be better than three. The point of a name is to describe something so that you don’t confuse your users, not to be cute.

If you obfuscate the naming, you obfuscate the value proposition, and people become easier to mislead into choosing an overly expensive model. Same as with Intel CPUs, or many many other hardware products.

Re: OpenAI o3-pro

#127

Post by littlestymaar »

DanMcInerney wrote: post #31

I'm really hoping GPT5 is a larger jump in metrics than the last several releases we've seen like Claude3.5 - Claude4 or o3-mini-high to o3-pro. Although I will preface that with the fact I've been building agents for about a year now and despite the benchmarks only showing slight improvement, I have seen that each new generation feels actively better at exactly the same tasks I gave the previous generation. It would…

> I'm really hoping GPT5 is a larger jump in metrics than the last several releases we've seen like Claude3.5 - Claude4 or o3-mini-high to o3-pro.

This kind of expectations explains why there hasn't been a GPT-5 so far, and why we get a dumb numbering scheme instead for no reason.

At least Claude eventually decided not to care anymore and release Claude 4 even if the jump from 3.7 isn't particularly spectacular. We're well into the diminishing returns at this point, so it doesn't really make sense to postpone the major version bump, it's not like they're going to make a big leap again anytime soon.

Re: OpenAI o3-pro

#128

Post by clbrmbr »

nipah wrote: post #102
saberience wrote: post #66

Earlier quoted context omitted.

I’m not sure the arcagi are interesting benchmarks, for one they are image based and for two most people I show them too have issues understanding them, and in fact I had issues understanding them. Given the models don’t even see the versions we get to see it doesn’t surprise me they have issues we these. It’s not hard to make benchmarks that are so hard that humans and Lims can’t do.

"most people I show them too have issues understanding them, and in fact I had issues understanding them" ??? those benchmarks are so extremely simple they have basically 100% human approval rates, unless you are saying "I could not grasp it immediately but later I was able to after understanding the point" I think you and your friends should see a neurologist. And I'm not mocking you, I mean seriously, those are tas…

You may be above average intelligence. Those challenges are like classic IQ tests and I bet have a significant distribution among humans.

Re: OpenAI o3-pro

#129

Post by buu700 »

baq wrote: post #124
mark_l_watson wrote: post #87

I am still not willing to upgrade to a Pro account. I pay $20 a month for both Gemini and ChatGPT, and for what I need this is currently enough. I have dreamed of having powerful AI ever since I read Bertram Raphael's great book Mind Inside Matter around 1978, getting hooked on AI research and sometimes practical applications for my life since then. I can easily afford $200 for a Pro account but I get this nagging fe…

LLMs would be transformative technology if all progress stopped today if only for their NLP capabilities, but the recent models obviously do so much more than that. Winter isn’t coming in that regard; what might happen if models won’t get smarter from here is a race to the bottom in token prices, which would still be not bad at all for token buyers.

Agreed. I've said exactly the same thing before. If GPT-4 from two years ago had turned out to be the endgame of LLM technology, and we collectively spent the following 20 years integrating those capabilities throughout the economy, even that would be a profound change to the world as we know it.

If we froze LLM technology at present-day capabilities and spent the next 20 years on that, I'd expect it to ultimately look transformative in a similar way to the Internet. I mean if you told me in fall 2022 that 2.5 years later I'd be building software by meta-prompting and meta-meta-prompting AI agents to write code overnight while I slept, I'd assume that we were fictional characters in a Black Mirror episode.

Re: OpenAI o3-pro

#130

Post by dtech »

FergusArgyll wrote: post #76
simonw wrote: post #36

Earlier quoted context omitted.

That's definitely not the case here. The new o3-pro is slow - it took two minutes just to draw me an SVG of a pelican riding a bicycle. o3-preview was much faster than that. https://simonwillison.net/2025/Jun/10/o3-pro/

Wow! pelican benchmark is now saturated

I wonder how much of that is because it's getting more and more included in training data.

We now need to start using walrusses riding rickshaws

Post reply on HN