Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o3-pro

help.openai.com

Post reply on HN
Print view View original

111–120 of 209 posts

Re: OpenAI o3-pro

#111

Post by radlad »

CamperBob2 wrote: post #101
Gerardo1 wrote: post #96

Earlier quoted context omitted.

I don't love that this is the conversation and when these models bake-in these silly scenarios with training data, everyone goes "see, pelican bike! super human intelligence!" The point is never the pelican. The point is that if a thing has information about pelicans, and has information about bicycles, then why can't it combine those ideas? Is it because it's not intelligent?

"I'm taking this talking dog right back to the pound. It told me to go long on AAPL. Totally overhyped"

Just because it's impressive doesn't mean it has "super human intelligence" though.

Re: OpenAI o3-pro

#112

Post by jsjohnst »

XCSme wrote: post #67
DanMcInerney wrote: post #31

I'm really hoping GPT5 is a larger jump in metrics than the last several releases we've seen like Claude3.5 - Claude4 or o3-mini-high to o3-pro. Although I will preface that with the fact I've been building agents for about a year now and despite the benchmarks only showing slight improvement, I have seen that each new generation feels actively better at exactly the same tasks I gave the previous generation. It would…

I remember the saying that from 90% to 99% is a 10x increase in accuracy, but 99% to 99.999% is a 1000x increase in accuracy. Even though it's a large10% increase first then only a 0.999% increase.

The saying goes:

From 90% to 99% is a 10x reduction in error rate, but 99% to 99.999% is a 1000x decrease in error rates.

Re: OpenAI o3-pro

#113

Post by petesergeant »

manmal wrote: post #3

The benchmarks don’t look _that_ much better than o3. Does that mean Pro models are just incrementally better than base models, or are we approaching the higher end of a sigmoid function, with performance gains leveling off?

I am starting to feel like hallucination is a fundamentally unsolvable problem with the current architecture, and is going to keep squeezing the benchmarks until something changes.

At this point I don't need smarter general models for my work, I need models that don't hallucinate, that are faster/cheaper, and that have better taste in specific domains. I think that's where we're going to see improvements moving forward.

Re: OpenAI o3-pro

#114

Post by petesergeant »

lhl wrote: post #94
manmal wrote: post #3

The benchmarks don’t look _that_ much better than o3. Does that mean Pro models are just incrementally better than base models, or are we approaching the higher end of a sigmoid function, with performance gains leveling off?

I've been using o3 extensively since release (and a lot of Deep Research). I also use a lot of Claude and Gemini 2.5 Pro (most of the times, for code I'll let all of them go at it and iterate on my fav results). So far I've only used o3-pro a bit today, and it's a bit too heavy to use interactively (fire it off, revisit in 10-15 minutes), but it seems to generate much cleaner/more well organized code and answers. I f…

I wonder if we'll start to see artisanal benchmarks. You -- and I -- have preferred models for certain tasks. There's a world in which we start to see how things score on the "simonw chattiness index", and come to rely on smaller more specific benchmarks I think

Re: OpenAI o3-pro

#115

Post by zmgsabst »

XCSme wrote: post #67
DanMcInerney wrote: post #31

I'm really hoping GPT5 is a larger jump in metrics than the last several releases we've seen like Claude3.5 - Claude4 or o3-mini-high to o3-pro. Although I will preface that with the fact I've been building agents for about a year now and despite the benchmarks only showing slight improvement, I have seen that each new generation feels actively better at exactly the same tasks I gave the previous generation. It would…

I remember the saying that from 90% to 99% is a 10x increase in accuracy, but 99% to 99.999% is a 1000x increase in accuracy. Even though it's a large10% increase first then only a 0.999% increase.

Sometimes it’s nice to frame it the other way, eg:

90% -> 1 error per 10

99% -> 1 error per 100

99.99% -> 1 error per 10,000

That can help to see the growth in accuracy, when the numbers start getting small (and why clocks are framed as 1 second lost per…).

Re: OpenAI o3-pro

#116

Post by kevinqi »

codingwagie wrote: post #40
DanMcInerney wrote: post #31

I'm really hoping GPT5 is a larger jump in metrics than the last several releases we've seen like Claude3.5 - Claude4 or o3-mini-high to o3-pro. Although I will preface that with the fact I've been building agents for about a year now and despite the benchmarks only showing slight improvement, I have seen that each new generation feels actively better at exactly the same tasks I gave the previous generation. It would…

I'm seeing big advances that arent shown in the benchmarks, I can simply build software now that I couldnt build before. The level of complexity that I can manage and deliver is higher.

yeah, the benchmarks are just a proxy. o3 was a step change where I started to really be able to build stuff I couldn't before

Re: OpenAI o3-pro

#117

Post by mkl »

paul7986 wrote: post #105

GPT needs way better image creation! Today I asked it to create a full image of a 2025 calendar highlighting all weekday workdays excluding federal holidays. At the bottom of legend tell me how many weekday work hours are available within criteria noted. It created the image showing each month but when you looked at each month it was so janky ... February 31st and other huge errors! I'm not using image creation to cr…

Wrong tool for the job. Try asking it to generate an SVG calendar with those features, or to generate Python code that produces an SVG calendar with those features.

Re: OpenAI o3-pro

#118

Post by bobbylarrybobby »

XCSme wrote: post #67
DanMcInerney wrote: post #31

I'm really hoping GPT5 is a larger jump in metrics than the last several releases we've seen like Claude3.5 - Claude4 or o3-mini-high to o3-pro. Although I will preface that with the fact I've been building agents for about a year now and despite the benchmarks only showing slight improvement, I have seen that each new generation feels actively better at exactly the same tasks I gave the previous generation. It would…

I remember the saying that from 90% to 99% is a 10x increase in accuracy, but 99% to 99.999% is a 1000x increase in accuracy. Even though it's a large10% increase first then only a 0.999% increase.

I think the proper way to compare probabilities/proportions is by odds ratios. 99:1 vs 99999:1. (So a little more than 1000x.) This also lets you talk about “doubling likelihood”, where twice as likely as 1/2=1:1 is 2:1=2/3, and twice as likely again is 4:1=4/5.

Re: OpenAI o3-pro

#119

Post by djrj477dhsnv »

Osyris wrote: post #5
mmsc wrote: post #2

I understand that things are moving fast and all, but surely the.. 8? models which are currently available is a bit .. overwhelming for users that just want to get answers to their questions of life? What's the end goal with having so many models available?

This is a much more expensive model to run and is only available to users who pay the most. I don't see an issue. However, the "plus" plan absolutely could use some trimming.

If it's better (and newer) than gpt4, it shouldn't have a lower version number.

Re: OpenAI o3-pro

#120

Post by datameta »

CamperBob2 wrote: post #63
jstummbillig wrote: post #60

Earlier quoted context omitted.

I love that we are in the timeline where we are somewhat seriously evaluating probably super human intelligence by their ability to draw a svg of a cycling pelican.

I still remember my jaw hitting the floor when the first DALL-E paper came out, with the baby daikon radish walking a dog. How the actual fuck ...? Now we're probably all too jaded to fully appreciate the next advance of that magnitude, whatever that turns out to be. E.g., the pelicans all look pretty cruddy including this one, but the fact that they are being delivered in .SVG is a bigger deal than the quality of th…

This makes me think of a reduction gear as a metaphor. At a high enough ratio, the torque is enormous but being put toward barely perceptible movement. There is the huge amount of computation happening to result in SVG that resembles a pelican on a bicycle.
Post reply on HN