Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI o3-pro

help.openai.com

Post reply on HN
Print view View original

101–110 of 209 posts

Re: OpenAI o3-pro

#101

Post by CamperBob2 »

Gerardo1 wrote: post #96
jstummbillig wrote: post #60

Earlier quoted context omitted.

I love that we are in the timeline where we are somewhat seriously evaluating probably super human intelligence by their ability to draw a svg of a cycling pelican.

I don't love that this is the conversation and when these models bake-in these silly scenarios with training data, everyone goes "see, pelican bike! super human intelligence!" The point is never the pelican. The point is that if a thing has information about pelicans, and has information about bicycles, then why can't it combine those ideas? Is it because it's not intelligent?

"I'm taking this talking dog right back to the pound. It told me to go long on AAPL. Totally overhyped"

Re: OpenAI o3-pro

#102

Post by nipah »

saberience wrote: post #66
croddin wrote: post #52

Earlier quoted context omitted.

There is still plenty of room for growth on the ARC-AGI benchmarks. ARC-AGI 2 is still "ARC-AGI-1: * Low: 44%, $1.64/task * Medium: 57%, $3.18/task * High: 59%, $4.16/task ARC-AGI-2: * All reasoning efforts: Takeaways: * o3-pro in line with o3 performance * o3's new price sets the ARC-AGI-1 Frontier" - https://x.com/arcprize/status/1932535378080395332

I’m not sure the arcagi are interesting benchmarks, for one they are image based and for two most people I show them too have issues understanding them, and in fact I had issues understanding them. Given the models don’t even see the versions we get to see it doesn’t surprise me they have issues we these. It’s not hard to make benchmarks that are so hard that humans and Lims can’t do.

"most people I show them too have issues understanding them, and in fact I had issues understanding them" ??? those benchmarks are so extremely simple they have basically 100% human approval rates, unless you are saying "I could not grasp it immediately but later I was able to after understanding the point" I think you and your friends should see a neurologist. And I'm not mocking you, I mean seriously, those are tasks extremely basic for any human brain and even some other mammals to do.

Re: OpenAI o3-pro

#103

Post by AtlasBarfed »

CamperBob2 wrote: post #56
AtlasBarfed wrote: post #51

Earlier quoted context omitted.

I'd like one to do my test use case: Port unix-sed from c to java with a full test suite and all options supported. Somewhere between "it answers questions of life" and "it beats PhDs at math questions", I'd like to see one LLM take this, IMO, rather "pure" language task and succeeed. It is complicated, but it isn't complex. It's string operations with a deep but not that deep expression system and flag set. It is we…

How does the latest Gemini 2.5 Pro Ultra Flash Max Hemi XLT release do on that task? It obviously demands a massive context window.

I'll check once I get the nitrous tanks and the aftermarket turbos overnighted from Japan arrive.

Re: OpenAI o3-pro

#104

Post by nipah »

AtlasBarfed wrote: post #51
mmsc wrote: post #2

I understand that things are moving fast and all, but surely the.. 8? models which are currently available is a bit .. overwhelming for users that just want to get answers to their questions of life? What's the end goal with having so many models available?

I'd like one to do my test use case: Port unix-sed from c to java with a full test suite and all options supported. Somewhere between "it answers questions of life" and "it beats PhDs at math questions", I'd like to see one LLM take this, IMO, rather "pure" language task and succeeed. It is complicated, but it isn't complex. It's string operations with a deep but not that deep expression system and flag set. It is we…

I have a very simple question with like, 5 lines at best, that basically no model, neither reasoning or simpler could grasp. For obvious reasons I'm not disclosing it here (because I fear data contamination in the long run), but it basically breaks the "reasoning" of those things. Unfortunately, I still can't try the o3-pro because the API version is not easily available, and I'm certainly not willing to pay for it in pro mode, but when it comes to the plus version (if it comes) I'll try. To this date, because of this question (and similar ones) I stand very unimpressed with those models, the marketing is a thousand times larger than reality, and I suspect people in general are surprisingly less capable of detecting intelligence than they think.

The normal o3 also managed to break 3 isolated installations of linux I was trying it with, a few days ago. The task was very simple, simply setup ubuntu with btrfs, timeshift and grub-btrfs and it managed to fail every single time (even when searching the web), so it was not impressive either.

Re: OpenAI o3-pro

#105

Post by paul7986 »

GPT needs way better image creation! Today I asked it to create a full image of a 2025 calendar highlighting all weekday workdays excluding federal holidays. At the bottom of legend tell me how many weekday work hours are available within criteria noted.

It created the image showing each month but when you looked at each month it was so janky ... February 31st and other huge errors!

I'm not using image creation to create 3d art for fun or art sake im trying to use it to create utility images to share for discussion with friends & co-workers. The above is just one of many ways it fails when creating utility images!

Re: OpenAI o3-pro

#106

Post by [deleted] »

paul7986 wrote: post #105

GPT needs way better image creation! Today I asked it to create a full image of a 2025 calendar highlighting all weekday workdays excluding federal holidays. At the bottom of legend tell me how many weekday work hours are available within criteria noted. It created the image showing each month but when you looked at each month it was so janky ... February 31st and other huge errors! I'm not using image creation to cr…

[deleted]

Re: OpenAI o3-pro

#107

Post by lobsterthief »

nickysielicki wrote: post #59
simonw wrote: post #24

Earlier quoted context omitted.

Sam Altman has said the same thing on Twitter a few times. https://x.com/sama/status/1911906570835022319 > how about we fix our model naming by this summer and everyone gets a few more months to make fun of us (which we very much deserve) until then?

I’d prefer for them to just fix it asap instead and then keep the existing endpoints around for a year as aliases.

They will definitely keep the endpoint aliases around for years. No real cost in doing so.

Re: OpenAI o3-pro

#108

Post by teruakohatu »

simonw wrote: post #36
chad1n wrote: post #28

The guys in the other thread who said that OpenAI might have quantized o3 and that's how they reduced the price might be right. This o3-pro might be the actual o3-preview from the beginning and the o3 might be just a quantized version. I wish someone benchmarks all of these models to check for drops in quality.

That's definitely not the case here. The new o3-pro is slow - it took two minutes just to draw me an SVG of a pelican riding a bicycle. o3-preview was much faster than that. https://simonwillison.net/2025/Jun/10/o3-pro/

Do you think a cycling pelican is still a valid cursory benchmark? By now surely discussions about it are in the training set.

There is quite a few on Google Image search.

On the other hand they still seem to struggle!

Re: OpenAI o3-pro

#109

Post by protocolture »

codingwagie wrote: post #40
DanMcInerney wrote: post #31

I'm really hoping GPT5 is a larger jump in metrics than the last several releases we've seen like Claude3.5 - Claude4 or o3-mini-high to o3-pro. Although I will preface that with the fact I've been building agents for about a year now and despite the benchmarks only showing slight improvement, I have seen that each new generation feels actively better at exactly the same tasks I gave the previous generation. It would…

I'm seeing big advances that arent shown in the benchmarks, I can simply build software now that I couldnt build before. The level of complexity that I can manage and deliver is higher.

I am finding that my ability to use it to code, aligns almost perfectly with increasing token memory.

Re: OpenAI o3-pro

#110

Post by corysama »

motorest wrote: post #88
alightsoul wrote: post #78

Earlier quoted context omitted.

mind telling examples?

Not OP, but a couple of days ago I managed to vibecode my way through a small app that pulled data from a few services and did a few validation checks. By itself its not very impressive, but my input was literally "this is how the responses from endpoint A,B and C look like. This field included somewhere in A must be somewhere in the response from B, and the response from C must feature this and that from response A…

I’m wrapping up doing literally the same thing. I did it step-by-step. But, for me there was also a process of figuring out how it should work.
Post reply on HN