Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

971–980 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#971

Post by akra »

mensetmanusman wrote: post #74

I’m super curious as to whether this technology completely destroys the middle class, or if everyone becomes better off because productivity is going to skyrocket.

The default position is that it will decrease the need of what the middle class can offer (skilled labor). All else being equal that increases the value of the other factors of production (the next bottleneck) such as capital and land.

Unless something changes, if I was a billionaire I would be ecstatic at the moment. Now even the impossible seems potentially possible if this delivers on its promises (e.g. go to Mars, build a utopia for my inner circle, etc). I no longer need other people to have everything. Previously there was no point in money if I didn't have a place to spend it/people to accept it. Now with real assets I can use AI/machines to do what I want - I no longer need "money" or more accurately other people to live a very wealthy life.

Again this is all else being equal. Lots of other things could change, but with increasing surveillance by use of technology I doubt large revolutions/etc will ever get the chance to get off the ground or have the scale to be effective.

Interesting times.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#972

Post by akra »

CliveBloomers wrote: post #135

Another meaningless benchmark, another month—it’s like clockwork at this point. No one’s going to remember this in a month; it’s just noise. The real test? It’s not in these flashy metrics or minor improvements. The only thing that actually matters is how fast it can wipe out the layers of middle management and all those pointless, bureaucratic jobs that add zero value. That’s the true litmus test. Everything else? I…

Its a common view from the "do'ers" (the people who made most of the value in the past; the hard workers, etc) that this will make management redundant. Sadly with a basic understanding of economics you can see this is probably wrong. The "do'ers" have given more power to the management class at their own expense with this solution - if I can get the AI to "do" all I need are the people who "decide what to do". Market power belongs with scarcity - all else being equal AI makes the barrier to development smaller meaning less scarcity on that side. In general technology developments have increased inequality especially since the 90's onwards.

Generally with AI think the top of society stand to gain a lot more than the middle/bottom of it for a whole host of reasons. If you think anything different your framework you use to make your conclusion is probably wrong at least in IMO.

I don't like saying this but there is a reason why the "AI bros", VC's, big tech CEO's, etc are all very very excited about this and many employees (some commenting here) are filled with dread/fear. The sales people, the managers, the MBA's, etc stand to gain a lot from this. Fear also serves as the best marketing tool; it makes people talk and spread OpenAI's news more so than everything else. Its a reason why targeting coding jobs/any jobs is so effective. I want to be wrong of course.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#973

Post by timabdulla »

slewis wrote: post #197
timabdulla wrote: post #168

Earlier quoted context omitted.

What's your explanation for why it can only get ~70% on SWE-bench Verified? I believe about 90% of the tasks were estimated by humans to take less than one hour to solve, so we aren't talking about very complex problems, and to boot, the contamination factor is huge: o3 (or any big model) will have in-depth knowledge of the internals of these projects, and often even know about the individual issues themselves (e.g.…

I've spent tons of time evaluating o1-preview on SWEBench-Verified. For one, I speculate OpenAI is using a very basic agent harness to get the results they've published on SWEBench. I believe there is a fair amount of headroom to improve results above what they published, using the same models. For two, some of the instances, even in SWEBench-Verified, require a bit of "going above and beyond" to get right. One examp…

So what percentage would you say falls to simple inability versus the other two factors you've mentioned?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#974

Post by edanm »

xanderlewis wrote: post #938
cornholio wrote: post #916

Earlier quoted context omitted.

But does it matter if it "really, really" reasons in the human sense, if it's able to prove some famous math theorem or come up with a novel result in theoretical physics? While beyond current motels, that would be the final test of AGI capability.

That would be important, but as far as I know it hasn’t happened (despite how often it’s intimated that we’re on the verge of it happening).

I've seen one Twitter thread from a mathematician who used an llm to come up with a new math result. Both coming up with the theorem statement and a unique proof,iirc.

Though to be clear, this wasn't a one shot thing - it was iirc a few months of back and forth chats with plenty of wrong turns too.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#975

Post by vicentwu »

nmca wrote: post #849
vicentwu wrote: post #714

"Note on "tuned": OpenAI shared they trained the o3 we tested on 75% of the Public Training set. They have not shared more details. We have not yet tested the ARC-untrained model to understand how much of the performance is due to ARC-AGI data." Really want to see the number of training pairs needed to achieve this socre. If it only takes a few pairs, say 100 pairs, I would say it is amazing!

75% of 400 is 300 :)

Trained with 300 raw pairs directly from the ARC training set without using any data augmentation process, such as generating many more pairs with some kind of ARC generator? That's amazing.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#976

Post by noFaceDiscoG668 »

bluecoconut wrote: post #201
bluecoconut wrote: post #141

Efficiency is now key. ~=$3400 per single task to meet human performance on this benchmark is a lot. Also it shows the bullets as "ARC-AGI-TUNED", which makes me think they did some undisclosed amount of fine-tuning (eg. via the API they showed off last week), so even more compute went into this task. We can compare this roughly to a human doing ARC-AGI puzzles, where a human will take (high variance in my subjective…

some other imporant quotes: "Average human off the street: 70-80%. STEM college grad: >95%. Panel of 10 random humans: 99-100%" -@fchollet on X So, considering that the $3400/task system isn't able to compete with STEM college grad yet, we still have some room (but it is shrinking, i expect even more compute will be thrown and we'll see these barriers broken in coming years) Also, some other back of envelope calculat…

Maybe another plane with a bunch of semiconductor people will disappear over Kazakhstan or something. Capitalist communisms gets bossier in stealth mode.

But sorry, blablabla, this shit is getting embarrassing.

> The question is now, can we close this "to human" gap

You won’t close this gap by throwing more compute at it. Anything in the sphere of creative thinking eludes most people in the history of the planet. People with PhDs in STEM end up working in IT sales not because they are good or capable of learning but because more than half of them can’t do squat shit, despite all that compute and all those algorithms in their brains.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#977

Post by rowanG077 »

miki123211 wrote: post #542
bluecoconut wrote: post #201

Earlier quoted context omitted.

some other imporant quotes: "Average human off the street: 70-80%. STEM college grad: >95%. Panel of 10 random humans: 99-100%" -@fchollet on X So, considering that the $3400/task system isn't able to compete with STEM college grad yet, we still have some room (but it is shrinking, i expect even more compute will be thrown and we'll see these barriers broken in coming years) Also, some other back of envelope calculat…

It's also worth keeping in mind that AIs are a lot less risky to deploy for businesses than humans. You can scale them up and down at any time, they can work 24/7 (including holidays) with no overtime pay and no breaks, they need no corporate campuses, office space, HR personnel or travel budgets, you don't have to worry about key employees going on sick/maternity leave or taking time off the moment they're needed mo…

AI do require overtime pay. In fact they are literally pay for use. If you use an AI 8 hours vs 16 hours a day is literally the difference between 2x cost.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#978

Post by 555watch »

Engineering-MD wrote: post #575

Can I just say what a dick move it was to do this as a 12 days of Christmas. I mean to be honest I agree with the arguments this isn’t as impressive as my initial impression, but they clearly intended it to be shocking/a show of possible AGI, which is rightly scary. It feels so insensitive to that right before a major holiday when the likely outcome is a lot of people feeling less secure in their career/job/life. Tha…

I don't know, maybe it's a bit off topic, but at least in cases that I'm imagining, I would always hire a human than fully rely on AI. Let the human consult with AI if needed, but still finalize the decision or result. The human will be thinking about the problem for months or years, even if passively during a vacation, an idea will occasionaly pop up. AI will think about its task for seconds, in case it missed some information or whatever, it will never wake up in the middle of the night thinking "s**, i forgot about X"

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#979

Post by attentionmech »

dyauspitr wrote: post #747
attentionmech wrote: post #21

Isn't this at the level now where it can sort of self improve. My guess is that they will just use it to improve the model and the cost they are showing per evaluation will go down drastically. So, next step in reasoning is open world reasoning now?

I don’t believe so. If it’s at the point where you could just plug it into a bunch of camera feeds around the world and it could only filter out a useful training set for itself out of that data then we truly would have AGI. I don’t think it’s there yet.

may be for some sub-domains like math and code it can do that since the verification process can be done / relatively tractable

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#980

Post by itfossil »

The amount of desperate rationalization in this thread is unbelievable. It's like watching people at a Pentecostal church start speaking in tongues in the hope that something wonderful will happen until it evolves into the realization that shit isn't going to happen and then slowly they just kind of putter out.

TLDR: The cacophony of fools is so loud now. Thank goodness it won't last.

Post reply on HN