Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

711–720 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#711

Post by raincole »

lumost wrote: post #696
duluca wrote: post #689

The first computers cost millions of dollars and filled entire rooms to accomplish what we would now consider simple computational tasks. That same computing power now fits into the width of a finger nail. I don’t get how technologists balk at the cost of experimental tech or assume current tech will run at the same efficiency for decades to come and melt the planet into a puddle. AGI won’t happen until you can fit e…

The concern here is mainly on practicality. The original mainframes did not command startup valuations counted in fractions of the US economy, they did qualify for billions in investment. This is a great milestone, but OpenAI will not be successful charging 10x the cost of a human to perform a task.

The cost of inference has be dropping by ~100x in the past 2 years.

https://a16z.com/llmflation-llm-inference-cost/

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#712

Post by hmottestad »

neom wrote: post #341
hmottestad wrote: post #323

Earlier quoted context omitted.

I took a look at those examples that o3 can't solve. Looks similar to an IQ-test. Took me less time to figure out the 3 examples that it took to read your post. I was honestly a bit surprised to see how visual the tasks were. I had thought they were text based. So now I'm quite impressed that o3 can solve this type of task at all.

I also took some time to look at the ones it couldn't solve. I stopped after this one: https://kts.github.io/arc-viewer/page6/#47996f11

That one's cool. All pink pixels need to be repaired so they match the symmetry in the picture.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#713

Post by serpix »

drdrey wrote: post #328
Balgair wrote: post #214

Complete aside here: I used to do work with amputees and prosthetics. There is a standardized test (and I just cannot remember the name) that fits in a briefcase. It's used for measuring the level of damage to the upper limbs and for prosthetic grading. Basically, it's got the dumbest and simplest things in it. Stuff like a lock and key, a glass of water and jug, common units of currency, a zipper, etc. It tests if y…

I think assembling Legos would be a cool robot benchmark: you need to parse the instructions, locate the pieces you need, pick them up, orient them, snap them to your current assembly, visually check if you achieved the desired state, repeat

I agree. Watching my toddler daughter build with small legos makes me understand how incredible fine motor skills are as even with small fingers some of the blocks are just too hard to snap together.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#714

Post by vicentwu »

"Note on "tuned": OpenAI shared they trained the o3 we tested on 75% of the Public Training set. They have not shared more details. We have not yet tested the ARC-untrained model to understand how much of the performance is due to ARC-AGI data."

Really want to see the number of training pairs needed to achieve this socre. If it only takes a few pairs, say 100 pairs, I would say it is amazing!

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#716

Post by madduci »

bluecoconut wrote: post #141

Efficiency is now key. ~=$3400 per single task to meet human performance on this benchmark is a lot. Also it shows the bullets as "ARC-AGI-TUNED", which makes me think they did some undisclosed amount of fine-tuning (eg. via the API they showed off last week), so even more compute went into this task. We can compare this roughly to a human doing ARC-AGI puzzles, where a human will take (high variance in my subjective…

Let's see when this will be released to the free tier. Looks promising, although I hope they will also be able to publish more details on this, as part of the "open" in their name

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#717

Post by patrickhogan1 »

otabdeveloper4 wrote: post #707
duluca wrote: post #689

The first computers cost millions of dollars and filled entire rooms to accomplish what we would now consider simple computational tasks. That same computing power now fits into the width of a finger nail. I don’t get how technologists balk at the cost of experimental tech or assume current tech will run at the same efficiency for decades to come and melt the planet into a puddle. AGI won’t happen until you can fit e…

Intelligence has nothing at all whatever to do with compute.

Do you think intelligence exists without prior experience? For instance, can someone instantly acquire a skill—like playing the piano—as if downloading it in The Matrix? Even prodigies like Mozart had prior exposure. His father, a composer and music teacher, introduced him to music from an early age. Does true intelligence require a foundation of prior knowledge?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#718

Post by Jensson »

hamburga wrote: post #627

I’m not sure if people realize what a weird test this is. They’re these simple visual puzzles that people can usually solve at a glance, but for the LLMs, they’re converted into a json format, and then the LLMs have to reconstruct the 2D visual scene from the json and pick up the patterns. If humans were given the json as input rather than the images, they’d have a hard time, too.

> If humans were given the json as input rather than the images, they’d have a hard time, too.

We shine light in text patterns at humans rather than inject the text directly into the brain as well, that is extremely unfair! Imagine how much better humans would be at text processing if we injected and extracted information from their brains using the neurons instead of eyes and hands.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#719

Post by waldrews »

lxgr wrote: post #694
duluca wrote: post #689

The first computers cost millions of dollars and filled entire rooms to accomplish what we would now consider simple computational tasks. That same computing power now fits into the width of a finger nail. I don’t get how technologists balk at the cost of experimental tech or assume current tech will run at the same efficiency for decades to come and melt the planet into a puddle. AGI won’t happen until you can fit e…

> take several data center’s worth of compute into a brain sized vessel. So the thing can move around process the world in real time How so? I'd imagine a robot connected to the data center embodying its mind, connected via low-latency links, would have to walk pretty far to get into trouble when it comes to interacting with the environment. The speed of light is about three orders of magnitude faster than the speed…

6 orders of magnitude if we use 120 m/s vs 300 km/s

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#720

Post by TechDebtDevin »

duluca wrote: post #689

The first computers cost millions of dollars and filled entire rooms to accomplish what we would now consider simple computational tasks. That same computing power now fits into the width of a finger nail. I don’t get how technologists balk at the cost of experimental tech or assume current tech will run at the same efficiency for decades to come and melt the planet into a puddle. AGI won’t happen until you can fit e…

Batteries..
Post reply on HN