Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI O3 breakthrough high score on ARC-AGI-PUB

arcprize.org

Post reply on HN
Print view View original

271–280 of 1001 posts

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#271

Post by rahimnathwani »

croemer wrote: post #255
Balgair wrote: post #214

Complete aside here: I used to do work with amputees and prosthetics. There is a standardized test (and I just cannot remember the name) that fits in a briefcase. It's used for measuring the level of damage to the upper limbs and for prosthetic grading. Basically, it's got the dumbest and simplest things in it. Stuff like a lock and key, a glass of water and jug, common units of currency, a zipper, etc. It tests if y…

> We had hand prosthetics that could play Mozart at 5x speed on a baby grand, but could not pick up a silver dollar or zip a jacket even a little bit. " I must be missing something, how can they be able to play Mozart at 5x speed with their prosthetics but not zip a jacket? They could press keys but not do tasks requiring feedback? Or did you mean they used to play Mozart at 5x speed before they became amputees?

Imagine a prosthetic 'hand' that has 5 regular fingers, rather than 4 fingers and a thumb. It would be able to play a piano just fine, but be unable to grasp anything small, like a zipper.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#272

Post by jack_pp »

AGI for me is something I can give a new project to and be able to use it better than me. And not because it has a huge context window, because it will update its weights after consuming that project. Until we have that I don't believe we have truly reached AGI.

Edit: it also tests the new knowledge, it has concepts such as trusting a source, verifying it etc. If I can just gaslight it into unlearning python then it's still too dumb.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#273

Post by Method-X »

Balgair wrote: post #214

Complete aside here: I used to do work with amputees and prosthetics. There is a standardized test (and I just cannot remember the name) that fits in a briefcase. It's used for measuring the level of damage to the upper limbs and for prosthetic grading. Basically, it's got the dumbest and simplest things in it. Stuff like a lock and key, a glass of water and jug, common units of currency, a zipper, etc. It tests if y…

Was it the Southampton hand assessment procedure?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#274

Post by numpad0 »

croemer wrote: post #255
Balgair wrote: post #214

Complete aside here: I used to do work with amputees and prosthetics. There is a standardized test (and I just cannot remember the name) that fits in a briefcase. It's used for measuring the level of damage to the upper limbs and for prosthetic grading. Basically, it's got the dumbest and simplest things in it. Stuff like a lock and key, a glass of water and jug, common units of currency, a zipper, etc. It tests if y…

> We had hand prosthetics that could play Mozart at 5x speed on a baby grand, but could not pick up a silver dollar or zip a jacket even a little bit. " I must be missing something, how can they be able to play Mozart at 5x speed with their prosthetics but not zip a jacket? They could press keys but not do tasks requiring feedback? Or did you mean they used to play Mozart at 5x speed before they became amputees?

Thumb not opposable?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#275

Post by KeplerBoy »

antirez wrote: post #118
obblekk wrote: post #42

Human performance is 85% [1]. o3 high gets 87.5%. This means we have an algorithm to get to human level performance on this task. If you think this task is an eval of general reasoning ability, we have an algorithm for that now. There's a lot of work ahead to generalize o3 performance to all domains. I think this explains why many researchers feel AGI is within reach, now that we have an algorithm that works. Congrat…

NNs are not algorithms.

Running inference on a model certainly is a algorithm.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#276

Post by hartator »

phil917 wrote: post #183

Direct quote from the ARC-AGI blog: “SO IS IT AGI? ARC-AGI serves as a critical benchmark for detecting such breakthroughs, highlighting generalization power in a way that saturated or less demanding benchmarks cannot. However, it is important to note that ARC-AGI is not an acid test for AGI – as we've repeated dozens of times this year. It's a research tool designed to focus attention on the most challenging unsolve…

> acid test

The css acid test? This can be gamed too.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#277

Post by 8note »

croemer wrote: post #255
Balgair wrote: post #214

Complete aside here: I used to do work with amputees and prosthetics. There is a standardized test (and I just cannot remember the name) that fits in a briefcase. It's used for measuring the level of damage to the upper limbs and for prosthetic grading. Basically, it's got the dumbest and simplest things in it. Stuff like a lock and key, a glass of water and jug, common units of currency, a zipper, etc. It tests if y…

> We had hand prosthetics that could play Mozart at 5x speed on a baby grand, but could not pick up a silver dollar or zip a jacket even a little bit. " I must be missing something, how can they be able to play Mozart at 5x speed with their prosthetics but not zip a jacket? They could press keys but not do tasks requiring feedback? Or did you mean they used to play Mozart at 5x speed before they became amputees?

zipping up a jacket is really hard to do, and requires very precise movements and coordination between hands.

playing mozart is much more forgiving in terms of the number of different motions you have to make in different directions, the amount of pressure to apply, and even the black keys are much bigger than large sized zipper tongues.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#278

Post by ben_w »

croemer wrote: post #255
Balgair wrote: post #214

Complete aside here: I used to do work with amputees and prosthetics. There is a standardized test (and I just cannot remember the name) that fits in a briefcase. It's used for measuring the level of damage to the upper limbs and for prosthetic grading. Basically, it's got the dumbest and simplest things in it. Stuff like a lock and key, a glass of water and jug, common units of currency, a zipper, etc. It tests if y…

> We had hand prosthetics that could play Mozart at 5x speed on a baby grand, but could not pick up a silver dollar or zip a jacket even a little bit. " I must be missing something, how can they be able to play Mozart at 5x speed with their prosthetics but not zip a jacket? They could press keys but not do tasks requiring feedback? Or did you mean they used to play Mozart at 5x speed before they became amputees?

Playing a piano involves pushing down on the right keys with the right force at the right time, but that could be pre-programmed well before computers. The self-playing piano in the saloon in Westworld wasn't a huge anachronism, such things slightly overlapped with the Wild West era: https://en.wikipedia.org/wiki/Player_piano

Picking up a 1mm thick metal disk from a flat surface requires the user gives the exact time, place, and force, and I'm not even sure what considerations it needs for surface materials (e.g. slightly squishy fake skin) and/or tip shapes (e.g. fake nails).

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#279

Post by croemer »

xvector wrote: post #171
RivieraKid wrote: post #139

It sucks that I would love to be excited about this... but I mostly feel anxiety and sadness.

Humanity is about to enter an even steeper hockey stick growth curve. Progressing along the Kardashev scale feels all but inevitable. We will live to see Longevity Escape Velocity. I'm fucking pumped and feel thrilled and excited and proud of our species. Sure, there will be growing pains, friction, etc. Who cares? There always is with world-changing tech. Always.

Longevity Escape Velocity? Even if you had orders of magnitude more people working on medical research, it's not a given that prolonging life indefinitely is even possible.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#280

Post by lossolo »

modeless wrote: post #13

Congratulations to Francois Chollet on making the most interesting and challenging LLM benchmark so far. A lot of people have criticized ARC as not being relevant or indicative of true reasoning, but I think it was exactly the right thing. The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning. It's obvious to everyone that…

> making the most interesting and challenging LLM benchmark so far.

This[1] is currently the most challenging benchmark. I would like to see how O3 handles it, as O1 solved only 1%.

1. https://epoch.ai/frontiermath/the-benchmark

Post reply on HN