OpenAI O3 breakthrough high score on ARC-AGI-PUB

Post by neom » Fri, Dec 20, 2024, 9:24 PM UTC

hmottestad wrote: post #323

highfrequency wrote: post #291

Very cool. I recommend scrolling down to look at the example problem that O3 still can’t solve. It’s clear what goes on in the human brain to solve this problem: we look at one example, hypothesize a simple rule that explains it, and then check that hypothesis against the other examples. It doesn’t quite work, so we zoom into an example that we got wrong and refine the hypothesis so that it solves that sample. We kee…

I took a look at those examples that o3 can't solve. Looks similar to an IQ-test. Took me less time to figure out the 3 examples that it took to read your post. I was honestly a bit surprised to see how visual the tasks were. I had thought they were text based. So now I'm quite impressed that o3 can solve this type of task at all.

Post by vjerancrnjak » Fri, Dec 20, 2024, 9:24 PM UTC

Post by cerved » Fri, Dec 20, 2024, 9:25 PM UTC

thisisthenewme wrote: post #313

I feel like AI is already changing how we work and live - I've been using it myself for a lot of my development work. Though, what I'm really concerned about is what happens when it gets smart enough to do pretty much everything better (or even close) than humans can. We're talking about a huge shift where first knowledge workers get automated, then physical work too. The thing is, our whole society is built around p…

Post by danielmarkbruce » Fri, Dec 20, 2024, 9:27 PM UTC

refulgentis wrote: post #71

modeless wrote: post #13

Congratulations to Francois Chollet on making the most interesting and challenging LLM benchmark so far. A lot of people have criticized ARC as not being relevant or indicative of true reasoning, but I think it was exactly the right thing. The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning. It's obvious to everyone that…

This emphasizes persons and a self-conceived victory narrative over the ground truth. Models have regularly made progress on it, this is not new with the o-series. Doing astoundingly well on it, and having a mutually shared PR interest with OpenAI in this instance, doesn't mean a pile of visual puzzles is actually AGI or some well thought out and designed benchmark of True Intelligence(tm). It's one type of visual pu…

Post by munchler » Fri, Dec 20, 2024, 9:27 PM UTC

skywhopper wrote: post #285

modeless wrote: post #13

Congratulations to Francois Chollet on making the most interesting and challenging LLM benchmark so far. A lot of people have criticized ARC as not being relevant or indicative of true reasoning, but I think it was exactly the right thing. The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning. It's obvious to everyone that…

"The fact that scaled reasoning models are finally showing progress on ARC proves that what it measures really is relevant and important for reasoning." Not sure I understand how this follows. The fact that a certain type of model does well on a certain benchmark means that the benchmark is relevant for a real-world reasoning? That doesn't make sense.