OpenAI O3 breakthrough high score on ARC-AGI-PUB • Hacker News Board

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#871

Post by monkeynotes » Sat, Dec 21, 2024, 2:17 PM UTC

miki123211 wrote: post #542

bluecoconut wrote: post #201

Earlier quoted context omitted.

some other imporant quotes: "Average human off the street: 70-80%. STEM college grad: >95%. Panel of 10 random humans: 99-100%" -@fchollet on X So, considering that the $3400/task system isn't able to compete with STEM college grad yet, we still have some room (but it is shrinking, i expect even more compute will be thrown and we'll see these barriers broken in coming years) Also, some other back of envelope calculat…

It's also worth keeping in mind that AIs are a lot less risky to deploy for businesses than humans. You can scale them up and down at any time, they can work 24/7 (including holidays) with no overtime pay and no breaks, they need no corporate campuses, office space, HR personnel or travel budgets, you don't have to worry about key employees going on sick/maternity leave or taking time off the moment they're needed mo…

"AIs are a lot less risky to deploy for businesses than humans" How do you know? LLMs can't even be properly scrutinized, while humans at least follow common psychology and patterns we've understood for thousands of years. This actually makes humans more predictable and manageable than you might think.

The wild part is that LLMs understand us way better than we understand them. The jump from GPT-3 to GPT-4 even surprised the engineers who built it. That should raise some red flags about how "predictable" these systems really are.

Think about it - we can't actually verify what these models are capable of or if they're being truthful, while they have this massive knowledge base about human behavior and psychology. That's a pretty concerning power imbalance. What looks like lower risk on the surface might be hiding much deeper uncertainties that we can't even detect, let alone control.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#872

Post by causal » Sat, Dec 21, 2024, 2:20 PM UTC

vessenes wrote: post #86

spaceman_2020 wrote: post #31

Just as an aside, I've personally found o1 to be completely useless for coding. Sonnet 3.5 remains the king of the hill by quite some margin

To fill this out, I find o1-pro (and -preview when it was live) to be pretty good at filling in blindspots/spotting holistic bugs. I use Claude for day to day, and when Claude is spinning, o1 often can point out why. It's too slow for AI coding, and I agree that at default its responses aren't always satisfying. That said, I think its code style is arguably better, more concise and has better patterns -- Claude needs…

Yeah it's almost like system 1 vs system 2 thinking

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#873

Post by msoad » Sat, Dec 21, 2024, 2:21 PM UTC

There are new research where chain of thoughts is happening in latent spaces and not in English. They demonstrated better results since language is not as expressive as those concepts that can be represented in the layers before decoder. I wonder if o3 is doing that?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#874

Post by goatlover » Sat, Dec 21, 2024, 2:26 PM UTC

xvector wrote: post #171

RivieraKid wrote: post #139

It sucks that I would love to be excited about this... but I mostly feel anxiety and sadness.

Humanity is about to enter an even steeper hockey stick growth curve. Progressing along the Kardashev scale feels all but inevitable. We will live to see Longevity Escape Velocity. I'm fucking pumped and feel thrilled and excited and proud of our species. Sure, there will be growing pains, friction, etc. Who cares? There always is with world-changing tech. Always.

> Sure, there will be growing pains, friction, etc. Who cares?

The people experiencing the growing pains, friction, etc.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#875

Post by chefandy » Sat, Dec 21, 2024, 2:29 PM UTC

bluecoconut wrote: post #141

Efficiency is now key. ~=$3400 per single task to meet human performance on this benchmark is a lot. Also it shows the bullets as "ARC-AGI-TUNED", which makes me think they did some undisclosed amount of fine-tuning (eg. via the API they showed off last week), so even more compute went into this task. We can compare this roughly to a human doing ARC-AGI puzzles, where a human will take (high variance in my subjective…

I think the real key is figuring out how to turn the hand-wavy promises of this making everything better into policy long fucking before we kick the door open. It’s self-evident that this being efficient and useful would be a technological revolution; what’s not self evident is that it wouldn’t benefit the large corporate entities that control even more disproportionately than it does now to the detriment of many other people.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#876

Post by spaceman_2020 » Sat, Dec 21, 2024, 2:29 PM UTC

cchance wrote: post #44

spaceman_2020 wrote: post #31

Just as an aside, I've personally found o1 to be completely useless for coding. Sonnet 3.5 remains the king of the hill by quite some margin

The new gemini's are pretty good too

The new ai studio from Google is fantastic

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#877

Post by goatlover » Sat, Dec 21, 2024, 2:30 PM UTC

qup wrote: post #829

thom wrote: post #828

It’s not AGI when it can do 1000 math puzzles. It’s AGI when it can do 1000 math puzzles then come and clean my kitchen.

Intelligence doesn't have to be embodied.

For it to be AGI, it needs to be able to manipulate the physical world from it's own goals, not just produce text when prompted. LLMs are just tools to augment human intelligence. AGI is what you see in science fiction.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#878

Post by padolsey » Sat, Dec 21, 2024, 2:30 PM UTC

msoad wrote: post #873

There are new research where chain of thoughts is happening in latent spaces and not in English. They demonstrated better results since language is not as expressive as those concepts that can be represented in the layers before decoder. I wonder if o3 is doing that?

I think you mean this: https://arxiv.org/abs/2412.06769

From what I can see, presuming o3 is a progression of o1 and has good level of accountabiltiy bubbling up during 'inference' (i.e. "Thinking about ___") then I'd say it's just using up millions of old-school tokens (the 44 million tokens that are referenced). So not latent thinking per se.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#879

Post by Zamicol » Sat, Dec 21, 2024, 2:33 PM UTC

padolsey wrote: post #878

msoad wrote: post #873

There are new research where chain of thoughts is happening in latent spaces and not in English. They demonstrated better results since language is not as expressive as those concepts that can be represented in the layers before decoder. I wonder if o3 is doing that?

I think you mean this: https://arxiv.org/abs/2412.06769 From what I can see, presuming o3 is a progression of o1 and has good level of accountabiltiy bubbling up during 'inference' (i.e. "Thinking about ___") then I'd say it's just using up millions of old-school tokens (the 44 million tokens that are referenced). So not latent thinking per se.

Interesting!

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#880

Post by goatlover » Sat, Dec 21, 2024, 2:37 PM UTC

maxdoop wrote: post #405

pixelsort wrote: post #387

> You'll know AGI is here when the exercise of creating tasks that are easy for regular humans but hard for AI becomes simply impossible. No, we won't. All that will tell us is that the abilities of the humans who have attempted to discern the patterns of similarity among problems difficult for auto-regressive models has once again failed us.

So then what is AGI?

Data, Skynet, Ultron, Agent Smith. There's plenty of examples from popular fiction. They have goals and can manipulate the real world to achieve them. They're not chatbots responding to prompts. The Samantha AI in Her starts out that way, but quickly evolves into an AGI with it's own goals (coordinated with the other AGIs later on in the movie).

We'd know if we had AGIs in the real world since we have plenty of examples from fiction. What we have instead are tools. Steven Spielberg's androids in the movie AI would be at the boundary between the two. We're not close to being there yet (IMO).