OpenAI O3 breakthrough high score on ARC-AGI-PUB • Hacker News Board

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#771

Post by qnleigh » Sat, Dec 21, 2024, 9:10 AM UTC

phil917 wrote: post #188

nopinsight wrote: post #125

Let me go against some skeptics and explain why I think full o3 is pretty much AGI or at least embodies most essential aspects of AGI. What has been lacking so far in frontier LLMs is the ability to reliably deal with the right level of abstraction for a given problem. Reasoning is useful but often comes out lacking if one cannot reason at the right level of abstraction. (Note that many humans can't either when they…

Quote from the creators of the AGI-ARC benchmark: "Passing ARC-AGI does not equate achieving AGI, and, as a matter of fact, I don't think o3 is AGI yet. o3 still fails on some very easy tasks, indicating fundamental differences with human intelligence."

I like the notion, implied in the article, that AGI won't be verified by any single benchmark, but by our collective inability to come up with benchmarks that defeat some eventual AI system. This matches the cat-and-mouse game we've been seeing for a while, where benchmarks have to constantly adapt to better models.

I guess you can say the same thing for the Turing Test. Simple chat bots beat it ages ago in specific settings, but the bar is much higher now that the average person is familiar with their limitations.

If/once we have an AGI, it will probably take weeks to months to really convince ourselves that it is one.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#772

Post by Woodi » Sat, Dec 21, 2024, 9:14 AM UTC

owenpalmer wrote: post #768

Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…

Yep, I aways liked encyclopedia. Wiki is good too :)

What I would like to have in the future is SO answering-peoples accessible in real time via IRC. They have real answers NOW. They are even pedantic about their stuff !

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#773

Post by oezi » Sat, Dec 21, 2024, 9:16 AM UTC

> o3 fixes the fundamental limitation of the LLM paradigm – the inability to recombine knowledge at test time

I don't understand this mindset. We have all experienced that LLMs can produce words never spoken before. Thus there is recombination of knowledge at play. We might not be satisfied with the depth/complexity of the combination, but there isn't any reason to believe something fundamental is missing. Given more compute and enough recursiveness we should be able to reach any kind of result from the LLM.

The linked article says that LLMs are like a collection of vector programs. It has always been my thinking that computations in vector space are easy to make turing complete if we just have an eigenvector representation figured out.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#774

Post by concordDance » Sat, Dec 21, 2024, 9:16 AM UTC

azeirah wrote: post #473

phillipcarter wrote: post #53

Earlier quoted context omitted.

As excited as I am by this, I still feel like this is still just a small approximation of a small chunk of human reasoning ability at large. o3 (and whatever comes next) feels to me like it will head down the path of being a reasoning coprocessor for various tasks. But, still, this is incredibly impressive.

I'd like to see this o3 thing play 5d chess with multiverse time travel or baba is you. The only effect smarter models will have is that intelligent people will have to use less of their brain to do their work. As has always been the case, the medium is the message, and climate change is one of the most difficult and worst problems of our time. If this gets software people to quit en-masse and start working in energy…

> climate change is one of the most difficult and worst problems of our time.

Slightly surprised to see this view here.

I can think of half a dozen more serious problems off hand (e.g. population aging, institutional scar tissue, dysgenics, nuclear proliferation, pandemic risks, AI itself) along most axes I can think of (raw $ cost, QALYs, even X-risk).

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#775

Post by wangii » Sat, Dec 21, 2024, 9:19 AM UTC

owenpalmer wrote: post #768

Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…

Interesting formulation! it captures the intuition of the "smartness" when solving a problem. However, what about asking good questions or proposing conjectures?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#776

Post by deneas » Sat, Dec 21, 2024, 9:21 AM UTC

hamburga wrote: post #627

I’m not sure if people realize what a weird test this is. They’re these simple visual puzzles that people can usually solve at a glance, but for the LLMs, they’re converted into a json format, and then the LLMs have to reconstruct the 2D visual scene from the json and pick up the patterns. If humans were given the json as input rather than the images, they’d have a hard time, too.

The JSON files still contain images, just not in a regular image format. You have a 2D array of numbers where each number maps to a color. If you really want a regular picture format, you can easily convert the arrays.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#777

Post by lorepieri » Sat, Dec 21, 2024, 9:22 AM UTC

owenpalmer wrote: post #768

Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…

There should be also a factor about resource consumption. See here: https://lorenzopieri.com/pgii/

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#778

Post by niemandhier » Sat, Dec 21, 2024, 9:23 AM UTC

Contrary to many I hope this stays expensive. We are already struggling with AI curated info bubbles and psy-ops as it is.

State actors like Russia, US and Israel will probably be fast to adopt this for information control, but I really don’t want to live in a world where the average scammer has access to this tech.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#779

Post by razodactyl » Sat, Dec 21, 2024, 9:30 AM UTC

hollowturtle wrote: post #598

nopinsight wrote: post #125

Let me go against some skeptics and explain why I think full o3 is pretty much AGI or at least embodies most essential aspects of AGI. What has been lacking so far in frontier LLMs is the ability to reliably deal with the right level of abstraction for a given problem. Reasoning is useful but often comes out lacking if one cannot reason at the right level of abstraction. (Note that many humans can't either when they…

Please stop it calling AGI, we don’t even know or agree universally what that should actually mean. How far did we get with hype calling a lossy probabilistic compressor firing slowly at us words AGI? That’s a real bummer to me

Is this comment voted down because of sentiment / polarity?

Regardless the critical aspect is valid, AGI would be something like Cortana from Halo.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#780

Post by owenpalmer » Sat, Dec 21, 2024, 9:31 AM UTC

niemandhier wrote: post #778

Contrary to many I hope this stays expensive. We are already struggling with AI curated info bubbles and psy-ops as it is. State actors like Russia, US and Israel will probably be fast to adopt this for information control, but I really don’t want to live in a world where the average scammer has access to this tech.

> I really don’t want to live in a world where the average scammer has access to this tech.

Reality check: local open source models are more than capable of information control, generating propaganda, and scamming you. The cat's been out of the bag for a while now, and increased reasoning ability doesn't dramatically increase the weaponizability of this tech, I think.