OpenAI O3 breakthrough high score on ARC-AGI-PUB • Hacker News Board

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#791

Post by croes » Sat, Dec 21, 2024, 10:18 AM UTC

yawnxyz wrote: post #551

ripped_britches wrote: post #544

Sad to see everyone so focused on compute expense during this massive breakthrough. GPT-2 originally cost $50k to train, but now can be trained for ~$150. The key part is that scaling test-time compute will likely be a key to achieving AGI/ASI. Costs will definitely come down as is evidenced by precedents, Moore’s law, o3-mini being cheaper than o1 with improved performance, etc.

I think the question everyone has in their minds isn't "when will AGI get here" or even "how soon will it get here" — it's "how soon will AGI get so cheap that everyone will get their hands on it" that's why everyone's thinking about compute expense. but I guess in terms of a "lifetime expense of a person" even someone who costs $10/hr isn't actually all that cheap, considering what it takes to grow a human into a fu…

We are nowhere near AGI.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#792

Post by fragmede » Sat, Dec 21, 2024, 10:20 AM UTC

lumost wrote: post #696

duluca wrote: post #689

The first computers cost millions of dollars and filled entire rooms to accomplish what we would now consider simple computational tasks. That same computing power now fits into the width of a finger nail. I don’t get how technologists balk at the cost of experimental tech or assume current tech will run at the same efficiency for decades to come and melt the planet into a puddle. AGI won’t happen until you can fit e…

The concern here is mainly on practicality. The original mainframes did not command startup valuations counted in fractions of the US economy, they did qualify for billions in investment. This is a great milestone, but OpenAI will not be successful charging 10x the cost of a human to perform a task.

How much does AWS charge for compute?

If it can be spun up with Terraform, I bet you they could.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#793

Post by [deleted] » Sat, Dec 21, 2024, 10:22 AM UTC

[deleted]

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#794

Post by dmezzetti » Sat, Dec 21, 2024, 10:22 AM UTC

owenpalmer wrote: post #768

Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…

We should wait until it's released before we anoint it. It's disheartening to see how we keep repeating the same pattern that gives in to hype over the scientific method.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#795

Post by catmanjan » Sat, Dec 21, 2024, 10:26 AM UTC

bubblyworld wrote: post #697

dimitri-vs wrote: post #488

Earlier quoted context omitted.

Have we really watered down the definition of AGI that much? LLMs aren't really capable of "learning" anything outside their training data. Which I feel is a very basic and fundamental capability of humans. Every new request thread is a blank slate utilizing whatever context you provide for the specific task and after the tread is done (or context limit runs out) it's like it never happened. Sure you can use database…

That's true for vanilla LLMs, but also keep in mind that there are no details about o3's architecture at the moment. Clearly they are doing something different given the huge performance jump on a lot of benchmarks, and it may well involve in-context learning.

Given every other iteration has basically just been the same thing but bigger, why should we think this?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#796

Post by matusp » Sat, Dec 21, 2024, 10:28 AM UTC

bluecoconut wrote: post #141

Efficiency is now key. ~=$3400 per single task to meet human performance on this benchmark is a lot. Also it shows the bullets as "ARC-AGI-TUNED", which makes me think they did some undisclosed amount of fine-tuning (eg. via the API they showed off last week), so even more compute went into this task. We can compare this roughly to a human doing ARC-AGI puzzles, where a human will take (high variance in my subjective…

I don't think this is only about efficiency. The model I have here is that this is similar to when we beat chess. Yes, it is impressive that we made progress on a class of problems, but is this class aligned with what the economy or the society needs?

Simple turn-based games such as chess turned out to be too far away from anything practical and chess-engine-like programs were never that useful. It is entirely possible that this will end up in a similar situation. ARC-like pattern matching problems or programming challenges are indeed a respectable challenge for AI, but do we need a program that is able to solve them? How often does something like that come up really? I can see some time-saving in using AI vs StackOverflow in solving some programming challenges, but is there more to this?

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#797

Post by lazide » Sat, Dec 21, 2024, 10:32 AM UTC

dmezzetti wrote: post #794

owenpalmer wrote: post #768

Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…

We should wait until it's released before we anoint it. It's disheartening to see how we keep repeating the same pattern that gives in to hype over the scientific method.

The scientific method doesn’t drive stock price (apparently).

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#798

Post by matusp » Sat, Dec 21, 2024, 10:34 AM UTC

owenpalmer wrote: post #768

Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…

Well put. You ask LLMs about ARC-like challenges and they are able to come up with a list of possible problem formulations even before you show them the input. The models already know that they might expect various object manipulations, symmetry problem, etc. The fact that the solution costs thousands of dollars says to me that the model iterates over many solutions while using this implicit knowledge and feedback it gets from running the program. It is still impressive, but I don't think this is what the ARC prize was supposed to be about.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#799

Post by gritzko » Sat, Dec 21, 2024, 10:44 AM UTC

miga89 wrote: post #738

How do the organisers keep the private test set private? Does openAI hand them the model for testing? If they use a model API, then surely OpenAI has access to the private test set questions and can include it in the next round of training? (I am sure I am missing something.)

That is the top question, actually. Given all the billions at stake.

Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB

#800

Post by onemetwo » Sat, Dec 21, 2024, 10:47 AM UTC

owenpalmer wrote: post #768

Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…

An intelligent system could take more advantage of an increase of knowledge than a dumb one, so I should propose a simple formula: the derivative of efficiency with respect to knowledge is proportional to intelligence.

$$ I = \frac{partial E}{partial K} \simeq \frac{\delta E}{\delta K} $$

In order to estimate $I$ you have to consider that efficiency and knowledge are task related, so you could take some weighted mean $sum_T C(E,K,T)*I(E,K,T)$ where $T$ is task category. I am thinking in $C(E,K,T)$ as something similar to thermal capacity or electrical resistance, the equivalent concept when applied to task. An intelligent agent in a medium of low resistance should fly while a dumb one would still crawl.