Can someone ELI5 how ARC-AGI-PUB is resistant to p-hacking?
OpenAI O3 breakthrough high score on ARC-AGI-PUB
801–810 of 1001 posts
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#802Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…
Well put. You ask LLMs about ARC-like challenges and they are able to come up with a list of possible problem formulations even before you show them the input. The models already know that they might expect various object manipulations, symmetry problem, etc. The fact that the solution costs thousands of dollars says to me that the model iterates over many solutions while using this implicit knowledge and feedback it…
> while using this implicit knowledge and feedback it gets from running the program.
What feedback, and what program, are you referring to?
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#803Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…
There should be also a factor about resource consumption. See here: https://lorenzopieri.com/pgii/
An interesting point from a philosophical perspective!
But if we'd take this into consideration would it mean that 1st world engineer is by definition less inteligent than 3rd world one?
I think the (completely reasonable) knee jerk reaction is a definsive one, but I can imagine absolutarian regime escapee working side-by-side an engineer groomed in expensive, air conditioned lecture rooms. In this imaginary scenario escapee, even if slower and less efficient at the problem at hand would have to be more intelligent generally.
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#804Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…
Where did someone ask that?
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#805I’m confused about the excitement. Are people just flat out ignoring the sentences below? I don’t see any breakthrough towards AGI here. I see a model doing great in another AI test but about to abysmally fail a variation of it that will come out soon. Also, aren’t these comparisons completely nonsense considering it’s o3 tuned vs other non-tuned? > Note on "tuned": OpenAI shared they trained the o3 we tested on 75%…
Me too. This looks to me like a holiday PR stunt. Get everybody to talk about AI during the Christmas parties.
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#806Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…
As a kid I absolutely hated math and loved physics and chemistry because solving anything in math requires vast specific K.
In comparison you can easily know everything there is to know about physics or chemistry and it's sufficient to solve interesting puzzles. In math every puzzle has it's own vast lore you need to know before you can have any chance at tackling it.
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#807Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…
An intelligent system could take more advantage of an increase of knowledge than a dumb one, so I should propose a simple formula: the derivative of efficiency with respect to knowledge is proportional to intelligence. $$ I = \frac{partial E}{partial K} \simeq \frac{\delta E}{\delta K} $$ In order to estimate $I$ you have to consider that efficiency and knowledge are task related, so you could take some weighted mean…
> An intelligent system could take more advantage of an increase of knowledge than a dumb one
Why?
> derivative of efficiency
Where did your efficiency variable come from?
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#808Someone asked if true intelligence requires a foundation of prior knowledge. This is the way I think about it. I = E / K where I is the intelligence of the system, E is the effectiveness of the system, and K is the prior knowledge. For example, a math problem is given to two students, each solving the problem with the same effectiveness (both get the correct answer in the same amount of time). However, student A happ…
As a kid I absolutely hated math and loved physics and chemistry because solving anything in math requires vast specific K. In comparison you can easily know everything there is to know about physics or chemistry and it's sufficient to solve interesting puzzles. In math every puzzle has it's own vast lore you need to know before you can have any chance at tackling it.
Physics and chemistry require experimentation to verify solutions. With math however, any new knowledge can be intuited and proven from previous proofs, so yes, the lore goes deep!
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#809How do the organisers keep the private test set private? Does openAI hand them the model for testing? If they use a model API, then surely OpenAI has access to the private test set questions and can include it in the next round of training? (I am sure I am missing something.)
If we really want to imagine a cold-war-style solution, the two teams could meet in an empty warehouse, bring one computer with the model, one with the benchmarks, and connect them with a USB cable.
In practice I assume they just gave them the benchmarks and took it on the honor system they wouldn't cheat, yeah. They can always cook up a new test set for next time, it's only 10% of the benchmark content anyway and the results are pretty close.
Re: OpenAI O3 breakthrough high score on ARC-AGI-PUB
#810[deleted]