The surprising effectiveness of test-time training for abstract reasoning [pdf]
1–10 of 29 posts
Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]
#2Context: ARC Prize 2024 just wrapped up yesterday. ARC Prize's goal is to be a north star towards AGI. The two major categories of this year's progress seem to fall into "program synthesis" and "test-time fine tuning". Both of these techniques are adopted by DeepMind's impressive AlphaProof system [1]. And I'm personally excited to finally see actual code implementation of these ideas [2]!
We still have a long way to go for the grand prize -- we'll be back next year. Also got some new stuff in the works for 2025.
Watch for the official ARC Prize 2024 paper coming Dec 6. We're going to be overviewing all the new AI reasoning code and approaches open sourced via the competition [3].
[1] https://deepmind.google/discover/blog/ai-solves-imo-problems...
Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]
#3I initially read that as "Tea-Time" training and my inner Brit got a little excited..
Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]
#4Test-Time Training is incredibly powerful. Most recently, it has been shown that Self-Attention can in fact be viewed through the lens of test-time training, with a kernel-smoother "learning" from context. Simply replacing that with more powerful models than a kernel-smoother result in very capable and scalable models!
Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]
#5Context: ARC Prize 2024 just wrapped up yesterday. ARC Prize's goal is to be a north star towards AGI. The two major categories of this year's progress seem to fall into "program synthesis" and "test-time fine tuning". Both of these techniques are adopted by DeepMind's impressive AlphaProof system [1]. And I'm personally excited to finally see actual code implementation of these ideas [2]! We still have a long way to…
I am a bit uncertain about the rules of the ARC-AGI contest, but would this program count? A good chunk of the logic of ARC is essentially hardcoded, including a Python function that checks whether or not the proposed solution makes sense.
The point of the contest is to measure intelligence in general-purpose AI systems: it does not seem in the spirit of the contest that this AI would completely fail if the test was presented on a hexagonal grid.
Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]
#6Is test time the same thing as inference time?
Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]
#7Context: ARC Prize 2024 just wrapped up yesterday. ARC Prize's goal is to be a north star towards AGI. The two major categories of this year's progress seem to fall into "program synthesis" and "test-time fine tuning". Both of these techniques are adopted by DeepMind's impressive AlphaProof system [1]. And I'm personally excited to finally see actual code implementation of these ideas [2]! We still have a long way to…
I am a bit uncertain about the rules of the ARC-AGI contest, but would this program count? A good chunk of the logic of ARC is essentially hardcoded, including a Python function that checks whether or not the proposed solution makes sense. The point of the contest is to measure intelligence in general-purpose AI systems: it does not seem in the spirit of the contest that this AI would completely fail if the test was…
The point in the contest is to measure an algorithms ability to solve ARC problems specifically, no one believes that it's general-purpose AI. They're highly contrived problems by design.
Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]
#8Is test time the same thing as inference time?
yes
Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]
#9Context: ARC Prize 2024 just wrapped up yesterday. ARC Prize's goal is to be a north star towards AGI. The two major categories of this year's progress seem to fall into "program synthesis" and "test-time fine tuning". Both of these techniques are adopted by DeepMind's impressive AlphaProof system [1]. And I'm personally excited to finally see actual code implementation of these ideas [2]! We still have a long way to…
I am a bit uncertain about the rules of the ARC-AGI contest, but would this program count? A good chunk of the logic of ARC is essentially hardcoded, including a Python function that checks whether or not the proposed solution makes sense. The point of the contest is to measure intelligence in general-purpose AI systems: it does not seem in the spirit of the contest that this AI would completely fail if the test was…
The contest is misnamed, solving ARC will not get us any closer to AGI.
Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]
#10Earlier quoted context omitted.
I am a bit uncertain about the rules of the ARC-AGI contest, but would this program count? A good chunk of the logic of ARC is essentially hardcoded, including a Python function that checks whether or not the proposed solution makes sense. The point of the contest is to measure intelligence in general-purpose AI systems: it does not seem in the spirit of the contest that this AI would completely fail if the test was…
The point in the contest is to measure an algorithms ability to solve ARC problems specifically, no one believes that it's general-purpose AI. They're highly contrived problems by design.
My point is that the contest really should be "can solve ARC problems without having anything about ARC problems in its pre-training data or hard-coded in the design of the program." Otherwise these claims from ARC-AGI are simply false:
Solving ARC-AGI represents a material stepping stone toward AGI. At minimum, solving ARC-AGI would result in a new programming paradigm. It would allow anyone, even those without programming knowledge, to create programs simply by providing a few input-output examples of what they want.
This would dramatically expand who is able to leverage software and automation. Programs could automatically refine themselves when exposed to new data, similar to how humans learn.
If found, a solution to ARC-AGI would be more impactful than the discovery of the Transformer. The solution would open up a new branch of technology.