Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The surprising effectiveness of test-time training for abstract reasoning [pdf]

mit.edu

Post reply on HN
Print view View original

11–20 of 29 posts

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#11

Post by naasking »

benchmarkist wrote: post #9
aithrowawaycomm wrote: post #5

Earlier quoted context omitted.

I am a bit uncertain about the rules of the ARC-AGI contest, but would this program count? A good chunk of the logic of ARC is essentially hardcoded, including a Python function that checks whether or not the proposed solution makes sense. The point of the contest is to measure intelligence in general-purpose AI systems: it does not seem in the spirit of the contest that this AI would completely fail if the test was…

The contest is misnamed, solving ARC will not get us any closer to AGI.

Why?

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#12

Post by antonvs »

aithrowawaycomm wrote: post #10
0x1064 wrote: post #7

Earlier quoted context omitted.

The point in the contest is to measure an algorithms ability to solve ARC problems specifically, no one believes that it's general-purpose AI. They're highly contrived problems by design.

My point is that the contest really should be "can solve ARC problems without having anything about ARC problems in its pre-training data or hard-coded in the design of the program." Otherwise these claims from ARC-AGI are simply false: Solving ARC-AGI represents a material stepping stone toward AGI. At minimum, solving ARC-AGI would result in a new programming paradigm. It would allow anyone, even those without prog…

> Otherwise these claims from ARC-AGI are simply false:

Currently, any claim about AGI other than "we're probably not anywhere close to strong AGI" is simply false.

Of course a lot depends on one's definition of AGI. From another perspective, one could argue that ChatGPT 4 and similar models are already AGI.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#14

Post by benchmarkist »

naasking wrote: post #11
benchmarkist wrote: post #9

Earlier quoted context omitted.

The contest is misnamed, solving ARC will not get us any closer to AGI.

Why?

Because it's a set of puzzles on a 2D grid. We don't live on a 2D grid so it's already on the wrong track. A set of puzzles for a 3D sphere wouldn't get us any closer to AGI either but at least it would be a more realistic representation of the world and how a general purpose problem solver should approach reality. Even Minecraft would be a better test and lately people have started testing LLMs in virtual worlds which is a much better test case than ARC.

Insofar as ARC is being used as a benchmark for code synthesis it might be somewhat successful but it doesn't seem like people are using code synthesis to solve the puzzles so it's not really clear how much success on ARC is going to advance the state of the art in AI and code synthesis according to a logical specification.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#15

Post by [deleted] »

benchmarkist wrote: post #14
naasking wrote: post #11

Earlier quoted context omitted.

Why?

Because it's a set of puzzles on a 2D grid. We don't live on a 2D grid so it's already on the wrong track. A set of puzzles for a 3D sphere wouldn't get us any closer to AGI either but at least it would be a more realistic representation of the world and how a general purpose problem solver should approach reality. Even Minecraft would be a better test and lately people have started testing LLMs in virtual worlds whi…

[deleted]

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#16

Post by razodactyl »

aithrowawaycomm wrote: post #5
mikeknoop wrote: post #2

Context: ARC Prize 2024 just wrapped up yesterday. ARC Prize's goal is to be a north star towards AGI. The two major categories of this year's progress seem to fall into "program synthesis" and "test-time fine tuning". Both of these techniques are adopted by DeepMind's impressive AlphaProof system [1]. And I'm personally excited to finally see actual code implementation of these ideas [2]! We still have a long way to…

I am a bit uncertain about the rules of the ARC-AGI contest, but would this program count? A good chunk of the logic of ARC is essentially hardcoded, including a Python function that checks whether or not the proposed solution makes sense. The point of the contest is to measure intelligence in general-purpose AI systems: it does not seem in the spirit of the contest that this AI would completely fail if the test was…

Majority of ARC can be gamed / hard-coded, no doubt about it.

The real pressure is the private hold-out set and the variations that can be added to counter this aspect.

A true AGI would be able to solve anything thrown at it which is where the authors are trying to lead AI engineering towards since LLMs have pretty much taken over.

If it starts getting too easy, they just reconsider and add harder problems.

It's like how we don't talk about the Turing Test anymore as it's no longer the best metric to determine real intelligence.

The authors are signalling to the industry that new ideas are needed and the monetary aspect is to show how serious they are about it.

It's good because as per above we have research being thrown at it which means we can iterate until we perhaps find another breakthrough.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#17

Post by razodactyl »

benchmarkist wrote: post #14
naasking wrote: post #11

Earlier quoted context omitted.

Why?

Because it's a set of puzzles on a 2D grid. We don't live on a 2D grid so it's already on the wrong track. A set of puzzles for a 3D sphere wouldn't get us any closer to AGI either but at least it would be a more realistic representation of the world and how a general purpose problem solver should approach reality. Even Minecraft would be a better test and lately people have started testing LLMs in virtual worlds whi…

You're on track in your arguments but don't underestimate how hard the puzzles in ARC actually are.

It takes a considerable amount of depth in reasoning to see and reason about the patterns / problems / solutions.

Try doing a few of them by hand to see what I mean.

Simulated worlds are complex enough to hide their own flaws just like LLMs are complex enough to lead us to believe they can reason when most of the time they are pattern matching.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#18

Post by naasking »

benchmarkist wrote: post #14
naasking wrote: post #11

Earlier quoted context omitted.

Why?

Because it's a set of puzzles on a 2D grid. We don't live on a 2D grid so it's already on the wrong track. A set of puzzles for a 3D sphere wouldn't get us any closer to AGI either but at least it would be a more realistic representation of the world and how a general purpose problem solver should approach reality. Even Minecraft would be a better test and lately people have started testing LLMs in virtual worlds whi…

> Because it's a set of puzzles on a 2D grid. We don't live on a 2D grid so it's already on the wrong track.

I don't see what this has to do with anything. Intelligence is about learning patterns and generalizing them into algorithmic understanding, where appropriate. The number of dimensions latent in the dataset is ultimately irrelevant. Humans live in a 4D world, or 3D if the holographic principle is true, and we regularly deal with mathematics 27 or more dimensions. LLMs build models with at least hundreds of thousands of dimensions.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#19

Post by Zondartul »

razodactyl wrote: post #17
benchmarkist wrote: post #14

Earlier quoted context omitted.

Because it's a set of puzzles on a 2D grid. We don't live on a 2D grid so it's already on the wrong track. A set of puzzles for a 3D sphere wouldn't get us any closer to AGI either but at least it would be a more realistic representation of the world and how a general purpose problem solver should approach reality. Even Minecraft would be a better test and lately people have started testing LLMs in virtual worlds whi…

You're on track in your arguments but don't underestimate how hard the puzzles in ARC actually are. It takes a considerable amount of depth in reasoning to see and reason about the patterns / problems / solutions. Try doing a few of them by hand to see what I mean. Simulated worlds are complex enough to hide their own flaws just like LLMs are complex enough to lead us to believe they can reason when most of the time…

ARC problems are too hard for me. I'm no longer sure I'm generally intelligent.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#20

Post by benchmarkist »

naasking wrote: post #18
benchmarkist wrote: post #14

Earlier quoted context omitted.

Because it's a set of puzzles on a 2D grid. We don't live on a 2D grid so it's already on the wrong track. A set of puzzles for a 3D sphere wouldn't get us any closer to AGI either but at least it would be a more realistic representation of the world and how a general purpose problem solver should approach reality. Even Minecraft would be a better test and lately people have started testing LLMs in virtual worlds whi…

> Because it's a set of puzzles on a 2D grid. We don't live on a 2D grid so it's already on the wrong track. I don't see what this has to do with anything. Intelligence is about learning patterns and generalizing them into algorithmic understanding, where appropriate. The number of dimensions latent in the dataset is ultimately irrelevant. Humans live in a 4D world, or 3D if the holographic principle is true, and we…

Show me an LLM that is doing any of the things you mentioned and furthermore I'm willing to bet none of that will be possible after ARC is solved either. How much money would you be willing to bet?
Post reply on HN