Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The surprising effectiveness of test-time training for abstract reasoning [pdf]

mit.edu

Post reply on HN
Print view View original

21–29 of 29 posts

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#21

Post by benchmarkist »

Zondartul wrote: post #19
razodactyl wrote: post #17

Earlier quoted context omitted.

You're on track in your arguments but don't underestimate how hard the puzzles in ARC actually are. It takes a considerable amount of depth in reasoning to see and reason about the patterns / problems / solutions. Try doing a few of them by hand to see what I mean. Simulated worlds are complex enough to hide their own flaws just like LLMs are complex enough to lead us to believe they can reason when most of the time…

ARC problems are too hard for me. I'm no longer sure I'm generally intelligent.

Humans are not generally intelligent. The adjective "general" in "AGI" does not mean it is equivalent to human intelligence, it means it's above and beyond human intelligence.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#22

Post by naasking »

benchmarkist wrote: post #20
naasking wrote: post #18

Earlier quoted context omitted.

> Because it's a set of puzzles on a 2D grid. We don't live on a 2D grid so it's already on the wrong track. I don't see what this has to do with anything. Intelligence is about learning patterns and generalizing them into algorithmic understanding, where appropriate. The number of dimensions latent in the dataset is ultimately irrelevant. Humans live in a 4D world, or 3D if the holographic principle is true, and we…

Show me an LLM that is doing any of the things you mentioned and furthermore I'm willing to bet none of that will be possible after ARC is solved either. How much money would you be willing to bet?

Not sure what's so controversial, it's well known that LLMs can trivially be viewed as operating in higher dimensional space:

https://gcptips.medium.com/a-geometric-perspective-on-large-...

As for generalizing to algorithms, LLMs don't yet do this as well as humans, but they do do it:

https://arxiv.org/abs/2309.02390

Finally, there's no intrinsic reason why an AI that can reliably solve deductive problems like ARC would be limited to two dimensions.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#23

Post by naasking »

benchmarkist wrote: post #21
Zondartul wrote: post #19

Earlier quoted context omitted.

ARC problems are too hard for me. I'm no longer sure I'm generally intelligent.

Humans are not generally intelligent. The adjective "general" in "AGI" does not mean it is equivalent to human intelligence, it means it's above and beyond human intelligence.

No that's super intelligence.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#24

Post by benchmarkist »

naasking wrote: post #22
benchmarkist wrote: post #20

Earlier quoted context omitted.

Show me an LLM that is doing any of the things you mentioned and furthermore I'm willing to bet none of that will be possible after ARC is solved either. How much money would you be willing to bet?

Not sure what's so controversial, it's well known that LLMs can trivially be viewed as operating in higher dimensional space: https://gcptips.medium.com/a-geometric-perspective-on-large-... As for generalizing to algorithms, LLMs don't yet do this as well as humans, but they do do it: https://arxiv.org/abs/2309.02390 Finally, there's no intrinsic reason why an AI that can reliably solve deductive problems like ARC wo…

Then you have no reason to argue with me.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#25

Post by benchmarkist »

naasking wrote: post #23
benchmarkist wrote: post #21

Earlier quoted context omitted.

Humans are not generally intelligent. The adjective "general" in "AGI" does not mean it is equivalent to human intelligence, it means it's above and beyond human intelligence.

No that's super intelligence.

Meaningless distinction.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#26

Post by naasking »

benchmarkist wrote: post #24
naasking wrote: post #22

Earlier quoted context omitted.

Not sure what's so controversial, it's well known that LLMs can trivially be viewed as operating in higher dimensional space: https://gcptips.medium.com/a-geometric-perspective-on-large-... As for generalizing to algorithms, LLMs don't yet do this as well as humans, but they do do it: https://arxiv.org/abs/2309.02390 Finally, there's no intrinsic reason why an AI that can reliably solve deductive problems like ARC wo…

Then you have no reason to argue with me.

The only position I took issue with, and still do, is my closing paragraph of my last post. Your argument for why ARC solvers wouldn't generalize doesn't even make sense.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#27

Post by naasking »

benchmarkist wrote: post #25
naasking wrote: post #23

Earlier quoted context omitted.

No that's super intelligence.

Meaningless distinction.

Not at all. Humans are fundamentally limited by our finite statespace and bandwidth. Classifying systems that are able to generalize at least as well as a human but that can exceed those limits as superintelligent is a meaningful distinction.

I agree that "equivalent to human intelligence" is not a robust way to define general intelligence, but humans are a general intelligence.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#28

Post by benchmarkist »

naasking wrote: post #26
benchmarkist wrote: post #24

Earlier quoted context omitted.

Then you have no reason to argue with me.

The only position I took issue with, and still do, is my closing paragraph of my last post. Your argument for why ARC solvers wouldn't generalize doesn't even make sense.

No point in arguing. If you think it will generalize then there is no reason to convince random people on the internet that ARC-AGI solver will get you closer to AGI.

Re: The surprising effectiveness of test-time training for abstract reasoning [pdf]

#29

Post by aithrowawaycomm »

benchmarkist wrote: post #21
Zondartul wrote: post #19

Earlier quoted context omitted.

ARC problems are too hard for me. I'm no longer sure I'm generally intelligent.

Humans are not generally intelligent. The adjective "general" in "AGI" does not mean it is equivalent to human intelligence, it means it's above and beyond human intelligence.

I think “general” should be taken to mean “has an average human child’s common sense and causal reasoning,” since common sense and causal reasoning are at some level shared by all vertebrates. It seems like the focus on “above and beyond human intelligence” is how you get AIs which appear to understand algebraic topology, yet utterly fail at counting problems designed for pigeons. It should be scientific malpractice to compare an AI to human intelligence without making any effort to compare it to rat/etc intelligence. (I guess investors wouldn’t lie happy if Sam Altman said “in 20 years I believe we’ll reach ARI.”)

In general tech folks are far too beholden to an instinctual and unscientific idea of intelligence as compared between humans, which mostly uses linguistic ability and surface knowledge as a proxy. This proxy might sometimes be useful in human group decision-making, but it is also how dumb confident people manage to fail upwards, and it works about as well for a computer as it does a rat (though it mismeasures in the opposite direction).

Post reply on HN