Why the Abstraction and Reasoning Corpus is interesting and important for AI
aiguide.substack.com
Why the Abstraction and Reasoning Corpus is interesting and important for AI
1–10 of 17 posts
Re: Why the Abstraction and Reasoning Corpus is interesting and important for AI
#2Related:
ARC 2 — https://news.ycombinator.com/item?id=32740353
The Measure of Intelligence — https://news.ycombinator.com/item?id=21547958
Neural Abstract Reasoner — https://news.ycombinator.com/item?id=25167182
A first lesson in meta-rationality (Bongard problems) — https://news.ycombinator.com/item?id=27411960
A HN dicussion about ARC — https://news.ycombinator.com/item?id=29867200
Re: Why the Abstraction and Reasoning Corpus is interesting and important for AI
#3Why on earth would you tell the model the task in english? The tasks have extremely simple JSON representations that would make for a way, way more natural input. The colorful grids are just to make it easier (for humans) to visualize the symmetries needed. Eg: Here is an (abbreviated) train/test example for task 007bbfb7 from https://github.com/fchollet/ARC/blob/master/data/training/00...
"train": [{
"input": [[0, 7, 7], [7, 7, 7], [0, 7, 7]],
"output": [[0, 0, 0, 0, 7, 7, 0, 7, 7], [0, 0, 0, 7, 7, 7, 7, 7, 7], [0, 0, 0, 0, 7, 7...
"test": [{
"input": [[7, 0, 7], [7, 0, 7], [7, 7, 0]],
"output": [[7, 0, 7, 0, 0, 0, 7, 0, 7], [7, 0, 7, 0, 0, 0, 7, 0, 7], [7, 7, 0, 0, 0...
Re: Why the Abstraction and Reasoning Corpus is interesting and important for AI
#4I would argue maybe that the author's premise depends on some faulty prompts.
With some gentle rephrasing, ChatGPT correctly interprets and manipulates the stack problem. I've observed ChatGPT may have some issues with non-standard sentence structure involving, especially with colons.
> Imagine a stack of items (from bottom to top) with a cat, laptop, television, and apple. The apple is moved between the laptop and the television. Which item is on top of the stack afterwards?
The author seems also to have misinterpreted that Stable Diffusion is the same sequence-sensitive model as Transformer LLMs, which it is not. While it is generative it has minimal capacity for abstraction, since the focus would rather be minimizing the "kernel" representation of each word in the weights.
DALL-E has a bit more luck, but requires much more verbose description to get anything like a screen on top of a cat, but that's presumably more due to the bias in the data set of things being displayed on screens, or cats getting on top of things.
Something like
> A cartoon image of a television above a cartoon image of a cat
https://imgur.com/q9NMY0W Illustrates that it's preference rather than capability.
Re: Why the Abstraction and Reasoning Corpus is interesting and important for AI
#5Why on earth would you tell the model the task in english? The tasks have extremely simple JSON representations that would make for a way, way more natural input. The colorful grids are just to make it easier (for humans) to visualize the symmetries needed. Eg: Here is an (abbreviated) train/test example for task 007bbfb7 from https://github.com/fchollet/ARC/blob/master/data/training/00... "train": [{ "input": [[0, 7…
> imagine it would be pretty easy to fine tune a davinci model on these and then test, no? Has anyone done it?
That was the a competition. Nobody scored well.
Re: Why the Abstraction and Reasoning Corpus is interesting and important for AI
#6Why on earth would you tell the model the task in english? The tasks have extremely simple JSON representations that would make for a way, way more natural input. The colorful grids are just to make it easier (for humans) to visualize the symmetries needed. Eg: Here is an (abbreviated) train/test example for task 007bbfb7 from https://github.com/fchollet/ARC/blob/master/data/training/00... "train": [{ "input": [[0, 7…
> imagine it would be pretty easy to fine tune a davinci model on these and then test, no? Has anyone done it? That was the a competition. Nobody scored well.
The competition was three years ago though.
Re: Why the Abstraction and Reasoning Corpus is interesting and important for AI
#7I would argue maybe that the author's premise depends on some faulty prompts. With some gentle rephrasing, ChatGPT correctly interprets and manipulates the stack problem. I've observed ChatGPT may have some issues with non-standard sentence structure involving, especially with colons. > Imagine a stack of items (from bottom to top) with a cat, laptop, television, and apple. The apple is moved between the laptop and t…
The author's prompts are all perfectly understandable to humans and expressed in a direct and clear way, and the AIs can't understand them.
This comment essentially says "the prompt is faulty because the AI didn't answer it correctly". This bakes the correctness of the AI into the tautology that it always answers "good" prompts, which are defined as those it answers correctly.
Re: Why the Abstraction and Reasoning Corpus is interesting and important for AI
#8The author mentions that Chollet's ARC tasks contain 200 unreleased "hidden" problems. This set off a light bulb for me: hidden problems are going to be increasingly important for benchmarking AI meaningfully. With RLHF, ChatGPT is memorizing the answer to every problem thrown at it. Eventually it may become impossible to test with any public question bank.
We are going to go through a period of fooling ourselves, as we always do, then we're going to hit a wall where models are preforming perfectly on benchmarks while still clearly lacking in genuine intelligence. And then we are going to need to test with a concern for security as serious as that used for the gaokao or SAT. Benchmarking will be a paid service performed under highly controlled conditions.
Re: Why the Abstraction and Reasoning Corpus is interesting and important for AI
#9I would argue maybe that the author's premise depends on some faulty prompts. With some gentle rephrasing, ChatGPT correctly interprets and manipulates the stack problem. I've observed ChatGPT may have some issues with non-standard sentence structure involving, especially with colons. > Imagine a stack of items (from bottom to top) with a cat, laptop, television, and apple. The apple is moved between the laptop and t…
The author's prompts are all perfectly understandable to humans and expressed in a direct and clear way, and the AIs can't understand them. This comment essentially says "the prompt is faulty because the AI didn't answer it correctly". This bakes the correctness of the AI into the tautology that it always answers "good" prompts, which are defined as those it answers correctly.
Well, humans don't know how to determine if a proposition is correct.
So it's no surprise we didn't teach AIs yet!
Edit: I don't understand why experts say it failed when they "asked it to reason". It doesn't have pictures of TVs on cats to produce variations of, right? How does "reason" enter into it?
Re: Why the Abstraction and Reasoning Corpus is interesting and important for AI
#10I would argue maybe that the author's premise depends on some faulty prompts. With some gentle rephrasing, ChatGPT correctly interprets and manipulates the stack problem. I've observed ChatGPT may have some issues with non-standard sentence structure involving, especially with colons. > Imagine a stack of items (from bottom to top) with a cat, laptop, television, and apple. The apple is moved between the laptop and t…
The author's prompts are all perfectly understandable to humans and expressed in a direct and clear way, and the AIs can't understand them. This comment essentially says "the prompt is faulty because the AI didn't answer it correctly". This bakes the correctness of the AI into the tautology that it always answers "good" prompts, which are defined as those it answers correctly.
Right but the author's premise is that the AI fails to reason or extend ideas, not that the AI fails to reason or extend ideas in the presence of extraneous, conflicting, or malformed instructions.
If you're prompting ChatGPT (which is already loaded with prompt-engineering cheez-whiz mind you) to reason as a computer, why use words like "imagine"? why phrase things as a question? I think the fragility the author is observing here is extraneous to the actual underlying technology, and a factor of the UI that's been stacked-on and purpose-built for softer-purpose use, not hard academic research such as this.
Case study, even GPT2 can reason about the stack https://transformer.huggingface.co/share/bQXbmeyBZD