Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I got the highest score on ARC-AGI again swapping Python for English

jeremyberman.substack.com

Post reply on HN
Print view View original

41–50 of 136 posts

Re: I got the highest score on ARC-AGI again swapping Python for English

#41

Post by karmakaze »

ozgung wrote: post #37
Akronymus wrote: post #31

Earlier quoted context omitted.

> I just don't see how you can argue with a straight face that this is "pattern matching". If that's pattern matching, then pattern matching is not an insult. IMO its still "just" a, very good, autocomplete. No actual reasoning, but lots of statistics on what is the next token to spit out.

No it's not "just a very good autocomplete". I don't know why people repeat this thing (it's wrong) but I find it an extremely counterproductive position. Some people just love to dismiss the capabilities of AI with a very shallow understanding of how it works. Why? It generates words one by one, like we all do. This doesn't mean it does just that and nothing else. It's the mechanics of how they are trained and how t…

I can't say for certain that our wetware isn't "just a very good autocomplete".

Re: I got the highest score on ARC-AGI again swapping Python for English

#43

Post by low_tech_love »

modeless wrote: post #5

I've been testing LLMs on Sokoban-like puzzles (in the style of ARC-AGI-3) and they are completely awful at them. It really highlights how poor their memory is. They can't remember abstract concepts or rules between steps, even if they discover them themselves. They can only be presented with lossy text descriptions of such things which they have to re-read and re-interpret at every step. LLMs are completely helpless…

Try to get your LLM of choice to find its way out of a labyrinth that you describe in text form. It's absolutely awful even with the simplest mazes. I'm not sure the problem here is memory, though? I think it has to do with spatial reasoning. I'd be willing to bet every company right now is working on spatial reasoning (at least up to 3D) and as soon as that is working, a huge amount of pieces will fall into place.

Re: I got the highest score on ARC-AGI again swapping Python for English

#44

Post by Bjartr »

ozgung wrote: post #37
Akronymus wrote: post #31

Earlier quoted context omitted.

> I just don't see how you can argue with a straight face that this is "pattern matching". If that's pattern matching, then pattern matching is not an insult. IMO its still "just" a, very good, autocomplete. No actual reasoning, but lots of statistics on what is the next token to spit out.

No it's not "just a very good autocomplete". I don't know why people repeat this thing (it's wrong) but I find it an extremely counterproductive position. Some people just love to dismiss the capabilities of AI with a very shallow understanding of how it works. Why? It generates words one by one, like we all do. This doesn't mean it does just that and nothing else. It's the mechanics of how they are trained and how t…

> like we all do

Do we though? Sure, we communicate sequentially, but that doesn't mean that our internal effort is piecewise and linear. A modern transformer LLM however is. Each token is sampled from a population exclusively dependent on the tokens that came before it.

Mechanistically speaking, it works similarly to autocomplete, but at a very different scale.

Now how much of an unavoidable handicap this incurs, if any, is absolutely up for debate.

But yes, taking this mechanistic truth and only considering it in a shallow manner underestimates the capability of LLMs by a large degree.

Re: I got the highest score on ARC-AGI again swapping Python for English

#45

Post by d_burfoot »

To me the reason ARC-AGI puzzles are difficult for LLMs and possible for humans is that they are expressed in a format for which humans have powerful preprocessing capabilities.

Imagine the puzzle layouts were expressed in JSON instead of as a pattern of visual blocks. How many humans could solve them in that case?

Re: I got the highest score on ARC-AGI again swapping Python for English

#46

Post by pessimizer »

0x20cowboy wrote: post #13

> LLMs are PhD-level reasoners in math and science, yet they fail at children's puzzles. How is this possible? Because they are not. Pattern matching questions on a contrived test is not the same thing as understanding or reasoning. It’s the same reason why most of the people who pass your leetcode tests don’t actually know how to build anything real. They are taught to the test not taught to reality.

> Pattern matching questions on a contrived test is not the same thing as understanding or reasoning.

Pattern matching is definitely the same thing as understanding and reasoning.

The problem is that LLMs can't recognize patterns that are longer than a few paragraphs, because the tokens would have to be far too long. LLMs are a thing we are lucky to have because we have very fast computers and very smart mathematicians making very hard calculations very efficient and parallelizable. But they sit on top of a bed of an enormous amount of human written knowledge, and can only stretch so far from that bed before completely falling apart.

Humans don't use tokenizers.

The goal right now is to build a scaffolding of these dummies in order to get really complicated work done, but that work is only ever going to accidentally be correct because of an accumulation of errors. This may be enough for a lot if we try it 1000x and run manually-tuned algos over the output to find the good ones. But this is essentially manual work, done in the traditional way.

edit: sorry, you're never going to convince me these things are geniuses when I chat to them for a couple of back and forth exchanges and they're already obviously losing track of everything, even what they just said. The good thing is that what they are is enough to do a lot, if you're a person who can be satisfied that they're not going to be your god anytime soon.

Re: I got the highest score on ARC-AGI again swapping Python for English

#47

Post by kenjackson »

Bjartr wrote: post #44
ozgung wrote: post #37

Earlier quoted context omitted.

No it's not "just a very good autocomplete". I don't know why people repeat this thing (it's wrong) but I find it an extremely counterproductive position. Some people just love to dismiss the capabilities of AI with a very shallow understanding of how it works. Why? It generates words one by one, like we all do. This doesn't mean it does just that and nothing else. It's the mechanics of how they are trained and how t…

> like we all do Do we though? Sure, we communicate sequentially, but that doesn't mean that our internal effort is piecewise and linear. A modern transformer LLM however is. Each token is sampled from a population exclusively dependent on the tokens that came before it. Mechanistically speaking, it works similarly to autocomplete, but at a very different scale. Now how much of an unavoidable handicap this incurs, if…

Our thinking is also based only on events that occurred previously in time. We don’t use events in the future.

Re: I got the highest score on ARC-AGI again swapping Python for English

#48

Post by pessimizer »

d_burfoot wrote: post #45

To me the reason ARC-AGI puzzles are difficult for LLMs and possible for humans is that they are expressed in a format for which humans have powerful preprocessing capabilities. Imagine the puzzle layouts were expressed in JSON instead of as a pattern of visual blocks. How many humans could solve them in that case?

Every one who had access to a computer that could convert json into something more readable for humans, and would know that was the first thing they needed to do?

You might as well have asked how many English speakers could solve the questions if they were in Chinese. All of them. They would call up someone who spoke Chinese, pay them to translate the questions, then solve them. Or failing that, they would go to the bookstore, buy books on learning Chinese, and solve them three years from now.

Re: I got the highest score on ARC-AGI again swapping Python for English

#49

Post by kenjackson »

d_burfoot wrote: post #45

To me the reason ARC-AGI puzzles are difficult for LLMs and possible for humans is that they are expressed in a format for which humans have powerful preprocessing capabilities. Imagine the puzzle layouts were expressed in JSON instead of as a pattern of visual blocks. How many humans could solve them in that case?

Bingo. We simply made a test for which we are well trained. We are constantly making real time decisions with our eyes. Interestingly certain monkeys are much better at certain visual pattern recognition than we are. They might laugh and think humans haven’t reached AGI yet.

Re: I got the highest score on ARC-AGI again swapping Python for English

#50

Post by jononor »

d_burfoot wrote: post #45

To me the reason ARC-AGI puzzles are difficult for LLMs and possible for humans is that they are expressed in a format for which humans have powerful preprocessing capabilities. Imagine the puzzle layouts were expressed in JSON instead of as a pattern of visual blocks. How many humans could solve them in that case?

We have powerful preprocessing blocks for images: Strong computer vision capabilities predates LLMs by several years. Image classification, segmentation, object detection, etc. All differential and trainable in same way as LLMs, including jointly. To the best of my knowledge, no team has shown really high scores by adding in a image preprocessing block?
Post reply on HN