Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arc Prize 2024 Winners and Technical Report

arcprize.org

Post reply on HN
Print view View original

21–30 of 59 posts

Re: Arc Prize 2024 Winners and Technical Report

#21

Post by szvsw »

aithrowawaycomm wrote: post #16
YeGoblynQueenne wrote: post #6

The first question I still have is what happened to core knowledge priors. The white paper that introduced ARC made a big todo about how core knowledge priors are necessary to solve ARC tasks but from what I can tell none of the best-performing (or at-all performing) systems have anything to do with core knowlege priors. So what happened to that assumption? Is it dead? The second question I still have is about the de…

Even the strongest possible interpretation of the results wouldn't conclude "ARC-AGI is dead" because none of the submissions came especially close to human-level performance; the criteria was 85% success but the best in 2024 was 55%. That said, I think there should be consideration via information thermodynamics: even with TTT these program-generating systems are using an enormous amount of bits compared to a human…

> That said, I think there should be consideration via information thermodynamics: even with TTT these program-generating systems are using an enormous amount of bits compared to a human mind, a tiny portion of which solves ARC quickly and easily using causality-first principles of reasoning.

This isn’t my area of expertise, but it seems plausible to me that what you said is completely erroneous or at the very least completely unverifiable at this point in time. How do you quantify how many bits it takes a human mind to solve one of the ARC problems?

That seems likely beyond the level of insight we have into the structure of cognition and information storage etc etc in wetware. I could of course be wrong and would love to be corrected if so! You mentioned a “tiny portion” of the human mind, but (as far as I’m aware), any given “small” part of human cognition still involves huge amounts of complexity and compute.

Maybe you are saying that the high level decision making a human goes through when solving can be represented with a relatively small number of pieces of information/logical operations (as opposed to a much lower level notion closer to the wetware of the quantity of information) but then it seems unfair to compare to the low level equivalent (weights & biases, FLOPs etc) in the ML system when there may be higher order equivalents.

I do appreciate the general notion of wanting to normalize against something though, and some notion of information seems like a reasonable choice, but practically out of our reach. Maybe something like peak power or total energy consumption would be a more reasonable choice, which we can at least get a lower and upper bounds on in the human case (metabolic rates are pretty well studied, and even if we don’t have a good idea of how much energy is involved in completing cognitive tasks we can at least get bounds for running the entire system in that period of time) and close to a precise value in the ML case.

Re: Arc Prize 2024 Winners and Technical Report

#22

Post by mikeknoop »

padswo1 wrote: post #17
mikeknoop wrote: post #2

Author here -- six months ago we launched ARC Prize, a huge $1M experiment, to test if we need new ideas for AGI. The ARC-AGI benchmark remains unbeaten and I think we can now definitely say "yes". One big update since June is that progress is no longer stalled. Coming into 2024, the public consensus vibe was that pure deep learning / LLMs would continue scaling to AGI. The fundamental architecture of these systems h…

I don’t think ARC has particularly advanced the research. The approaches that are successful were developed elsewhere and then applied to ARC. Happy to be shown somewhere this is not the case. In the case of TTT, I wouldn’t really describe that as a ‘new AGI reasoning approach’. People have been fine tuning deep learning models on specific tasks for a long time. The fundamental instinct driving the creation of ARC -…

Correct, fine-tuning is not new. It's long been used to augment foundational LLMs with private data. Eg. private enterprise data. We do this at Zapier, for instance.

The new and surprising thing about test-time training (TTT) is how effective it is an approach to deal with novel abstract reasoning problems like ARC-AGI.

TTT was pioneered by Jack Cole last year and popularized this year by several teams, including this winning paper: https://ekinakyurek.github.io/papers/ttt.pdf

Re: Arc Prize 2024 Winners and Technical Report

#23

Post by tuukkah »

optimalsolver wrote: post #13
fchollet wrote: post #8

Earlier quoted context omitted.

What all top models do is recombine at test time the knowledge they already have. So they all possess Core Knowledge priors. Techniques to acquire them vary: * Use a pretrained LLM and hope that relevant programs will be memorized via exposure to text data (this doesn't work that well) * Pretrain a LLM on ARC-AGI-like data * Hardcode the priors into a DSL > Which is to say, a data augmentation approach The key bit is…

>This demonstrates empirically that ARC-AGI cannot be solved purely via memorization and interpolation Now that the current challenge is over, and a successor dataset is in the works, can we see how well the leading LLMs perform against the private test set?

I think the "semi-private" numbers here already measure that: https://arcprize.org/2024-results

For example, Claude 3.5 gets 14% in semi-private eval vs 21% in public eval. I remember reading an explanation of "semi-private" earlier but cannot find it now.

Re: Arc Prize 2024 Winners and Technical Report

#25

Post by mikeknoop »

tbalsam wrote: post #20
mikeknoop wrote: post #18

Earlier quoted context omitted.

> I'd heartily recommend maybe taking down the marketing vibrance down a notch and keep things a bit more measured, it's not entirely a meme, though some of the more-serious researchers don't take it as seriously as a result. This is fair critique. ARC Prize's 2024 messaging was sharp to break through the noise floor -- ARC has been around since 2019 but most only learned about it this summer. Now that it has garnere…

I feel rather consternated that this response effectively boils down to "yes, we know we overhyped this to get people's attention, and now that we have it we can be more honest about it". Fighting for place in the attention economy is understandable, being deceptive about it is not. This is part of the ethical morass of why some more serious researchers aren't touching the benchmark. People are not going to take it s…

I think we agree; to clarify, sharp messaging isn't inaccurate messaging. And I believe the story is not overhyped given the evidence: the benchmark resisted a $1M prize pool for ~6 months. But I concede we did obsess about the story to give it the best chance of survival in the marketplace of ideas against the incumbent AI research meme (LLM scaling). Now that the AI research field is coming around to the idea that something beyond deep learning is needed, the story matters less, and the benchmark, and future versions, can stand on their utility as a compass towards AGI.

Re: Arc Prize 2024 Winners and Technical Report

#26

Post by iwsk »

tbalsam wrote: post #20
mikeknoop wrote: post #18

Earlier quoted context omitted.

> I'd heartily recommend maybe taking down the marketing vibrance down a notch and keep things a bit more measured, it's not entirely a meme, though some of the more-serious researchers don't take it as seriously as a result. This is fair critique. ARC Prize's 2024 messaging was sharp to break through the noise floor -- ARC has been around since 2019 but most only learned about it this summer. Now that it has garnere…

I feel rather consternated that this response effectively boils down to "yes, we know we overhyped this to get people's attention, and now that we have it we can be more honest about it". Fighting for place in the attention economy is understandable, being deceptive about it is not. This is part of the ethical morass of why some more serious researchers aren't touching the benchmark. People are not going to take it s…

we live in a society

Re: Arc Prize 2024 Winners and Technical Report

#27

Post by iwsk »

mrandish wrote: post #15
celeritascelery wrote: post #3

What surprises me about this is how poorly general-purpose LLMs do. The best one is OpenAI o1-preview at 18%. This is significantly worse than the purpose-built models like ARChitects (which scored 53.5). This model used TTT to train on the ARC-AGI task specification (amoung other things). It seems that even if someone creates a model that can "solve" ARC, it still is not indicative of AGI since it is not "general" a…

> It seems that even if someone creates a model that can "solve" ARC, it still is not indicative of AGI since it is not "general" anymore I recently explained why I like ARC to a non-technical friend this way: "When an AI solves ARC it won't be proof of AGI. It's the opposite. As long as ARC remains unsolved I'm confident we're not even close to AGI." For the sake of being provocative, I'd even argue that ARC remaini…

in other words, solving ARC is necessary but not sufficient for AGI

Re: Arc Prize 2024 Winners and Technical Report

#28

Post by tbalsam »

mikeknoop wrote: post #25
tbalsam wrote: post #20

Earlier quoted context omitted.

I feel rather consternated that this response effectively boils down to "yes, we know we overhyped this to get people's attention, and now that we have it we can be more honest about it". Fighting for place in the attention economy is understandable, being deceptive about it is not. This is part of the ethical morass of why some more serious researchers aren't touching the benchmark. People are not going to take it s…

I think we agree; to clarify, sharp messaging isn't inaccurate messaging. And I believe the story is not overhyped given the evidence: the benchmark resisted a $1M prize pool for ~6 months. But I concede we did obsess about the story to give it the best chance of survival in the marketplace of ideas against the incumbent AI research meme (LLM scaling). Now that the AI research field is coming around to the idea that…

> Now that the AI research field is coming around to the idea that something beyond deep learning is needed,

I have not heard this from anyone that I work with! It would be a curious violation of info theory were this to be the case.

Certainly, some things cannot efficiently be learned from data. This is a case where some other kind of inductive bias or prior is needed (again, from info theory) -- but replacing deep learning entirely would be rather silly.

Part of the reason that a number of researchers don't take the benchmark more seriously is because it's meant to cripple the results. For example, in the name of reducing brute force search, the compute was severely limited! This turned many off to begin with. The general contention as I understand was to let compute be a reasonable amount, but this would not play well with the numbers game. Because if you restrict compute beyond a reasonable point, it makes the numbers artificially low for people who don't know what's going on behind the scenes. And this ends up biasing the results unreasonably to favor the original messaging, (i.e., "We need something other than deep learning.")

If it was structured with a reasonable amount of compute, and instead, time-accuracy gates were used for prizes, it would be much more open. But people do not use it because the game is rigged to begin with!

Unfortunately due to that, plus the consistent goal-post moving of the benchmark is why it's generally not really held with staying power in the research community -- the messaging changes based upon what is convenient for publicity, and there's unfortunately been a history of similar things in the past in the pedigree leading up to the ARC prize itself.

It is not entirely unsalvageable, but there really needs to be a turnaround of how the competition and prize is managed in order to win back people's trust. Placing a thumb on the scales to confirm a prior bias/previous messaging may work for a little while, but over time it robs the metric of its usability over time as the greater research community loses trust.

Re: Arc Prize 2024 Winners and Technical Report

#29

Post by aithrowawaycomm »

szvsw wrote: post #21
aithrowawaycomm wrote: post #16

Earlier quoted context omitted.

Even the strongest possible interpretation of the results wouldn't conclude "ARC-AGI is dead" because none of the submissions came especially close to human-level performance; the criteria was 85% success but the best in 2024 was 55%. That said, I think there should be consideration via information thermodynamics: even with TTT these program-generating systems are using an enormous amount of bits compared to a human…

> That said, I think there should be consideration via information thermodynamics: even with TTT these program-generating systems are using an enormous amount of bits compared to a human mind, a tiny portion of which solves ARC quickly and easily using causality-first principles of reasoning. This isn’t my area of expertise, but it seems plausible to me that what you said is completely erroneous or at the very least…

I was speaking loosely but the operative term is "information thermodynamics": comparing bits of AI output versus bits of intentional human thought, ignoring statistical/physical bits related to ANN inference or biological neuron activity. The "tiny chunk of the human mind" thing was a distraction I shouldn't have included.

These AI output as tokens hundreds of potential solutions, whereas a human solving a very tricky ARC problem might need at most a few dozen cases to run through. There's a big mess of ANN linear algebra / human subconscious thought and I agree these messes can't be compared (or even identified in the human case). But we can compare the efficiency of the solution. It is possible that subconsciously humans "generate" hundreds of solutions that are mostly discarded, but I don't think the brain is fast enough to do that at the speed of conscious thought: it's a 50bn core processor but each core is only 200Hz and they aren't general-purpose CPUs. It also seems inconsistent with how humans solve these problems.

I believe energy usage would be even more misleading: in terms of operations/second a human brain is comparable to a 2020s supercomputer running at 30MW, but it only consumes 300 watts. (I was thinking about this with the "tiny portion" comment but it is irrelevant.)

Re: Arc Prize 2024 Winners and Technical Report

#30

Post by a_wild_dandan »

Reasons that I can't take this benchmark seriously:

1. Existing brute force algorithms solve 40% of this "reasoning" and "generalization" test.

2. AGI must evidently fit on a single 16GB, decade-old GPU?

3. If ARC fails blind people, it's not a reasoning test. Reasoning is independent of visual acuity. So ARC is at best a vision processing then reasoning test. SotA model "failure" is meaningless. ("But what about the other format, JSON?" Yeah, I would love to see the human solve rate on that...)

Post reply on HN