Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ARC-AGI Leaderboard

arcprize.org

Post reply on HN
Print view View original

71–80 of 156 posts

Re: ARC-AGI Leaderboard

#71

Post by rf15 »

IanCal wrote: post #64
rf15 wrote: post #47

Earlier quoted context omitted.

Yes! Impressive, isn't it? I see how it has improved for some minor points, that the big models can cover more finetuning ground, but my big gripes are still the same - you could do the same back then with multiple models and more targeted finetuning.

Utter nonsense. There’s no way you could get models as smart by fine tuning. I couldn’t throw a problem like “build a pokemon database with UI to teach my son sql” and get a working system, nice ui, tests (which it iterated on) examples and explanations in one shot. There weren’t thinking tokens. Maths is now dramatically better, making actual contributions when before they were mostly mocked for making extremely bas…

My company had such a system four years ago, for internal work, somewhat more limited in scope (one language). What you are seeing as the frontier is not necessarily the best you can have - just because people don't try to push it to market as a product doesn't mean it's not there.

Edit: we do have a system that uses LLM and fixes the above issues largely (tracking of state, calculations and objects, still flawed in finer details). No, we don't sell, it's experimental fun and not really ready in terms of setup/ux/etc.

It codes really well for our case though.

Re: ARC-AGI Leaderboard

#72

Post by rf15 »

danielbln wrote: post #56
anonzzzies wrote: post #49

Earlier quoted context omitted.

> you could do the same back then with multiple models and more targeted finetuning Are you one of those anonymous billionaires as if you did this a few years ago, you would've been famous and rich.

OP is delusional or deliberately optuse. I work in the space and stare down these systems 12h/day, and saying the systems haven't meaningfully improved is ludicrous.

OP is largely pissed with what OAI/Antrophic are trying to sell as meaningful improvements and the market-bending money they ask for it. I work in the space and we trained LLM models on conceptual tokens, not language tokens, for example. See Symbolic AI and all the attempts at hybrid models.

Also, uh, fame and riches are not really my thing. Middle income is fine. My mistake was speaking up here because I got carelessly annoyed because I have skin in the game, research-wise. I'm sorry for that.

Re: ARC-AGI Leaderboard

#73

Post by rf15 »

Barbing wrote: post #51
rf15 wrote: post #45

Earlier quoted context omitted.

I mean calculations, not mathematical proofs

If they use Python to fill the gap, and the end user doesn’t have to know or care, is it unfair to assess this as progress and attribute the progress to the _system_? OK, the core technology that is the language model still can’t math as well as you’d hope, but how about the end result users see from the system when they interface with it? “Did you know humans are better at flying today than they were a thousand year…

You are correct, the frameworks around it have improved. In that regard, my assessment is unfair: I only judge the underlying technology and what is sold by the sota providers, with the premise of what it's like when you start fresh. You can achieve a lot by coding around the issues, but that's kinda against the point of 'AI', is it?

Re: ARC-AGI Leaderboard

#74

Post by rf15 »

Narciss wrote: post #67
rf15 wrote: post #47

Earlier quoted context omitted.

Yes! Impressive, isn't it? I see how it has improved for some minor points, that the big models can cover more finetuning ground, but my big gripes are still the same - you could do the same back then with multiple models and more targeted finetuning.

I have no idea how you could try and hold this argument without being facetious.

Am I missing something that my original points no longer hold for their products? Did it get meaningfully solved? Are your experiences flawless on that front?

Re: ARC-AGI Leaderboard

#75

Post by kypro »

yladiz wrote: post #69
r0ze-at-hn wrote: post #63

Earlier quoted context omitted.

The exclusion of harness's feels really weird given that companies are recognizing the value of what harness's can do. By excluding them the benchmark is becoming less relevant.

You could argue that if you allowed a harness, and that harness was specific for ARC, then you don’t have AGI, you have something that is definitely not general.

I think that's a strange way to look at it. The brain also has different regions with different functions, and part of what makes us humans intelligent is that we can use tools like pen and paper to keep notes and help us solve problems.

Similarly LLMs are not just massive uniform artificial neural nets, and now we also have harnesses, which I'd personally view more of an extension of the model itself. The harness is both the executive and also what allows it to keep notes, use a calculator, or maybe even create scripts to help it solve complex deterministic problems.

I think it's unfair to give a human a very complex maths problem and say that they're not intelligent if they can't solve it without pen and paper or a calculate. At least expecting a human to solve complex problems this way doesn't really measure anything useful in the real world.

Re: ARC-AGI Leaderboard

#76

Post by thops-barrier »

rf15 wrote: post #74
Narciss wrote: post #67

Earlier quoted context omitted.

I have no idea how you could try and hold this argument without being facetious.

Am I missing something that my original points no longer hold for their products? Did it get meaningfully solved? Are your experiences flawless on that front?

You're moving the goalposts. Initially it was "no meaningful improvement" and now suddenly it has morphed into "they're not flawless".

I'm pretty sure you're just baiting for engagement though so well done, ya got me.

Re: ARC-AGI Leaderboard

#77

Post by kypro »

ARC-AGI-3 launched a few months ago which would suggest that prior models likely had no knowledge of ARC-AGI-3 or training on similar challenges.

I could be wrong, but given the large outsized jump solely in the ARC-AGI-3 score, it would suggest that the model didn't become significantly more intelligent overall, but significantly better at solving those specific problems.

This could mean one of two things (I think):

- Opus 5 was not benchmaxxed on ARC-AGI-3, but has benefited significantly from discussions about the various challenges and mechanisms deployed in ARC-AGI-3 such that it has far better heuristics to solve its challenges.

- Anthropic looking for buzz around their latest model picked a well regarded benchmark with significant room for improvement and focused some of Opus 5's training compute on ARC-AGI-3-style problems.

Or it could be some combination of both. Personally, given how much of an outlier the ARC-AGI-3 jump is I struggle to see it being the product of a significant improvement in general intelligence.

Re: ARC-AGI Leaderboard

#78

Post by usernametaken29 »

r0ze-at-hn wrote: post #63
dinp wrote: post #62

The last time I checked, for the arc agi 3 leaderboard, the models are given a simple prompt and the game input and asked to play the game, no harness/tools. If harnesses were allowed, I would expect the benchmark to be saturated. There were a few harness attempts, but they could only be evaluated on the public set, so it's not an apples to apples comparison. My guess is, the large score jump for Opus 5 is mainly bec…

The exclusion of harness's feels really weird given that companies are recognizing the value of what harness's can do. By excluding them the benchmark is becoming less relevant.

It’s because of inductive bias. Harnesses will massively skew results towards working solutions. You might think that’s a good thing but what it might mean that sometimes it becomes enough to run brute force search or a simple parameter search over the harness. Creating the harness is the actual work, because you’re selecting what are the levers to pull. There were some attempts of LLMs generating harnesses on the fly in ARC 2, but they were all mostly based on one handcrafted DSL that was copied over and over again. As it stands harnesses are not allowed because they’re simply not a meaningful measure. What you’d like is to measure how the model performs if it saw this benchmark for the very first time… but then again everyone knows the game is rigged, millions are at stake, and the AI companies fine tune and cheat on the benchmarks any way they can.

Re: ARC-AGI Leaderboard

#80

Post by glimshe »

rf15 wrote: post #34
throwaw12 wrote: post #14

Why Anthropic models are always leapfrogging these benchmarks, but in real life work I do feel like after 3 weeks I am back to Claude Opus 4.5? (regardless of the model I use, Fable was exception for 1 day when it was released)

I've worked with these systems for four years now and they have not meaningfully improved in that time frame. We still have: - statistical correlation between two things will always cause one thing to lead to the other, no matter how much you prompt it to not have that connection (to be expected with a stochastic system) - Math completely fails in longer contexts - "thinking" token generation being on the correct tra…

Messages like this in the training data are how LLMs learn to say absurd things with total confidence.
Post reply on HN