Are these typically the type of tasks that are genuinely worth tens of thousands of dollars?
Not at all. They are mini games.
ARC-AGI Leaderboard
151–156 of 156 posts
Re: ARC-AGI Leaderboard
#152ARC-AGI is a terrible benchmark for testing LLMs because LLMs are not made, trained, or tuned for playing games. They are trained on text to respond well to text based questions and do tasks involving modifying text files. They are not designed for playing games, looking at games, or visual puzzles. Also translating games into text input for the LLM skews the test completely. Imagine trying to get a human to solve vi…
Teams are more than welcome to use a non-LLM approach (or hybrid) if they consider that to be more suitable.
Re: ARC-AGI Leaderboard
#153Earlier quoted context omitted.
OP is delusional or deliberately optuse. I work in the space and stare down these systems 12h/day, and saying the systems haven't meaningfully improved is ludicrous.
OP is largely pissed with what OAI/Antrophic are trying to sell as meaningful improvements and the market-bending money they ask for it. I work in the space and we trained LLM models on conceptual tokens, not language tokens, for example. See Symbolic AI and all the attempts at hybrid models. Also, uh, fame and riches are not really my thing. Middle income is fine. My mistake was speaking up here because I got carele…
I still enjoy the symbolic ai space. Any examples of interesting progress there?
Re: ARC-AGI Leaderboard
#154Appears to be benchmaxxing https://x.com/quietnning/status/2080786711861407883
Hello, sorry. I know this is not relevant to what you are talking about. However I have been unable to contact you by any other means and have been trying to contact you for more than a month
Assuming that you are "kas", please check the dulst discord. It is important.
Re: ARC-AGI Leaderboard
#155Earlier quoted context omitted.
For frontier models, not local. https://schema-harness.github.io/
Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
Run policy search long enough with enough exploration and you can solve any of these games. But solving 110 games in 9 hours with a single RTX Pro 6000 doesn't seem likely. And if you did, you could keep the solution secret in exchange for the mountain of VC you would get to productize the approach. Not expecting it.
Re: ARC-AGI Leaderboard
#156Earlier quoted context omitted.
Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
If you stop and think about the problem it really is quite simple. Just need to build a graph of the game state and then run A* to get to the end.
You are not given the rules or the winning conditions. You are only given a potentially windowed and/or degenerate visualizer of the underlying game state along with the UI and told to just figure it out. And you as a human will, in a couple moves. An LLM? Not so much. But they do eventually solve them. And given enough moves, victory is inevitable, but you are penalized for taking more moves than a human, yet also slightly punished if you find a better solution by capping your reward to 115%.