ARC AGI-3 saturated by Astra! https://arcprize.org/leaderboard
AGI means that the answers to these benchmarks are now accessible, and the defense capabilities of the testing organizations are now negligible.
GPT-6 Astra
791–800 of 1001 posts
Re: GPT-6 Astra
#792I have nothing to say about the actual model, but unrelated--why do so many of these demos include people buying things autonomously? Even if I did trust an AI to get everything right, it's not like the AI can read my mind. If I was ordering food normally and without AI, I would want more control over the process--looking over the options, prices, thinking about what I really want. People don't know what they really…
Despite access to """"""AGI""""""" all the marketing teams at these companies can only dream up 2 things, buying plane tickets and online shopping autonomously. Sometimes they're feeling extra spicy and throw in sorting emails or something along those lines. I suspect it's because it's tailored towards VCs and other similar rich ghouls as a replacement for their overworked and underpaid secretaries
All those stupid Bell Labs researchers not inventing Uber or Tinder. How come they didn't just build the obviously popular and profitable businesses that became possible once they invented the internet?
Re: GPT-6 Astra
#793I have nothing to say about the actual model, but unrelated--why do so many of these demos include people buying things autonomously? Even if I did trust an AI to get everything right, it's not like the AI can read my mind. If I was ordering food normally and without AI, I would want more control over the process--looking over the options, prices, thinking about what I really want. People don't know what they really…
Maybe we are not the real audience. I wonder if they are trying to convince the advertising industry/investors that in the future it won't be Google search that stands between the consumer and the product, but rather their LLM.
Re: GPT-6 Astra
#794What is going to become of life for those of us who do not work at AI labs and are unlikely to be hired by AI labs, despite all the years we put into learning coding, math, etc, as we were told to do? Those of us who made the mistake of studying anything other than machine learning. How will we make a living? (We don't live in a world that seems likely to distribute gains widely instead of largely to the handful of a…
You'll be fine. All these AIs still need direction, drive, and prompts. And it'll still be important to evaluate their output and correct it. And you'll be more productive and able to do more tasks quicker than usual with you helpful human expertise guiding the AI which is far more general.
You're the taste and decision maker. The AI is like a really precise, fast Google, Wikipedia, codegen tool
Re: GPT-6 Astra
#795I have nothing to say about the actual model, but unrelated--why do so many of these demos include people buying things autonomously? Even if I did trust an AI to get everything right, it's not like the AI can read my mind. If I was ordering food normally and without AI, I would want more control over the process--looking over the options, prices, thinking about what I really want. People don't know what they really…
(This holds generally) maybe because the people most likely to be convinced to buy this particular thing are the people who buy a lot of things. The people who only buy things they need, they’re a harder demographic to convince via ads, and they tend to pay less since they rarely need extras. Every ad caters to whales.
Re: GPT-6 Astra
#796[flagged]
Re: GPT-6 Astra
#797It's fun, but every new model release makes me even less interested to create cool stuff. Like, what's the point, if the next AI can do it in 5 seconds?
„The depressing thing about tennis is that no matter how good I get, I'll never be as good as a wall.“ -Mitch Hedberg
I don’t like being this person, but you can very easily beat a wall at tennis.
Re: GPT-6 Astra
#798The launch video is incredibly cringe.
Humans adapt to change oddly fast. If you saw that video in 2022 you'd be 100% positive you were watching a sci-fi movie.
Re: GPT-6 Astra
#799The ARC-AGI-3 scorecard is extremely misleading given that it clearly states itself that "with [the responses API] harness, we estimate Sol would score in the ballpark of ~30%." but it shows a score of 7.8% for GPT-5.6 Sol presumably since if they updated the percentage for GPT-5.6 Sol to the score it would receive with the responses API harness they used for GPT-6 Astra they'd have to do the same for the percentage…
If you define AGI as "can do the work of a human sitting at a computer, end to end", then I'd say comparing yourself to it on a specific skill is the wrong test. Can you hand it a role and walk away for a day/week/month? I can’t yet. I think that I'd want at least two things it doesn't have: the ability to retain what it learned yesterday (without me carrying it in the context window and thus micromanaging it), and t…
To be fair, I think you can't hand a role over to someone you just hired and walk away for a week. No matter how much of a SME they are. Let's not forget human onboarding takes months. With the advantage of their knowledge not going into the void multiple times a day. That's likely the last missing piece, a solid system of memories that produces the same effect as short/long term memories in a person.
Re: GPT-6 Astra
#800I can’t help but notice how much this echoes Francois Chollet’s On the Measure of Intelligence: https://arxiv.org/abs/1911.01547 Most of frontier-model progress still looks like skill acquisition optimization: broader benchmark coverage and performance, more domains absorbed into the training distribution, and increasingly strong performance within that surface area. It seems more about coverage-driven competence. So…
>The harder question, in Chollet’s framing, is: how efficiently can a system learn to do something genuinely new?
The more diverse stuff it knows, the easier it will be to learn something new.