A Man Out to Prove How Dumb AI Still Is
theatlantic.com
A Man Out to Prove How Dumb AI Still Is
1–10 of 66 posts
Re: A Man Out to Prove How Dumb AI Still Is
#2> When I spoke with him earlier this year, Chollet told me that AI companies have long been “intellectually lazy“
s/intellectually lazy/hype maxing for fundraising/
Re: A Man Out to Prove How Dumb AI Still Is
#3Re: A Man Out to Prove How Dumb AI Still Is
#4> To hit 87 percent on the original ARC-AGI test, o3 spent roughly 14 minutes per puzzle and, by my calculations, may have required hundreds of thousands of dollars in computing and electricity
> the bot came up with more than 1,000 possible answers per grid before selecting a final submission.
Yeah, AGI is right around the corner… /s
Re: A Man Out to Prove How Dumb AI Still Is
#5>Chollet, a French computer scientist and one of the industry’s sharpest skeptics
I feel like this description really buries the lede on Chollet's expertise. (For those who don't know, he's the creator of and lead contributor[0] to Keras)
Re: A Man Out to Prove How Dumb AI Still Is
#6> When I spoke with him earlier this year, Chollet told me that AI companies have long been “intellectually lazy“ s/intellectually lazy/hype maxing for fundraising/
I think it's fascinating that his impossible benchmark got defeated, but because the Keras guy doesn't like LLMs, it is possible to mishear algorithmic distaste as saying people shipping this are "lazy" and "hype maxing."
Re: A Man Out to Prove How Dumb AI Still Is
#7Calling Francois Chollet just "A Man" in the title (or "The Man" in the actual article as of writing) is crazy work, he's been deeply involved in ML for ages including creating Keras.
Re: A Man Out to Prove How Dumb AI Still Is
#8> When I spoke with him earlier this year, Chollet told me that AI companies have long been “intellectually lazy“ s/intellectually lazy/hype maxing for fundraising/
I think it's fascinating that his impossible benchmark got defeated, but because the Keras guy doesn't like LLMs, it is possible to mishear algorithmic distaste as saying people shipping this are "lazy" and "hype maxing."
Arc agi 1 that "got defeated" was published even before first mainstream llms and still stood the test of time
Re: A Man Out to Prove How Dumb AI Still Is
#9>Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a “mini” version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesperson declined to say whether the company plans to run the benchmark with o3.) Other flagship models from OpenAI, Anthropic, and Google have achieved roughly 1 percent, if not lower. Human testers average about 60 percent.
Arc AGI is the main reason why I don't trust static bench marks.
If you don't have an essentially infinite set to draw your validation data from then a large enough model will memorize it as part of its developer teams KPIs.
Forget all these fancy benchmarks. If you want to saturate any model today give it a string and a grammar and ask it to generate the string from the grammar. I've had _every_ model fail this on regular grammars with strings of more than 4 characters long.
LLMs are the solution to natural language, which is a huge deal. They aren't the solution to reasoning which is still best solved with what used to be called symbolic AI before it started working, e.g. sat solvers.
Re: A Man Out to Prove How Dumb AI Still Is
#10Not to dismiss Chollet’s work, but I’m starting to think he need prove nothing to even the muggles. For example, nearly any endurance athlete stands a good chance of being a Strava user. If you run in those circles, have you heard a single person with anything good to say about Strava’s “Athletic Intelligence”? Garmin is rolling out a beta right now that includes “AI Insights” or summat. Same deal: useless summaries like “you ran 5 miles today, which contributes to your aerobic base”. I could do better with a database and some if/else statements. And Garmin wants a subscription for this. (It’s included in Strava’s subscription, but I suppose you’re still paying for it.) And so now the memes tend toward “dumb AI insight of the day” on many online forums.
Seems to me that a lot of folks are enjoying having an LLM rewrite their email or whatever, but I wonder how many are actually buying the rest of it? The companies themselves sure aren’t helping.