Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Man Out to Prove How Dumb AI Still Is

theatlantic.com

Post reply on HN
Print view View original

11–20 of 66 posts

Re: A Man Out to Prove How Dumb AI Still Is

#11

Post by mdp2021 »

> In 2019, Chollet created the Abstraction and Reasoning Corpus for Artificial General Intelligence, or ARC-AGI—an exam designed to show the gulf between AI models’ memorized answers and the “fluid intelligence” that people have

There are a number of skill signals we demand from an intelligence.

Mind you: some of them are achieved - like the ability to interpret pronouns (Hinton's "the trophy will not enter the case: it's too big" vs "the trophy will not enter the case: it's too small").

Others, we meet occasionally when we are not researching said requirements systematically: one example is that detective game described at https://news.ycombinator.com/item?id=43284420 - a simple game of logic that intelligences are required to be able to solve (...and yet, again some rebutted that humans would fail etc.).

It remains important though that those working modules are not clustered (solving specific tasks and remaining unused otherwise): they must be intellectual keys adapted into use in the most general cases they can be be helpful in. That's important in intelligence. So, even the ability to solve "revealing" tasks is not enough - the way in which the ability works is crucial.

Re: A Man Out to Prove How Dumb AI Still Is

#12

Post by HenryBemis » 

  1a) it's not AI, it's LLM. The companies who create/train/operate them may (wink-wink) pitch them as "AI" with half-truths, but we (here) know it's LLMs "all the way down"
  1b) just like I disliked the "autopilot" in Teslas because it was never autopilot.
  2) I know that I wanted to write some software tools, and I have been successful at this for the past many months, and I got top-shelve tools, that work, do their tasks, send alerts, etc. etc. And I am not the only one. So if the purpose is to "show it's a stupid AI".. well.. it's not AI.. so yeah. If the purpose is "it is not perfect", yes, because it draws a hand with 10 fingers. What else is new?
LLMs are a tool, still under development, still early in the curve, they can do A-B-C well but not X-Y-Z well (or at all). Congratulations :)

Re: A Man Out to Prove How Dumb AI Still Is

#13

Post by HenryBemis »

j_bum wrote: post #4

> To hit 87 percent on the original ARC-AGI test, o3 spent roughly 14 minutes per puzzle and, by my calculations, may have required hundreds of thousands of dollars in computing and electricity > the bot came up with more than 1,000 possible answers per grid before selecting a final submission. Yeah, AGI is right around the corner… /s

Let's all hope two things: 1) AGI is not near 2) Nobody ever allows it to have physical presence 2b) should that happen, expect the one idiot ask it to make "as many paperclips as possible" :)

Re: A Man Out to Prove How Dumb AI Still Is

#14

Post by mdp2021 »

noosphr wrote: post #9

>Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a “mini” version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesper…

> I've had _every_ model fail this

That seems to be because LLMs don't seem to be able to follow procedures (e.g. reliably counting).

Re: A Man Out to Prove How Dumb AI Still Is

#16

Post by whiplash451 »

noosphr wrote: post #9

>Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a “mini” version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesper…

Show me the results of your symbolic AI on ARC 2.

Re: A Man Out to Prove How Dumb AI Still Is

#17

Post by whiplash451 »

refulgentis wrote: post #6
echelon wrote: post #2

> When I spoke with him earlier this year, Chollet told me that AI companies have long been “intellectually lazy“ s/intellectually lazy/hype maxing for fundraising/

I think it's fascinating that his impossible benchmark got defeated, but because the Keras guy doesn't like LLMs, it is possible to mishear algorithmic distaste as saying people shipping this are "lazy" and "hype maxing."

Francois never said he dislike LLMs. In fact, he said he expected them to be part of the solution to ARC.

I don’t know where this persistent myth comes from, but it has to go.

Re: A Man Out to Prove How Dumb AI Still Is

#18

Post by gambiting »

noosphr wrote: post #9

>Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a “mini” version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesper…

>> If you want to saturate any model today give it a string and a grammar and ask it to generate the string from the grammar.

I'm not sure I understand what that means - could you explain please?

Re: A Man Out to Prove How Dumb AI Still Is

#19

Post by anthomtb »

mikestew wrote: post #10

Not to dismiss Chollet’s work, but I’m starting to think he need prove nothing to even the muggles. For example, nearly any endurance athlete stands a good chance of being a Strava user. If you run in those circles, have you heard a single person with anything good to say about Strava’s “Athletic Intelligence”? Garmin is rolling out a beta right now that includes “AI Insights” or summat. Same deal: useless summaries…

I use Strava for mountain biking and the Athletic Intelligence is just comical.

"This ride was longer and harder than usual" - no sh*t, the map, elevation profile and my legs have already informed me.

"You set 3 new PRs" - I can see that with one flick of the thumb thank you.

"Consider a rest day" - consider? None of my job, spouse, equipment or body is too keen on doing that again for a while.

Re: A Man Out to Prove How Dumb AI Still Is

#20

Post by elpocko »

gambiting wrote: post #18
noosphr wrote: post #9

>Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a “mini” version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesper…

>> If you want to saturate any model today give it a string and a grammar and ask it to generate the string from the grammar. I'm not sure I understand what that means - could you explain please?

I'm not GP, but here goes:

LLMs are token-based, which are words or word fragments; they have limited ability to work on a letter-by-letter basis. They can't reliably count letters in a sentence, for example. "give it a string and a grammar and ask it to generate the string from the grammar" can't be done by inference alone because of this: they would generate tokens that don't match the grammar.

But you can use a grammar-based sampler and it'll generate valid strings just fine. llama.cpp can easily do this if you provide an EBNF grammar specification.

Post reply on HN