Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Man Out to Prove How Dumb AI Still Is

theatlantic.com

Post reply on HN
Print view View original

31–40 of 66 posts

Re: A Man Out to Prove How Dumb AI Still Is

#31

Post by janalsncm »

mdp2021 wrote: post #26
janalsncm wrote: post #22

Earlier quoted context omitted.

> best solved with what used to be called symbolic AI before it started working Right, the current paradigm of requiring an LLM to do arbitrary digit multiplication will not work and we shouldn’t need to. If your task is “do X” and it can be reliably accomplished with “write a python program to do X” that’s good enough as far as I’m concerned. It’s preferable, in fact. Btw Chollet has said basically as much. He calls…

> that’s good enough as far as I’m concerned But in that case, why an LLM. If we want Question-Answer machines to be reliable, they must have the skills which include "counting" just as a basic example.

The purpose of the LLM would be to translate natural language into computer language, not to do the calculation itself.

Re: A Man Out to Prove How Dumb AI Still Is

#32

Post by janalsncm »

Ologn wrote: post #25
janalsncm wrote: post #22

Earlier quoted context omitted.

> best solved with what used to be called symbolic AI before it started working Right, the current paradigm of requiring an LLM to do arbitrary digit multiplication will not work and we shouldn’t need to. If your task is “do X” and it can be reliably accomplished with “write a python program to do X” that’s good enough as far as I’m concerned. It’s preferable, in fact. Btw Chollet has said basically as much. He calls…

Most human ten year olds in school can add two large numbers together. If a connectionist network is supposed to model the human brain, it should be able to do that. Maybe LLMs can do a lot of things, but if they can't do that, then they're an incomplete model of the human brain.

Artificial neural nets are pretty far from brains. We don’t use them because they are like brains, we use them because they can approximate arbitrary functions given sufficient data. In other words, they work.

For what it’s worth, people are also pretty bad at math compared to calculators. We are slow and error prone. That’s ok.

What I was (poorly) trying to say is that I don’t care if the neural net solves the problem if it can outsource it to a calculator. People do the same thing. What is important is reliably accomplishing the goal.

Re: A Man Out to Prove How Dumb AI Still Is

#33

Post by sionisrecur »

noosphr wrote: post #9

>Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a “mini” version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesper…

> If you don't have an essentially infinite set to draw your validation data from then a large enough model will memorize it as part of its developer teams KPIs.

Sounds like a use-case for property testing: https://en.wikipedia.org/wiki/Software_testing#Property_test...

Re: A Man Out to Prove How Dumb AI Still Is

#34

Post by pixl97 »

j_bum wrote: post #4

> To hit 87 percent on the original ARC-AGI test, o3 spent roughly 14 minutes per puzzle and, by my calculations, may have required hundreds of thousands of dollars in computing and electricity > the bot came up with more than 1,000 possible answers per grid before selecting a final submission. Yeah, AGI is right around the corner… /s

I also remember when a computer about as powerful as the watch on my wrist cost millions of dollars and filled multiple large rooms.

Re: A Man Out to Prove How Dumb AI Still Is

#36

Post by YurgenJurgensen »

janalsncm wrote: post #22
noosphr wrote: post #9

>Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a “mini” version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesper…

> best solved with what used to be called symbolic AI before it started working Right, the current paradigm of requiring an LLM to do arbitrary digit multiplication will not work and we shouldn’t need to. If your task is “do X” and it can be reliably accomplished with “write a python program to do X” that’s good enough as far as I’m concerned. It’s preferable, in fact. Btw Chollet has said basically as much. He calls…

As soon as the companies behind these systems stop marketing them as do-anything machines, I will stop judging them on their ability to do everything.

The ChatGPT input field still says ‘Ask anything’, and that is what I shall do.

Re: A Man Out to Prove How Dumb AI Still Is

#37

Post by dgfitz »

HenryBemis wrote: post #12

1a) it's not AI, it's LLM. The companies who create/train/operate them may (wink-wink) pitch them as "AI" with half-truths, but we (here) know it's LLMs "all the way down" 1b) just like I disliked the "autopilot" in Teslas because it was never autopilot. 2) I know that I wanted to write some software tools, and I have been successful at this for the past many months, and I got top-shelve tools, that work, do their ta…

> LLMs are a tool, still under development, still early in the curve…

I completely agree, they are a tool, and a decently useful tool. They are not early in the curve, they’re about flat at this point.

Re: A Man Out to Prove How Dumb AI Still Is

#38

Post by fritzo »

whiplash451 wrote: post #16
noosphr wrote: post #9

>Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a “mini” version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesper…

Show me the results of your symbolic AI on ARC 2.

ARC 2 is brand new, but neurosymbolic approaches have performed well on the original ARC, e.g. https://arxiv.org/abs/2411.02272

Re: A Man Out to Prove How Dumb AI Still Is

#39

Post by globnomulous »

noosphr wrote: post #9

>Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a “mini” version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesper…

I tried my own test recently:

"Write a history of the Greek language but reverse it, so that one would need to read it from right to left and bottom to top."

ChatGPT wrote the history and showed absolutely no awareness, let alone, "understanding" of the second half of the prompt.

Post reply on HN