A Man Out to Prove How Dumb AI Still Is • Hacker News Board

Re: A Man Out to Prove How Dumb AI Still Is

#61

Post by amelius » Sat, Apr 05, 2025, 10:32 AM UTC

mhast wrote: post #58

globnomulous wrote: post #48

Earlier quoted context omitted.

?siht ekil kool rewsna eht diD Edit: realized just now that my summary of the 'test' failed to specify the request fully: the letters need to be reversed, too. Maybe I'm just bad with AI tools, because I didn't even get a response that 'this like looked' (i.e. reversed the order of the words).

LLMs work with tokens, not letters. So that's not going to work. It might work in an agent system where it can make and execute code to solve problems.

By the way, why _do_ llms work with tokens and not letters? Is that some kind of optimization, or is there a more fundamental reason for it?

Re: A Man Out to Prove How Dumb AI Still Is

#62

Post by globnomulous » Sat, Apr 05, 2025, 2:48 PM UTC

Timwi wrote: post #60

tkcranny wrote: post #43

Earlier quoted context omitted.

As much I think AI is overhyped too, that is a prime use case that would be better solved by passing the text to a tool, rather than jam a complex transformations like that into its latent space.

A real intelligence would recognize that this task is better solved with an automated tool and actually do so. ChatGPT is capable of writing and executing Python code, but it doesn't occur to it to use that in cases like this.

Thanks, that was essentially the test. I've gotten into a number of disagreements with people on HN about whether LLMs are 'just' token predictors, whether they 'understand' (whatever we mean by that), whether there's a guiding intelligence, whether they're 'just' language calculators, etc.

As someone else in this thread nicely put it, the tools are being sold as a hop, skip, and jump away from AGI. They clearly aren't. ChatGPT tells us to "ask anything." I did that. There is no 'there' there with these tools. They aren't even dumb.

Re: A Man Out to Prove How Dumb AI Still Is

#63

Post by whiplash451 » Sat, Apr 05, 2025, 3:57 PM UTC

fritzo wrote: post #38

whiplash451 wrote: post #16

Earlier quoted context omitted.

Show me the results of your symbolic AI on ARC 2.

ARC 2 is brand new, but neurosymbolic approaches have performed well on the original ARC, e.g. https://arxiv.org/abs/2411.02272

Yet LLMs won on ARC 1. So what’s your point exactly?

Re: A Man Out to Prove How Dumb AI Still Is

#64

Post by yencabulator » Sat, Apr 05, 2025, 5:40 PM UTC

jameshart wrote: post #45

Ologn wrote: post #25

Earlier quoted context omitted.

Most human ten year olds in school can add two large numbers together. If a connectionist network is supposed to model the human brain, it should be able to do that. Maybe LLMs can do a lot of things, but if they can't do that, then they're an incomplete model of the human brain.

Most human ten year olds can add two large numbers together with the aid of a scratchpad and a pen. You need tools other than a single dimensional vector of text to do some of these things.

AI apologists need to decide whether they are claiming LLMs are almost-AGI, or not.

This backlash of pointing out LLM failures is a reaction to the overblown hype. We don't expect a statistical-language-processing-gadget to do math well, but then people need to stop claiming they're something other than statistical-language-processing-gadgets.

Re: A Man Out to Prove How Dumb AI Still Is

#65

Post by globnomulous » Sun, Apr 06, 2025, 1:20 AM UTC

amelius wrote: post #61

mhast wrote: post #58

Earlier quoted context omitted.

LLMs work with tokens, not letters. So that's not going to work. It might work in an agent system where it can make and execute code to solve problems.

By the way, why _do_ llms work with tokens and not letters? Is that some kind of optimization, or is there a more fundamental reason for it?

I actually would love to see the output of an LLM that used letters or syllables or phonemes. The output when it makes mistakes would be absolutely wild.

Re: A Man Out to Prove How Dumb AI Still Is

#66

Post by j_bum » Mon, Apr 07, 2025, 4:02 AM UTC

pixl97 wrote: post #34

j_bum wrote: post #4

> To hit 87 percent on the original ARC-AGI test, o3 spent roughly 14 minutes per puzzle and, by my calculations, may have required hundreds of thousands of dollars in computing and electricity > the bot came up with more than 1,000 possible answers per grid before selecting a final submission. Yeah, AGI is right around the corner… /s

I also remember when a computer about as powerful as the watch on my wrist cost millions of dollars and filled multiple large rooms.

lol point taken.

I would be willing to bed we’ll see something rambling natural intelligence created in this century, but I think this brute force approach shows that we’re just not there yet.