Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why can't transformers learn multiplication?

arxiv.org

Post reply on HN
Print view View original

61–70 of 111 posts

Re: Why can't transformers learn multiplication?

#61

Post by CaveTech »

hodgehog11 wrote: post #55
CaveTech wrote: post #53

Earlier quoted context omitted.

The latter, and I would disagree that “this works and scales well” in the general sense. It clearly has very finite bounds by the fact we haven’t achieved agi by running an llm in a loop.. The approach of “try a few more things before stopping” is a great strategy akin to taking a few more stabs at RNG. It’s not the same as saying keep trying until you get there - you won’t.

> It clearly has very finite bounds by the fact we haven’t achieved agi by running an llm in a loop.. That's one hell of a criterion. Test-time inference undergoes a similar scaling law to pretraining, and has resulted in dramatically improved performance on many complex tasks. Law of diminishing returns kicks in of course, but this doesn't mean it's ineffective. > akin to taking a few more stabs at RNG Assuming I un…

I never made a claim that it's ineffective, just that it's of limited effectiveness. The diminishing returns kick in quickly, and it's not applicable in more domains than it is applicable.

Re: Why can't transformers learn multiplication?

#62

Post by fooker »

nico wrote: post #10

Would love to see an architecture that learned more like humans. Start with just imitating one letter, then a few more, than some syllables, then full words, then sentences, etc. Progressively adding on top of previous knowledge Also, it’s interesting that one of the big goals/measures of models is their capacity to “generalize”, but the training methods optimize for loss/accuracy, and only after training test for ge…

There's an interesting question here.

Would a single human/entity learn more in ..say.. three million years or would short lived ones evolving over three million years and then ~20 years of education learn more?

The current AI tech cycle is focusing on the first, but we don't really know if there are benefits of both.

There's no obvious way to combine these yet.

Re: Why can't transformers learn multiplication?

#63

Post by AvAn12 »

Computers are already fast and efficient at multiplication - optimized long ago. Transformers are fast and efficient at working with sequences of tokens. Tools are not universal. A hammer is not a good violin bow. A MRI machine is not a good relational database. This extends to the natural world too. A zebra is not a good dairy animal. And a human poet may or may not be a good surgeon. It’s good to explore what things can do beyond their intrinsic nature - but expect to encounter limits eventually.

Re: Why can't transformers learn multiplication?

#64

Post by Nifty3929 »

jakefromstatecs wrote: post #57
Nifty3929 wrote: post #45

Numbers aren't language, or even sequences of tokens, or vectors. There is an inherent numeric-ness and logic to math that I don't think we can represent well using LLMs and transformers. 3 isn't about the word "three" - it is a quantity or a measurement. And 3x4 is a specific numerical operation that is not really contained in that sequence of symbols.

Math is just symbol manipulation with a set of rules, no?

No. Math and especially numbers are not just symbol manipulation. Geometry is a counter-example. So is multiplication, for that matter.

Maybe you could say that algebra is just symbol manipulation.

And in any case - "set of rules" is exactly what transformers aren't good at. Transformers are good at capturing the essence of what you meant and responding in a sensible, but not rule-bound way. This works well for language problems.

Perhaps you could argue that transformers are just a set of rules (weights/parameters) being applied, and you might similarly argue that numbers reduce to logical symbols like S(0), S(S(0)), but then I'd argue that you're missing the point.

Re: Why can't transformers learn multiplication?

#65

Post by ddingus »

CaveTech wrote: post #36
ddingus wrote: post #31

A while back I saw a post where people ran a model over and over to accomplish a code base port from one language to another. In their prompt, they told it to leave itself a note and to accomplish something each time. Then they put the model in a loop and it worked. In one instance, a model removed itself from the loop by editing a file or some other basic means. To me, iterative tasks like like multiply and long div…

The amount of paths in the wrong direction are infinitely more than then number in the right direction. You'll quickly realize this doesn't actually scale.

I am not sure it needs to scale.

Re: Why can't transformers learn multiplication?

#66

Post by ddingus »

CaveTech wrote: post #53
hodgehog11 wrote: post #46

Earlier quoted context omitted.

I'm a bit confused by this; are you referring to vanishing/exploding gradients during training or iteration at inference? If the former, this is only true if you take too many steps. If the latter, we already know this works and scales well.

The latter, and I would disagree that “this works and scales well” in the general sense. It clearly has very finite bounds by the fact we haven’t achieved agi by running an llm in a loop.. The approach of “try a few more things before stopping” is a great strategy akin to taking a few more stabs at RNG. It’s not the same as saying keep trying until you get there - you won’t.

Achieving agi is not a requirement to working well.

Re: Why can't transformers learn multiplication?

#67

Post by rurban »

Even worse: Why cannot programming languages learn arithmetic?

Most languages and its stdlib's cannot deal with numbers properly at all. Most overflow without errors. Most integers cannot keep precision, most cannot promote types properly.

I only know of Common Lisp, Scheme, Python 3, Ruby, Erlang, Haskell, Raku which can handle numbers properly by default. Python extremely slow though.

Re: Why can't transformers learn multiplication?

#68

Post by malfist »

ares623 wrote: post #59
onlyrealcuzzo wrote: post #49

Earlier quoted context omitted.

You don't need general intelligence to make good memes to keep people scrolling through Instagram. You don't need general intelligence to make a decent coding tool like Cursor. You don't need general intelligence to improve SERPs. You don't need general intelligence to sell a subscription for a decent AI assistant. There's tons of value already added without anything general.

Yes but $500B and counting for memes wasn’t what was sold

I remember reading somewhere someone said "the problem with AI is it's a $50b industry pretending its a $10t industry"

Re: Why can't transformers learn multiplication?

#69

Post by malfist »

hodgehog11 wrote: post #46
CaveTech wrote: post #36

Earlier quoted context omitted.

The amount of paths in the wrong direction are infinitely more than then number in the right direction. You'll quickly realize this doesn't actually scale.

I'm a bit confused by this; are you referring to vanishing/exploding gradients during training or iteration at inference? If the former, this is only true if you take too many steps. If the latter, we already know this works and scales well.

How do you know if you've taken too many steps beforehand?

Re: Why can't transformers learn multiplication?

#70

Post by krackers »

wat10000 wrote: post #28
skinner_ wrote: post #27

Earlier quoted context omitted.

If being probabilistic prevented learning deterministic functions, transformers couldn’t learn addition either. But they can, so that can't be the reason.

People are probabilistic, and I've been informed that people are able to perform multiplication.

Are you sure? I bet you if you pull 10 people off the street and ask them to multiply 5 digit by 5 digit numbers by hand, you won't have a 100% success rate.
Post reply on HN