Very cool that it was able to generalise from small numbers to larger ones with such high accuracy.
Transformers Can Do Arithmetic with the Right Embeddings
21–30 of 215 posts
Re: Transformers Can Do Arithmetic with the Right Embeddings
#22What is the point of this work? 99% on 100-digit arithmetic means there's a 0% chance anyone will ever use a Transformer as an ALU or anything of the kind. We already know how to hard-code a (literally) infinitely more accurate addition machine. And not only addition: all four arithmetic operations. The technique proposed in the article -imposing a strong inductive bias for addition- kiind of works for multiplication…
By such arguments, what is the point of any research, at all?
If everyone was using horses what would you had said about the first prototype car? Probably a very slow and clumsy and failureprone thing.
Re: Transformers Can Do Arithmetic with the Right Embeddings
#23It seems like a hack to be honest. Problem at hand is not to make transformers do addition of 100 digit numbers. Problem is the current systems can’t reason about things, math included. Optimizing for a certain use case is not gonna take us where we wanna be. We want to have a system that can learn to reason.
We're as humanity building a reasoning machine bottom up. It can't reason... yet. Expecting a magical switch that will make it reason about anything and everything is unreasonable. Starting with arithmetic makes perfect sense.
Re: Transformers Can Do Arithmetic with the Right Embeddings
#24Since models are very good at writing very short computer programs, and computer programs are very good at mathematical calculations, would it not be considerably more efficient to train them to recognise a "what is x + y" type problem, and respond with the answer to "write and execute a small javascript program to calculate x + y, then share the result"?
Re: Transformers Can Do Arithmetic with the Right Embeddings
#25I think understanding mathematics is what LLM really need at the moment far more important than video generation that is just another form of CGI [1]. After deep learning and transformer, understanding mathematics and its proofs not just arithmetic will be the next game changer for LLM and a turning point for humanity. [1] Why LLMs like ChatGPT and Google Bard are bad at math: https://www.xda-developers.com/why-llms-…
> understanding mathematics and its proofs not just arithmetic will be the next game changer for LLM
Why?
I definitely agree that such capabilities would represent a major advance (and very likely go together with game changing increases of capabilities in other areas). I also think using AI to write formal math proofs in e.g. Lean is very cool.
However, by itself, it seems like this capability wouldn't be very useful, commercially for example. Do you think this capability is exceptionally informative merely because it has to go together with other capabilities? It's not impossible to have a (maybe somewhat limited) formal math AI that will remain mostly irrelevant to the everyday world (like FormalGeo).
Re: Transformers Can Do Arithmetic with the Right Embeddings
#26Since models are very good at writing very short computer programs, and computer programs are very good at mathematical calculations, would it not be considerably more efficient to train them to recognise a "what is x + y" type problem, and respond with the answer to "write and execute a small javascript program to calculate x + y, then share the result"?
[deleted]
Re: Transformers Can Do Arithmetic with the Right Embeddings
#27Since models are very good at writing very short computer programs, and computer programs are very good at mathematical calculations, would it not be considerably more efficient to train them to recognise a "what is x + y" type problem, and respond with the answer to "write and execute a small javascript program to calculate x + y, then share the result"?
From a getting answers perspective yes, from an understanding LLMs perspective no. If you read the avstract you can see how this goes beyond arithmetic and helps with longform reasoning
Re: Transformers Can Do Arithmetic with the Right Embeddings
#28That's great, 99% math is absolutely good enough for real world problems /s
Re: Transformers Can Do Arithmetic with the Right Embeddings
#29Since models are very good at writing very short computer programs, and computer programs are very good at mathematical calculations, would it not be considerably more efficient to train them to recognise a "what is x + y" type problem, and respond with the answer to "write and execute a small javascript program to calculate x + y, then share the result"?
That's in fact what ChatGPT does ... because 99% accurate math is not useful to anyone.
Re: Transformers Can Do Arithmetic with the Right Embeddings
#30That's great, 99% math is absolutely good enough for real world problems /s
It's probably on-par or better than humans get unaided. Hell, I'd bet due to transcription errors it's better than what humans get in a lot of settings, even when aided by a calculator.