How large is the data set they're training on? I suspect there are many math equations including these or similar numbers. As the result shows, the model is generally right about the first a few digits, but frequently wrong about the last digit. This may due to the fact that the data set can hardly cover the exact numbers in the questions, but it's likely to cover the first a few digits.
I asked GPT-NeoX-20B a hundred arithmetic questions
31–40 of 235 posts
Re: I asked GPT-NeoX-20B a hundred arithmetic questions
#32It seems to me that carries are where this trips up. Which is weirdly human. I wonder if there are enough examples to learn each digit pair addition or subtraction, but not enough to learn every contextual action.
Re: I asked GPT-NeoX-20B a hundred arithmetic questions
#33Maybe we're already past singularity and the AI is simply pretending to be bad at this in order to avoid making humans feel insecure. /s
Re: I asked GPT-NeoX-20B a hundred arithmetic questions
#34What? You think this is poor performance? This totally blows my mind. I would never have guessed that GPT could get ANY of these right. I mean, is there a data point in the dataset used to train where you can read 2241 + 19873 = 22114? Quite unlikely... And those multiplications. It's consistently getting the number of digits right and the first two numbers correct. How the hell does this happen? Sure, it's sometimes…
>I mean, is there a data point in the dataset used to train where you can read 2241 + 19873 = 22114? Quite unlikely...
But there might be something like xxx1 + xxxx3 = xxxx4 in the dataset so it can learn the pattern.
Re: I asked GPT-NeoX-20B a hundred arithmetic questions
#35This just shows that this model did not learn anything. Humans do not see billions of examples to add numbers. We see just few and can apply learned notation and procedures to infinity with 100% precision. GPT-3 learned mathematical intuition. Humans can hardly learn multiplication table over months and repetitions of same examples, and that table hardly matters at all. GPT-3 is just plainly wrong objective they tryi…
The vast majority of humans don't just see a few examples and figure it out. They're taught an algorithm. Eventually they may also come up with another algorithm, but they're taught one first. They also don't have "100% precision". Many, many humans are incredibly bad at math, and even the ones that are good at it often make mistakes.
>They also don't have "100% precision". Many, many humans are incredibly bad at math,
Many humans are bad at surgery this does not mean that an AI that is slightly better then the average human is an accomplishment.
On the other hand someone could write the algorithms for math and teach an AI when and how to use it. The rules of math are clear you don't need a bad search algorithm to approximate them for a extremely limited subset of inputs.
Re: I asked GPT-NeoX-20B a hundred arithmetic questions
#36If you want to play with the model, you can (with difficulty) for free at https://goose.ai/playground. You have to log in, but thankfully you can via google.
The playground crashes every minute, and the defaults ruin your outputs (temperature 1, really? 0.7 to 0.8 is a necessity, with top-k 40), and they turned off autocorrect on mobile, presumably because they hate you and your family for owning an iPad, but you can indeed play with it.
The outputs feel pretty magical, too. With the settings above, it started printing... an IRC conversation? https://gist.github.com/shawwn/9a201990196b61cd21847487185dd... This is impressive, because I'm not sure we explicitly included any IRC logs in The Pile.
re: the current title "GPT-3's answers to arithmetic questions": We've come full circle. I used to give Eleuther a hard time for confusing people. But now that people confuse themselves, they should declare victory. It's as close to success as an open source effort could hope for. And with only years of work -- not too shabby.
You can join them: https://www.eleuther.ai/faq/
GPT-NeoX-20B paper: http://eaidata.bmk.sh/data/GPT_NeoX_20B.pdf
Re: I asked GPT-NeoX-20B a hundred arithmetic questions
#37What? You think this is poor performance? This totally blows my mind. I would never have guessed that GPT could get ANY of these right. I mean, is there a data point in the dataset used to train where you can read 2241 + 19873 = 22114? Quite unlikely... And those multiplications. It's consistently getting the number of digits right and the first two numbers correct. How the hell does this happen? Sure, it's sometimes…
>I mean, is there a data point in the dataset used to train where you can read 2241 + 19873 = 22114? Quite unlikely... But there might be something like xxx1 + xxxx3 = xxxx4 in the dataset so it can learn the pattern.
That's the astonishing bit
Re: I asked GPT-NeoX-20B a hundred arithmetic questions
#38What? You think this is poor performance? This totally blows my mind. I would never have guessed that GPT could get ANY of these right. I mean, is there a data point in the dataset used to train where you can read 2241 + 19873 = 22114? Quite unlikely... And those multiplications. It's consistently getting the number of digits right and the first two numbers correct. How the hell does this happen? Sure, it's sometimes…
Regardless of the base, be it text or encoded numbers, text, as encoded is just a different base of representing a number('325' is '3' * (2^32) + '2' * (2^16) + '5' * 2^8 = 51 * 2^32 + 50 * 2^16 + 53 * 2^8). Neural networks can approximate polynomials, and additions/substractions & multiplications can be approximated - in this case, the base is just not '10' but 'ASCII'. I think that if you tried to train it only with arithmetic expressions (in text) it should get even better, it should just approximate the underlying operation - it does not need to understand the text.
Re: I asked GPT-NeoX-20B a hundred arithmetic questions
#39A really intelligent AI would just use a calculator.
Re: I asked GPT-NeoX-20B a hundred arithmetic questions
#40What? You think this is poor performance? This totally blows my mind. I would never have guessed that GPT could get ANY of these right. I mean, is there a data point in the dataset used to train where you can read 2241 + 19873 = 22114? Quite unlikely... And those multiplications. It's consistently getting the number of digits right and the first two numbers correct. How the hell does this happen? Sure, it's sometimes…
>I mean, is there a data point in the dataset used to train where you can read 2241 + 19873 = 22114? Quite unlikely... But there might be something like xxx1 + xxxx3 = xxxx4 in the dataset so it can learn the pattern.
They should have asked questions like:
What is twothousandfortyone plus nineteenthousandeighthundredseventythree?