Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I asked GPT-NeoX-20B a hundred arithmetic questions

twitter.com

Post reply on HN
Print view View original

81–90 of 235 posts

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#81

Post by ur-whale »

Yup, glorified, extremely large, lookup table, but has zero clues of how the world works.

Or to put it another way: if I haven't seen the result before I'm very likely to get it wrong.

And inferring the rules of arithmetic from examples : nope.

We're all very impressed with the large language models because human language was always very hard to get a handle one.

Now that we've made progress in manipulating natural language, we are very quick to believe that the darn thing is somehow "intelligent" because we've always attached some sort o mystical quality to language.

This demonstrates that the capacity of these systems to reason and craft high order rules from enough examples is exactly nil.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#84

Post by tehsauce »

The latest version of gpt-3 spooked me the other day with:

Me: Can here you here still here understand here what here I here am here asking here if here I here put here words here in here between?

Gpt: Yes, I can understand what you are asking if you put words in between.

Me: What was the word that I put in between?

Gpt: The word that you put in between is "here."

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#85

Post by varjag »

wccrawford wrote: post #27
nikolayasdf123 wrote: post #8

This just shows that this model did not learn anything. Humans do not see billions of examples to add numbers. We see just few and can apply learned notation and procedures to infinity with 100% precision. GPT-3 learned mathematical intuition. Humans can hardly learn multiplication table over months and repetitions of same examples, and that table hardly matters at all. GPT-3 is just plainly wrong objective they tryi…

The vast majority of humans don't just see a few examples and figure it out. They're taught an algorithm. Eventually they may also come up with another algorithm, but they're taught one first. They also don't have "100% precision". Many, many humans are incredibly bad at math, and even the ones that are good at it often make mistakes.

GPT3 surely has several algorithms for addition in its training corpus. Just unable to make good use of them.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#86

Post by dr_zoidberg »

vbuterin wrote: post #79
mordymoop wrote: post #65

When you toss “2241 + 19873 =” into an applet that shows you the default tokenization scheme GPT-3 uses, you get this: (224)(1)( +)( 198)(73)( =) I’ve heard it remarked before that, while tokenization is obviously an unavoidable part of a model with an architecture like GPT, this is a very silly way of tokenizing number strings for the purposes of learning or doing arithmetic. Indeed, I think a lot of GPT-3’s puzzlin…

When I do mental arithmetic my brain frequently tokenizes into digit pairs or triples if I can recognize pairs and triples that have specific properties. "224" is actually a really nice object to recognize because it's 7 * 32, and if you can recognize other multiples of 32 it frequently gives you shortcuts. It's less useful for addition because you would need to get lucky and get a multiple of 32 (or 7) on both sides…

I think that part of why the tokenization is a proble for math here is that it doesn't seem to be carrying overflow into the left token. Anyway, I haven't worked with GPT in detail to do a deeper analysis than that hunch, so take my comment with a couple of salt grains.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#88

Post by ceejayoz »

jcims wrote: post #71
sillysaurusx wrote: post #36

If you want to play with the model, you can (with difficulty) for free at https://goose.ai/playground . You have to log in, but thankfully you can via google. The playground crashes every minute, and the defaults ruin your outputs (temperature 1, really? 0.7 to 0.8 is a necessity, with top-k 40), and they turned off autocorrect on mobile, presumably because they hate you and your family for owning an iPad, but you ca…

For what it's worth you can get access to the OpenAI playground. You might have to put yourself on a waiting list. https://beta.openai.com/playground I tried a couple different things to pick it apart. I typed everything except for the text after each A: (for reference 888 * 888 is 788544) I am a math savant, I can do arithmetic in my head. I will provide an answer, then show how I arrived at the answer. Q: What is 8…

> How did you arrive at that answer?

This sort of question - "you don't remember what I just asked you?" - is one of the most glaring failings in Apple's Siri and I hope this leads to it being solved soon.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#89

Post by ceejayoz »

wildmanx wrote: post #61
NateEag wrote: post #57

Earlier quoted context omitted.

Not really. Average person on the street is going to correctly say "geez, I dunno. Can I use my phone?" If you don't forbid them to, then they'll whip it out and get 98% correct (I figure they'll typo a few). This model didn't have enough understanding to do that (since it literally has no understanding at all).

I don't know where you live, but 98% correct is not what would happen around here. Edit: Oh, 98% _with_ a calculator. What if you force them to do it by hand?

> What if you force them to do it by hand?

Why?

GPT-3 isn't doing it that way. The importance is understanding the question, not being able to do math in your head.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#90

Post by spiderfarmer »

tehsauce wrote: post #84

The latest version of gpt-3 spooked me the other day with: Me: Can here you here still here understand here what here I here am here asking here if here I here put here words here in here between? Gpt: Yes, I can understand what you are asking if you put words in between. Me: What was the word that I put in between? Gpt: The word that you put in between is "here."

That's shockingly good.
Post reply on HN