Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I asked GPT-NeoX-20B a hundred arithmetic questions

twitter.com

Post reply on HN
Print view View original

161–170 of 235 posts

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#161

Post by mlb_hn »

moyix wrote: post #148

Hey! As the author of the gist, just wanted to clear up what seem to be a few misconceptions: - This isn't GPT-3, it's the recently-released open-source and open-weights model from EleutherAI, GPT-NeoX-20B. GPT-3 is much larger (175 billion parameters vs NeoX's 20 billion). - It's well-known that language models don't tend to be good at math by default (Gwern, among others, pointed this out back in June 2020). It see…

I get the tokenization argument and it may influence it a bit, but I suspect the n-digit math issue has to do more with search the way it samples (in the bpe link gwern references some experiements I'd done with improving n-digit math by chunking using commas, http://gptprompts.wikidot.com/logic:math). I think since it samples left to right on the first pass, it's not able to predict well if things carry from right to left.

I think can mitigate the search issue a bit if you have the prompt double-check itself after the fact (e.g. https://towardsdatascience.com/1-1-3-wait-no-1-1-2-how-to-ha...). Works different depending on the size of the model tho.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#162

Post by Isinlor »

YeGoblynQueenne wrote: post #56
Isinlor wrote: post #13

Earlier quoted context omitted.

Take a look at this paper: Deep Symbolic Regression for Recurrent Sequences https://arxiv.org/abs/2201.04600 If you look at embedding visualization it is very clear that the model learns order of numbers. (Interactive demo: http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com... ) There is also: Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets https://arxiv.org/abs/2201.02177 Again, l…

The "Deep Symbolic Regression" paper reports very poor generalisation results that break off after a small n (where n is the number of tokens in the predicted sequence). It works some of the time for n = 1 (predicts the next token) but accuracy drops off for n = 10. No results are reported for N > 10 as far as I can tell in the "Out of Domain Generalization" section (which is the meat and potatoes of the "generalizat…

I don't think you will find any human that will extrapolate sequence generated with more than 10 operators. And longer input sequences are actually easier to handle - fig 1. the right most graph.

If you think you can do better than their program then:

Seq1: [0, 1, 2, 3, 6, 7, 13, 26, 32, 58, 116, 142, 258, 516]

Seq2: [2, 2, 3, 5, 10, 12, 22, 44, 54, 98, 196, 240, 436, 872]

Seq3: [3, 1, 8, 9, 18, 19, 37, 74, 92, 166, 332, 406, 738, 1476]

Their program is able to guess correct continuation with one more sequence element.

SHA1 hash for verification: bef5e213340f91258b3b9a0042c9c083dd91cb80

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#163

Post by kordlessagain »

moyix wrote: post #148

Hey! As the author of the gist, just wanted to clear up what seem to be a few misconceptions: - This isn't GPT-3, it's the recently-released open-source and open-weights model from EleutherAI, GPT-NeoX-20B. GPT-3 is much larger (175 billion parameters vs NeoX's 20 billion). - It's well-known that language models don't tend to be good at math by default (Gwern, among others, pointed this out back in June 2020). It see…

What I did was train GPT-3 that I was asking a math question and then have it run some JavaScript to do the math with the text it thought was math formulas to get the answer.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#165

Post by edouard-harris »

baalimago wrote: post #159

I'll be impressed when the AI consults with an ordinary calculator for the correct answer

This already exists: Google's recently-published LaMDA dialogue model [1] is trained to consult a calculator for arithmetic questions and consistently succeeds at it.

[1] https://arxiv.org/abs/2201.08239v2

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#166

Post by jjoonathan »

olyjohn wrote: post #160
jjoonathan wrote: post #101

Earlier quoted context omitted.

Like religion, I suspect you will have many different machine answers to choose from.

I suspect we will have 2 of them. We will start out with lots of them, but then a couple of them will start making the most money, and resort to underhanded tactics, bribery, and lobbying, and put the others out of business.

Sounds about right. We'll replace our lizard overlords with robot overlords.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#167

Post by moyix »

kordlessagain wrote: post #163
moyix wrote: post #148

Hey! As the author of the gist, just wanted to clear up what seem to be a few misconceptions: - This isn't GPT-3, it's the recently-released open-source and open-weights model from EleutherAI, GPT-NeoX-20B. GPT-3 is much larger (175 billion parameters vs NeoX's 20 billion). - It's well-known that language models don't tend to be good at math by default (Gwern, among others, pointed this out back in June 2020). It see…

What I did was train GPT-3 that I was asking a math question and then have it run some JavaScript to do the math with the text it thought was math formulas to get the answer.

Yeah it's interesting how moving up a level of abstraction works well here! "Write me a function that multiplies two numbers" works much better than trying to get it to multiply the numbers themselves. There's a recent-ish paper exploring this:

https://arxiv.org/abs/2108.07732

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#168

Post by newsbinator »

andai wrote: post #128
newsbinator wrote: post #124

Earlier quoted context omitted.

If you want to get to AGI, it doesn't really have to be a remote possibility with current techniques. The only thing current techniques would have to do is build something that improves itself intelligently enough. Then the next iteration and the next, until you get to an iteration in which AGI is a remote possibility with current techniques and implementations. We're currently far from AGI, but I'm not sure we're fa…

I still think we'd be better off making a thing that makes smarter humans.

That would be AGI by another name.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#169

Post by moyix »

mlb_hn wrote: post #161
moyix wrote: post #148

Hey! As the author of the gist, just wanted to clear up what seem to be a few misconceptions: - This isn't GPT-3, it's the recently-released open-source and open-weights model from EleutherAI, GPT-NeoX-20B. GPT-3 is much larger (175 billion parameters vs NeoX's 20 billion). - It's well-known that language models don't tend to be good at math by default (Gwern, among others, pointed this out back in June 2020). It see…

I get the tokenization argument and it may influence it a bit, but I suspect the n-digit math issue has to do more with search the way it samples (in the bpe link gwern references some experiements I'd done with improving n-digit math by chunking using commas, http://gptprompts.wikidot.com/logic:math ). I think since it samples left to right on the first pass, it's not able to predict well if things carry from right…

Yup, quite possible that this has something to do with it. There is other work showing that giving LMs a "scratchpad" for intermediate computations allows them to do much better not just at arithmetic but also things like predicting the output of some code: https://arxiv.org/abs/2112.00114
Post reply on HN