Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I asked GPT-NeoX-20B a hundred arithmetic questions

twitter.com

Post reply on HN
Print view View original

191–200 of 235 posts

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#191

Post by chaxor »

mordymoop wrote: post #94
vbuterin wrote: post #79

Earlier quoted context omitted.

When I do mental arithmetic my brain frequently tokenizes into digit pairs or triples if I can recognize pairs and triples that have specific properties. "224" is actually a really nice object to recognize because it's 7 * 32, and if you can recognize other multiples of 32 it frequently gives you shortcuts. It's less useful for addition because you would need to get lucky and get a multiple of 32 (or 7) on both sides…

Sure - I think we all learn tricks like that. But you learned that pattern of tokenization, it wasn't arbitrarily foisted on you. What GPTs have to deal with is more like, you are fed an arithmetic problem via colored slips of paper, and you just have to remember that this particular shade of chartreuse means "224", which you happen to have memorized equals 7 * 32, etc., but then the next slip of paper is off-white w…

The tokens in most gpt models are small like this, but they still 'learn tokenization' very similar to what you just mentioned. It's part of the multi headed attention.

It learns what level of detail in the tokenization is needed for given tasks. For example, If you're not interested in parsing the problem for actually doing the computation for example, you don't pay attention to the finer tokenization'. If you do need that level of detail, you use those finer groupings. Some of the difficulty a few years ago was trying to extend these models to handle longer contexts (or just variable contexts which can go to very long), but that also seems close to solved now too. So you're not exactly giving much insight with this observation.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#192

Post by chaxor »

williamtrask wrote: post #44
qumpis wrote: post #37

Earlier quoted context omitted.

That's the astonishing bit

It really isn’t. You see a lot of things when reading 500 billion tokens

Yeah, I'm *totally* unimpressed.

I, for one, learn all my math without ever seeing any math or logic examples at all.

"Teacher, what is this '34+12' stuff - I've already developed a complete grand unification theory on my own - I don't need examples of what you call 'addition'" - apparently everyone unimpressed by nlp today

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#193

Post by moyix »

YeGoblynQueenne wrote: post #189
moyix wrote: post #187

Earlier quoted context omitted.

Invent is probably the wrong word since it implies agency, sure. Maybe "discover" or "luck upon", since whatever it's doing was formed by updating a pile of floating point weights with gradient descent? I think it certainly makes sense to ask what the higher level "algorithm" at work here is, though. Electrons flow through wires and transistors in (say) an adder [1]; looking at the wires and transistors you won't see…

Don't worry, I won't yell at you :) I'm fine with "invent" actually, despite the implication of agency (I'm used to the terminology "predicate invention" [1]; although maybe I should actually re-examine the motivation behind it). I'm more interested in the representation issue. I had a look at the quoted article on CNNs earlier. I think there is a very fine line between claiming that a CNN's weights represent an algo…

> I think there is a very fine line between claiming that a CNN's weights represent an algorithm and that its weights can be _interpreted_ as an algorithm.

Yeah, I agree this is an issue. It feels a bit reminiscent of Searle's Waterfall argument, and so I'm inclined to turn to Scott Aaronson's response here [1; Section 6] – basically, how much work is the interpretation itself doing? If you actually tried to use the "algorithm" to do what your interpretation says it should do, how much work would you have to put into that mapping? If the work required amounts to just implementing the algorithm and effectively ignoring the CNN (or waterfall), then the interpretation is what was doing all the work.

IMO the Curve Circuits case passes this test, since they show that you can mechanically take out the learned weights, drop in the weights derived algorithm they reverse engineered, and the model still works about as well.

> In any case, I would like to understand how a language model can represent an algorithm.

Likewise! :)

[1] https://www.scottaaronson.com/papers/philos.pdf

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#194

Post by ravi-delia »

YeGoblynQueenne wrote: post #189
moyix wrote: post #187

Earlier quoted context omitted.

Invent is probably the wrong word since it implies agency, sure. Maybe "discover" or "luck upon", since whatever it's doing was formed by updating a pile of floating point weights with gradient descent? I think it certainly makes sense to ask what the higher level "algorithm" at work here is, though. Electrons flow through wires and transistors in (say) an adder [1]; looking at the wires and transistors you won't see…

Don't worry, I won't yell at you :) I'm fine with "invent" actually, despite the implication of agency (I'm used to the terminology "predicate invention" [1]; although maybe I should actually re-examine the motivation behind it). I'm more interested in the representation issue. I had a look at the quoted article on CNNs earlier. I think there is a very fine line between claiming that a CNN's weights represent an algo…

I mean it's just a generic blob of machine learning. It learned to do math because we write about math. It can't do arbitrarily long computations since it's only so recursive, but getting the basics down isn't too hard. It probably only struggles because it can't see digits, and can't do digit-wise operations (same with rhyming actually!).

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#195

Post by ravi-delia »

mlb_hn wrote: post #161
moyix wrote: post #148

Hey! As the author of the gist, just wanted to clear up what seem to be a few misconceptions: - This isn't GPT-3, it's the recently-released open-source and open-weights model from EleutherAI, GPT-NeoX-20B. GPT-3 is much larger (175 billion parameters vs NeoX's 20 billion). - It's well-known that language models don't tend to be good at math by default (Gwern, among others, pointed this out back in June 2020). It see…

I get the tokenization argument and it may influence it a bit, but I suspect the n-digit math issue has to do more with search the way it samples (in the bpe link gwern references some experiements I'd done with improving n-digit math by chunking using commas, http://gptprompts.wikidot.com/logic:math ). I think since it samples left to right on the first pass, it's not able to predict well if things carry from right…

I feel like attention would largely mitigate that, no? Has anyone looked at what the weights are while doing addition?

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#196

Post by gwern »

mlb_hn wrote: post #173
moyix wrote: post #169

Earlier quoted context omitted.

Yup, quite possible that this has something to do with it. There is other work showing that giving LMs a "scratchpad" for intermediate computations allows them to do much better not just at arithmetic but also things like predicting the output of some code: https://arxiv.org/abs/2112.00114

definitely. also works on text translation/comprehension like emojis! https://aidungeon.medium.com/introducing-ai-dungeon-translat... . For actual benchmarks, scratchpad improves GPT-Davinci WIC from 50% accuracy (chance) to nearly 70%. I think the check and validate is a different sort of scratchpad but maybe not. Seems like at least 3 types - soe for pulling implicit info out of the network viz wic, sometimes for i…

The big caveat here is that the inner monologue papers generally work with GPT-3-175b, LaMDA, or Gopher, all of which are much bigger than 20b, and they generally show phase transitions (https://old.reddit.com/r/mlscaling/comments/sjzvl0/d_instanc...) in the monologue capability: below a critical size, inner monologue doesn't work at all, performing worse than baseline even, no matter how they scale, and only past the critical size does inner monologue suddenly start working much better. So it's possible (has anyone checked?) that GPT-NeoX-20b just isn't large enough to do inner monologue.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#197

Post by moffkalast »

dlkf wrote: post #19
nikolayasdf123 wrote: post #8

This just shows that this model did not learn anything. Humans do not see billions of examples to add numbers. We see just few and can apply learned notation and procedures to infinity with 100% precision. GPT-3 learned mathematical intuition. Humans can hardly learn multiplication table over months and repetitions of same examples, and that table hardly matters at all. GPT-3 is just plainly wrong objective they tryi…

I'll preface this by saying that I am 100% in the camp that thinks these language models are neither intelligent nor a promising avenue towards understanding intelligence. But your conclusion here is entirely wrong: the model clearly is learning something. From eyeballing this, the model is right about 10% of the time. If it were spitting out random digits the accuracy would effectively be zero. So exactly what is it…

I think fundamentally these models compress the learning data into network weights and connections, so in effect if the learning data was 6 + 10 = 16 and 9 + 10 = 19, then you give it 7 + 10 it'll interpolate between what it's seen or something of the sort, giving you something approximately right. It's also not lossless compression so what it may have actually inside is 9 + 10 = 18 so yeah.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#198

Post by moffkalast »

williamtrask wrote: post #5
spupe wrote: post #3

Thank you for this. Technically it's not GPT-3, but GPT-NeoX-20B, although they are based on a similar architecture. The poor performance is most likely due to not having a large database of math problems to draw from. Github, for example, is part of the dataset that is used to train both GPT-3 and GPT-Neo variants, which is partly why they can generate meaningful code (sometimes). I wonder how a model finetuned for…

Poor performance is more likely due to how transformer neural networks view numbers. It memorises them like words instead of modeling their numerical structure. Thus even if it’s seen the number 3456 and 3458, it knows nothing of 3457. Totally different embedding. It’s like a kid memorising a multiplication table instead of learning the more general principle of multiplication (related: this illusion is why big model…

> this illusion is why big models are so popular. Memorise more stuff

It's all just a compressed lookup table that can handle in-betweens.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#200

Post by moffkalast »

guantanamo_bob wrote: post #121
tehsauce wrote: post #84

The latest version of gpt-3 spooked me the other day with: Me: Can here you here still here understand here what here I here am here asking here if here I here put here words here in here between? Gpt: Yes, I can understand what you are asking if you put words in between. Me: What was the word that I put in between? Gpt: The word that you put in between is "here."

Must :clap: have :clap: learned :clap: that :clap: from :clap: all :clap: the :clap: clapping :clap: that :clap: goes :clap: on :clap:.

But does GPT-3 know what based is based on? Or why kids love the taste of Cinnamon Toast Crunch?
Post reply on HN