Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I asked GPT-NeoX-20B a hundred arithmetic questions

twitter.com

Post reply on HN
Print view View original

181–190 of 235 posts

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#181

Post by rasen58 »

jonathankoren wrote: post #179
0xTJ wrote: post #48

Earlier quoted context omitted.

I think you're completely wrong. This shows that the model learned a lot about at-a-glance math. Sure if you sit down with pen and paper you can get the answer, but few people could do these reliably in their head. But what you can do is figure the order of magnitude, and get a rough answer for the first few digits and last digits, each with their chance of being wrong. If anything, this shows that it learned math de…

No. A million times no. It’s a language model. It doesn’t understand math at all. It doesn’t even understand language. All it did was spit out something that looks like math. It’s fancy automatic writing. I’ll concede that if you tokenized the equations correctly, you might be able to get a language model to learn arithmetic, since it’s just symbol manipulation; but to make the leap that a general text model has lear…

Isn’t your comment that you wrote here also just a bunch of “symbol manipulation“?

It definitely hasn’t learned math but it definitely has learned general concepts

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#182

Post by YeGoblynQueenne »

moyix wrote: post #148

Hey! As the author of the gist, just wanted to clear up what seem to be a few misconceptions: - This isn't GPT-3, it's the recently-released open-source and open-weights model from EleutherAI, GPT-NeoX-20B. GPT-3 is much larger (175 billion parameters vs NeoX's 20 billion). - It's well-known that language models don't tend to be good at math by default (Gwern, among others, pointed this out back in June 2020). It see…

How can a language model invent algorithms for arithmetic? How would an algorithm be represented in a language model? Isn't that the first thing to ask, before starting to look for algorithms?

For example, if I take a stroll on the beach, am I likely to see any algorithms coalescing in the grains of sand?

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#183

Post by YeGoblynQueenne »

Isinlor wrote: post #177
YeGoblynQueenne wrote: post #174

Earlier quoted context omitted.

I don't think I understand what you mean. Aren't all the sequences on the Online Encyclopedia of Integer Sequences created by humans? We clearly have the tools to extrapolate sequences from examples, rather than just eyballing them and trying to guess them. For instance: we have maths. So I must have misunderstood your meaning?

If you look at the 3 sequences I gave you, can you guess following elements of the sequence? We can create sequences, but guessing underlying patterns is a lot more difficult. Humans will have very hard time if you go beyond around 10 operators in a pattern used to generate a sequence. My guess is that their model will be better at it than me or you.

Ah, I think I see what you mean: you are saying that because it's better than humans at predicting the next element in a sequence it's good at generalising. Is that correct, or am I misrepresenting your point?

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#184

Post by YeGoblynQueenne »

Isinlor wrote: post #78
YeGoblynQueenne wrote: post #59

Earlier quoted context omitted.

As far as I can tell from a quick heuristic perusal, the "Generalization Beyond Overfitting" paper reports "generalisation" _on the validation set_. That's not particularly impressive and it's not particularly "generalisation" either. Actually, I really don't grokk this (if I may). I often see deep learning work reporting generalisation on the validation set. What's up with that? Why is generalisation on the validati…

The point of the paper is to show that NN can still learn long after fully memorizing the train dataset. This behavior goes against current paradigm of thinking about training NNs. It is just very unexpected, similarly as double descent is unexpected from classical statistics point of view that more parameters lead to more over-fitting. They could have split validation test set into validation and test sets, but I do…

>> The point of the paper is to show that NN can still learn long after fully memorizing the train dataset.

That is the claim in the paper. I don't understand how it is supported by measuring results on the validation set.

Figure 3 looks nice but it doesn't say anything on its own. I don't know what's the best way to interpret it. The paper offers some interpretation that convinces you, but not me. Sorry, this kind of work is too fuzzy for me. What happened to good, old-fasion proofs?

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#185

Post by wallacoloo »

ceejayoz wrote: post #88
jcims wrote: post #71

Earlier quoted context omitted.

For what it's worth you can get access to the OpenAI playground. You might have to put yourself on a waiting list. https://beta.openai.com/playground I tried a couple different things to pick it apart. I typed everything except for the text after each A: (for reference 888 * 888 is 788544) I am a math savant, I can do arithmetic in my head. I will provide an answer, then show how I arrived at the answer. Q: What is 8…

> How did you arrive at that answer? This sort of question - "you don't remember what I just asked you?" - is one of the most glaring failings in Apple's Siri and I hope this leads to it being solved soon.

it could be interesting to gauge how entwined the “how did you arrive at that answer” process is with the answering itself. i.e. which paths do they share? even at this early a stage: is there some structure which is used to determine the operand(s) that’s leveraged in both of these prompts? is the “how did you X” answer leveraging most of the “X” circuitry and just bailing out early? or does it deviate as early as post-tokenization?

philosophers would like to know.

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#187

Post by moyix »

YeGoblynQueenne wrote: post #182
moyix wrote: post #148

Hey! As the author of the gist, just wanted to clear up what seem to be a few misconceptions: - This isn't GPT-3, it's the recently-released open-source and open-weights model from EleutherAI, GPT-NeoX-20B. GPT-3 is much larger (175 billion parameters vs NeoX's 20 billion). - It's well-known that language models don't tend to be good at math by default (Gwern, among others, pointed this out back in June 2020). It see…

How can a language model invent algorithms for arithmetic? How would an algorithm be represented in a language model? Isn't that the first thing to ask, before starting to look for algorithms? For example, if I take a stroll on the beach, am I likely to see any algorithms coalescing in the grains of sand?

Invent is probably the wrong word since it implies agency, sure. Maybe "discover" or "luck upon", since whatever it's doing was formed by updating a pile of floating point weights with gradient descent?

I think it certainly makes sense to ask what the higher level "algorithm" at work here is, though. Electrons flow through wires and transistors in (say) an adder [1]; looking at the wires and transistors you won't see an algorithm for addition, but there is certainly one present, codified in the arrangement of those wires and transistors. But maybe we can reverse engineer whatever the LM is doing by a combination of probing it with experiments like these and (maybe) inspecting the learned weights. The Curve Circuits paper did this for reverse engineering a curve detector learned by a convolutional neural network: https://distill.pub/2020/circuits/curve-circuits/

I also don't mean to imply that it's a good algorithm, or one that generalizes to arbitrary numbers, etc. Maybe it's just (effectively) a lookup table and some special cases!

[1] Please don't yell at me for this metaphor, I bailed out of physics after scraping out a B- in E&M ;)

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#188

Post by supermdguy »

nlh wrote: post #98

I discovered something like this in real-world usage. I’ve have GitHub Copilot running in VSCode and I’ve been experimenting with how it works when doing plain text accounting (using ledger / hledger). The ledger files are somewhat “code”-like so it’s been super interesting to see how it works. The short answer: it works really quite well! ..except for the math part :) I have a long ledger of transactions, and I can…

I also have Copilot running, and I was surprised when it had pretty good autosuggestions when writing proofs in Latex! A lot of times it has subtle logical errors in the proof, but the syntax is always correct. And there have been a few times when it gives a sentence or two that's exactly right

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#189

Post by YeGoblynQueenne »

moyix wrote: post #187
YeGoblynQueenne wrote: post #182

Earlier quoted context omitted.

How can a language model invent algorithms for arithmetic? How would an algorithm be represented in a language model? Isn't that the first thing to ask, before starting to look for algorithms? For example, if I take a stroll on the beach, am I likely to see any algorithms coalescing in the grains of sand?

Invent is probably the wrong word since it implies agency, sure. Maybe "discover" or "luck upon", since whatever it's doing was formed by updating a pile of floating point weights with gradient descent? I think it certainly makes sense to ask what the higher level "algorithm" at work here is, though. Electrons flow through wires and transistors in (say) an adder [1]; looking at the wires and transistors you won't see…

Don't worry, I won't yell at you :)

I'm fine with "invent" actually, despite the implication of agency (I'm used to the terminology "predicate invention" [1]; although maybe I should actually re-examine the motivation behind it).

I'm more interested in the representation issue. I had a look at the quoted article on CNNs earlier. I think there is a very fine line between claiming that a CNN's weights represent an algorithm and that its weights can be _interpreted_ as an algorithm. I feel that the article leans too heavily on the interpretation side and doesn't make enough of an effort to show that the CNNs weight really represent an algorithm, rather than having activations in subsequent layers and therefore with a natural ordering.

In any case, I would like to understand how a language model can represent an algorithm.

_____________

[1] https://link.springer.com/referenceworkentry/10.1007/978-0-3...

Re: I asked GPT-NeoX-20B a hundred arithmetic questions

#190

Post by ggrrhh_ta »

kripke wrote: post #126
ggrrhh_ta wrote: post #62

Earlier quoted context omitted.

I pointed it out above; even though it is text, the ASCII representation is just a different base for the numbers - base 2^8 - ('325' is '3' * (2^32) + '2' * (2^16) + '5' * 2^8 = 51 * 2^32 + 50 * 2^16 + 53 * 2^8); it should approximate those polynomial functions very well.

Does GPT know about ASCII? My understanding was that these models use a dictionary of (initially) random vectors as input and learn their own text representation.

In that case, I would say that GPT's performance in arithmetic is something that we see because we are looking for it or want to find it but that is not there. It is an illusion. If we have no theory of why would it an arithmetic capability would emerge from GPT, then, there is no scientific discovery; at most, there a field survey, a taxonomist work, but no understanding is generated.
Post reply on HN