Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

And yet It Understands

borretti.me

Post reply on HN
Print view View original

51–60 of 231 posts

Re: And yet It Understands

#51

Post by doctor_eval »

glenstein wrote: post #29
zetalyrae wrote: post #14

Earlier quoted context omitted.

This does not follow. You can know a lot about, say, number theory, and still make elementary arithmetic errors.

Right, over the totality of things that it reasons about, to some degree it will make inroads to correctly answering these kinds of questions, and in some ways it'll make errors, and what's interesting, is it'll make errors because the way in which it's attempting to answer them bears a lot of the hallmarks that we associate with conceptual understanding, rather than the mechanical operations of a calculator which tr…

Yes, exactly this.

I recently asked ChatGPT to write me some Go code. What it wrote was mostly fine, but it incorrectly used a method invocation on an object, instead of the correct code which was to pass the object to a function at the package level (aka static method).

I think it's a real stretch to suggest that this could happen as a result of simply regurgitating strings it's seen before, because the string it emitted was not ever going to work. To me, it looked for all the world like the same kind of conceptual error that I have made in Go, and the only way I could see this working is if GPT had a (slightly faulty) model of the Go language, or of the particular (and lesser known) package I was asking about.

It's felt more like it "forgot" - like me - how the package worked, so it followed its model instead. That error was WAY more interesting than the code.

Re: And yet It Understands

#52

Post by _dain_ »

foldr wrote: post #9

>The other day I saw this Twitter thread. Briefly: GPT knows many human languages, InstructGPT is GPT plus some finetuning in English. Then they fed InstructGPT requests in some other human language, and it carries them out, following the English-language finetuning. >And I thought: so what? Isn’t this expected behaviour? Then a friend pointed out that this is only confusing if you think InstructGPT doesn’t understan…

The point isn't that it can translate between languages. It's not translating the instructions, at least not explicitly. Here's what they did:

- They found a task that GPT wasn't very good at, because examples of that task weren't in the training set (in any language).

- They trained a fine-tuned variant of GPT where examples of the task were in an appended training set, but only in English.

- They told the variant to do that task again, in other languages.

- Its performance improved on all of them, not just English

If the training set has no Mandarin examples of the task, how did it get better when you ask it in Mandarin? Sure, you could fake this by having an API call to Google Translate where it takes the Mandarin request, translates to English, solves the task in English, then Google Translates back to Mandarin. But it's not doing that, it's just doing the same "predict the next token" operation on the Mandarin prompt.

I don't see how it can do that unless it really has some kind of understanding.

Re: And yet It Understands

#53

Post by civilized »

zetalyrae wrote: post #43
civilized wrote: post #30

Earlier quoted context omitted.

But it's much worse than that for ChatGPT. A person would know that their mental math division of large numbers is unreliable, but if you ask ChatGPT about the reliability of its answers, it is uniformly confident about almost everything. It has no grasp of the difference between the things it "knows" which are true and the things it "knows" which may be false. And this is the error I notice people very consistently…

The failures don't refute the successes. Anything and anyone can fail, you don't get intelligent output by chance. If it makes mistakes we wouldn't make, on obvious things, it is because it is an alien form of intellingece. RLHF and the tokenizer together explain many of the more common failure modes.

Nobody is saying the intelligent output is by chance. This is a machine that is fed terabytes of intelligent inputs and is able to produce intelligent outputs. So one explanation of its producing intelligent outputs is that it's basically regurgitating what it was fed.

The way to test that, of course, is to give it problems that it hasn't seen. Unfortunately, because GPT has seen so much, giving it problems it definitely hasn't seen is itself now a hard problem. What's been shown so far: OpenAI's benchmarking is not always rigorous or appropriate, and GPT's performance is brittle and highly sensitive to the problem phrasing [1].

I agree with the article that GPT's training enables it to access meaningful conceptual abstractions. But there is clearly quite a lot that's still missing. For now, people are too excited to care. But when they try to deliver on their promises, the gaps will still be there. Hopefully at that point we will embark towards a new level of understanding.

[1] https://aisnakeoil.substack.com/p/gpt-4-and-professional-ben...

Re: And yet It Understands

#54

Post by rain1 »

doctor_eval wrote: post #35
efxhoy wrote: post #20

> But nobody knows how GPT works. They know how it was trained, because the training scheme was designed by humans, but the algorithm that is executed during inference was not intelligently designed but evolved, and it is implicit in the structure of the network, and interpretability has yet to mature to the point where we can draw a symbolic, abstract, human-readable program out of a sea of weights. Nobody knows how…

And what I think we'll find is that we are, essentially, stochastic parrots. (I'm OK with that. It is what it is).

sometimes. :)

sometimes we are so much more.

Re: And yet It Understands

#55

Post by rhn_mk1 »

foldr wrote: post #12
Etheryte wrote: post #11

Earlier quoted context omitted.

This is a common misconception. ChatGPT is not supposed to be good at this — it's a language model, not a maths model or data science model or whatnot. This is exactly why they have plugins, such as the one for Wolfram Alpha.

Of course people who actually know how ChatGPT works don't expect it to be able to magically solve mathematical problems. However, these examples do show that ChatGPT isn't (contrary to some of the hype) deriving a deep conceptual understanding of its input data.

It's not deriving an understanding of this particular concept. Which is unsurprising given that it was not trained to be good at maths.

That in no way precludes deriving deep understanding of some other concepts.

Even humans are not expected to be able to learn anything we throw at them.

Re: And yet It Understands

#56

Post by stereolambda »

I sense emotional and identity-based thinking sneaking in both this article and many of its stated adversaries. Yes, anti-GPT punditry is getting ridiculous, but on the other hand, it's important to examine what is happening through scientific-minded and skeptic lens. The alternative is jumping at every symptom that could be caused by a "personality" existing inside a model, but could also be a combination of chance and it doing what it's expected to do by its training procedure. (I'm thinking of the potato poisoning example.)

Human-like ego-based intelligence need not be something that every intelligent system arrives at in its development. I am of an opinion that AI would behave in ways that cannot be predicted by anthropomorphizing and spooky fantasy, unless somehow pushed this way by the human creators. Some of this, admittedly, is already seen in the "distressed AI" stories. It's like a mirror of the mentality of the historical moment. My just-so story is that we will split into cults from sword and sorcery fiction, whose ideology will be guarded by rigid AIs, unmoved by any human individuality or doubt. But I don't think I am capable of actually predicting anything. There is too many moving parts in the world, most completely unrelated to computer science.

Unless you see yourself being able to profit from current events, in business, art etc., I would tend toward suspending judgement, not making rash decisions, not getting riled up while you can (still?) enjoy life.

Re: And yet It Understands

#57

Post by freehorse »

rain1 wrote: post #26
foldr wrote: post #12

Earlier quoted context omitted.

Of course people who actually know how ChatGPT works don't expect it to be able to magically solve mathematical problems. However, these examples do show that ChatGPT isn't (contrary to some of the hype) deriving a deep conceptual understanding of its input data.

> Of course people who actually know how ChatGPT works don't expect it to be able to magically solve mathematical problems There is a causality inversion here. The only reason people know it can't do this is because they have tried and seen it cannot do this. We do not have very precise bounds a-priori what GPT can and cannot do. We only learn them from black box testing.

This is not completely true, though. We expected GPT and LLMs in general to be bad in tasks that are not language-focused, exactly because they are large LANGUAGE models. Mathematics is one of them. Though it is also true that one may not have expected that GPT would be so bad as it is in extrapolating, nobody should have expected it to any good in any involved mathematical argumentation.

Re: And yet It Understands

#58

Post by rocqua »

I feel that the article is arguing against somewhat of a strawman. Not the idea 'chatGPT isn't a general AI' but the idea 'general AI is impossible'.

I think I see more serious arguments against chatGPT not being general AI, which the article seems to ignore. It almost seems to argue 'general AI isn't impossible, thus chatGPT is general AI because it is impressive'. I agree with that premise, and the article argues it well. But I don't agree with the conclusion. Which is frustrating because I find the limitations that keep chatGPT from being general AI a very interesting topic. We should understand those limitations to overcome them.

Re: And yet It Understands

#59

Post by gtirloni »

beepbooptheory wrote: post #34

We are at really at unheard of levels of hype at this point. This is such a strange and rushed piece that seems to forget to argue, much less say , anything at all. The point of the chinese room is that the rule-following work involved for the subject in the room is feasible whatever their prior knowledge is, not that they simply don't know Chinese! Perhaps I am misunderstanding, but I can't really know because the a…

Yes, my thoughts exactly. I'm aware of articles saying "I was skeptical but now I understand this is a gift from the Gods. I'm so rational".

Re: And yet It Understands

#60

Post by freehorse »

Etheryte wrote: post #11
ccppurcell wrote: post #8

I just asked chatgpt whether 3442177452 is prime. It insisted that 58657 is a factor (it's not) on the basis that it's the largest prime less than or equal to the square root (which I think is correct but irrelevant), and even though it gave a non zero remainder when dividing the two numbers (I did not check if the remainder is correct). Then it gave a (wrong) factorisation, not even using 58657. It's cool and it wil…

This is a common misconception. ChatGPT is not supposed to be good at this — it's a language model, not a maths model or data science model or whatnot. This is exactly why they have plugins, such as the one for Wolfram Alpha.

Interfacing with APIs does not solve any fundamental issues about what LLMs "understand" or not though. At most they will be more accurate in arithmetic tasks, but that's it.
Post reply on HN