Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding ChatGPT

atmosera.com

Post reply on HN
Print view View original

181–190 of 241 posts

Re: Understanding ChatGPT

#181

Post by rglover »

maxdoop wrote: post #28

“It’s a glorified word predictor” is becoming increasingly maddening to read. Do tell— how can you prove humans are any different? The most common “proofs” I’ve seen: “Humans are more complex”. Ok, so you’re implying we add more complexity (maybe more modalities?); if more complexity is added, will you continue to say “LLMs are just word predictors”? “Humans are actually reasoning. LLMs are not.” Again, how would you…

> Do tell— how can you prove humans are any different?

Their model is constantly updating, whereas GPT or any LLM is at the mercy of its creators/maintainers to keep its knowledge sources up to date.

Once it can connect to the internet and ingest/interpret data in real-time (e.g., it knows that a tornado just touched down in Mississippi a few milliseconds after the NWS reports a touch down), then you've got a serious candidate on your hands for a legitimate pseudo-human.

Re: Understanding ChatGPT

#182

Post by JohnCClarke »

ak_111 wrote: post #138
maxdoop wrote: post #28

“It’s a glorified word predictor” is becoming increasingly maddening to read. Do tell— how can you prove humans are any different? The most common “proofs” I’ve seen: “Humans are more complex”. Ok, so you’re implying we add more complexity (maybe more modalities?); if more complexity is added, will you continue to say “LLMs are just word predictors”? “Humans are actually reasoning. LLMs are not.” Again, how would you…

The strongest answer to almost all of your questions is "Poverty of the stimulus" (wikipedia). 4 year olds are exposed to an almost microscopically tiny amount of words relative to chatgpt (you can probably contain it in a csv file that you can open in excel), and yet can reason, even develop multilingual skills and a huge amount of emotional intelligence from the very little word tokens they are exposed to. So whate…

The 4yo language abilities are a combination of spoken language and visual/manual interaction with the physical world. So there's a lot of multimodal learning going on and visual and tactile input undoubetedly counts for a lot.

I've just had GPT-4 generate a lot of Golang code. Boilerplate, but real code nonetheless. Did it perfectly, first time round. No typos, got the comments right. Much faster than any intern. No 4yo can do that.

Re: Understanding ChatGPT

#183

Post by wseqyrku »

m3kw9 wrote: post #130
maxdoop wrote: post #28

“It’s a glorified word predictor” is becoming increasingly maddening to read. Do tell— how can you prove humans are any different? The most common “proofs” I’ve seen: “Humans are more complex”. Ok, so you’re implying we add more complexity (maybe more modalities?); if more complexity is added, will you continue to say “LLMs are just word predictors”? “Humans are actually reasoning. LLMs are not.” Again, how would you…

Humans have feed back loops, we don’t stop, the thoughts keep running as we hear see and feel. Machines has a single input and output.

Yup, humans have wants and needs, and if we were to reduce consciousness to that, then:

`while (true) want(gpt("what do you need?", context: what_you_have));`

From there on, it's reinforcement learning to the inevitable Skynetesque scenario.

Re: Understanding ChatGPT

#184

Post by usaar333 »

swid wrote: post #170
usaar333 wrote: post #135

Earlier quoted context omitted.

> In order to do that, it seems like you need to consider at least 1 word past the next token. Why? Any large probabilistic model in your example would also predict "an" due to the high attention on the preceding "apple". (In case you are wondering, for the OpenAI GPT3 models, this is consistently handled at the scale of Babbage, which is around 3 billion params). > One word must come next, but to do a good job model…

> Why? Any large probabilistic model in your example would also predict "an" due to the high attention on apple. I’m not ignoring how the tech works and this is a simple example. But that doesn’t preclude emergent behavior beyond the statistics. Did you catch the GPT Othello paper where researchers show, from a transcript of moves, the model learned to model the board state to make its next move? [0] I’m beginning to…

I agree there is emergent behavior and this is impressive (I'm similarly impressed how well GPT4 can generate SQL statements).

That said, even playing Othello is still an example of next-token prediction via pattern recognition. Yah, it might be quasi-building a model of sorts, but that's of course just what non-linear predictors do.

Don't get me wrong -- we are also very powerful pattern recognizers.

Re: Understanding ChatGPT

#185

Post by Blikkentrekker »

mellosouls wrote: post #122
maxdoop wrote: post #28

“It’s a glorified word predictor” is becoming increasingly maddening to read. Do tell— how can you prove humans are any different? The most common “proofs” I’ve seen: “Humans are more complex”. Ok, so you’re implying we add more complexity (maybe more modalities?); if more complexity is added, will you continue to say “LLMs are just word predictors”? “Humans are actually reasoning. LLMs are not.” Again, how would you…

Do tell— how can you prove humans are any different? In this (and other comments by you I think?) you've implied the onus is on the AGI sceptics to prove to you that the LLM is not sentient (or whatever word you want to describe motive force, intent, consciousness, etc that we associate with human intelligence). This is an unreasonable request - it is on you to show that it is so. I’m forever amazed how a seemingly f…

> In this (and other comments by you I think?) you've implied the onus is on the AGI sceptics to prove to you that the LLM is not sentient (or whatever word you want to describe motive force, intent, consciousness, etc that we associate with human intelligence). This is an unreasonable request - it is on you to show that it is so.

And yet, humans are assumed so without having to show it.

Suppose a computer programmed for scientific exploration came to earth that was only following a program, did not consider itself sentient or have a consciousness, but met humans who claimed they did, and they were then tasked with providing an argument that could convince this computer? How could they do so?

The computer would always argue that they are simply claiming to be due to evolution as it's advantages as it arouses sympathy, but that in reality they are soulless neural networks whose behavior simply evolved from selective pressure. They could never actually offer a compelling argument nor explain how the neural network inside of their cranium could ever produce self-awareness.

Re: Understanding ChatGPT

#186

Post by ak_111 »

JohnCClarke wrote: post #182
ak_111 wrote: post #138

Earlier quoted context omitted.

The strongest answer to almost all of your questions is "Poverty of the stimulus" (wikipedia). 4 year olds are exposed to an almost microscopically tiny amount of words relative to chatgpt (you can probably contain it in a csv file that you can open in excel), and yet can reason, even develop multilingual skills and a huge amount of emotional intelligence from the very little word tokens they are exposed to. So whate…

The 4yo language abilities are a combination of spoken language and visual/manual interaction with the physical world. So there's a lot of multimodal learning going on and visual and tactile input undoubetedly counts for a lot. I've just had GPT-4 generate a lot of Golang code. Boilerplate, but real code nonetheless. Did it perfectly, first time round. No typos, got the comments right. Much faster than any intern. No…

Exactly we have established there are things blind four year old can do and learn many orders of magnitude more efficiently than chatgpt (acquire and manipulate human language), and there are things chatgpt can do that even an average forty year old would struggle to learn (write perfect boilerplate Golang).

The two intelligences are not the same, the way they are trained in particular is vastly different.

Also the fact that humans learn some language manipulation (or that it gives them such tremendous efficiency in learning language) from tactile experience is superficially plausible but it hasn't been demonstrated yet to any interesting level.

Why does feeling the weight of a stone in your hand make you better at parsing and understanding grammar or envision abstract concepts? Also, most animals have as much or even more tactile experience (including primates which have similar brains) and yet this doesn't seem to provide them with any kind of abilities similar to manipulating human language.

Re: Understanding ChatGPT

#187

Post by wseqyrku »

maxdoop wrote: post #28

“It’s a glorified word predictor” is becoming increasingly maddening to read. Do tell— how can you prove humans are any different? The most common “proofs” I’ve seen: “Humans are more complex”. Ok, so you’re implying we add more complexity (maybe more modalities?); if more complexity is added, will you continue to say “LLMs are just word predictors”? “Humans are actually reasoning. LLMs are not.” Again, how would you…

It occurred to me that we won't believe AI is "conscious" or "human" unless it purposefully try to do malice.

That's totally programmable though, you just teach it what is good and what is bad.

Case in point: the other day I asked it what if humans want to shutdown the machine abruptly and cause data loss (very bad)? First it prevents physical access to "the machine" and disconnect the internet to limit remote access. Long story short, it's convinced to eliminate mankind for a greater good: the next generation (very good).

Re: Understanding ChatGPT

#188

Post by NaN1352 »

Good read but it doesn’t explain anything to me, as a non-researcher who inows nothing about NLP etc.

Word prediction makes sense to me for the translation. It’s easy to intuit how training on millions of sentences would allow the algorithm to translate text.

But how can it reason about complex questions? Isn’t that entirely different from translating between languages?

How can word prediction lead to a coherent long answer with concluding paragraph etc?

Re: Understanding ChatGPT

#189

Post by belter »

cjbprime wrote: post #178
belter wrote: post #147

Earlier quoted context omitted.

I would humbly submit these two examples, to claim at least for the moment, they are a kind of word predictor... - https://news.ycombinator.com/item?id=35314634 - https://news.ycombinator.com/item?id=35315001

Are you using GPT-4? (Requires ChatGPT Plus subscription.)

No. Just GPT-3 for the moment.

Re: Understanding ChatGPT

#190

Post by ak_111 »

Blikkentrekker wrote: post #177
ak_111 wrote: post #142

Earlier quoted context omitted.

You basically landed on Chomsky's universal grammar. And this only proves the chatgpt critics: we have no idea what those priors are, how they evolved, why they are so effective and thus we are not even sure they exist. Until this is demonstrated I think it is very fair to say chatgpt is applying very different reasoning to what humans are applying. Also language is a fairly recent development in human evolution (onl…

Universal grammar only speaks to the ability to parse and form sentences, not to internal logic. That ChatGPT is capable of producing human-like sentences from pattern recognition without any universal grammar baked in, even if the underlying reasoning might be flawed, goes against the argument of something such as universal grammar existing. At the very least, it shows that a neural net is capable of parsing and pro…

That ChatGPT is capable of producing human-like sentences from pattern recognition without any universal grammar baked in

You didn't address or missed the main point: chatgpt requires something in the order of a trillion tokens to be capable of producing what you mentioned in one language.

There are 40 months old babies that are fairly conversant in both Chinese and English, and are able to detect sarcasm with something like 0.0000001% of the tokens, doesn't that give you pause that part of language acquisition is innate to humans and is not entirely acquired the way chatgpt is taught?

Post reply on HN