Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What is ChatGPT doing and why does it work?

writings.stephenwolfram.com

Post reply on HN
Print view View original

171–180 of 518 posts

Re: What is ChatGPT doing and why does it work?

#171

Post by mastercheif »

lossolo wrote: post #79
visarga wrote: post #53

Earlier quoted context omitted.

> 'with enough data its easier to model than we expected' > a self-similar process is ultimately responsible for granting us our own intelligence In my view, intelligence essentially resides within language, specifically in the corpus of language. Both humans and AIs can be effectively colonized by language, as there are innumerable concepts and observations that are transmitted from one mind to another, and now even…

Intelligence exists without language, language is only a way to describe the world around us and transfer information. We personally can't experience intelligence without language because you know language all your life and can't remember a moment in which you didn't. But there were humans in history that didn't knew any language and they were intelligent, there are animals that do not know any language and are intel…

In college I tried a medication called Topamax (Topiramate) for migraine prevention. Topamax has a low-occurrence side effect of “language impairment”. After 10 days or so it became clear that I was particularly susceptible to this phenomenon.

It was a terrifying experience, but it was also a valuable one as it changed the way I view intelligence.

When I was in the thick of it, my writing and speech skills had devolved to that of a primary school aged child. I’ll never forget trying to type a text message and struggling to come up with simple words like “and”. My speech slowed down considerably and I was having trouble with verbal dictation.

The terrifying thing was that my internal world was still as complex and meaningful as it was before. All of the emotions I felt were real and legitimate. My cognition outside of communication was intact, I could do math just fine and conceptualize and abstract problems.

In spite of this, I was unable to convey what I was feeling and thinking to the outside world. It felt like I was trapped inside my own body.

I thankfully made a full recovery. However, my intuitive understanding of the link between language and intelligence was completely severed. While I believe there’s likely a high degree of correlation between the two in populations, on an individual level one’s language skills mostly represent one’s ability to communicate with the outside world, not their ability to understand complex information or process it.

See the following for more info on Topamax/Topiramate language impairment: https://link.springer.com/article/10.1007/s10072-008-0906-5

Re: What is ChatGPT doing and why does it work?

#172

Post by dumpsterdiver »

eclipxe wrote: post #150
criddell wrote: post #149

Off topic: I’m looking for a good web clipper to grab just the article and related images from a page like this and save it (as html or a pdf) for me to read later. I’d like it on iPadOS because that’s where I like to read and write. I tried reader mode, but it lost a lot of the images. Any suggestions? Edit: I was able to get a good PDF using the OneNote web clipper on my desktop.

Take a screen shot. On iPad/iPhone you can save the screenshot of a webpage as a full PDF of the page. Make sure you are in Safari.

Awesome tip. Thank you.

Re: What is ChatGPT doing and why does it work?

#173

Post by GreedClarifies »

inasio wrote: post #131
scottlawson wrote: post #112

I spent about an hour reading this. I know most people probably won't be able to do the same because of time constraints in their day, but seriously this is the most awesome post about large language models I've read. Take a second to think about how much time it must have taken to write this, to prepare all of these examples. This post gave me another way to look at something I already partially understood pieces of…

I'm typically a bit wary of Wolfram's posts, but it is very often the case that the content is good enough that I don't mind a bit of self-aggrandizing. This one though I found to be just pure great, super easy and clear intro into the "magic" of LLMs. Thank you.

Best comment!

He's clearly brilliant and I read his posts, but usually I have to grit my teeth at least a couple of times due to his self-aggrandizing. But there was almost none of that in this one.

Great stuff.

Re: What is ChatGPT doing and why does it work?

#174

Post by vore »

kilgnad wrote: post #145
vore wrote: post #139

Earlier quoted context omitted.

I mean, that is the fundamental architecture of the model and how it's explicitly designed: given an input embedding and output embedding, the model is trained over a massive corpus of text to optimize the probability for generating the next token in the sequence. Any other interpretation is secondary to that, no matter how impressive the results are.

There is no rule that implies the explicit design dominates the emergent effect. Evolutionary biology strives to make us logical creatures to fulfill the singular goal of passing on genetic material. Your sentience and your humanity is a side effect of this singular goal. So what dominates the description of who YOU are? Human or vessel for genetic material? I'll say that YOU are human and therefore more then just a…

They really aren't more than just statistical word generators because we can and do build what are exactly statistical word generators and get the behavior of ChatGPT (because that is what ChatGPT is).

For what it's worth, this isn't a knock on ChatGPT, but more just how amazing how far you can get with straightforward concepts.

Re: What is ChatGPT doing and why does it work?

#175

Post by grugagag »

kilgnad wrote: post #156
Rury wrote: post #148

Earlier quoted context omitted.

How is this significant? I mean you can view a moving car as stationary, and reality is the thing that moves around it, or you can view reality as stationary, but the car moves through it. Or you can make analogies between electricity and water. (Frankly, I like viewing neural nets as a numerical method for doing nonlinear multivariate regression, and ChatGPT is nothing but a resulting equation(s) of such). In the en…

Well you're just a jumble of electrons and protons interacting with each other. That's a 100% true interpretation is it not? It's also a view that misses the point that you are a molecular intelligence made of DNA that continually mutates and reconstructs it's physical form with generational copies to increase fitness in an ever changing environment. But that viewpoint also misses the point that you're a human with w…

> Because of fear. Because of bias.

Care to expand a bit more on what you think those fears may be? Or that bias?

Re: What is ChatGPT doing and why does it work?

#176

Post by jeremyjh »

hinkley wrote: post #120
jmoak3 wrote: post #87

Earlier quoted context omitted.

I saw a great comment here, and I will repeat it without the attribution it deserves: We may have realized it's easier to build a brain than to understand one

One of the things one might want to get out of this is a programming language that feels like human speech but is unambiguous to computers. If the understanding is the hard part, that seems much less likely.

Its not programming anymore its prompting. Prompting it to write and run the program that does what you want.

Re: What is ChatGPT doing and why does it work?

#177

Post by est »

I've read many articles explains how gpt-3 work, and did a great job.

What buffles me is the context consistency. ChatGPT was a huge leap compared to previous models. I have never seen it failed once. I often use "this" or "that" in my conversation with ChatGPT and it would guess 100% correct what I am refering to. Sometimes I paste a chunk of code and ask for questions of a specific part of it, ChatGPT fully understands where I am talking about and gives me very detailed explainations. It's astonishing and I never knew how it worked so well.

Also the title suggests "and why does it work" but I failed to find the reason why ChatGPT worked as in contrast that gpt-3/2/1 never really worked (well)

Re: What is ChatGPT doing and why does it work?

#178

Post by grugagag »

Rury wrote: post #165
kilgnad wrote: post #156

Earlier quoted context omitted.

Well you're just a jumble of electrons and protons interacting with each other. That's a 100% true interpretation is it not? It's also a view that misses the point that you are a molecular intelligence made of DNA that continually mutates and reconstructs it's physical form with generational copies to increase fitness in an ever changing environment. But that viewpoint also misses the point that you're a human with w…

I still don't see your point behind your first 4 paragraphs. How you decide to treat your fellow humans is up to you. Just as you can decide to view your fellow humans however you want. It's entirely possible to treat them like "humans" while still viewing them as nothing but jumbles of molecules and atoms. So again why does perspective matter here (particularly with ChatGPT being a statistical word generator)? Your…

> To me, it sounds like you're the fearful one.

What do you think that fear may be?

Re: What is ChatGPT doing and why does it work?

#179

Post by jeremyjh »

jmoak3 wrote: post #87
codeulike wrote: post #32

Wow this is 19,000 words. I like his summary at the end: At some level it’s a great example of the fundamental scientific fact that large numbers of simple computational elements can do remarkable and unexpected things. And this: ... But it’s amazing how human-like the results are. And as I’ve discussed, this suggests something that’s at least scientifically very important: that human language (and the patterns of th…

I saw a great comment here, and I will repeat it without the attribution it deserves: We may have realized it's easier to build a brain than to understand one

But if you ever a have a conversation with it you know it isn't a brain. I'm not talking about detection here; its whole point is to generate credible text so it is going to evade detection well. But can't you just tell from talking to it that there is nothing there?

Re: What is ChatGPT doing and why does it work?

#180

Post by dnaddie »

Rury wrote: post #165
kilgnad wrote: post #156

Earlier quoted context omitted.

Well you're just a jumble of electrons and protons interacting with each other. That's a 100% true interpretation is it not? It's also a view that misses the point that you are a molecular intelligence made of DNA that continually mutates and reconstructs it's physical form with generational copies to increase fitness in an ever changing environment. But that viewpoint also misses the point that you're a human with w…

I still don't see your point behind your first 4 paragraphs. How you decide to treat your fellow humans is up to you. Just as you can decide to view your fellow humans however you want. It's entirely possible to treat them like "humans" while still viewing them as nothing but jumbles of molecules and atoms. So again why does perspective matter here (particularly with ChatGPT being a statistical word generator)? Your…

(hit some ratelimiter on the other account, so using this account to reply)

Ok let me make this more clear.

I choose how to view things, yes this is true. But if I choose to treat human beings as jumbles of molecules, most people would consider that viewpoint flawed, inaccurate and slightly insane. Other humans would think that I'm in denial about some really obvious macro effects of configuring molecules in a way such that it forms a human.

I can certainly choose to view things this way, but do you see how such a singular viewpoint is sort of stubborn and unreasonable? This is why solely viewing LLMs as simple statistical word generators is unreasonable. Yes it's technically correct, but it is missing a lot.

There's another aspect to this too. What I'm seeing, to stay inline with the analogy, is people saying that the "human" viewpoint is entirely invalid. They are saying that the jumble of molecules only forms something that looks like a human, a "chinese room" if you will. They are saying the ONLY correct viewpoint is to view the jumble of molecules as a jumble of molecules. Nothing more.

So to bring the analogy back around to chatGPT. MANY people are saying that chatGPT is nothing more then a word generator. It does not have intelligence, it does not understand anything. I am disagreeing with this perspective because OP just linked a scientific paper CLEARLY showing that LLMs are building a realistic model of the information you are feeding it.

Post reply on HN