Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What is ChatGPT doing and why does it work?

writings.stephenwolfram.com

Post reply on HN
Print view View original

91–100 of 518 posts

Re: What is ChatGPT doing and why does it work?

#91

Post by SketchySeaBeast »

bheadmaster wrote: post #30
SketchySeaBeast wrote: post #22

Earlier quoted context omitted.

In my case it's because my internal dialogue is saying the sentence before I get to the end of it. I usually have the entire sentence in my inner dialogue before I even start typing. Will I edit during typing? Sure, but I have a first version in my head before I start.

> I usually have the entire sentence in my inner dialogue before I even start typing Interesting. Perhaps the question then becomes, does your inner dialogue simply chain the words one after another, or does it come up with sentences as whole?

Why would it? It would make more sense to start with concepts ("dog", "brown"), and then pull it into words ("That dog is brown") - given that we're trying to convey a thought that seems to be far more likely than us chaining things - we start with the intent to communicate something, the markov chain does not.

Re: What is ChatGPT doing and why does it work?

#92

Post by [deleted] »

jw1224 wrote: post #42
codeulike wrote: post #32

Wow this is 19,000 words. I like his summary at the end: At some level it’s a great example of the fundamental scientific fact that large numbers of simple computational elements can do remarkable and unexpected things. And this: ... But it’s amazing how human-like the results are. And as I’ve discussed, this suggests something that’s at least scientifically very important: that human language (and the patterns of th…

> Yeah I've been thinking along these lines. ChatGPT is telling us something about language or thought, we just havent got to the bottom of what it is yet. Something along the lines of 'with enough data its easier to model than we expected'. I’ve been thinking similarly, and am coming to understand and accept we’ll never get to the bottom of it :) The universe is fractal-like in nature. It shouldn’t be a surprise, th…

[deleted]

Re: What is ChatGPT doing and why does it work?

#93

Post by thegabriele »

osswid wrote: post #77

In the days when Sussman was a novice Minsky once came to him as he sat hacking at the PDP-6. "What are you doing?", asked Minsky. "I am training a randomly wired neural net to play Tic-Tac-Toe." "Why is the net wired randomly?", asked Minsky. "I do not want it to have any preconceptions of how to play" Minsky shut his eyes, "Why do you close your eyes?", Sussman asked his teacher. "So that the room will be empty." A…

I keep on stumbling on this story, yet i can't seem to be able to grasp it.

We have senses so our brain has some sort of a priori knowledge of the world? We're all almost blind when we born.

Re: What is ChatGPT doing and why does it work?

#95

Post by akomtu »

GPT looks a lot like an IIR filter that transforms a sequence of vectors. Edit: IIR filters are linear functions of N past inputs and N past output - the latter gives them "memory" and non-trivial abilities to process signals. GPT is mostly linear, uses 8192 past inputs and outputs. I'd be tempted to introduce the 3rd sequence - an "internal buffer" with 8192 tokens - that GPT updates even with null inputs, the process that corresponds to "thinking".

Re: What is ChatGPT doing and why does it work?

#96

Post by bottlepalm »

The easiest way for ChatGPT to generate good output is to plain understand it. Given the vast amount of input data fed into it, it has no choice, but to start reducing the input into fundamental rules which is basically what understanding is. Understanding is a form of compression. More efficient for a neural network to understand a concept than memorize permutations.

Same with statistics and markov chains, people for years tried to generate chat bots with those, but they never worked well.

Re: What is ChatGPT doing and why does it work?

#97

Post by goatlover »

bottlepalm wrote: post #96

The easiest way for ChatGPT to generate good output is to plain understand it. Given the vast amount of input data fed into it, it has no choice, but to start reducing the input into fundamental rules which is basically what understanding is. Understanding is a form of compression. More efficient for a neural network to understand a concept than memorize permutations. Same with statistics and markov chains, people fo…

Disagree when it comes to anything in the real world how we experience things.

Re: What is ChatGPT doing and why does it work?

#99

Post by Dr_Birdbrain »

bottlepalm wrote: post #96

The easiest way for ChatGPT to generate good output is to plain understand it. Given the vast amount of input data fed into it, it has no choice, but to start reducing the input into fundamental rules which is basically what understanding is. Understanding is a form of compression. More efficient for a neural network to understand a concept than memorize permutations. Same with statistics and markov chains, people fo…

I used to believe this, but then I saw the demo of ChatGPT attempting to play chess.

The rules of chess are much simpler than any structure of the real world we might hope for it to understand—and yet it failed to learn the rules of chess.

Based on this, I am pretty sure it has not learned any meaningful structure.

Post reply on HN