Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How does ChatGPT work?

news.ycombinator.com

Post reply on HN
Print view View original

151–160 of 185 posts

Re: Ask HN: How does ChatGPT work?

#151

Post by dtagames »

pushedx wrote: post #34
dtagames wrote: post #3

ChatGPT is really very simple. Imagine you could analyze a million books and identify all the words within them -- not the meanings of the words, just the actual letters they contain. Now, when someone asks you about the history of France (or why the sky is blue), you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question! It's like a kid in the…

Ask ChatGPT to write a song lyric for your favorite band in the style of your favorite poet, with a specific unusual topic. Then claim that the model is “really very simple”

At the end of the day, all computing is very simple. What's impressing you here is the size of the dataset and the speed of the retrieval, brought about by advances in hardware. Also, we like the well-formed writing which is created by filtering and massaging the parroted text into a new set of sentences.

It is impressive, and people who don't know how it really works will think it's capable of all kinds of things it's not. Of course, this has been true forever. Just ask anyone who writes code about what non-devs think software can do.

Re: Ask HN: How does ChatGPT work?

#152

Post by dtagames »

6nf wrote: post #33
dtagames wrote: post #3

ChatGPT is really very simple. Imagine you could analyze a million books and identify all the words within them -- not the meanings of the words, just the actual letters they contain. Now, when someone asks you about the history of France (or why the sky is blue), you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question! It's like a kid in the…

> it cannot apply logic Have you tried?

The tool can parrot back to you "logical" arguments which it has already scanned. It cannot figure out the logic behind them, nor seen when it's wrong, nor correct any errors, nor draw new logical conclusions.

Re: Ask HN: How does ChatGPT work?

#153

Post by soulofmischief »

gentoo wrote: post #53
sillysaurusx wrote: post #37

Earlier quoted context omitted.

Then why were they the first ones to exploit it so effectively? I don’t think it was standard for GPT models.

I don't think anything about high-performance GPT models is standard, since they are only a couple years old and only a handful of organizations have developed them

The technique in question has little to do with GPT itself; it involves using ML to generate more training data in an automated fashion, creating a generative training loop, which as another commenter mentioned, is also the basis behind general adversarial networks.

Re: Ask HN: How does ChatGPT work?

#154

Post by Sharlin »

dotancohen wrote: post #145
Sharlin wrote: post #119

Earlier quoted context omitted.

On the other hand, airplanes do fly . But I guess that just reinforces the point that language is a map, not the territory, and natural language in particular is full of inconsistencies and fuzzy boundaries between poorly defined categories. English happened to evolve so that it ended up using the same word for the thing both winged creatures and airplanes do, but only living creatures swim , boats and ships and subm…

> On the other hand, airplanes do fly. In my language, the verb that airplanes do is different than the verb that birds and insects do. As like the submarine, it is based on a different physical mechanism than are the natural animals in that space. For that matter, in English, what is the verb that submarines do? Surely it's not "sail".

Submarines do dive, but only when moving vertically downward. I don't think there is a single verb for what submarines do, they can move, travel, maneuver, navigate, or operate (underwater) depending on context.

Re: Ask HN: How does ChatGPT work?

#155

Post by layer8 »

oars wrote: post #59

High quality answers like in this thread are why I come to HN. Although I hope these high quality answers don't all just come from ChatGPT one day.

For better or worse, Ask HN is still wildly more useful than Ask ChatGPT.

Re: Ask HN: How does ChatGPT work?

#156

Post by dotancohen »

Sharlin wrote: post #154
dotancohen wrote: post #145

Earlier quoted context omitted.

> On the other hand, airplanes do fly. In my language, the verb that airplanes do is different than the verb that birds and insects do. As like the submarine, it is based on a different physical mechanism than are the natural animals in that space. For that matter, in English, what is the verb that submarines do? Surely it's not "sail".

Submarines do dive , but only when moving vertically downward. I don't think there is a single verb for what submarines do, they can move , travel , maneuver , navigate , or operate (underwater) depending on context.

I see, thank you.

Re: Ask HN: How does ChatGPT work?

#158

Post by Epa095 »

tluyben2 wrote: post #74
mclightning wrote: post #41

Earlier quoted context omitted.

That's super interesting. When GPT-3 came out, I wrote an article inspired by it. That we could one day build an AI that acts like AGI, by a crazy vast amount of multimedia training data, collected by willing users to participate in ever improving AI interactions; https://medium.com/swlh/bicameral-mind-humanoid-robot-with-g...

When I was a first year AI student beginning of the 90s I asked my professor what would happen if we just made a massive neural network and trained it with all information in the world. He said it cannot happen as it it impossible.

If an elderly but distinguished scientist says that something is possible, he is almost certainly right; but if he says that it is impossible, he is very probably wrong.

Arthur C. Clarke

Re: Ask HN: How does ChatGPT work?

#159

Post by ravi-delia »

jameshart wrote: post #133
ravi-delia wrote: post #54

Earlier quoted context omitted.

In learning to predict the next token, the model has to pick up lots of little bits of world knowledge. I'm sure someone would disagree with the phrasing of "understand", but it certainly operates with more complexity than, say, a markov chain. It has seen lots of python, and in order to predict better, it has developed internal models of how python works. Think of how much better you'd do predicting the next charact…

It’s not ‘more complexity’ than a Markov chain - it essentially is a Markov chain, just looking at a really deep sequence of preceding tokens to decide the probabilities for what comes next. And it’s not just looking that up in a state machine, it’s ‘calculating’ it based on weights. But in terms of ‘take sequence of input tokens; use them to decide probable next token’, it’s functionally indistinguishable from a Mar…

I look at deep sequences of tokens and predict what comes next- can you milk me? Once you've broadened "basically a markov chain" to "any function from a sequence of tokens to a probability distribution of tokens" there's a lot of explanatory power lost. If you had to characterize the difference between brute force mappings based on pure frequencies and model which selectively calculates probabilities based on underlying structure, wouldn't you say the latter had more complexity?

You don't have to believe the hype, but if you think you can get GPT performance out of anything remotely resembling a markov chain, I encourage you to try.

Re: Ask HN: How does ChatGPT work?

#160

Post by Epa095 »

mjburgess wrote: post #99
miltondts wrote: post #90

Earlier quoted context omitted.

What I don't understand is where is the memory? How does GPT-3 or ChatGPT remember so much information with just that architecture? It would seem that the maximum it could remember is 2048 words. EDIT: Maybe it's 2048 x 96? Still seems low for what it can do.

300bn weights, at 4bytes/weight is 1.2TB

Yes, but how does it remember the stuff you told it earlier in the conversation? Those 1.2TB is the trained model, and I assume that those weights are not changed by the conversation?
Post reply on HN