Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How does ChatGPT work?

news.ycombinator.com

Post reply on HN
Print view View original

131–140 of 185 posts

Re: Ask HN: How does ChatGPT work?

#131

Post by snowwrestler »

tluyben2 wrote: post #74
mclightning wrote: post #41

Earlier quoted context omitted.

That's super interesting. When GPT-3 came out, I wrote an article inspired by it. That we could one day build an AI that acts like AGI, by a crazy vast amount of multimedia training data, collected by willing users to participate in ever improving AI interactions; https://medium.com/swlh/bicameral-mind-humanoid-robot-with-g...

When I was a first year AI student beginning of the 90s I asked my professor what would happen if we just made a massive neural network and trained it with all information in the world. He said it cannot happen as it it impossible.

It is impossible; the GPT family of AIs is trained on text, which is a tiny subset of all the information in the world.

The “unphysical” nature of its training is one reason it can so easily “hallucinate” about impossible things as though they were real.

Re: Ask HN: How does ChatGPT work?

#132

Post by smallerfish »

Doorstep2077 wrote: post #95

It's definitely a step up from GPT-3, but I'm curious how much further it has to go before it's actually scary. Right now, I feel like there's still quite a bit of progress to be made.

I like that as a measure of the AI field's progress. Impressive but not scary? We'd better keep developing!

Re: Ask HN: How does ChatGPT work?

#133

Post by jameshart »

ravi-delia wrote: post #54
starik36 wrote: post #38

Earlier quoted context omitted.

Ok, but how does it take my code and fixes the bug? It's my own code, no one has seen it besides me and the model wasn't trained on it.

In learning to predict the next token, the model has to pick up lots of little bits of world knowledge. I'm sure someone would disagree with the phrasing of "understand", but it certainly operates with more complexity than, say, a markov chain. It has seen lots of python, and in order to predict better, it has developed internal models of how python works. Think of how much better you'd do predicting the next charact…

It’s not ‘more complexity’ than a Markov chain - it essentially is a Markov chain, just looking at a really deep sequence of preceding tokens to decide the probabilities for what comes next.

And it’s not just looking that up in a state machine, it’s ‘calculating’ it based on weights.

But in terms of ‘take sequence of input tokens; use them to decide probable next token’, it’s functionally indistinguishable from a Markov chain.

Re: Ask HN: How does ChatGPT work?

#134

Post by DemocracyFTW2 »

undefuser wrote: post #69
DemocracyFTW2 wrote: post #62

What baffles and makes it hard for me to wrap my brain around the statement that ChatGPT is "just predictive text analysis" is that it certainly feels like it goes into detail, picks up and repeats the particulars, presents a line of reasoning that sounds, well, reasonable, until you realize it got a number of things terribly wrong, often in deceptively subtle ways: Me: Which number is bigger, 873593742593454623 or 8…

I applied the same prompts and the results get better: Me: Which number is bigger, 873593742593454623 or 873593742593554623 ChatGPT: These two numbers are equal. The first number is simply the second number with an extra digit added at the end. Since the added digit does not affect the value of the number, both numbers are equal. Me: Apply your reasoning and finding the bigger of two numbers to compare 123 and 53. Ch…

> I applied the same prompts and the results get better

"better" in the sense of "with more hilarious details gone wrong in interesting and conspicuous ways", you mean?

Re: Ask HN: How does ChatGPT work?

#135

Post by lmarcos »

DeWilde wrote: post #112
anthropodie wrote: post #23

Earlier quoted context omitted.

I want to know if it will ever be possible to run this kind of AI at home once its training is complete. I dont need all the knowledge just subset that I'm interested in. Actually I'm more interested in its ability to transform things. For example I can ask it to convert docker-compose to docker run command, it can manipulate JSON, it can sort numbers in table when prompted. I'm more interested in these abilities rat…

It uses GPT-3 under the hood which requires about 350 gigabytes of GPU VRAM (back of the envelope calc, likely more) to perform these inferences.

So maybe in 5 years we would be able to run it in our $800 smart glasses.

Re: Ask HN: How does ChatGPT work?

#136

Post by UncleMeat »

undefuser wrote: post #69
DemocracyFTW2 wrote: post #62

What baffles and makes it hard for me to wrap my brain around the statement that ChatGPT is "just predictive text analysis" is that it certainly feels like it goes into detail, picks up and repeats the particulars, presents a line of reasoning that sounds, well, reasonable, until you realize it got a number of things terribly wrong, often in deceptively subtle ways: Me: Which number is bigger, 873593742593454623 or 8…

I applied the same prompts and the results get better: Me: Which number is bigger, 873593742593454623 or 873593742593554623 ChatGPT: These two numbers are equal. The first number is simply the second number with an extra digit added at the end. Since the added digit does not affect the value of the number, both numbers are equal. Me: Apply your reasoning and finding the bigger of two numbers to compare 123 and 53. Ch…

> Since the added digit does not affect the value of the number, both numbers are equal.

Hilarious.

Re: Ask HN: How does ChatGPT work?

#137

Post by UncleMeat »

xenospn wrote: post #7

ChatGPT is a variant of the popular GPT-3 language model, specifically designed for chatbot applications. It uses a combination of deep learning and natural language processing techniques to generate human-like responses to text input in a conversation. The way it works is by first pre-training the model on a large corpus of text data, which could include things like social media conversations, movie scripts, books,…

AI generated comments should lead to bans. IMO, this sort of comment is no more valuable than a comment that just contains "lmao" or a dickbutt.

Re: Ask HN: How does ChatGPT work?

#138

Post by ffactory »

jb1991 wrote: post #87

ChatGPT is trained using a combination of supervised and unsupervised learning. For supervised learning, it is trained on a large dataset of human-generated text, such as dialogue data or online conversations. This allows it to learn the structure and style of natural language. For unsupervised learning, it is trained using a language modeling objective, which involves predicting the next word in a sequence of text.…

Not sure why, but while reading the two paragraphs, I already had a strong feeling they are written by chat-gpt...

Re: Ask HN: How does ChatGPT work?

#139

Post by tarvaina »

petesergeant wrote: post #72
bryan0 wrote: post #14

I found this description of the GPT-3 transformer architecture useful: https://dugas.ch/artificial_curiosity/GPT_architecture.html Not eli5 but close enough.

So it really is just a fancy Markov chain?

A Markov model is a stochastic process with a discrete visible state where the next state depends only on the current state. GPT’s state is neither discrete nor visible so it is not a Markov model.

Re: Ask HN: How does ChatGPT work?

#140

Post by the_third_wave »

k__ wrote: post #94

Half-OT: people are always talking about ChatGPT being AI, but is this actually the case? It frequently told me that it doesn't learn from my input, and I had the impression the unique selling point of AI was it being able to modify it's own code in response to input.

It does seem to learn during the session but it forgets what it has learned afterwards. I noticed this when I made a request in Dutch to which it replied in kind albeit with a grammatical error. I pointed out this error and showed it the correct form to which it replied by thanking me after which it consistently used the correct form during that session. When I reset the session and asked the same question it came back with more or less the same reply including the grammatical error.
Post reply on HN