Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How does ChatGPT work?

news.ycombinator.com

Post reply on HN
Print view View original

81–90 of 185 posts

Re: Ask HN: How does ChatGPT work?

#81

Post by throwaway290 »

chopete3 wrote: post #27

This blog explains some of the key innovations they added on top of GPT-3, especially the natural language understanding (following instructions). https://openai.com/blog/instruction-following/ In the first few paragraphs they show GPT-3 as equally dumb like all other language models that came before it and why they built instructgpt. >> Here is the summary (ChatGPT summarized) They present their approach to the prob…

To be clear there is no 'understanding' going on, and the article doesn't use that word either.

Re: Ask HN: How does ChatGPT work?

#82

Post by hugh-avherald »

starik36 wrote: post #38
akelly wrote: post #18

The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…

Ok, but how does it take my code and fixes the bug? It's my own code, no one has seen it besides me and the model wasn't trained on it.

How do you fix a bug? You've never seen it before.

Re: Ask HN: How does ChatGPT work?

#83

Post by FiberBundle »

zorr wrote: post #66
ravi-delia wrote: post #54

Earlier quoted context omitted.

In learning to predict the next token, the model has to pick up lots of little bits of world knowledge. I'm sure someone would disagree with the phrasing of "understand", but it certainly operates with more complexity than, say, a markov chain. It has seen lots of python, and in order to predict better, it has developed internal models of how python works. Think of how much better you'd do predicting the next charact…

In my (limited) experience it seems to perform even better for typed languages (for example Kotlin/Java/Swift) compared to Python. The Python code it provided often had subtle type issues when working with dates. While the Kotlin date-related code it provided was more accurate and correct in terms of types. Which makes sense since the additional type information likely leads to a much better "internal model of how Ko…

> zorr 1 hour ago | root | parent | next [–]

In my (limited) experience it seems to perform even better for typed languages (for example Kotlin/Java/Swift) compared to Python. The Python code it provided often had subtle type issues when working with dates. While the Kotlin date-related code it provided was more accurate and correct in terms of types. Which makes sense since the additional type information likely leads to a much better "internal model of how Kotlin works"

I think another possibility here is that they might have used an execution environment to check whether the code the model came up with actually compiles and used that as additional input during training. Some sort of execution environment seems to me to also be a possible explanation for how they managed the model to emulate a terminal so well.

Re: Ask HN: How does ChatGPT work?

#84

Post by 411111111111111 »

sillysaurusx wrote: post #37
soulofmischief wrote: post #25

Earlier quoted context omitted.

I don't mean to downplay how incredible the tech is, but I'm not sure I'd call this approach genius as it's the industry standard.

Then why were they the first ones to exploit it so effectively? I don’t think it was standard for GPT models.

I think the issue here was with the term genius, which makes it sounds like what was a completely new paradigme and revolutionary.

OpenAIs success mainly stems from extremely well executed previous concepts while mostly ignoring cost. And as they're pretty much the most successful public player in this domain, they've got the first-mover advantage which they're currently very succesfully leveraging. At least thats how it looks from the perspecitve of an armchair analysts, which wouldn't have been able to achieve the same -- even if I had the same resources and time.

The actual result is absolutely incredible however, regardless wherever the road to this end was genius or not

Re: Ask HN: How does ChatGPT work?

#85

Post by timonoko »

karmasimida wrote: post #79
timonoko wrote: post #42

How does the non-english languages part work? I thought maybe they use Google translator, but remembered that Russians have trained it to not to understand "russophobic" sentences. -- Mitä tarkoittaa ryssänvastainen, explain in English. -- Ryssänvastainen means "anti-Russian" or "anti-Russian sentiment." It refers to an attitude or behavior that is hostile or opposed to Russia or Russian interests.

Just vast amount of webpages in other languages.

You cannot learn this shit from just by reading webpages under hundred years. Where this insight comes from, I truly do not understand.

-- Mikä on ero sanoilla "talossa" ja "talosta". Explain in English.

-- The difference between the words "talossa" and "talosta" in Finnish is that "talossa" is the inessive case of the word "talo," which means "in the house," whereas "talosta" is the elative case of the same word, which means "out of the house." In other words, "talossa" indicates a location within a house, while "talosta" indicates movement away from a house.

Re: Ask HN: How does ChatGPT work?

#87

Post by jb1991 »

ChatGPT is trained using a combination of supervised and unsupervised learning. For supervised learning, it is trained on a large dataset of human-generated text, such as dialogue data or online conversations. This allows it to learn the structure and style of natural language. For unsupervised learning, it is trained using a language modeling objective, which involves predicting the next word in a sequence of text. This allows it to learn the broader patterns and characteristics of language, and to generate text that is fluent and coherent.

ChatGPT and GPT-3 are both large language models trained by OpenAI, but they have some important differences. GPT-3 is a more general-purpose language model, which means it is trained on a broader range of data and can generate a wider range of responses. It is also much larger than ChatGPT, with 175 billion parameters compared to ChatGPT's 2.6 billion parameters. This makes GPT-3 more powerful and capable of generating more realistic and diverse text, but also makes it more expensive and resource-intensive to use.

In case you are curious, the above information was written entirely by ChatGPT when asking it about itself.

Re: Ask HN: How does ChatGPT work?

#88

Post by ribit »

I understand the basic idea of predicting the words in a sequence, but what totally eludes me is how this relates to the prompt. After all, you don't give it a sequence to continue, you give it a direct request. Is there some special processing going on here or do they really just take the prompt as is and encode it?

Re: Ask HN: How does ChatGPT work?

#90

Post by miltondts »

bryan0 wrote: post #14

I found this description of the GPT-3 transformer architecture useful: https://dugas.ch/artificial_curiosity/GPT_architecture.html Not eli5 but close enough.

What I don't understand is where is the memory? How does GPT-3 or ChatGPT remember so much information with just that architecture? It would seem that the maximum it could remember is 2048 words.

EDIT: Maybe it's 2048 x 96? Still seems low for what it can do.

Post reply on HN