Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How does ChatGPT work?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 185 posts

Re: Ask HN: How does ChatGPT work?

#41

Post by mclightning »

akelly wrote: post #18

The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…

That's super interesting. When GPT-3 came out, I wrote an article inspired by it. That we could one day build an AI that acts like AGI, by a crazy vast amount of multimedia training data, collected by willing users to participate in ever improving AI interactions;

https://medium.com/swlh/bicameral-mind-humanoid-robot-with-g...

Re: Ask HN: How does ChatGPT work?

#42

Post by timonoko »

How does the non-english languages part work?

I thought maybe they use Google translator, but remembered that Russians have trained it to not to understand "russophobic" sentences.

-- Mitä tarkoittaa ryssänvastainen, explain in English.

-- Ryssänvastainen means "anti-Russian" or "anti-Russian sentiment." It refers to an attitude or behavior that is hostile or opposed to Russia or Russian interests.

Re: Ask HN: How does ChatGPT work?

#43

Post by hanniabu »

akelly wrote: post #18

The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…

How does step 1 work? It seems incredibly inefficient to check your word combo against every single segment of text they have. How does it do this efficiently?

Re: Ask HN: How does ChatGPT work?

#44

Post by seydor »

hanniabu wrote: post #43
akelly wrote: post #18

The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…

How does step 1 work? It seems incredibly inefficient to check your word combo against every single segment of text they have. How does it do this efficiently?

https://en.wikipedia.org/wiki/Transformer_(machine_learning_...

Re: Ask HN: How does ChatGPT work?

#45

Post by amilios »

subroutine wrote: post #17

How does it know when to stop when asked for a description or summary? Sometimes it outputs a few sentences, sometimes a few paragraphs. Does it know how much output it has already provided when deciding on the next token? How does it decide to start a new sentence or paragraph, or if it's 'satisfied' with its current response?

In the same way that it predicts a given token like 'we' or 'write', it predicts a special end-output token, something analogous to 'EOF'. So at some point as it's generating new tokens and coming up with probability distributions for the next token (conditioned on its own output so far), selecting from this distribution (either by greedily picking the highest probability token or sampling), it will select this 'EOF'-style token, and end its own output. Does that make sense?

So to answer your question about knowing how much text it has output so far, not exactly. It is not 'stateful' exactly, it's simply conditioned on previous text and gives a new probability distribution over its vocabulary for the next token. Whether the 'previous text' is its own output, or provided by a human user (a 'prompt'), it isn't necessarily aware of this by default (unless the user vs. system responses are delineated somehow with special tokens for example, which with ChatGPT they may be/are likely to be. My point is that these models don't come with this awareness built in, you have to add it. Fundamentally they simply condition their probability distribution on a piece of existing text).

Re: Ask HN: How does ChatGPT work?

#46

Post by krackers »

throwaway1851 wrote: post #20
krackers wrote: post #5

Earlier quoted context omitted.

> you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question! This is not sufficient to explain how LLMs are able to synthesize novel, coherent poems or song lyrics. What you're describing seems closer to a markov model. So far I have yet to see a good explanatin of _why_ transformer models seem to have this emergent behavior as you scale it up.…

I don't really see how it's different from, say, a large convolutional neural network learning progressively higher-order features as you progress through the layers of the network. At the lowest layers it's learning simple edge filters, which get combined into shapes, which get combined into filters that activate on faces, which get combined in ways that can be recognized as "family portrait", and so on. Of course,…

Thanks, this is a nice explanation! Also found a paper related to emergence in LLMs: https://arxiv.org/pdf/2206.07682.pdf

Re: Ask HN: How does ChatGPT work?

#47

Post by seydor »

timonoko wrote: post #42

How does the non-english languages part work? I thought maybe they use Google translator, but remembered that Russians have trained it to not to understand "russophobic" sentences. -- Mitä tarkoittaa ryssänvastainen, explain in English. -- Ryssänvastainen means "anti-Russian" or "anti-Russian sentiment." It refers to an attitude or behavior that is hostile or opposed to Russia or Russian interests.

The model says it has been trained in a variety of languages, but its current version is english. Its translations are bad, probably because it uses a translation model on top of its output. Sometimes it is obvious it is translating english articles from the mistakes it makes

Re: Ask HN: How does ChatGPT work?

#48

Post by seydor »

xenospn wrote: post #7

ChatGPT is a variant of the popular GPT-3 language model, specifically designed for chatbot applications. It uses a combination of deep learning and natural language processing techniques to generate human-like responses to text input in a conversation. The way it works is by first pre-training the model on a large corpus of text data, which could include things like social media conversations, movie scripts, books,…

thats only half the truth, in which the AI sneakily omits the role of human reviewers that worked tirelessly to align its output and make it useful. Shame on the AI

Re: Ask HN: How does ChatGPT work?

#49

Post by osigurdson »

ChatGPT is great. I use it a lot. But... it is still necessary to use google for many things. ChatGPT is somewhat out of date and strangest thing is, it will almost always provide an answer (right or wrong). So, for the most part, everything has to be manually checked.

Re: Ask HN: How does ChatGPT work?

#50

Post by stevenhuang »

chopete3 wrote: post #29
chopete3 wrote: post #27

This blog explains some of the key innovations they added on top of GPT-3, especially the natural language understanding (following instructions). https://openai.com/blog/instruction-following/ In the first few paragraphs they show GPT-3 as equally dumb like all other language models that came before it and why they built instructgpt. >> Here is the summary (ChatGPT summarized) They present their approach to the prob…

The interesting thing was - there were 0 comments on HN when this blog was posted on Jan 27th. It was such a fundamental breakthrough NLU. NLU has been a such a holy grail of language models.

NLU?
Post reply on HN