I'd love a recap of the tech for someone that remembers how ANNs work but not transformers (ELI5?). Why is ChatGPT so much better, too? and how big of a weight network are we talking about that it retains such a diverse knowledge on things?
Ask HN: How does ChatGPT work?
1–10 of 185 posts
Re: Ask HN: How does ChatGPT work?
#2No post body was provided.
Re: Ask HN: How does ChatGPT work?
#3ChatGPT is really very simple. Imagine you could analyze a million books and identify all the words within them -- not the meanings of the words, just the actual letters they contain.
Now, when someone asks you about the history of France (or why the sky is blue), you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question!
It's like a kid in the 80's who thinks the answer to an essay question is to copy it from an encyclopedia, only the "encyclopedia" is very large and contains multiple sources.
So, the big take away needs to be that there is absolutely no understanding, no cognizance of any kind, no language comprehension going on. The answers look good because they contain all the same words as the most popular answers people have already written which the system scanned.
So ChatGPT turns out to great for parsing and summarizing documents, if that's something you need. But, since it doesn't know fact from fiction, it cannot apply logic or math, and it cannot perform reasoning or analysis, it's not good for finding out facts or discerning truth.
Another great failing of LLM software is that the user being spoken to is generic. The answers are not modeled for you, they're the same models for everyone. But a human teacher does their job by being exactly the opposite of this -- someone who is finely tuned to the needs and understandings of their audience. A good journalist or writer does the same.
Re: Ask HN: How does ChatGPT work?
#4ChatGPT is really very simple. Imagine you could analyze a million books and identify all the words within them -- not the meanings of the words, just the actual letters they contain. Now, when someone asks you about the history of France (or why the sky is blue), you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question! It's like a kid in the…
I don't think this is an accurate depiction and I think there's a lot more going on there than you're saying there is. The models also involve Codex, which is able to effectively do pattern matching with more context you give it so that it can emit novel code. I've seen this happen numerous times with Copilot, where it works very well the more open documents you have, especially if those documents are related. True, it doesn't have a full semantic understanding of your codebase, but it's good enough to usually generate the right code once it's been "seeded" with something like an updated type definition that you then need to plumb through various parts of a codebase.
Re: Ask HN: How does ChatGPT work?
#5ChatGPT is really very simple. Imagine you could analyze a million books and identify all the words within them -- not the meanings of the words, just the actual letters they contain. Now, when someone asks you about the history of France (or why the sky is blue), you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question! It's like a kid in the…
> you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question!
This is not sufficient to explain how LLMs are able to synthesize novel, coherent poems or song lyrics. What you're describing seems closer to a markov model. So far I have yet to see a good explanatin of _why_ transformer models seem to have this emergent behavior as you scale it up. It's easy to see by construction that the stacked attention layers will work well to predict the masked token in "Capital of France is [MASK]", or how you can throw an additional head on top for sentiment analysis (which I recall was implemented by literally adding a dummy token at the start into which information about the entire sentence gets embedded). But it's not obvious (to me at least) that this would somehow generalize into being able to do the things chatGPT does.
Re: Ask HN: How does ChatGPT work?
#6ChatGPT is really very simple. Imagine you could analyze a million books and identify all the words within them -- not the meanings of the words, just the actual letters they contain. Now, when someone asks you about the history of France (or why the sky is blue), you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question! It's like a kid in the…
OP question mentions specific network types, while reply uses a vague "million-book library" metaphor, but at least mentions LLM?
Also arbitrarily sorts human { teacher, journalist, writer } as "people who do custom work with each client" which makes a bit of sense for a Teacher in a small-class environment. But Writers and Journalists write at SCALE, and while each may speak with gusto to individuals just as well as they do their job, tailoring custom content to individuals is not their job.
Not to mention that individual-tailored model files are very handy with AI, share-able as those diles may be.
I call ChatGPT on you!!! And also do not know anything about your question, OP...
Re: Ask HN: How does ChatGPT work?
#7ChatGPT is a variant of the popular GPT-3 language model, specifically designed for chatbot applications. It uses a combination of deep learning and natural language processing techniques to generate human-like responses to text input in a conversation.
The way it works is by first pre-training the model on a large corpus of text data, which could include things like social media conversations, movie scripts, books, etc. This allows the model to learn the general structure and patterns of language.
Then, when given an input in the form of a question or statement, the model uses its pre-trained knowledge to generate a response. It does this by predicting the next word in the sentence, and then continuing to predict subsequent words until it reaches the end of the response.
Overall, the goal of ChatGPT is to enable chatbots to have more natural, human-like conversations with users.
(I asked ChatGPT to tell me how it works)
Re: Ask HN: How does ChatGPT work?
#8ChatGPT is really very simple. Imagine you could analyze a million books and identify all the words within them -- not the meanings of the words, just the actual letters they contain. Now, when someone asks you about the history of France (or why the sky is blue), you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question! It's like a kid in the…
I don't think this is an accurate depiction and I think there's a lot more going on there than you're saying there is. The models also involve Codex, which is able to effectively do pattern matching with more context you give it so that it can emit novel code. I've seen this happen numerous times with Copilot, where it works very well the more open documents you have, especially if those documents are related. True,…
Yes, I use Copilot, too for that very reason. But we need to be very careful about words like "semantic" and "understanding" as the method is neither.
We'll get the best use out of these technologies if we don't ascribe to them magical qualities they don't have.
Poke around. You'll find out it just statistical math with tokens (letters and punctuation). No meaning is ascribed to anything.
Re: Ask HN: How does ChatGPT work?
#9ChatGPT is a variant of the popular GPT-3 language model, specifically designed for chatbot applications. It uses a combination of deep learning and natural language processing techniques to generate human-like responses to text input in a conversation. The way it works is by first pre-training the model on a large corpus of text data, which could include things like social media conversations, movie scripts, books,…
Marginally funny, utterly useless and counterproductive is what it is (this particular use of chatGPT that is).
Re: Ask HN: How does ChatGPT work?
#10It's still transformer underneath, but openai researchers have figured out how to improve it through engineering efforts and improved training data. I believe it's not easy for outsiders without large model pretraning experience like most of us to understand the tunning details.