Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How does ChatGPT work?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 185 posts

Re: Ask HN: How does ChatGPT work?

#21

Post by typon »

There were several key insights that have made something like ChatGPT possible relative to traditional neural networks.

* A fixed (but large) vocabulary of sub-word like tokens as inputs.

* Attention mechanism for learning the correlation of words in a fixed sequence window.

* Implementing this attention mechanism in the form of matrix multiplies rather than some other complex math - it allows it to be parallelized and run on GPUs fast.

* Having enough layers of these encoders to have a huge amount of parameter space. ~175B parameters in the case of ChatGPT.

* Feed the model a lot of data - in this case, pretty much the entire internet as text.

* Self supervised learning: we take sentences from the internet and mask out some words and force the network to predict the missing word. Turns out this works extremely well. We don't use the traditional supervised learning inputs -> (prediction, label) paradigm that was the standard 10 years ago.

* RLHF (Reinforcement learning from human feedback). Take generated text from GPT-3 (the underlying generative model) and ask humans to rate different completions. Retrain the model from those ratings.

* A massive compute infrastructure that is able to train this model in a reasonable amount of time, allowing for iteration on hyperparameters. For example, what's the optimal attention head-size? How many encoder layers are good? What should the sequence length be? What should the embedding dimension be? etc. In OpenAI's case, they used thousands of GPUs and thousands of CPUs provided by Microsoft/Azure.

In summary, relatively simple model, parallelizable on GPUs, trained on a lot of data.

Re: Ask HN: How does ChatGPT work?

#22

Post by jupp0r »

I asked ChatGPT and here is its answer:

Sure, I'd be happy to help. An artificial neural network (ANN) is a type of machine learning algorithm that is designed to mimic the way the human brain processes information. At its core, an ANN is composed of a large number of interconnected processing nodes, called neurons, which are organized into multiple layers. These layers are interconnected in a way that allows them to pass information back and forth, with each layer building on the information provided by the previous layer to make a decision or generate a response.

Transformers are a type of ANN that was introduced in 2017 by a team of researchers at Google. They are called "transformers" because they use a technique called "self-attention" to transform the input data before passing it through the network. This allows them to better capture the underlying relationships in the data, which makes them particularly well-suited for tasks like natural language processing and machine translation.

ChatGPT is a type of transformer that has been specifically designed for use in chatbots and other conversational AI systems. It is an extension of the GPT-3 model, which was developed by the same team of researchers at OpenAI. GPT-3 is a large language model that has been trained on a massive amount of text data and is able to generate human-like responses to a wide range of questions. By building on the capabilities of GPT-3, ChatGPT is able to provide even more sophisticated and natural-sounding responses in a conversational setting.

As for the size of the network, GPT-3 is a very large model, with 175 billion parameters. This means that it has a massive amount of interconnected neurons, which allows it to capture a wide range of knowledge and generate responses to a wide variety of questions. Of course, having a large network also requires a lot of computational power, which is why GPT-3 and other large language models are typically run on powerful servers or specialized hardware.

Re: Ask HN: How does ChatGPT work?

#23

Post by anthropodie »

akelly wrote: post #18

The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…

I want to know if it will ever be possible to run this kind of AI at home once its training is complete. I dont need all the knowledge just subset that I'm interested in.

Actually I'm more interested in its ability to transform things. For example I can ask it to convert docker-compose to docker run command, it can manipulate JSON, it can sort numbers in table when prompted. I'm more interested in these abilities rather than just getting answers for which I already have Google

Re: Ask HN: How does ChatGPT work?

#24

Post by the_gipsy »

dtagames wrote: post #3

ChatGPT is really very simple. Imagine you could analyze a million books and identify all the words within them -- not the meanings of the words, just the actual letters they contain. Now, when someone asks you about the history of France (or why the sky is blue), you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question! It's like a kid in the…

> So, the big take away needs to be that there is absolutely no understanding, no cognizance of any kind, no language comprehension going on.

Okay, but what is understanding, cognizance, comprehension? And so on for further precision of definitions, until you reach that vague nebulous area that is consciousness. We simply haven't figured that out, period.

So we cannot really say that this AI does not have those qualities, unless its output is obviously showing it. Which ChatGPT does not, most of the time.

Re: Ask HN: How does ChatGPT work?

#25

Post by soulofmischief »

akelly wrote: post #18

The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…

I don't mean to downplay how incredible the tech is, but I'm not sure I'd call this approach genius as it's the industry standard.

Re: Ask HN: How does ChatGPT work?

#26

Post by benatkin »

xenospn wrote: post #7

ChatGPT is a variant of the popular GPT-3 language model, specifically designed for chatbot applications. It uses a combination of deep learning and natural language processing techniques to generate human-like responses to text input in a conversation. The way it works is by first pre-training the model on a large corpus of text data, which could include things like social media conversations, movie scripts, books,…

Here's an answer I wrote myself:

It gives people a heavily filtered and throttled interface to access to their language model.

---

Edit: I asked chat.openai.com about it:

Would "It gives people a heavily filtered and throttled interface to access to their language model." be a fair way to describe chat.openai.com?

Yes, that is an accurate description of chat.openai.com. The site provides users with a limited and controlled interface to access a language model, which is a type of artificial intelligence that is capable of processing and generating natural language. This allows users to have conversations with the language model and get responses based on the information it has been trained on.

Re: Ask HN: How does ChatGPT work?

#27

Post by chopete3 »

This blog explains some of the key innovations they added on top of GPT-3, especially the natural language understanding (following instructions).

https://openai.com/blog/instruction-following/

In the first few paragraphs they show GPT-3 as equally dumb like all other language models that came before it and why they built instructgpt.

>> Here is the summary (ChatGPT summarized) They present their approach to the problem, which involves using a recurrent neural network to encode both the instruction and the environment, and then using a reinforcement learning algorithm to learn how to execute the instruction. They demonstrate that their method can learn to solve a variety of instruction following tasks.

Some snippets from the blog:

>> InstructGPT is then further fine-tuned on a dataset labeled by human labelers. The labelers comprise a team of about 40 contractors whom we hired through Upwork and ScaleAI.

>> We hired about 40 contractors, guided by their performance on a screening test meant to judge how well they could identify and respond to sensitive prompts, and their agreement rate with researchers on a labeling task with detailed instructions. We kept our team of contractors small because it's easier to have high-bandwidth communication with a smaller set of contractors who are doing the task full-time.

Re: Ask HN: How does ChatGPT work?

#29

Post by chopete3 »

chopete3 wrote: post #27

This blog explains some of the key innovations they added on top of GPT-3, especially the natural language understanding (following instructions). https://openai.com/blog/instruction-following/ In the first few paragraphs they show GPT-3 as equally dumb like all other language models that came before it and why they built instructgpt. >> Here is the summary (ChatGPT summarized) They present their approach to the prob…

The interesting thing was - there were 0 comments on HN when this blog was posted on Jan 27th. It was such a fundamental breakthrough NLU. NLU has been a such a holy grail of language models.

Re: Ask HN: How does ChatGPT work?

#30

Post by wizofaus »

Rather strangely, it would seem - I just had this response: "However, I am a language model and do not have the ability to edit or revise my responses once they have been generated"

Except I've had no problem getting it to do just that previously...

I'm curious about its training data too, as I've managed to find a few things it knows nothing about (despite them having wikipedia pages and multiple dedicated websites about, and having been around 10+ years).

Post reply on HN