Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How does ChatGPT work?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 185 posts

Re: Ask HN: How does ChatGPT work?

#91

Post by spion »

starik36 wrote: post #38
akelly wrote: post #18

The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…

Ok, but how does it take my code and fixes the bug? It's my own code, no one has seen it besides me and the model wasn't trained on it.

This is why this paper was so exciting when it came out https://arxiv.org/abs/2005.14165

Make a large enough model and train it with all sorts of data and it will be able to encode generalized concepts which can then be applied to specific tasks (given only a few examples of the task, or even just a query / question, rather than an example)

Re: Ask HN: How does ChatGPT work?

#94

Post by k__ »

Half-OT: people are always talking about ChatGPT being AI, but is this actually the case?

It frequently told me that it doesn't learn from my input, and I had the impression the unique selling point of AI was it being able to modify it's own code in response to input.

Re: Ask HN: How does ChatGPT work?

#96

Post by timonoko »

timonoko wrote: post #85
karmasimida wrote: post #79

Earlier quoted context omitted.

Just vast amount of webpages in other languages.

You cannot learn this shit from just by reading webpages under hundred years. Where this insight comes from, I truly do not understand. -- Mikä on ero sanoilla "talossa" ja "talosta". Explain in English. -- The difference between the words "talossa" and "talosta" in Finnish is that "talossa" is the inessive case of the word "talo," which means "in the house," whereas "talosta" is the elative case of the same word, wh…

Too simple example above. The idea is that you can modify the baseword ad inifinitum, which the machine seems to grasp.

-- Explain the difference between Finnish words "taloissani" and "taloistani".

The words "taloissani" and "taloistani" are both possessive forms of the word "talo" which means "house" in Finnish. The difference between the two words lies in their grammatical case.

"Taloissani" is in the inessive case, which is used to indicate location within something. In this case, it would be translated as "in my houses."

"Taloistani" is in the elative case, which is used to indicate movement out of something. In this case, it would be translated as "out of my houses."

In Finnish, the ending of a word changes depending on its grammatical case, and this is how the difference between "taloissani" and "taloistani" is expressed.

Re: Ask HN: How does ChatGPT work?

#97

Post by rakejake »

stevenhuang wrote: post #55
dtagames wrote: post #3

ChatGPT is really very simple. Imagine you could analyze a million books and identify all the words within them -- not the meanings of the words, just the actual letters they contain. Now, when someone asks you about the history of France (or why the sky is blue), you could simply pluck out of your library the most common strings of word that seem to follow the words that were in your question! It's like a kid in the…

> Another great failing of LLM software is that the user being spoken to is generic. The answers are not modeled for you, they're the same models for everyone. But a human teacher does their job by being exactly the opposite of this -- someone who is finely tuned to the needs and understandings of their audience. A good journalist or writer does the same. ChatGPT can do this if you just ask. E.g., explain as if I'm 5…

This is true but does not refute OP's point. There is an entire subreddit for ELI5 that is probably part of its training data. So if you ask for ELI5, that pattern IS part of the training corpus.

Re: Ask HN: How does ChatGPT work?

#98

Post by [deleted] »

dzink wrote: post #31
akelly wrote: post #18

The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…

ChatGPT seems to be/result in some amount of caching of responses - there is very little variation when to asking the same question multiple times. CharacterAI produces a lot more variety in comparison, making it more helpful for brainstorming. That said ChatGPT is likely closer to the truth, even if not perfect, for searches. The innovation happening lately is incredible.

[deleted]

Re: Ask HN: How does ChatGPT work?

#99

Post by mjburgess »

miltondts wrote: post #90
bryan0 wrote: post #14

I found this description of the GPT-3 transformer architecture useful: https://dugas.ch/artificial_curiosity/GPT_architecture.html Not eli5 but close enough.

What I don't understand is where is the memory? How does GPT-3 or ChatGPT remember so much information with just that architecture? It would seem that the maximum it could remember is 2048 words. EDIT: Maybe it's 2048 x 96? Still seems low for what it can do.

300bn weights, at 4bytes/weight is 1.2TB

Re: Ask HN: How does ChatGPT work?

#100

Post by samus »

tluyben2 wrote: post #74
mclightning wrote: post #41

Earlier quoted context omitted.

That's super interesting. When GPT-3 came out, I wrote an article inspired by it. That we could one day build an AI that acts like AGI, by a crazy vast amount of multimedia training data, collected by willing users to participate in ever improving AI interactions; https://medium.com/swlh/bicameral-mind-humanoid-robot-with-g...

When I was a first year AI student beginning of the 90s I asked my professor what would happen if we just made a massive neural network and trained it with all information in the world. He said it cannot happen as it it impossible.

Considering the computational resources available at the time, he was not that wrong. Research into artificial neuronal networks has always been held back by available computational power.
Post reply on HN