Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How does ChatGPT work?

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 185 posts

Re: Ask HN: How does ChatGPT work?

#121

Post by Sharlin »

ribit wrote: post #88

I understand the basic idea of predicting the words in a sequence, but what totally eludes me is how this relates to the prompt. After all, you don't give it a sequence to continue, you give it a direct request. Is there some special processing going on here or do they really just take the prompt as is and encode it?

A direct request is just a specific sort of a sequence to complete. There are extra training steps involved in ChatGPT that are designed to make it particularly good at predicting sequences that look like semantically valid, coherent dialogue, but fundamentally there's nothing special about dialogue. It's just tokens after tokens. For example, even a fairly simple model trained on only play and movie scripts would be naturally good at completing pieces of dialogue because that would be all that it knows!

Re: Ask HN: How does ChatGPT work?

#123

Post by acapybara »

anthropodie wrote: post #23
akelly wrote: post #18

The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…

I want to know if it will ever be possible to run this kind of AI at home once its training is complete. I dont need all the knowledge just subset that I'm interested in. Actually I'm more interested in its ability to transform things. For example I can ask it to convert docker-compose to docker run command, it can manipulate JSON, it can sort numbers in table when prompted. I'm more interested in these abilities rat…

Yeah buddy!

Look up fine tuning GPT-J in 8 bit mode.

People have made domain-specific models that perform well (IIRC, better than GPT-3 in their domain).

The team behind Stable Diffusion is also working on one that's supposed to be pretty good.

Re: Ask HN: How does ChatGPT work?

#124

Post by Sharlin »

k__ wrote: post #94

Half-OT: people are always talking about ChatGPT being AI, but is this actually the case? It frequently told me that it doesn't learn from my input, and I had the impression the unique selling point of AI was it being able to modify it's own code in response to input.

There is no exact definition of what "AI" is. But there have been millions and millions of software systems in the world that could classified as "AI" since the 60s, and approximately none of them have had any means of altering their code on the fly, so no.

Re: Ask HN: How does ChatGPT work?

#125

Post by bozhark »

DeWilde wrote: post #112
anthropodie wrote: post #23

Earlier quoted context omitted.

I want to know if it will ever be possible to run this kind of AI at home once its training is complete. I dont need all the knowledge just subset that I'm interested in. Actually I'm more interested in its ability to transform things. For example I can ask it to convert docker-compose to docker run command, it can manipulate JSON, it can sort numbers in table when prompted. I'm more interested in these abilities rat…

It uses GPT-3 under the hood which requires about 350 gigabytes of GPU VRAM (back of the envelope calc, likely more) to perform these inferences.

7x NVIDIA RTX A6000's so ~$32,550

Re: Ask HN: How does ChatGPT work?

#126

Post by amelius »

dsr3 wrote: post #111
amelius wrote: post #103

Earlier quoted context omitted.

That's not genius, that's called unsupervised learning and it is an entire subfield.

I think number 3 is a description of Generative Adversarial Network (GAN).

Ok, regardless, it is not really new. ML researchers are doing these kinds of things all the time.

By the way, according to some people, GANs are also a kind of unsupervised learning: https://stackoverflow.com/questions/44445778/are-gans-unsupe...

Re: Ask HN: How does ChatGPT work?

#127

Post by weird-eye-issue »

dotancohen wrote: post #58
ofrzeta wrote: post #35

Earlier quoted context omitted.

> Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst Recently I saw an image where Indian women sat in front of computers and the caption said they were classifying "AI" responses. I guess that's true and this kind of work is the new outsourced cheap labour in the AI age.

That Indian woman's idea of acceptable and not acceptable AI responses surely vary from that of a San Fransisco tech worker, or Cape Town motorcycle mechanic, or an English teacher from Liverpool. I really doubt the mechanical turk method is applicable or even useful for the current state of AI-generated text.

It could be a first pass

Re: Ask HN: How does ChatGPT work?

#128

Post by nsb1 »

akelly wrote: post #18

The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…

You forgot the fresh cup of really hot tea :)

https://hitchhikers.fandom.com/wiki/Infinite_Improbability_D...

Re: Ask HN: How does ChatGPT work?

#129

Post by tluyben2 »

Al-Khwarizmi wrote: post #117
tluyben2 wrote: post #113

Earlier quoted context omitted.

Agreed, but as a professor I believe one needs to be looking in the future. It was not that far out, but yeah it was an AI winter. We were stuck until 2012 basically. That's a long time.

Easier said than done. I'm a professor in an AI field, and I can tell you that neither myself nor the colleagues I regularly have scientific discussions with could imagine ten years ago that something like ChatGPT would be possible in 2022. I suppose there might be a minority who called it, but recent advances in deep learning absolutely whooshed past the predictions of the overwhelming majority of people in the fiel…

Ah yes, that's what I was trying to say. I was the worst sceptic of AI; I never did anything with AI with it after getting my masters. I just went for money, programming and managing programming.

For me [1] this is the most mindboggling thing I have seen in my life and I don't think people realise what it means. And yes, it wooshed passed anything I thought possible in my lifetime. I hate that it's 'not be evil', 'anti thought crime' etc but it is really incredible what it does.

[0] https://twitter.com/luyben/status/1600663169353015297 [1] https://brainfisheatfishbrain.com/about/

Re: Ask HN: How does ChatGPT work?

#130

Post by weird-eye-issue »

rakejake wrote: post #97
stevenhuang wrote: post #55

Earlier quoted context omitted.

> Another great failing of LLM software is that the user being spoken to is generic. The answers are not modeled for you, they're the same models for everyone. But a human teacher does their job by being exactly the opposite of this -- someone who is finely tuned to the needs and understandings of their audience. A good journalist or writer does the same. ChatGPT can do this if you just ask. E.g., explain as if I'm 5…

This is true but does not refute OP's point. There is an entire subreddit for ELI5 that is probably part of its training data. So if you ask for ELI5, that pattern IS part of the training corpus.

It is more complex than that, it isn't just scraping /r/ELI5 lol

Ask it a question about JavaScript and tell it to respond as a 1920s gangster and it will happily oblige and do a great job. And just to make sure we are on the same page, JavaScript was not available in the 20s

Post reply on HN