Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How does ChatGPT work?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 185 posts

Re: Ask HN: How does ChatGPT work?

#111

Post by dsr3 »

amelius wrote: post #103
akelly wrote: post #18

The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…

That's not genius, that's called unsupervised learning and it is an entire subfield.

I think number 3 is a description of Generative Adversarial Network (GAN).

Re: Ask HN: How does ChatGPT work?

#112

Post by DeWilde »

anthropodie wrote: post #23
akelly wrote: post #18

The way they went from GPT-3 to ChatGPT is really quite genius. My understanding is that it's something like this: 1. Start with GPT-3, which predicts the next word in some text and is trained on all the text on the internet 2. Take thousands of prompts, generate several responses for each of them, and have human reviewers rank the responses for each prompt from best to worst 3. The GPT model needs a massive amount o…

I want to know if it will ever be possible to run this kind of AI at home once its training is complete. I dont need all the knowledge just subset that I'm interested in. Actually I'm more interested in its ability to transform things. For example I can ask it to convert docker-compose to docker run command, it can manipulate JSON, it can sort numbers in table when prompted. I'm more interested in these abilities rat…

It uses GPT-3 under the hood which requires about 350 gigabytes of GPU VRAM (back of the envelope calc, likely more) to perform these inferences.

Re: Ask HN: How does ChatGPT work?

#113

Post by tluyben2 »

samus wrote: post #100
tluyben2 wrote: post #74

Earlier quoted context omitted.

When I was a first year AI student beginning of the 90s I asked my professor what would happen if we just made a massive neural network and trained it with all information in the world. He said it cannot happen as it it impossible.

Considering the computational resources available at the time, he was not that wrong. Research into artificial neuronal networks has always been held back by available computational power.

Agreed, but as a professor I believe one needs to be looking in the future. It was not that far out, but yeah it was an AI winter. We were stuck until 2012 basically. That's a long time.

Re: Ask HN: How does ChatGPT work?

#114

Post by dotancohen »

swayson wrote: post #32

Yannic Kilcher did an explainer recently on his YT channel. https://www.youtube.com/watch?v=0A8ljAkdFtg Yannic explains these models pretty well.

I just skimmed the video. It seems full of information, but this video skips. It seems like either the author composed his sentences by slicing together various clips, or a few frames have been removed between some words. I tried opening the video in NewPipe as well, same problem.

Re: Ask HN: How does ChatGPT work?

#115

Post by gillesjacobs »

xvector wrote: post #108
hackernewds wrote: post #107

Earlier quoted context omitted.

you can do that today in the free release?

I think he wants to self host. It sucks to have no ownership of such a powerful tool I would pay upwards of $3000 to be able to self host something like this.

Rest assured someone is working on a self-hosted (distilled) model. Stable Diffusion has shown there is a viable market for open, consumer-hardware inferencable models.

Re: Ask HN: How does ChatGPT work?

#117

Post by Al-Khwarizmi »

tluyben2 wrote: post #113
samus wrote: post #100

Earlier quoted context omitted.

Considering the computational resources available at the time, he was not that wrong. Research into artificial neuronal networks has always been held back by available computational power.

Agreed, but as a professor I believe one needs to be looking in the future. It was not that far out, but yeah it was an AI winter. We were stuck until 2012 basically. That's a long time.

Easier said than done.

I'm a professor in an AI field, and I can tell you that neither myself nor the colleagues I regularly have scientific discussions with could imagine ten years ago that something like ChatGPT would be possible in 2022. I suppose there might be a minority who called it, but recent advances in deep learning absolutely whooshed past the predictions of the overwhelming majority of people in the field.

Re: Ask HN: How does ChatGPT work?

#118

Post by k__ »

mjburgess wrote: post #101
k__ wrote: post #94

Half-OT: people are always talking about ChatGPT being AI, but is this actually the case? It frequently told me that it doesn't learn from my input, and I had the impression the unique selling point of AI was it being able to modify it's own code in response to input.

That's basically never been the case for any statistical AI system which are all subject to "catestrophic forgetting". The weights of the system which are its memory are set by having their values determined over an entire training set. This is, effectively, a compression process which takes eg., most of the internet, and compresses it to 1TB of weights. If you update those weights by a process which doesn't "zip eve…

Just because it's a bad idea doesn't mean it's the right definition of AI.

I always thought of the current systems of static machine learning models.

Re: Ask HN: How does ChatGPT work?

#119

Post by Sharlin »

dotancohen wrote: post #109
sebzim4500 wrote: post #102

Earlier quoted context omitted.

There should be a rule on HN that anyone claiming definitively whether or not AIs 'know' or 'understand' something should have to define those words rigorously.

The question goes back decades - back then the question was phrased "Can computers think" and the rebuttal was "Can submarines swim?"

On the other hand, airplanes do fly. But I guess that just reinforces the point that language is a map, not the territory, and natural language in particular is full of inconsistencies and fuzzy boundaries between poorly defined categories. English happened to evolve so that it ended up using the same word for the thing both winged creatures and airplanes do, but only living creatures swim, boats and ships and submarines do not. In other languages, submarines do swim.

Re: Ask HN: How does ChatGPT work?

#120

Post by rightbyte »

undefuser wrote: post #69
DemocracyFTW2 wrote: post #62

What baffles and makes it hard for me to wrap my brain around the statement that ChatGPT is "just predictive text analysis" is that it certainly feels like it goes into detail, picks up and repeats the particulars, presents a line of reasoning that sounds, well, reasonable, until you realize it got a number of things terribly wrong, often in deceptively subtle ways: Me: Which number is bigger, 873593742593454623 or 8…

I applied the same prompts and the results get better: Me: Which number is bigger, 873593742593454623 or 873593742593554623 ChatGPT: These two numbers are equal. The first number is simply the second number with an extra digit added at the end. Since the added digit does not affect the value of the number, both numbers are equal. Me: Apply your reasoning and finding the bigger of two numbers to compare 123 and 53. Ch…

By picking both possible answers you are always partially correct!

I have noticed this too. I guess it is an artifact of just being probable text output.

Post reply on HN