Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

anfalmushtaq.com

Post reply on HN
Print view View original

61–70 of 91 posts

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#61

Post by quantumspandex »

mtkd wrote: post #58
quantumspandex wrote: post #56

Andrej's video is great but the explanation on the RL part is a bit vague to me. How exactly do we train on the right answers? Do we collect the reasoning traces and train on them like supervised learning or do we compute some scores and use them as a loss function ? Isn't the reward then very sparse? What if LLMs can't generate any right answers cause the problems are too hard? Also how can the training of LLMs be p…

Details on how DS used GRPO for RL rewards https://medium.com/@sahin.samia/the-math-behind-deepseek-a-d...

Thanks!

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#62

Post by quantumspandex »

epr wrote: post #59
quantumspandex wrote: post #56

Andrej's video is great but the explanation on the RL part is a bit vague to me. How exactly do we train on the right answers? Do we collect the reasoning traces and train on them like supervised learning or do we compute some scores and use them as a loss function ? Isn't the reward then very sparse? What if LLMs can't generate any right answers cause the problems are too hard? Also how can the training of LLMs be p…

https://arxiv.org/abs/1707.06347

Will have a look. Thanks!

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#63

Post by quantumspandex »

fenomas wrote: post #60
quantumspandex wrote: post #56

Andrej's video is great but the explanation on the RL part is a bit vague to me. How exactly do we train on the right answers? Do we collect the reasoning traces and train on them like supervised learning or do we compute some scores and use them as a loss function ? Isn't the reward then very sparse? What if LLMs can't generate any right answers cause the problems are too hard? Also how can the training of LLMs be p…

As I understood that part, in RL for LLMs you take questions for which the model already sometimes emits correct answers, and then repeatedly infer while reinforcing the activations the model made during correct responses, which lets it evolve its own ways of more reliably reaching the right answer. (Hence the analogy to training AlphaGo, wherein you take a model that sometimes wins games, and then play a bunch of ga…

AlphaGo seems more like an automated process to me because you can start from nothing except the algorithm and the rules. Since a Go game only has 2 outcomes most of the time, and the model can play with itself, it is guaranteed to learn something during self-play.

In the LLM case you have to have an already capable model to do RL. Also I feel like the problem selection part is important to make sure it's not too hard. So there's still much labor involved.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#65

Post by p0w3n3d »

On 53 minutes from the original video, he shows how exact is the quotation of an LLM based on the text it was learning from. I wonder how did the bigtech convince the courts that this is not copyright violation (especially when ChatGPT was quoting some GPL code). I can imagine that the same thing would happen opposite, if I trained a model to draw a disney character, and my ass would be sued in a fraction of a second.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#66

Post by [deleted] »

m11a wrote: post #37
albert_e wrote: post #12

OT: What is a good place to discuss the original video -- once it has dropped out of the HN front-page? I am going through the video myself -- roughly halfway through -- and have a fw things to bring up. Here they are now that we have a fresh opportunity to discuss: 1 - MATH and LLMs I am curious why many of the examples Andrej chose to pose to the LLM were "computational" questions -- for instance "what is 2+2" or s…

> I am curious to know more about the limitations / perils of using LLMs to train/evaluate other LLMs. At the extreme, there is this paper on ‘inbred LLMs’: https://www.nature.com/articles/s41586-024-07566-y

[deleted]

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#67

Post by avbanks »

p0w3n3d wrote: post #65

On 53 minutes from the original video, he shows how exact is the quotation of an LLM based on the text it was learning from. I wonder how did the bigtech convince the courts that this is not copyright violation (especially when ChatGPT was quoting some GPL code). I can imagine that the same thing would happen opposite, if I trained a model to draw a disney character, and my ass would be sued in a fraction of a second…

This is still being litigated I believe.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#68

Post by p0w3n3d »

albert_e wrote: post #12

OT: What is a good place to discuss the original video -- once it has dropped out of the HN front-page? I am going through the video myself -- roughly halfway through -- and have a fw things to bring up. Here they are now that we have a fresh opportunity to discuss: 1 - MATH and LLMs I am curious why many of the examples Andrej chose to pose to the LLM were "computational" questions -- for instance "what is 2+2" or s…

> I am curious to know more about the limitations / perils of using LLMs to train/evaluate other LLMs.

the entropy goes up in such case (up means less information). The result will be as if someone recompressed mpeg with another lossy compression. You can sometimes see the results on the internet.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#69

Post by sambull »

p0w3n3d wrote: post #65

On 53 minutes from the original video, he shows how exact is the quotation of an LLM based on the text it was learning from. I wonder how did the bigtech convince the courts that this is not copyright violation (especially when ChatGPT was quoting some GPL code). I can imagine that the same thing would happen opposite, if I trained a model to draw a disney character, and my ass would be sued in a fraction of a second…

It's a interesting question, one I wonder now that our federal data is being exfiltrated to AI companies. If they train their models on the data; how does the law tell them to 'unlearn'?

Destroying the copies they took will be what the courts ordered, but the data will still be there.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#70

Post by fenomas »

2-3-7-43-1807 wrote: post #52
thomasahle wrote: post #48

I find Meta’s approach to hallucinations delightfully counter intuitive. Basically they (and presumably OpenAI and others): - Extract a snippet of training data. - Generate a factual question about it using Llama 3. - Have Llama 3 generate an answer. - Score the response against the original data. - If incorrect, train the model to recognize and refuse incorrect responses. In a way this is obvious in hindsight, but i…

> In a way this is obvious in hindsight, but it goes against ML engineers natural tendency when detecting a wrong answer: Teaching the model the right answer. But the answer space for LLMs is infinite and unbounded. So, no effort will be complete and you will always end up with the question of how to deal with uncertainty. But I admit this is a bit of hindsight 20/20.

Karpathy's point in the video is that the models don't need to be exhaustively told what they don't know - they already have a good understanding of the extents of their knowledge. Older models just didn't use that understanding; they answered every question confidently because they'd only been trained on confident answers.
Post reply on HN