Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

anfalmushtaq.com

Post reply on HN
Print view View original

71–80 of 91 posts

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#71

Post by fenomas »

quantumspandex wrote: post #63
fenomas wrote: post #60

Earlier quoted context omitted.

As I understood that part, in RL for LLMs you take questions for which the model already sometimes emits correct answers, and then repeatedly infer while reinforcing the activations the model made during correct responses, which lets it evolve its own ways of more reliably reaching the right answer. (Hence the analogy to training AlphaGo, wherein you take a model that sometimes wins games, and then play a bunch of ga…

AlphaGo seems more like an automated process to me because you can start from nothing except the algorithm and the rules. Since a Go game only has 2 outcomes most of the time, and the model can play with itself, it is guaranteed to learn something during self-play. In the LLM case you have to have an already capable model to do RL. Also I feel like the problem selection part is important to make sure it's not too har…

Yes, IIUC those points are correct - you need relatively capable models, and well-crafted questions. The comparison with AlphaGo is that the processes are analogous, not identical - the key point being that in both cases the model is choosing its own path towards a goal, not just imitating the path that a human labeler took.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#73

Post by fenomas »

albert_e wrote: post #12

OT: What is a good place to discuss the original video -- once it has dropped out of the HN front-page? I am going through the video myself -- roughly halfway through -- and have a fw things to bring up. Here they are now that we have a fresh opportunity to discuss: 1 - MATH and LLMs I am curious why many of the examples Andrej chose to pose to the LLM were "computational" questions -- for instance "what is 2+2" or s…

For point 1, he gets into that more later in the video - e.g. specifically on counting, and about how/when to have models invoke tools instead of doing math themselves, etc.

Also for the second point, check later in the video when he talks about RL and (simulated) RLHF - he gets into the feedback loops of models training each other and the collapse that follows.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#74

Post by fhd2 »

darkwater wrote: post #26
fhd2 wrote: post #5

Earlier quoted context omitted.

Because practically speaking, you can fine tune them I suppose? But that's also true for binaries, games are a good example of where people pushed this quite far. Based on what little experience I have in ML, I'd say it's about the same thing. Whereas an API is more akin to a piece of software you can't tinker with in any way. Guess the bar is just lower in the LLM space :P

So a proprietary program with a lot of knobs and configuration files is kind of opensource?

By what appears to be the logic for "open source AI", a locally executable proprietary program would be "open source" (because you can meddle with the executable). To me, that's mostly just "not SaaS". But somehow, a different definition appears to have stuck for LLMs than for other types of software.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#75

Post by fenomas »

p0w3n3d wrote: post #65

On 53 minutes from the original video, he shows how exact is the quotation of an LLM based on the text it was learning from. I wonder how did the bigtech convince the courts that this is not copyright violation (especially when ChatGPT was quoting some GPL code). I can imagine that the same thing would happen opposite, if I trained a model to draw a disney character, and my ass would be sued in a fraction of a second…

Note that he's inferring from a base model there, which are fairly capable of regurgitating their (highly-weighted) inputs since they do nothing but predict pre-training tokens. For instruct services like ChatGPT, if they regurgitate something I'd think it would more likely be their fine-tuning data, which is usually owned by the provider (and also kept secret).

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#76

Post by 2-3-7-43-1807 »

FeepingCreature wrote: post #57
2-3-7-43-1807 wrote: post #53

Earlier quoted context omitted.

> for instance "what is 2+2" or some numerical puzzles that needed algebraic thinking there is only one algebraic approach to solving something like 2+2 and that is counting! 2+2 = (((0 + 1) + 1) + 1) + 1). but llms are infamously bad at counting. which is why 2+2 isn't an algebraic problem to an llm. it's pattern matching or linguistic reasoning token by token.

LLMs are bad at counting because nobody counts in text, we count in our heads which is not in the training material.

i don't think it's just about the training material. it's also about keeping track of the precise number of tokens. you'd have to have dedicated tokens for 1+1+1+1 and another one for 1+1+1+1+1 etc.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#77

Post by 2-3-7-43-1807 »

fenomas wrote: post #70
2-3-7-43-1807 wrote: post #52

Earlier quoted context omitted.

> In a way this is obvious in hindsight, but it goes against ML engineers natural tendency when detecting a wrong answer: Teaching the model the right answer. But the answer space for LLMs is infinite and unbounded. So, no effort will be complete and you will always end up with the question of how to deal with uncertainty. But I admit this is a bit of hindsight 20/20.

Karpathy's point in the video is that the models don't need to be exhaustively told what they don't know - they already have a good understanding of the extents of their knowledge. Older models just didn't use that understanding; they answered every question confidently because they'd only been trained on confident answers.

i don't think that's what he meant and also don't think that is accurate to say they already have an understanding. i'm not even basing my criticism on the anthropomorphization but on the fact that there will be activation constellation that correlate with uncertainty but you have to train them to channel this into an actual response expressing uncertainty ... only then it makes sense to speak of understanding uncertainty.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#78

Post by fenomas »

2-3-7-43-1807 wrote: post #77
fenomas wrote: post #70

Earlier quoted context omitted.

Karpathy's point in the video is that the models don't need to be exhaustively told what they don't know - they already have a good understanding of the extents of their knowledge. Older models just didn't use that understanding; they answered every question confidently because they'd only been trained on confident answers.

i don't think that's what he meant and also don't think that is accurate to say they already have an understanding. i'm not even basing my criticism on the anthropomorphization but on the fact that there will be activation constellation that correlate with uncertainty but you have to train them to channel this into an actual response expressing uncertainty ... only then it makes sense to speak of understanding uncert…

Sorry, I don't follow what you're disagreeing with. I was summarizing what Karpathy talks about in the vicinity of 1:31:00 - where he talks (I assume notionally) about a specific neuron lighting lighting up to indicate uncertainty, and how empirically this turns out probably to be the case.

Edit: concretely, we can presume that OpenAI didn't specifically train ChatGPT to know that "Orson Kovacs" isn't a famous person, right? That's all I'm saying here - that they trained it how to say it doesn't know things, and it took care of the rest.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#79

Post by 2-3-7-43-1807 »

fenomas wrote: post #78
2-3-7-43-1807 wrote: post #77

Earlier quoted context omitted.

i don't think that's what he meant and also don't think that is accurate to say they already have an understanding. i'm not even basing my criticism on the anthropomorphization but on the fact that there will be activation constellation that correlate with uncertainty but you have to train them to channel this into an actual response expressing uncertainty ... only then it makes sense to speak of understanding uncert…

Sorry, I don't follow what you're disagreeing with. I was summarizing what Karpathy talks about in the vicinity of 1:31:00 - where he talks (I assume notionally) about a specific neuron lighting lighting up to indicate uncertainty, and how empirically this turns out probably to be the case. Edit: concretely, we can presume that OpenAI didn't specifically train ChatGPT to know that "Orson Kovacs" isn't a famous person…

i think i misinterpreted your first sentence.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#80

Post by fenomas »

2-3-7-43-1807 wrote: post #79
fenomas wrote: post #78

Earlier quoted context omitted.

Sorry, I don't follow what you're disagreeing with. I was summarizing what Karpathy talks about in the vicinity of 1:31:00 - where he talks (I assume notionally) about a specific neuron lighting lighting up to indicate uncertainty, and how empirically this turns out probably to be the case. Edit: concretely, we can presume that OpenAI didn't specifically train ChatGPT to know that "Orson Kovacs" isn't a famous person…

i think i misinterpreted your first sentence.

Hrrm, I'm reading back and I may have misinterpreted your first post too. If so apologies!
Post reply on HN