Live data from Hacker News

TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

anfalmushtaq.com

81–90 of 91 posts

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#81
post #47

Earlier quoted context omitted.

It's not a purity contest, it's a clarity contest. If Meta and Deepseek want to operate the way they have been, where they release baked models and whitepapers, that's fine - and you're right, it's certainly more than they're obligated to release. They just shouldn't be calling it "open source" when the source is literally not open.

Eh, I can kinda see it. It depends on your definitions of words. People have been muddying the waters with what "open source" means anyway. I have known it to mean code released under an open source license. Other people use it to mean programs where the source is available regardless of license. I would use "source available" to describe that, but some people strongly disagree with my definitions. If I write a progr…

> If I write a program, then obfuscate it and then release the obfuscated code under an open source license, would you consider it open source(I would)?

I wouldn't. Most definitions of open source say something like "in the form used for editing". You can release a built binary under an unrestrictive license, but that does not mean that you've opened the source. It's literally the plain meaning of the words: the source, as in where the thing comes from, needs to be open for it be meaningful.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#82
post #34

OT: What is a good place to discuss the original video -- once it has dropped out of the HN front-page? I am going through the video myself -- roughly halfway through -- and have a fw things to bring up. Here they are now that we have a fresh opportunity to discuss: 1 - MATH and LLMs I am curious why many of the examples Andrej chose to pose to the LLM were "computational" questions -- for instance "what is 2+2" or s…

Regarding 1 - MATH: Somewhere in the video he says that LLMs have expert (only slightly fuzzy) knowledge about a lot of topics, but fail with simple math questions. Many non-technical people anthropomorphize LLMs and don't know that they can't think or calculate like a real calculator. LLMs compute tokens and you can improve the performance, if you don't put too much computation into a single result token. I think it…

I've definitely done things like "give me a time stamp" then took too long to realize the time it gave made no sense. You get used to it working well when it does, and then it doesn't, and it's hard to switch the skepticism back on in response.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#83

Earlier quoted context omitted.

LLMs are bad at counting because nobody counts in text, we count in our heads which is not in the training material.

i don't think it's just about the training material. it's also about keeping track of the precise number of tokens. you'd have to have dedicated tokens for 1+1+1+1 and another one for 1+1+1+1+1 etc.

Internal representation is multidimensional vectors. A typical 4096 in q4 one can name every particle in the universe and have over 4000 dimensions left for other purposes

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#84
post #83

Earlier quoted context omitted.

i don't think it's just about the training material. it's also about keeping track of the precise number of tokens. you'd have to have dedicated tokens for 1+1+1+1 and another one for 1+1+1+1+1 etc.

Internal representation is multidimensional vectors. A typical 4096 in q4 one can name every particle in the universe and have over 4000 dimensions left for other purposes

i don't think that is a valid argument.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#85
post #75
post #65

On 53 minutes from the original video, he shows how exact is the quotation of an LLM based on the text it was learning from. I wonder how did the bigtech convince the courts that this is not copyright violation (especially when ChatGPT was quoting some GPL code). I can imagine that the same thing would happen opposite, if I trained a model to draw a disney character, and my ass would be sued in a fraction of a second…

Note that he's inferring from a base model there, which are fairly capable of regurgitating their (highly-weighted) inputs since they do nothing but predict pre-training tokens. For instruct services like ChatGPT, if they regurgitate something I'd think it would more likely be their fine-tuning data, which is usually owned by the provider (and also kept secret).

what I mean is if we can describe an LLM as a lossy compression (which are words spoken by Andrej), we could define what was done during inferring as uncompressing the compressed data, and at this moment shit would hit the fan.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#86
post #20

I have read many articles about LLMs, and understand how it works in general, but one thing always bothers me: why other models did't work as good as SOTA ones? What's the history and reason behind the current model architecture?

You mean the history of pre-transformer language models, and reason for the transformer architecture ?

Once upon a time ....

Language modelling in general grew out of attempts to build grammars for natural languages, which then gave rise to statistical approaches to modelling languages based on "n-gram" models (use last n words to predict next word). This was all before modern neural networks.

Language modelling (pattern recognition) is a natural fit for neural networks, and in particular recurrent neural networks (RNNs) seemed like a good fit because they have a feedback loop allowing an arbitrarily long preceding context (not just last n words) to be used predicting the next word. However, in practice RNNs didn't work very well since they tended to forget older context in favor of more recent words. To address this "forgetting" problem, LSTMs were designed, which are a variety of RNN that explicitly retain state and learn what to retain and what to forget, and using LSTMs for language models was common before transformers.

While LSTMs were better able to control what part of their history to retain and forget, the next shortcoming to be addressed was that in natural language the next word doesn't depend uniformly on what came before, and can be better predicted by paying more attention to certain words that are more important in the sentence structure (subjects, verbs, etc) than others. This was addressed by adding an attention mechanism ("Bahdanau attention") that learnt to weight preceding words by varying amounts when predicting the next word.

While attention was an improvement, a major remaining problem with LSTMs was that they are inefficient to train due to their recurrent/sequential nature, which is a poor match for today's highly parallel hardware (GPUs, etc). This inefficiency was the motivation for the modern transformer architecture, described in the "Attention is all you need" paper.

The insight that gave rise to the transformer was that the structure of language is really as much parallel as it is sequential, which you can visualize with linguist's sentence parse trees where each branch of the tree is largely independent of other branches at the same level. This structure suggests that language can be understood by a hierarchy (levels of branches) of parallel processing whereby small localized regions of the sentence are analyzed and aggregated into ever larger regions. Both within and across regions (branches), the successful attention mechanism can be used ("Attention is all you need").

However, the idea of hierarchical parallel processing + attention didn't immediately give rise to the transformer architecture ... The researcher who's idea this was (Jakob Uszkoreit) had initially implemented it using some architecture that I've never seen described, and had not been able to get predictive/modelling performance to beat the LSTM+attention approach that it was hoping to replace. At this point another researcher, Noam Shazeer (now back at Google and working on their Gemini model), got involved and worked his magic to turn the idea into a realization - the transformer architecture - whose language modelling performance was indeed an improvement. Actually, there seems to have been a bit of a "throw the kitchen sink" at it approach, as well as Shazeer's insight as to what would work, so there was then an ablation process to identify and strip away all unecessary parts of this new architecture to essentially give the transformer as we now know it.

So this is the history and reason/motivation behind the transformer architecture (the basis of all of today's LLMs), but the prediction performance and emergent intelligence of large models built using this architecture seems to have been quite a surprise. It's interesting to go back and read the early GPT-1, GPT-2 and GPT-3 papers (ChatGPT was intitally based on GPT-3.5) and see the increasing realization of how capable the architecture was.

I think there are a couple of major reasons why older architectures didn't work as well as the transformer.

1) The training efficiency of the transformer, it's primary motivation, has allowed it to be scaled up to enormous size, and a lot of the emergent behavior only becomes apparent at scale.

2) I think the details of the transformer architecture - interaction of key-based attention with hierarchical processing, etc, somewhat accidentally created an architecture capable of much more powerful learning than it's creators had anticipated. One of the most powerful mechanism in the way trained transformers operate is "induction heads" whereby the attention mechanism of two adjacent layers of the transformer learn to co-operate to implement a very powerful analogical copying operation that is the basis of much of what they do. These induction heads are an emergent mechanism - the result of training the transformer rather than something directly built into the architecture.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#87

Earlier quoted context omitted.

Open source becomes really complicated once assets with unclear license are involved in any way. Lots of people for example would say that Jedi Knight 2 is open source because Raven Software released the source code and tools needed to build the game. But that alone doesn't mean you can run it, because you still need to get a hold of all the assets (models, textures, sounds) which may or may not still be property of…

Games like that, or the open-source clones of commercial games that require original assets to play (e.g. OpenXCOM), actually give a very clear analogy here: open source does not mean open assets . The software code is under a separate license from the data it processes. Emulators like Dolphin are kind of in this situation too - the program is open, the data it processes is not. And that's fine! It's still valuable t…

If it has already been established that open source doesn't mean open assets, why would we change that now? After all, training data is literally nothing but assets - except that you don't need them to run the application. So in that sense open LLMs are more open than these games.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#88

Earlier quoted context omitted.

> for instance "what is 2+2" or some numerical puzzles that needed algebraic thinking there is only one algebraic approach to solving something like 2+2 and that is counting! 2+2 = (((0 + 1) + 1) + 1) + 1). but llms are infamously bad at counting. which is why 2+2 isn't an algebraic problem to an llm. it's pattern matching or linguistic reasoning token by token.

LLMs are bad at counting because nobody counts in text, we count in our heads which is not in the training material.

This result --

https://x.com/yuntiandeng/status/1889704768135905332

Is this a consequence of the fact that "multiplication tables" For kindergarteners are available online (in training data) abundantly ... typically up to 12 times or 13 times table as plain text ?

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#89

Earlier quoted context omitted.

Games like that, or the open-source clones of commercial games that require original assets to play (e.g. OpenXCOM), actually give a very clear analogy here: open source does not mean open assets . The software code is under a separate license from the data it processes. Emulators like Dolphin are kind of in this situation too - the program is open, the data it processes is not. And that's fine! It's still valuable t…

If it has already been established that open source doesn't mean open assets, why would we change that now? After all, training data is literally nothing but assets - except that you don't need them to run the application. So in that sense open LLMs are more open than these games.

But the training data isn't open...

I agree that open source doesn't mean open assets, but neither does open assets mean open source. You could make a linguistic argument that the training data is part of the "source" of the model (as in, from whence it came), but in any case the point is moot because neither the training data nor the code is open.

Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy

#90

I'm still seeking an answer to what DeepSeek really is , especially in the context of their $5M versus ChatGPT's >$1B (source: internet). What did they do versus not do?

Hey, Anfal here. I am actually the author of this article. I have had some really good discussion with a few really intelligent fellows I know and they tie very deeply into deepseek more. I'll create a post about it soon.
Post reply on HN