TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
anfalmushtaq.com
TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
1–10 of 91 posts
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#2For a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it.
See The Open Source AI Definition from OSI: https://opensource.org/ai
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#3For a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. See The Open Source AI Definition from OSI: https://opensource.org/ai
Yes. I cannot comprehend this to this day. A model weights data + runner is how different from a closed source executable? Why do everyone call these open source?
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#4For a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. See The Open Source AI Definition from OSI: https://opensource.org/ai
Sadly, the OSAID also does not require training data to be available. :(
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#5For a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. See The Open Source AI Definition from OSI: https://opensource.org/ai
Yes. I cannot comprehend this to this day. A model weights data + runner is how different from a closed source executable? Why do everyone call these open source?
Because practically speaking, you can fine tune them I suppose?
But that's also true for binaries, games are a good example of where people pushed this quite far. Based on what little experience I have in ML, I'd say it's about the same thing. Whereas an API is more akin to a piece of software you can't tinker with in any way.
Guess the bar is just lower in the LLM space :P
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#6For a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. See The Open Source AI Definition from OSI: https://opensource.org/ai
Is it reasonable to expect companies to redistribute 100TB of copyrighted content they used for their LLM, just on the off-chance someone has a few million laying around and wants to reproduce the model from scratch?
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#7For a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. See The Open Source AI Definition from OSI: https://opensource.org/ai
Yes. I cannot comprehend this to this day. A model weights data + runner is how different from a closed source executable? Why do everyone call these open source?
Because typically adapting or improving traditional code to your needs is very difficult without access to the source code and build files.
For an LLM you can finetune and enhance, distill and embed given just the model weights, the runtime, and a permissive license. Having more is better. Well written detailed model release papers help a lot. Training code and training data are a great bonus.
However, I find the purity contest a bit too dismissive of the great contributions to the AI dev ecosystem that Meta and Deepseek have brought us. Without these, there wouldn't be the open ecosystem we have today.
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#8For a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. See The Open Source AI Definition from OSI: https://opensource.org/ai
Is it reasonable to expect companies to redistribute 100TB of copyrighted content they used for their LLM, just on the off-chance someone has a few million laying around and wants to reproduce the model from scratch?
The RL-only (no SFT) approaches might remove that issue. Problem sets should be smaller (and mechanically creatable) than the entire western corpus.
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#9For a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. See The Open Source AI Definition from OSI: https://opensource.org/ai
Yes. I cannot comprehend this to this day. A model weights data + runner is how different from a closed source executable? Why do everyone call these open source?
Open source becomes really complicated once assets with unclear license are involved in any way. Lots of people for example would say that Jedi Knight 2 is open source because Raven Software released the source code and tools needed to build the game. But that alone doesn't mean you can run it, because you still need to get a hold of all the assets (models, textures, sounds) which may or may not still be property of LucasArts or its successors. Even if you have them, it's actually unclear if it is legal to use them this way. So while there are tons of people working on mods and conversions, noone in their right mind would distribute all the source assets.
Much in the same way, no sane company will touch the legal nightmare of releasing LLM training data scraped from public websites. Even releasing the LLM alone might be infringement, there are literally court cases being fought over this right now.
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#10Great write up of what is presumably a truly great lecture. Debating trying to follow the original now.