I'm still seeking an answer to what DeepSeek really is, especially in the context of their $5M versus ChatGPT's >$1B (source: internet). What did they do versus not do?
TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
21–30 of 91 posts
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#22Earlier quoted context omitted.
Yes. I cannot comprehend this to this day. A model weights data + runner is how different from a closed source executable? Why do everyone call these open source?
Open source becomes really complicated once assets with unclear license are involved in any way. Lots of people for example would say that Jedi Knight 2 is open source because Raven Software released the source code and tools needed to build the game. But that alone doesn't mean you can run it, because you still need to get a hold of all the assets (models, textures, sounds) which may or may not still be property of…
Games like that, or the open-source clones of commercial games that require original assets to play (e.g. OpenXCOM), actually give a very clear analogy here: open source does not mean open assets. The software code is under a separate license from the data it processes. Emulators like Dolphin are kind of in this situation too - the program is open, the data it processes is not.
And that's fine! It's still valuable to have access to the source code, even if the "batteries" aren't included. Of course, if you really want to call it an open source model you should include the source for the data scraping/cleaning stages too; then the only thing missing would be the compute time and risk of acquiring dubiously-legal inputs.
I personally prefer a taxonomy like:
* Open weights: you can download the artifact and run it locally, not just use it through an application like chatgpt or an API.
* Open source: the code that created the artifact is provided in the same format that the authors used to work on it.
* Open data: the dataset that the source code was used on is available for download.
All three of those could be individually licensed or released, for 8 possible combinations. In the analogy to games, they would correspond to the licenses on the retail binary, the source code of the game, and the original uncompressed art assets or Blender projects, respectively.
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#23Earlier quoted context omitted.
Yes. I cannot comprehend this to this day. A model weights data + runner is how different from a closed source executable? Why do everyone call these open source?
Because typically adapting or improving traditional code to your needs is very difficult without access to the source code and build files. For an LLM you can finetune and enhance, distill and embed given just the model weights, the runtime, and a permissive license. Having more is better. Well written detailed model release papers help a lot. Training code and training data are a great bonus. However, I find the pur…
It's not a purity contest, it's a clarity contest. If Meta and Deepseek want to operate the way they have been, where they release baked models and whitepapers, that's fine - and you're right, it's certainly more than they're obligated to release. They just shouldn't be calling it "open source" when the source is literally not open.
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#24OT: What is a good place to discuss the original video -- once it has dropped out of the HN front-page? I am going through the video myself -- roughly halfway through -- and have a fw things to bring up. Here they are now that we have a fresh opportunity to discuss: 1 - MATH and LLMs I am curious why many of the examples Andrej chose to pose to the LLM were "computational" questions -- for instance "what is 2+2" or s…
I believe Andrej Karpathy runs a discord, which is linked on his website [1]. I haven't participated personally, but from what I've seen, it's very active.
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#25[deleted]
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#26Earlier quoted context omitted.
Yes. I cannot comprehend this to this day. A model weights data + runner is how different from a closed source executable? Why do everyone call these open source?
Because practically speaking, you can fine tune them I suppose? But that's also true for binaries, games are a good example of where people pushed this quite far. Based on what little experience I have in ML, I'd say it's about the same thing. Whereas an API is more akin to a piece of software you can't tinker with in any way. Guess the bar is just lower in the LLM space :P
So a proprietary program with a lot of knobs and configuration files is kind of opensource?
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#27For a model to be ‘fully’ open source you need more than the model itself and a way to run it. You also need the data and the program that can be used to train it. See The Open Source AI Definition from OSI: https://opensource.org/ai
Is it reasonable to expect companies to redistribute 100TB of copyrighted content they used for their LLM, just on the off-chance someone has a few million laying around and wants to reproduce the model from scratch?
Perhaps that's not reasonable to expect, but Meta apparently kind of did it anyway, if not in a way that helps reproduce their LLM: https://arstechnica.com/tech-policy/2025/02/meta-torrented-o...
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#28i saw a good thread today: https://x.com/0xmetaschool/status/1888873661840634111
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#29I'm still seeking an answer to what DeepSeek really is , especially in the context of their $5M versus ChatGPT's >$1B (source: internet). What did they do versus not do?
This one's for you, from Diana Hu (GP @ YC)
Re: TL;DR of Deep Dive into LLMs Like ChatGPT by Andrej Karpathy
#30I'm still seeking an answer to what DeepSeek really is , especially in the context of their $5M versus ChatGPT's >$1B (source: internet). What did they do versus not do?
There's a great deal of coverage about DeepSeek on Zvi's newsletter here: https://thezvi.substack.com/archive?sort=new - the first post on R1 is from January 22