81–90 of 127 posts
Earlier quoted context omitted.
Ok, we've changed the URL to that from https://twitter.com/karpathy/status/1795484547267834137 above. Thanks!
Could you mention what the link has been changed from too? Sometimes it helps with context when reading the comments. Thanks!
Earlier quoted context omitted.
Ok, we've changed the URL to that from https://twitter.com/karpathy/status/1795484547267834137 above. Thanks!
sounds good. both work, (though) I think HN has a bit of an anti-twitter bias.
Earlier quoted context omitted.
First, love the videos and other work you've been doing. The micrograd videos are a great way to show people this is all math in the end, and I've linked to specific timestamps in that video and others more times than I can count. For why I think we have a anti-twitter bias... Twitter doesn't show replies or any further context without being logged in. Most people will have accounts but I know a lot here deleted thei…
fwiw I totally understand the sentiment! it's actually a bit sad to me that so much of our content is moving from the shared, open web to platforms like twitter, unfortunately there seems to be too much value add around built-in discoverability, comments, ease of authoring, for many people revenue sharing, etc.
Earlier quoted context omitted.
My understanding and suspicion is mostly less than you think. Llama 3 architecture has the following changes on GPT-2: 1. delete the absolute positional encoding and replace with RoPE 2. delete all biases in all layers (in LayerNorms, they turn into RMSNorm) 3. GeLU -> SwiGLU non-linearity in the MLP 4. longer context length 5. architecture hyperparameter changes, e.g. slightly different aspect ratios And there was a…
Note llama's feed forward is a bit different too: self.w2(F.silu(self.w1(x)) * self.w3(x)) I.e. the nonlinearity is a gate. https://github.com/meta-llama/llama3/blob/14aab0428d3ec3a959...
Earlier quoted context omitted.
It’s not about learning. It’s about owning. Exactly the reason OpenAI stopped being open. Having GPT-4-quality LLMs created by anyone with a gaming PC would be pretty radical.
And you won’t get there. Those models are far too large for a 2024 GPU. Llama-3 70b is arguably close to GPT-4 but is still too large for gaming GPUs (and probably for many years of GPU updates)
At some point they cross, and you will be able to run a GPT4-quality LLM on a consumer GPU. At some point after that, you’ll be able to run a GPT4-quality LLM on a 2024 consumer GPU if you can find one.
Important to emphasize, I’m not saying “GPT-4”. Llama-3 was trained on 24k GPU clusters. “Able to do the exact same processing at 1/24k the compute” is different from “Able to get equivalent performance at 1/24k compute”. Even then, given a long enough time scale, the former is possible.
Earlier quoted context omitted.
https://huggingface.co/datasets/HuggingFaceFW/fineweb has 15T cleaned and deduplicated english web data tokens.
Holy crap, Does huggingface charge for bandwidth if you're downloading 45 terabytes??
Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!
I just hope than in a couple of years we'll see a submission here titled "Reproduce GPT-4 on legacy RTX 4090." Because currently even with open source (?) models we are still consumers, and the training is still the domain of the rich.
Earlier quoted context omitted.
Last time I tried GPT-2 on CPU (which I think was shortly before chatGPT was launched), I was getting about 0.2 tokens/sec. CPU utilization was low though, so running inference in parralel gave better results. I was using 2 x E5-2660's.
DDR5 helps a lot. You can actually run stuff like LLaMA at >1 tok/s on the CPU with high-end gaming hardware these days.
Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!