See, C does it very well.
Great stuff. Karpathy has a gift for teaching.
Llm.c – LLM training in simple, pure C/CUDA
101–110 of 189 posts
Re: Llm.c – LLM training in simple, pure C/CUDA
#102Earlier quoted context omitted.
Memory speed is more or less directly proportional to how close the memory is to the processor, with the fastest memory being literally inside the processor (SRAM cache), followed by memory on the same package as the processor (HBM GPUs, Apple M-series), followed by soldered down discrete memory chips (regular GPUs, games consoles), followed by socketed DIMMs in distant last place. There's not really any getting arou…
FYI, most discrete GPUs with discrete memory packages soldered to the board near the GPU are running at substantially higher memory frequencies than the on-package DRAM in Apple's chips. But running GDDR at those speeds costs a lot of power.
I watched a presentation on this today. The presenter focused on the soldering and proximity as well. Is this really the only difference or is this transistor based memory (like L1, L2, etc.)? I get the proximity factor of course (1ft / ns EE rule of thumb). In any case, soldering and proximity don't seem like breakthrough innovations (but maybe I am wrong).
Re: Llm.c – LLM training in simple, pure C/CUDA
#103Earlier quoted context omitted.
George Hotz is attempting to solve this: https://github.com/tinygrad/tinygrad
He loudly gave up on AMD after they did not fix a blocker he had for 5+ months and gave him the runaround the entire time when he asked for the code to fix it himself. He is still shipping the AMD tinybox with huge warning labels.
Didn't they recently announce that everything was open sourced? Would be cool if he took another look at it once all of the souce code is available (if not already).
Re: Llm.c – LLM training in simple, pure C/CUDA
#104> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch. It almost hurts, to read that PyTorch is faster. But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU. We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?
[dead]
Re: Llm.c – LLM training in simple, pure C/CUDA
#105Earlier quoted context omitted.
FYI, most discrete GPUs with discrete memory packages soldered to the board near the GPU are running at substantially higher memory frequencies than the on-package DRAM in Apple's chips. But running GDDR at those speeds costs a lot of power.
I watched a presentation on this today. The presenter focused on the soldering and proximity as well. Is this really the only difference or is this transistor based memory (like L1, L2, etc.)? I get the proximity factor of course (1ft / ns EE rule of thumb). In any case, soldering and proximity don't seem like breakthrough innovations (but maybe I am wrong).
Gpu ram is typically gddr6 or gddr6x which is a different standard to the chips used in ddr5 for example. GPUs have terrible latency to ram, but enormous throughput, and I assume the chips are internally optimized for that. Many aspects of a design change when you choose different latency or clockspeed targets translating into different power / area calculations.
Re: Llm.c – LLM training in simple, pure C/CUDA
#106Kind of amazing that something that can be expressed in ~1000 lines of code has completely turned the world on its head.
Re: Llm.c – LLM training in simple, pure C/CUDA
#107I've seen his nano GPT implemented using JAX, now we have C/CUDA. I'd love to see if nano GPT could be doable in Mojo. I took a stab at a Mojo conversion of his Wavenet project (Andrej's zero to hero course) and I gotta say... python has so many nice features lol. Stating the obvious I know but what you see done in 6 lines of python takes so much more work in other languages.
How in Mojo do you support GPU data parallelism and all the benefits it brings ?
Re: Llm.c – LLM training in simple, pure C/CUDA
#108Wow, and this is done after a recent trip to Bhutan to clear his head! I follow karpathy on twitter and he posted that 2 weeks without constantly looking and checking his phone kind of turns off the constantly on radio in his head.
Re: Llm.c – LLM training in simple, pure C/CUDA
#109OT but question from someone curious..... is Cuda still entrenched as the only option for doing AI or is there growing support for AMD/Intel/Other ways of doing AI?
Modular Mojo is the most well funded and full of respectable players for making an alternative possible
Check out Hidet [1]. Not as well funded, but delivers Python based ML acceleration with GPU support (unlike Mojo).
Re: Llm.c – LLM training in simple, pure C/CUDA
#110It should be rewritten in Rust. (Just joking)
I think you just cracked AI safety.