Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llm.c – LLM training in simple, pure C/CUDA

github.com

Post reply on HN
Print view View original

91–100 of 189 posts

Re: Llm.c – LLM training in simple, pure C/CUDA

#91

Post by blackeyeblitzar »

karpathy wrote: post #90
blackeyeblitzar wrote: post #41

It would be great if someone created a tutorial around this explaining exactly how it works and how to do a test training run. I’m aware it’s not feasible to train a “real” model on personal hardware but it would be nice to have a practical learning experience. I’m not sure if there are good alternatives for that.

I wrote this, which might be a bit helpful: https://github.com/karpathy/llm.c/blob/master/doc/layernorm/... But if you don't have the background, I'd recommend my YouTube videos, see the Zero To Hero playlist: https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...

Thank you so much for responding. I will definitely check these out and also pass it on to others who might be interested.

Re: Llm.c – LLM training in simple, pure C/CUDA

#93

Post by viraptor »

jsheard wrote: post #60
qwertox wrote: post #48

> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch. It almost hurts, to read that PyTorch is faster. But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU. We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?

Memory speed is more or less directly proportional to how close the memory is to the processor, with the fastest memory being literally inside the processor (SRAM cache), followed by memory on the same package as the processor (HBM GPUs, Apple M-series), followed by soldered down discrete memory chips (regular GPUs, games consoles), followed by socketed DIMMs in distant last place. There's not really any getting arou…

That's true it's got an impact, but I think there's still space available for "slightly slower with 2x memory" models. For many local uses, new cards are way past the "fast enough" line, but having 64gb on them would be really beneficial.

It's love to see some experiments / different SKUs in this area, given people are already diy-ing extra memory on NVIDIA. (https://hackaday.com/2021/01/29/add-an-extra-8gb-of-vram-to-... there were stable experiments later on, but I don't have a link now)

Re: Llm.c – LLM training in simple, pure C/CUDA

#94

Post by viraptor »

vineyardmike wrote: post #88
idkwhatimdoin wrote: post #64

If I was starting from scratch, what resources should I start with to build up an understanding of what this code does and how to read it? It's quite dense and my knowledge of LLMs is quite minimal. Are these terse variable names standard in LLM-land?

Terse variables are a C thing. “What resources would I need” -> you’re literally commenting on a teachers content. Karpathy (the author) has a very informative YouTube channel where he goes step by step through everything. He has a ton of repos and tutorials. Dig a little. If all else fails… Google it.

> Terse variables are a C thing.

They're a math / toy code thing. Large C projects have long descriptive names just like other languages.

Re: Llm.c – LLM training in simple, pure C/CUDA

#96

Post by schneehertz »

viraptor wrote: post #93
jsheard wrote: post #60

Earlier quoted context omitted.

Memory speed is more or less directly proportional to how close the memory is to the processor, with the fastest memory being literally inside the processor (SRAM cache), followed by memory on the same package as the processor (HBM GPUs, Apple M-series), followed by soldered down discrete memory chips (regular GPUs, games consoles), followed by socketed DIMMs in distant last place. There's not really any getting arou…

That's true it's got an impact, but I think there's still space available for "slightly slower with 2x memory" models. For many local uses, new cards are way past the "fast enough" line, but having 64gb on them would be really beneficial. It's love to see some experiments / different SKUs in this area, given people are already diy-ing extra memory on NVIDIA. ( https://hackaday.com/2021/01/29/add-an-extra-8gb-of-vram-…

Graphics card manufacturers believe that selling high-memory consumer graphics cards will affect the market for commercial computing cards, so they will not do so, that's all.

Re: Llm.c – LLM training in simple, pure C/CUDA

#97

Post by satokema »

idkwhatimdoin wrote: post #64

If I was starting from scratch, what resources should I start with to build up an understanding of what this code does and how to read it? It's quite dense and my knowledge of LLMs is quite minimal. Are these terse variable names standard in LLM-land?

As siblings have said, his video series are quite good. But if you're just looking at this repo only, you probably want to look at the python reference implementation. (The C is designed to exactly replicate its functionality.)

Re: Llm.c – LLM training in simple, pure C/CUDA

#98

Post by towelpluswater »

andrewstuart wrote: post #22

OT but question from someone curious..... is Cuda still entrenched as the only option for doing AI or is there growing support for AMD/Intel/Other ways of doing AI?

Modular Mojo is the most well funded and full of respectable players for making an alternative possible

Re: Llm.c – LLM training in simple, pure C/CUDA

#100

Post by yinser »

cb321 wrote: post #81
yinser wrote: post #37

I've seen his nano GPT implemented using JAX, now we have C/CUDA. I'd love to see if nano GPT could be doable in Mojo. I took a stab at a Mojo conversion of his Wavenet project (Andrej's zero to hero course) and I gotta say... python has so many nice features lol. Stating the obvious I know but what you see done in 6 lines of python takes so much more work in other languages.

For a prior generation of karpathy-splaining this is this Nim port: https://github.com/Vindaar/llama2nim - maybe of interest if you are interested in Mojo.

Thank you!
Post reply on HN