Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llm.c – LLM training in simple, pure C/CUDA

github.com

Post reply on HN
Print view View original

141–150 of 189 posts

Re: Llm.c – LLM training in simple, pure C/CUDA

#141

Post by MAMAMassakali »

karpathy wrote: post #90
blackeyeblitzar wrote: post #41

It would be great if someone created a tutorial around this explaining exactly how it works and how to do a test training run. I’m aware it’s not feasible to train a “real” model on personal hardware but it would be nice to have a practical learning experience. I’m not sure if there are good alternatives for that.

I wrote this, which might be a bit helpful: https://github.com/karpathy/llm.c/blob/master/doc/layernorm/... But if you don't have the background, I'd recommend my YouTube videos, see the Zero To Hero playlist: https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...

Thank you so much for the Zero To Hero playlist!

Re: Llm.c – LLM training in simple, pure C/CUDA

#142

Post by ItsBob »

LatticeAnimal wrote: post #54
qwertox wrote: post #48

> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch. It almost hurts, to read that PyTorch is faster. But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU. We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?

> We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it? I imagine it would incur a non trivial latency and cost penalty. The memory modules are placed pretty close to the compute die right now. Cooling would also have to change (the memory modules produce a lot of heat). But there is also no reason for any of the GP…

I don't disagree but (I know nothing about this btw...) would it not benefit in terms of, say, a L3 cache kind of thing?

Imagine you could stick 2 x 64GB DDR5 DIMMS on the GPU in sockets, would that not be faster to access than the motherboard DIMMS? It won't be as fast as on-die memory of course but could it not act like a sort of halfway house?

Re: Llm.c – LLM training in simple, pure C/CUDA

#144

Post by zzbn00 »

Very nice.

In my experience much of the complexity of numerical software is to enable the search for the algorithm that works well with the problem/data you have. Once you know the exact algorithm you want, it is possible to make a nice clean minimalistic implementation, but that does not mean such an implementation would have been easy at the beginning.

Re: Llm.c – LLM training in simple, pure C/CUDA

#145

Post by bootsmann »

milansuk wrote: post #143

This is an implementation of a transformer and in README it's presented as text->text. Tokens are just integers going in and out. Is it possible to use it to train other types of LLMs(text->image, image->text, speech->text, etc.)?

The transformer itself just takes arrays of numbers and turns them into arrays of numbers. What you are interested in is the process that happens before and after the transformer.

Re: Llm.c – LLM training in simple, pure C/CUDA

#147

Post by sirsinsalot »

Karpathy's code, teaching and contribution to the body of knowledge in this area really is admirable.

Sadly I am a generalist, but if I were a specialist, I would hope to contribute as openly and widely as Karpathy.

Not clout chasing, click-bait, "top 5 javascript frameworks of 2023!" ... just high quality output that marks a specialist.

Sorry to gush.

Re: Llm.c – LLM training in simple, pure C/CUDA

#148

Post by holoduke »

osigurdson wrote: post #106

Kind of amazing that something that can be expressed in ~1000 lines of code has completely turned the world on its head.

Speed of hardware did. Back in 80a they already knew the principles of llm training. It only took one week to train 10.000 tokens.

Re: Llm.c – LLM training in simple, pure C/CUDA

#149

Post by bootsmann »

antirez wrote: post #140

Is this able to replace PyTorch, ... in normal practice? No. Does this show that in general the most used ML frameworks are a mess? Yes.

This is a bit an apples and oranges comparison. Pytorch is a research framework not a transformer inference library.
Post reply on HN