Quick question, is this just pure C code that can be loaded into an Nvidia gpu and run (via the python code)? I scanned the C and didn't see anything CUDA related (maybe I missed something, I'm not a GPU programmer!). K mentions something about a direct CUDA implementation coming soon, how would that be different than what this is?
Llm.c – LLM training in simple, pure C/CUDA
161–170 of 189 posts
Re: Llm.c – LLM training in simple, pure C/CUDA
#162Quick question, is this just pure C code that can be loaded into an Nvidia gpu and run (via the python code)? I scanned the C and didn't see anything CUDA related (maybe I missed something, I'm not a GPU programmer!). K mentions something about a direct CUDA implementation coming soon, how would that be different than what this is?
It’s not, if you look at his X account, he talks about his work adding on the CUDA parts
Re: Llm.c – LLM training in simple, pure C/CUDA
#163I'd like to think he took the name from my llm.f90 project https://github.com/rbitr/llm.f90 It was originally based off of Karpathy's llama2.c but I renamed it when I added support for other architectures. Probable a coincidence :)
I'm the creator behind https://github.com/nlpodyssey/rwkv.f90 . How about joining forces?
I'll send you an email
Re: Llm.c – LLM training in simple, pure C/CUDA
#164Earlier quoted context omitted.
Which important concept or algorithm can't be expressed in ≤1000 lines? Seems like a pretty common theme among groundbreaking ideas.
Most modern A/V codecs won't fit in that limit by several orders of magnitude. Even standard-compliant JPEG decoder would be hard to squeeze without some serious codegolfing. Discarding some barely used features gets you close to that limit, though [1]. Smallest popular TCP/IP stack [2] is ~20kLoC. [1] https://github.com/richgel999/picojpeg [2] https://savannah.nongnu.org/projects/lwip/
JPEGDEC seems to be about 500 lines: https://github.com/bitbank2/JPEGDEC
Re: Llm.c – LLM training in simple, pure C/CUDA
#165Earlier quoted context omitted.
How in Mojo do you support GPU data parallelism and all the benefits it brings ?
You don't. Mojo doesn't support GPUs at the moment, which says a lot about a language which claims to be AI first.
If you want CUDA up front go write PyTorch. No one is stopping you. Modular’s goal was to leverage MLIR first and bring GPUs in later. They’re barely a year old company.
Re: Llm.c – LLM training in simple, pure C/CUDA
#166Earlier quoted context omitted.
Candle focuses on inference though.
What is referecing ?
Inference means using the neural net, as opposed to training it.
During inference you feed an input into the NN and it passes through it in "forwards" direction (i.e. from input to output), being modified according to the "weights" that were learnt during training, to derive the output.
During training, each training sample is first fed forwards through the NN, the same way as for inference, but then the output of the model (which at the beginning of training will be random/wrong) is compared to the correct/desired output for that training sample, and a corresponding error value will then be fed backwards (from output to input) through the NN according to the "backpropagation" mechanism to update the weights.
Training is a lot more involved than inference since it involves this backpropagation step.
Re: Llm.c – LLM training in simple, pure C/CUDA
#167I'd like to think he took the name from my llm.f90 project https://github.com/rbitr/llm.f90 It was originally based off of Karpathy's llama2.c but I renamed it when I added support for other architectures. Probable a coincidence :)
In f90? That’s pretty cool.
On a related note, IMO it would be pretty cool if we could get an LLM implementation that provides and RCI interface like all the old computational codes used to.
Re: Llm.c – LLM training in simple, pure C/CUDA
#168Fantastic -- gotta love Andrej. I am sick of the ball and chain that is Python and all of its environment dependencies. It is nice to shed all the weight and get down to the metal.
Re: Llm.c – LLM training in simple, pure C/CUDA
#169Earlier quoted context omitted.
This is a bit an apples and oranges comparison. Pytorch is a research framework not a transformer inference library.
This post is about training not inference. And llama.cpp has similarly simple LoRa training code. There is nothing in neural networks themselves so complex to justify the amount of complexity the Python-ML community piled up. MLX, for instance, is a similarly general purpose research framework that is a fraction of the size.
Sure neural networks in of themselves are conceptually simple, and not difficult to code. Andrew Ng's original Coursera class is all you need to go from zero knowledge to building MATLAB based neural nets in this same hard coded style.
However, there is a huge difference in functionality (hence complexity) in a framework such as PyTorch vs hardcoding a single NN. It's a bit like the difference between writing a toy compiler in CompSci class vs a production one that supports optimization, multiple targets, etc, etc.
The first step in convenience beyond hardcoding models, was frameworks like the original Torch, and original TensorFlow. Those frameworks let you explicitly assemble a neural net out of modular "lego blocks" (tensor operations), then just call model.forward() or model.backward() - no need to yourself write the forwards and backwards functions.
What PyTorch (successor to Torch) did was increase the complexity of the framework, but bring massive ease-of-use to the developer, by getting rid of the explicit lego-block assembly process, and instead let the developer just write arbitrary Python code corresponding to what they want the model to do, and then PyTorch itself build the model internally and therefore is able to infer the backward function. This extra functionality/ease-of-use, but with corresponding internal complexity, is what differentiated PyTorch from TensorFlow, made it so succesful, and caused most developers to switch to it.
There is also a lot of other functionality in PyTorch that adds to the complexity - supporting multiple back ends, custom CUDA/etc kernels beyond what is provided by cuDNN, etc, etc.
Re: Llm.c – LLM training in simple, pure C/CUDA
#170OT but question from someone curious..... is Cuda still entrenched as the only option for doing AI or is there growing support for AMD/Intel/Other ways of doing AI?
George Hotz is attempting to solve this: https://github.com/tinygrad/tinygrad
I'm not sure if he's "attempting to solve it" so much as he's looking for yet another way to keep himself famous.
The guy did one good jailbreak for the iPhone, and as near as I can tell, the rest of his work has been a lot of boasting, half-assed hyped-up implementations (e.g: his self-driving car), and trying to befriend other powerful people in tech (see: his promise to single-handedly fix Musk's Twitter). He might be a smart dude, but he vastly overrates his own accomplishments, and doesn't finish near anything he starts.