Llm.c – LLM training in simple, pure C/CUDA • Hacker News Board

Re: Llm.c – LLM training in simple, pure C/CUDA

#21

Post by dartos » Mon, Apr 08, 2024, 9:08 PM UTC

simonw wrote: post #18

QuadmasterXLII wrote: post #10

Earlier quoted context omitted.

107MB of cPython defeated Go to try for self Step 1 download 2.4GB of CUDA

The size of CUDA really is astonishing. Any chance someone might figure out how to slim that down?

Nvidia is the only one who could, since they own it.

Re: Llm.c – LLM training in simple, pure C/CUDA

#22

Post by andrewstuart » Mon, Apr 08, 2024, 9:08 PM UTC

OT but question from someone curious..... is Cuda still entrenched as the only option for doing AI or is there growing support for AMD/Intel/Other ways of doing AI?

Re: Llm.c – LLM training in simple, pure C/CUDA

#23

Post by triyambakam » Mon, Apr 08, 2024, 9:08 PM UTC

When Lex recently talked to Andre, Andre said that he gets positively obsessed with a problem and says "this must exist". I imagine this must be one of those outputs.

Re: Llm.c – LLM training in simple, pure C/CUDA

#24

Post by dartos » Mon, Apr 08, 2024, 9:09 PM UTC

eclectic29 wrote: post #5

[flagged]

Next level troll?

Re: Llm.c – LLM training in simple, pure C/CUDA

#25

Post by QuadrupleA » Mon, Apr 08, 2024, 9:09 PM UTC

eclectic29 wrote: post #5

[flagged]

RAII sucks the way C++ does it. Magical background BS with massive unintended complexity consequences requiring obtuse intricate crap like the copy-and-swap idiom, a mudball of pointer and reference types, etc. They should have added a defer/scope-exit statement and been done with it.

Re: Llm.c – LLM training in simple, pure C/CUDA

#26

Post by BraverHeart » Mon, Apr 08, 2024, 9:09 PM UTC

andrewstuart wrote: post #22

OT but question from someone curious..... is Cuda still entrenched as the only option for doing AI or is there growing support for AMD/Intel/Other ways of doing AI?

George Hotz is attempting to solve this: https://github.com/tinygrad/tinygrad

Re: Llm.c – LLM training in simple, pure C/CUDA

#27

Post by ipsum2 » Mon, Apr 08, 2024, 9:10 PM UTC

maxdoop wrote: post #8

[flagged]

Nothing wrong with critiques.

Re: Llm.c – LLM training in simple, pure C/CUDA

#28

Post by dvt » Mon, Apr 08, 2024, 9:12 PM UTC

homarp wrote: post #6

eclectic29 wrote: post #5

[flagged]

Rewrite in (safe) Rust, I am sure many will enjoy to compare how it runs.

Candle already exists[1], and it runs pretty well. Can use both CUDA and Metal backends (or just plain-old CPU).

[1] https://github.com/huggingface/candle

Re: Llm.c – LLM training in simple, pure C/CUDA

#29

Post by jiggawatts » Mon, Apr 08, 2024, 9:13 PM UTC

im3w1l wrote: post #13

api wrote: post #3

Earlier quoted context omitted.

Python has been popular for this because it’s convenient to quickly hack on and experiment with, not because it’s the most efficient thing.

The overhead really isn't that bad is it? Since the the python code is mostly about saying multiply matrix A with matrix B, and then that actual computation is done by optimized low level code.

I suspect that this has a high chance of running afoul of Ahmdal’s Law. Even if you can parallelise the bulk of the computation, the serial parts remain single-threaded and start to dominate the total runtime.

Re: Llm.c – LLM training in simple, pure C/CUDA

#30

Post by gpderetta » Mon, Apr 08, 2024, 9:13 PM UTC

geph2021 wrote: post #16

brcmthrowaway wrote: post #4

Very sad, shouldve used an agnostic framework instead of CUDA

As far as I can tell, its optional dependency is Open MP, not CUDA. Doesn't seem directly dependent on CUDA.

Yes, a quick skim of the code only shows openmp dependency. The C/CUDA reference might have meant to be C/OMP .

Although I wonder if it would work well with GCC PTX OMP offloading.