Candle is a minimalist ML framework for Rust with a focus on performance (including GPU support) and ease of use https://github.com/huggingface/candle
Not barely as minimal as Karpathy his implementation
Llm.c – LLM training in simple, pure C/CUDA
121–130 of 189 posts
Re: Llm.c – LLM training in simple, pure C/CUDA
#122Kind of amazing that something that can be expressed in ~1000 lines of code has completely turned the world on its head.
Echoes of DeCSS ;)
Re: Llm.c – LLM training in simple, pure C/CUDA
#123> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch. It almost hurts, to read that PyTorch is faster. But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU. We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?
NVIDIA hates that trick.
Re: Llm.c – LLM training in simple, pure C/CUDA
#124I've seen his nano GPT implemented using JAX, now we have C/CUDA. I'd love to see if nano GPT could be doable in Mojo. I took a stab at a Mojo conversion of his Wavenet project (Andrej's zero to hero course) and I gotta say... python has so many nice features lol. Stating the obvious I know but what you see done in 6 lines of python takes so much more work in other languages.
How in Mojo do you support GPU data parallelism and all the benefits it brings ?
You don't. Mojo doesn't support GPUs at the moment, which says a lot about a language which claims to be AI first.
Re: Llm.c – LLM training in simple, pure C/CUDA
#125Kind of amazing that something that can be expressed in ~1000 lines of code has completely turned the world on its head.
Which important concept or algorithm can't be expressed in ≤1000 lines? Seems like a pretty common theme among groundbreaking ideas.
Re: Llm.c – LLM training in simple, pure C/CUDA
#126Candle is a minimalist ML framework for Rust with a focus on performance (including GPU support) and ease of use https://github.com/huggingface/candle
Candle focuses on inference though.
Candle dev here, we also support training/backdrop! We certainly focus on optimizing inference performance but hopefully that should improve the training efficiency too.
Re: Llm.c – LLM training in simple, pure C/CUDA
#127Earlier quoted context omitted.
How in Mojo do you support GPU data parallelism and all the benefits it brings ?
You don't. Mojo doesn't support GPUs at the moment, which says a lot about a language which claims to be AI first.
They only made Mojo available outside the preview circle about a couple of months ago, and it is yet to run on Windows laptops of researchers.
I love the attitude of considering 0.x languages production ready for all imaginable kinds of workloads.
Re: Llm.c – LLM training in simple, pure C/CUDA
#128Earlier quoted context omitted.
That's true it's got an impact, but I think there's still space available for "slightly slower with 2x memory" models. For many local uses, new cards are way past the "fast enough" line, but having 64gb on them would be really beneficial. It's love to see some experiments / different SKUs in this area, given people are already diy-ing extra memory on NVIDIA. ( https://hackaday.com/2021/01/29/add-an-extra-8gb-of-vram-…
Graphics card manufacturers believe that selling high-memory consumer graphics cards will affect the market for commercial computing cards, so they will not do so, that's all.
Nice room for a new player to disrupt then
Re: Llm.c – LLM training in simple, pure C/CUDA
#129Candle is a minimalist ML framework for Rust with a focus on performance (including GPU support) and ease of use https://github.com/huggingface/candle
Love Candle! I actually ported Karpathy's previous GPT tutorial to candle, including training [0]
[0] https://www.perceptivebits.com/building-gpt-from-scratch-in-...
Re: Llm.c – LLM training in simple, pure C/CUDA
#130Kind of amazing that something that can be expressed in ~1000 lines of code has completely turned the world on its head.
Which important concept or algorithm can't be expressed in ≤1000 lines? Seems like a pretty common theme among groundbreaking ideas.
That's a good question. Unfortunately I think you're asking to compute the Kolmogorov complexity of every interesting concept we have that doesn't yet have an implementation less than n=1000 lines, which is equivalent to the halting problem (modulo unbounded memory).
If you could exhaustively list all the interesting algorithms (hard but feasible) you could potentially prove a lower bound for each one's complexity by writing a shorter than n implementation (hard, probably infeasiblel and show positively that GP's prop isn't true. On the other hand showing that it was true would require either some very clever proof which can't apply to all programs, but somehow only these interesting ones (very likely impossible) or enumerate all C^n programs where C is the number of possible lines (something like 64^80) and show that none of them implements at least one of the interesting algorithms (absurdly impossible).