OT but question from someone curious..... is Cuda still entrenched as the only option for doing AI or is there growing support for AMD/Intel/Other ways of doing AI?
There are some stirrings but don't hold your breath
Llm.c – LLM training in simple, pure C/CUDA
31–40 of 189 posts
Re: Llm.c – LLM training in simple, pure C/CUDA
#32[flagged]
I see it as an excuse to talk about it more, and I'm happy to read it all
Re: Llm.c – LLM training in simple, pure C/CUDA
#33[flagged]
[deleted]
Re: Llm.c – LLM training in simple, pure C/CUDA
#34Earlier quoted context omitted.
Python has been popular for this because it’s convenient to quickly hack on and experiment with, not because it’s the most efficient thing.
The overhead really isn't that bad is it? Since the the python code is mostly about saying multiply matrix A with matrix B, and then that actual computation is done by optimized low level code.
For that stuff, yeah you're correct.
What I've seen is issues with the implementation of those libraries in a project.
I don't remember exactly, but I was playing with someone's wrapper for some kind of machine learning snake game and it was taking way longer than it should have on back of the napkin math.
The issue was using either a dict or a list in a hot loop and changing it to the other sped it up like 1000x.
So it's easy to think "yeah this library is optimized" but then you build something on top of it that is not obviously going to slow it down.
But, that's the Python tradeoff.
Re: Llm.c – LLM training in simple, pure C/CUDA
#35Very sad, shouldve used an agnostic framework instead of CUDA
Are there any strong LLMs trained without CUDA?
Yes, there are several. See this blog post from Databricks describing the landscape of LLMs trained on AMD hardware for example: https://www.databricks.com/blog/training-llms-scale-amd-mi25...
The most interesting one IMO is OLMo from AI2, which is truly open. You can read their blog post about it (https://blog.allenai.org/hello-olmo-a-truly-open-llm-43f7e73...) but basically it is open everything - they released everything you need to reproduce their weights (training data, training code, evaluation code, and weights) with a friendly (Apache) license.
Re: Llm.c – LLM training in simple, pure C/CUDA
#36Earlier quoted context omitted.
107MB of cPython defeated Go to try for self Step 1 download 2.4GB of CUDA
The size of CUDA really is astonishing. Any chance someone might figure out how to slim that down?
Talking directly to the kernel / driver / firmware.
As others have said, George Hotz is doing his best in reverse-engineering and skipping layers.
Re: Llm.c – LLM training in simple, pure C/CUDA
#37I've seen his nano GPT implemented using JAX, now we have C/CUDA. I'd love to see if nano GPT could be doable in Mojo. I took a stab at a Mojo conversion of his Wavenet project (Andrej's zero to hero course) and I gotta say... python has so many nice features lol. Stating the obvious I know but what you see done in 6 lines of python takes so much more work in other languages.
Re: Llm.c – LLM training in simple, pure C/CUDA
#38OT but question from someone curious..... is Cuda still entrenched as the only option for doing AI or is there growing support for AMD/Intel/Other ways of doing AI?
See my comment on this here: https://news.ycombinator.com/item?id=39973816
Re: Llm.c – LLM training in simple, pure C/CUDA
#39Very sad, shouldve used an agnostic framework instead of CUDA
[deleted]
Re: Llm.c – LLM training in simple, pure C/CUDA
#40Earlier quoted context omitted.
107MB of cPython defeated Go to try for self Step 1 download 2.4GB of CUDA
The size of CUDA really is astonishing. Any chance someone might figure out how to slim that down?
Taking a peek inside the package it seems to mostly be the libraries - CuFFT alone is about 350MB for example, twice over for the debug and release versions. I'm guessing those are probably fat binaries pre-compiled for every generation of Nvidia hardware rather than just the PTX bytecode, which would help to speed up fresh builds, at the expense of being huge.