Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llm.c – LLM training in simple, pure C/CUDA

github.com

Post reply on HN
Print view View original

51–60 of 189 posts

Re: Llm.c – LLM training in simple, pure C/CUDA

#51

Post by 0cf8612b2e1e »

bongodongobob wrote: post #34
im3w1l wrote: post #13

Earlier quoted context omitted.

The overhead really isn't that bad is it? Since the the python code is mostly about saying multiply matrix A with matrix B, and then that actual computation is done by optimized low level code.

For that stuff, yeah you're correct. What I've seen is issues with the implementation of those libraries in a project. I don't remember exactly, but I was playing with someone's wrapper for some kind of machine learning snake game and it was taking way longer than it should have on back of the napkin math. The issue was using either a dict or a list in a hot loop and changing it to the other sped it up like 1000x. So…

That sounds irrelevant to Python and just a matter of slow code cropping up in libraries until someone runs a profiler.

Re: Llm.c – LLM training in simple, pure C/CUDA

#52

Post by dnlzro »

geph2021 wrote: post #16
brcmthrowaway wrote: post #4

Very sad, shouldve used an agnostic framework instead of CUDA

As far as I can tell, its optional dependency is Open MP, not CUDA. Doesn't seem directly dependent on CUDA.

The plan is to eventually implement with CUDA:

"Currently, I am working on [...] direct CUDA implementation, which will be significantly faster and probably come close to PyTorch."

Re: Llm.c – LLM training in simple, pure C/CUDA

#53

Post by 0cf8612b2e1e »

patrick-fitz wrote: post #50

https://twitter.com/karpathy/status/1777427944971083809 > And once this is a in a bit more stable state: videos on building this in more detail and from scratch. Looking forward to watching the videos.

I love his videos. They are dense, but I get a lot out of them.

Re: Llm.c – LLM training in simple, pure C/CUDA

#54

Post by LatticeAnimal »

qwertox wrote: post #48

> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch. It almost hurts, to read that PyTorch is faster. But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU. We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?

> We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?

I imagine it would incur a non trivial latency and cost penalty. The memory modules are placed pretty close to the compute die right now. Cooling would also have to change (the memory modules produce a lot of heat).

But there is also no reason for any of the GPU manufacturers to do this. A skew with twice as much memory can go for a lot more than the difference in memory cost alone

Re: Llm.c – LLM training in simple, pure C/CUDA

#55

Post by dwroberts »

QuadmasterXLII wrote: post #10
tosh wrote: post #2

> LLM training in simple, pure C/CUDA. There is no need for 245MB of PyTorch or 107MB of cPython

107MB of cPython defeated Go to try for self Step 1 download 2.4GB of CUDA

A bunch of install methods for torch via pip include ~1.5GB of lib/ because of CUDA. libtorch_cuda.so is like 800MB on its own

Re: Llm.c – LLM training in simple, pure C/CUDA

#57

Post by fsloth »

QuadmasterXLII wrote: post #10
tosh wrote: post #2

> LLM training in simple, pure C/CUDA. There is no need for 245MB of PyTorch or 107MB of cPython

107MB of cPython defeated Go to try for self Step 1 download 2.4GB of CUDA

I don't think it's about the byte size, but the inherent complexity of the implementation. 1000 lines of C code is extremely simple by any standard. Whereas a sundry collection of Python and PyTorch libraries is anything but.

Re: Llm.c – LLM training in simple, pure C/CUDA

#58

Post by hcarvalhoalves »

bongodongobob wrote: post #34
im3w1l wrote: post #13

Earlier quoted context omitted.

The overhead really isn't that bad is it? Since the the python code is mostly about saying multiply matrix A with matrix B, and then that actual computation is done by optimized low level code.

For that stuff, yeah you're correct. What I've seen is issues with the implementation of those libraries in a project. I don't remember exactly, but I was playing with someone's wrapper for some kind of machine learning snake game and it was taking way longer than it should have on back of the napkin math. The issue was using either a dict or a list in a hot loop and changing it to the other sped it up like 1000x. So…

> The issue was using either a dict or a list in a hot loop and changing it to the other sped it up like 1000x.

The programmer using the wrong data structure is not a problem with the language.

Re: Llm.c – LLM training in simple, pure C/CUDA

#60

Post by jsheard »

qwertox wrote: post #48

> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch. It almost hurts, to read that PyTorch is faster. But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU. We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?

Memory speed is more or less directly proportional to how close the memory is to the processor, with the fastest memory being literally inside the processor (SRAM cache), followed by memory on the same package as the processor (HBM GPUs, Apple M-series), followed by soldered down discrete memory chips (regular GPUs, games consoles), followed by socketed DIMMs in distant last place. There's not really any getting around it, the bandwidth that GPUs crave just isn't compatible with modularity.

Even CPUs are starting to move their memory closer to the core in the name of performance, as mentioned Apple is already doing it, Intel is making Xeons with on-chip memory now, and they have a version aimed at consumers on their roadmap.

Post reply on HN