Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llm.c – LLM training in simple, pure C/CUDA

github.com

Post reply on HN
Print view View original

71–80 of 189 posts

Re: Llm.c – LLM training in simple, pure C/CUDA

#71

Post by wtallis »

jsheard wrote: post #60
qwertox wrote: post #48

> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch. It almost hurts, to read that PyTorch is faster. But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU. We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?

Memory speed is more or less directly proportional to how close the memory is to the processor, with the fastest memory being literally inside the processor (SRAM cache), followed by memory on the same package as the processor (HBM GPUs, Apple M-series), followed by soldered down discrete memory chips (regular GPUs, games consoles), followed by socketed DIMMs in distant last place. There's not really any getting arou…

FYI, most discrete GPUs with discrete memory packages soldered to the board near the GPU are running at substantially higher memory frequencies than the on-package DRAM in Apple's chips. But running GDDR at those speeds costs a lot of power.

Re: Llm.c – LLM training in simple, pure C/CUDA

#72

Post by kleiba »

eclectic29 wrote: post #19
kleiba wrote: post #7

Earlier quoted context omitted.

Better in which way?

Automatic resource management thanks to RAII. To me this is the most important thing.

But in this particular application, if you look e.g. through the training code, there's very little going on in terms of resource management. A handful of mallocs and some file handling.

That doesn't mean that you can have bugs even in a small number of instances, but if automatic resource management is your main argument, perhaps llm.c isn't the most prominent case that could benefit from it.

Re: Llm.c – LLM training in simple, pure C/CUDA

#73

Post by SunlitCat »

LatticeAnimal wrote: post #54
qwertox wrote: post #48

> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch. It almost hurts, to read that PyTorch is faster. But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU. We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?

> We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it? I imagine it would incur a non trivial latency and cost penalty. The memory modules are placed pretty close to the compute die right now. Cooling would also have to change (the memory modules produce a lot of heat). But there is also no reason for any of the GP…

And especially doing "interesting" combinations of gpu and memory.

Like lower end gpu with 16 GB of VRAM, but offering just 8 / 12 GB of VRAM in the middle class and then again 16 GB in the upper class of gpu selection.

Re: Llm.c – LLM training in simple, pure C/CUDA

#74

Post by llm_nerd »

im3w1l wrote: post #13
api wrote: post #3

Earlier quoted context omitted.

Python has been popular for this because it’s convenient to quickly hack on and experiment with, not because it’s the most efficient thing.

The overhead really isn't that bad is it? Since the the python code is mostly about saying multiply matrix A with matrix B, and then that actual computation is done by optimized low level code.

It depends on how you define overhead. Runtime overhead and memory usage is absolutely marginal, and the tightest, most perfect implementation will have trouble beating it.

Instead people are trying to optimize install size of dependencies, which while maybe a fun hacking project...who really cares?

Re: Llm.c – LLM training in simple, pure C/CUDA

#75

Post by taminka »

andrewstuart wrote: post #22

OT but question from someone curious..... is Cuda still entrenched as the only option for doing AI or is there growing support for AMD/Intel/Other ways of doing AI?

there are obv alternatives from both intel and amd, performant blas/dnn packages, but small teams don’t use them bc cuda is easier to use and has more support, and larger teams don’t use them bc they have deals w/ nvidia or not enough GPUs are available or they’re after the absolute best performance (which is still nvidia) or bc of other stuff like unstable drivers or smth

Re: Llm.c – LLM training in simple, pure C/CUDA

#76

Post by 0cf8612b2e1e »

CamperBob2 wrote: post #67
hcarvalhoalves wrote: post #58

Earlier quoted context omitted.

> The issue was using either a dict or a list in a hot loop and changing it to the other sped it up like 1000x. The programmer using the wrong data structure is not a problem with the language.

It really is with Python. There are simply too many containers and container-like concepts. Lists, arrays, sets, dicts...

What modern language doesn’t have those?

Go kind of cheats and has maps play double duty as sets.

Re: Llm.c – LLM training in simple, pure C/CUDA

#77

Post by tayo42 »

idkwhatimdoin wrote: post #64

If I was starting from scratch, what resources should I start with to build up an understanding of what this code does and how to read it? It's quite dense and my knowledge of LLMs is quite minimal. Are these terse variable names standard in LLM-land?

Check out his zero to hero series. Which builds this with python and later pytorch, then probably his other mini C based projects.

Re: Llm.c – LLM training in simple, pure C/CUDA

#78

Post by tverbeure »

jsheard wrote: post #60
qwertox wrote: post #48

> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch. It almost hurts, to read that PyTorch is faster. But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU. We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?

Memory speed is more or less directly proportional to how close the memory is to the processor, with the fastest memory being literally inside the processor (SRAM cache), followed by memory on the same package as the processor (HBM GPUs, Apple M-series), followed by soldered down discrete memory chips (regular GPUs, games consoles), followed by socketed DIMMs in distant last place. There's not really any getting arou…

For data rates, as in bandwidth per IO pin, distance is really only a secondary factor. HBM memory, for example, runs at substantially lower data rates than GDDR, yet it sits right next to the GPU die compared to centimeters for the GDDR. And high-speed serial links run at speeds that are an order of magnitude higher than even the internal register files of a CPU.

Re: Llm.c – LLM training in simple, pure C/CUDA

#79

Post by teruakohatu »

flockonus wrote: post #68

Question, apologize if slightly off-topic, it's something I'd like to use this project for: Is there an example of how to train GPT-2 on time series, in particular with covariates? As my understanding of LLM goes at a basic level it's predicting the next token from previous tokens, which sounds directionally similar to time series (perhaps letting aside periodicity).

Yes general LLM models can be used for time series forecasting:

https://github.com/KimMeen/Time-LLM

Re: Llm.c – LLM training in simple, pure C/CUDA

#80

Post by littlestymaar »

0cf8612b2e1e wrote: post #51
bongodongobob wrote: post #34

Earlier quoted context omitted.

For that stuff, yeah you're correct. What I've seen is issues with the implementation of those libraries in a project. I don't remember exactly, but I was playing with someone's wrapper for some kind of machine learning snake game and it was taking way longer than it should have on back of the napkin math. The issue was using either a dict or a list in a hot loop and changing it to the other sped it up like 1000x. So…

That sounds irrelevant to Python and just a matter of slow code cropping up in libraries until someone runs a profiler.

But then again if your program have places where choosing the right Python primitive is important for performance, then using python is affecting performance here since even the best algorithm in Python would be slower than the equivalent C.

Most of the time it doesn't matter because there's nothing hoy on the Python side, but if there is, then Python is going to be slowing your stuff down.

Post reply on HN