Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llm.c – LLM training in simple, pure C/CUDA

github.com

Post reply on HN
Print view View original

151–160 of 189 posts

Re: Llm.c – LLM training in simple, pure C/CUDA

#151

Post by antirez »

bootsmann wrote: post #149
antirez wrote: post #140

Is this able to replace PyTorch, ... in normal practice? No. Does this show that in general the most used ML frameworks are a mess? Yes.

This is a bit an apples and oranges comparison. Pytorch is a research framework not a transformer inference library.

This post is about training not inference. And llama.cpp has similarly simple LoRa training code. There is nothing in neural networks themselves so complex to justify the amount of complexity the Python-ML community piled up. MLX, for instance, is a similarly general purpose research framework that is a fraction of the size.

Re: Llm.c – LLM training in simple, pure C/CUDA

#154

Post by flohofwoe »

davedx wrote: post #146

On one hand, really nice to see the whole thing in 1000 lines of C code. On the other hand, that malloc function low key terrifies me. :)

Better to be explicit than hiding unsafe memory accesses under C++ stdlib classes like std::vector which don't do range checking either in operator[]. And in this sort of code, automatically injected runtime range checks would most likely hurt performance enough to matter.

I would still run the code through the Clang static analyzer and a couple of test runs in ASAN and UBSAN to be sure that nothing slipped through.

Re: Llm.c – LLM training in simple, pure C/CUDA

#155

Post by flohofwoe »

rurban wrote: post #116

https://github.com/robjinman/Richard uses Vulkan, thus is portable across GPU's and much faster. It also has more kernels. In simple C++

Or rather GLSL... The C++ code looks like it's mostly just scaffolding to kick off the actually important GPU work, and for that it's a surprising amount of code. Quite typical both for Vulkan and C++ though ;)

Re: Llm.c – LLM training in simple, pure C/CUDA

#156

Post by magnat »

KeplerBoy wrote: post #125
osigurdson wrote: post #106

Kind of amazing that something that can be expressed in ~1000 lines of code has completely turned the world on its head.

Which important concept or algorithm can't be expressed in ≤1000 lines? Seems like a pretty common theme among groundbreaking ideas.

Most modern A/V codecs won't fit in that limit by several orders of magnitude.

Even standard-compliant JPEG decoder would be hard to squeeze without some serious codegolfing. Discarding some barely used features gets you close to that limit, though [1].

Smallest popular TCP/IP stack [2] is ~20kLoC.

[1] https://github.com/richgel999/picojpeg

[2] https://savannah.nongnu.org/projects/lwip/

Re: Llm.c – LLM training in simple, pure C/CUDA

#157

Post by einsteinx2 »

airspresso wrote: post #128
schneehertz wrote: post #96

Earlier quoted context omitted.

Graphics card manufacturers believe that selling high-memory consumer graphics cards will affect the market for commercial computing cards, so they will not do so, that's all.

Nice room for a new player to disrupt then

Problem is, making a board design using an existing GPU chip and sticking more RAM into it is (relatively) simple but of course none of the GPU chip makers would allow partners to do that. Making your own GPU chip that’s competitive with Nvidia or AMD’s current offerings is a massive undertaking and pretty much impossible for a newcomer.

Just look at how much trouble Intel has had breaking into the discrete GPU market or even just how hard it’s been for AMD to compete with Nvidia even with decades of experience in the market.

And if some newcomer could make a competitive GPU with large memory capacity they’d be crazy not to sell it at datacenter prices, maybe just undercutting the others but a few grand but still way more expensive than any consumer GPU you can buy today, even a 4090.

Re: Llm.c – LLM training in simple, pure C/CUDA

#158

Post by nextaccountic »

Y_Y wrote: post #130
KeplerBoy wrote: post #125

Earlier quoted context omitted.

Which important concept or algorithm can't be expressed in ≤1000 lines? Seems like a pretty common theme among groundbreaking ideas.

That's a good question. Unfortunately I think you're asking to compute the Kolmogorov complexity of every interesting concept we have that doesn't yet have an implementation less than n=1000 lines, which is equivalent to the halting problem (modulo unbounded memory). If you could exhaustively list all the interesting algorithms (hard but feasible) you could potentially prove a lower bound for each one's complexity by…

You are right but I think that there's a more interesting question: do humans stumble upon those large interesting/great algorithms in practice?

The key point here is that we are looking at algorithms already discovered in human history rather than enumerating all possible interesting algorithms. Of course there is an interesting algorithm that is very large, but humans don't discover them in practice. If you look up a list of greatest algorithms in history, they will be rather small in length. Many of them can be sketched in a whiteboard

I think that what is happening here is that our minds just can't hold billions of concepts at once. So if you have an algorithm with billions of things, it was most likely produced by a machine. Handcrafted things, on the other hand, are smaller in comparison

Another thing is that our minds like conceptual simplicity and view simplicity as a kind of beauty. So if we have a great algorithm but it is too large, we look for ways to express them in succinct ways (the right abstractions can help with that, and also help with understanding the algorithm better). We end up succeeding because the algorithms themselves had low Kolmogorov complexity (and thus, if they are too large they probably can be further compressed)

Re: Llm.c – LLM training in simple, pure C/CUDA

#159

Post by HarHarVeryFunny »

antirez wrote: post #140

Is this able to replace PyTorch, ... in normal practice? No. Does this show that in general the most used ML frameworks are a mess? Yes.

> Does this show that in general the most used ML frameworks are a mess? Yes.

Not really ... there is little to no overlap with what a framework like PyTorch does. There is no tensor class, no autograd, etc. Just malloc, a bunch of hand calculated pointers into that chunk of memory, and hand written gradient functions. I assume the intent here is to be educational by stripping away the layers of abstraction to make it clearer what is going on.

Frankly though, this code (all that pointer math!) is a mess too, maybe written this way to make it easy to port to cuDNN which is at a similarly low level (other than having tensor descriptors which make the memory layout more flexible).

If you want to write your own tensor class and reusable NN framework, then the lines of code go up very rapidly. I did one in C++ a while back, and the tensor class alone was 20K LOC.

Re: Llm.c – LLM training in simple, pure C/CUDA

#160

Post by epr »

magnat wrote: post #156
KeplerBoy wrote: post #125

Earlier quoted context omitted.

Which important concept or algorithm can't be expressed in ≤1000 lines? Seems like a pretty common theme among groundbreaking ideas.

Most modern A/V codecs won't fit in that limit by several orders of magnitude. Even standard-compliant JPEG decoder would be hard to squeeze without some serious codegolfing. Discarding some barely used features gets you close to that limit, though [1]. Smallest popular TCP/IP stack [2] is ~20kLoC. [1] https://github.com/richgel999/picojpeg [2] https://savannah.nongnu.org/projects/lwip/

A JPEG decoder or TCP stack are very clearly not individual concepts though. There's obviously some subjectivity as to what constitutes a single "concept" or "algorithm", but I'm not sure either of those two examples are in a gray area.

A single concept might be implementing just ARP or a discrete cosine transform. If you wanted to do a full TCP stack or JPEG decoder, that would make a lot more sense after building their internal components one by one.

Post reply on HN