Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llm.c – LLM training in simple, pure C/CUDA

github.com

Post reply on HN
Print view View original

171–180 of 189 posts

Re: Llm.c – LLM training in simple, pure C/CUDA

#171

Post by frankjr »

naasking wrote: post #164
magnat wrote: post #156

Earlier quoted context omitted.

Most modern A/V codecs won't fit in that limit by several orders of magnitude. Even standard-compliant JPEG decoder would be hard to squeeze without some serious codegolfing. Discarding some barely used features gets you close to that limit, though [1]. Smallest popular TCP/IP stack [2] is ~20kLoC. [1] https://github.com/richgel999/picojpeg [2] https://savannah.nongnu.org/projects/lwip/

JPEGDEC seems to be about 500 lines: https://github.com/bitbank2/JPEGDEC

More like 5000 https://github.com/bitbank2/JPEGDEC/blob/master/src/jpeg.inl

Re: Llm.c – LLM training in simple, pure C/CUDA

#172

Post by _giorgio_ »

milansuk wrote: post #143

This is an implementation of a transformer and in README it's presented as text->text. Tokens are just integers going in and out. Is it possible to use it to train other types of LLMs(text->image, image->text, speech->text, etc.)?

Yes, anything can be an input token.

Patch of pixels ---> token Fragment of input Audio ---> token etc

Re: Llm.c – LLM training in simple, pure C/CUDA

#173

Post by tlb »

einsteinx2 wrote: post #157
airspresso wrote: post #128

Earlier quoted context omitted.

Nice room for a new player to disrupt then

Problem is, making a board design using an existing GPU chip and sticking more RAM into it is (relatively) simple but of course none of the GPU chip makers would allow partners to do that. Making your own GPU chip that’s competitive with Nvidia or AMD’s current offerings is a massive undertaking and pretty much impossible for a newcomer. Just look at how much trouble Intel has had breaking into the discrete GPU marke…

It's not doable at the board level. High-end GPUs use HBM in the same package, connected using a silicon interposer.

Re: Llm.c – LLM training in simple, pure C/CUDA

#174

Post by antirez »

HarHarVeryFunny wrote: post #169
antirez wrote: post #151

Earlier quoted context omitted.

This post is about training not inference. And llama.cpp has similarly simple LoRa training code. There is nothing in neural networks themselves so complex to justify the amount of complexity the Python-ML community piled up. MLX, for instance, is a similarly general purpose research framework that is a fraction of the size.

Sure neural networks in of themselves are conceptually simple, and not difficult to code. Andrew Ng's original Coursera class is all you need to go from zero knowledge to building MATLAB based neural nets in this same hard coded style. However, there is a huge difference in functionality (hence complexity) in a framework such as PyTorch vs hardcoding a single NN. It's a bit like the difference between writing a toy c…

I know all this things. Again: look at MLX.

Re: Llm.c – LLM training in simple, pure C/CUDA

#175

Post by toxik »

waynecochran wrote: post #168

Fantastic -- gotta love Andrej. I am sick of the ball and chain that is Python and all of its environment dependencies. It is nice to shed all the weight and get down to the metal.

Yeah, as long as you don’t want to change the network architecture.

Edit: and you trust that Andrei didn’t screw up anywhere while hand rolling all the gradient calculations.

Re: Llm.c – LLM training in simple, pure C/CUDA

#176

Post by toxik »

holoduke wrote: post #148
osigurdson wrote: post #106

Kind of amazing that something that can be expressed in ~1000 lines of code has completely turned the world on its head.

Speed of hardware did. Back in 80a they already knew the principles of llm training. It only took one week to train 10.000 tokens.

Got a reference on that claim?

Re: Llm.c – LLM training in simple, pure C/CUDA

#177

Post by EricLeer »

flockonus wrote: post #68

Question, apologize if slightly off-topic, it's something I'd like to use this project for: Is there an example of how to train GPT-2 on time series, in particular with covariates? As my understanding of LLM goes at a basic level it's predicting the next token from previous tokens, which sounds directionally similar to time series (perhaps letting aside periodicity).

Yes there are many attempts in applying a transformers to timeseries forecasting. For instance (but there are many more): - Timegpt https://arxiv.org/abs/2310.03589 - Chronos https://github.com/amazon-science/chronos-forecasting

These kind of papers often talk the world, but often lack a proper baseline model. They only compare against very simple (naive forecast), or non tuned models. In my experience a gradient boosting model will probably solve 95% of your forecasting problems, and trying to get fancy with a transformer (or even just a simple neural net) is more trouble then it is worth.

Re: Llm.c – LLM training in simple, pure C/CUDA

#179

Post by osigurdson »

KeplerBoy wrote: post #125
osigurdson wrote: post #106

Kind of amazing that something that can be expressed in ~1000 lines of code has completely turned the world on its head.

Which important concept or algorithm can't be expressed in ≤1000 lines? Seems like a pretty common theme among groundbreaking ideas.

This gave me an interesting thought... How many algorithms can be expressed in 1000 lines (say 5 - 80 chars per line) of correct C code. Obviously the number is huge if just look at char combinations (62^80E3 maybe) but if reduced to compiling / correct and unique (i.e. not including the same algorithm with differing variable names for example) the number would obviously be much less. If in the range of Who knows, there might be a warp drive in there somewhere!

Re: Llm.c – LLM training in simple, pure C/CUDA

#180

Post by bootsmann »

antirez wrote: post #174
HarHarVeryFunny wrote: post #169

Earlier quoted context omitted.

Sure neural networks in of themselves are conceptually simple, and not difficult to code. Andrew Ng's original Coursera class is all you need to go from zero knowledge to building MATLAB based neural nets in this same hard coded style. However, there is a huge difference in functionality (hence complexity) in a framework such as PyTorch vs hardcoding a single NN. It's a bit like the difference between writing a toy c…

I know all this things. Again: look at MLX.

Show me the profiling in MLX
Post reply on HN