Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llm.c – LLM training in simple, pure C/CUDA

github.com

Post reply on HN
Print view View original

131–140 of 189 posts

Re: Llm.c – LLM training in simple, pure C/CUDA

#131

Post by vineyardmike »

idkwhatimdoin wrote: post #120
vineyardmike wrote: post #88

Earlier quoted context omitted.

Terse variables are a C thing. “What resources would I need” -> you’re literally commenting on a teachers content. Karpathy (the author) has a very informative YouTube channel where he goes step by step through everything. He has a ton of repos and tutorials. Dig a little. If all else fails… Google it.

> you’re literally commenting on a teachers content. How am I supposed to know that? > Karpathy (the author) has a very informative YouTube channel where he goes step by step through everything. Or that, without knowing that he's a teacher? > Terse variables are a C thing. I didn't realize variables had to be so short in C. Glad I write C++ professionally where they've added support for longer variable names. > If al…

> How am I supposed to know that?

You’re not supposed to know that. You asked a question, and this is you being told the answer.

It’s very convenient that the author of the post is quite literally the world’s most prolific teacher on this topic. Makes it easy to find Karpathy. You shouldn’t be expected to otherwise know that (or else why ask if you knew).

> I didn't realize variables had to be so short in C. Glad I write C++ professionally where they've added support for longer variable names.

This feels like a joke but old C compilers did have variable length limits. This is part of why C historically had shorter variables than other more modern languages.

Sorry if it came off rude, the internet is hard to communicate over.

https://publications.gbdirect.co.uk/c_book/chapter2/keywords...

Re: Llm.c – LLM training in simple, pure C/CUDA

#133

Post by [deleted] »

Y_Y wrote: post #130
KeplerBoy wrote: post #125

Earlier quoted context omitted.

Which important concept or algorithm can't be expressed in ≤1000 lines? Seems like a pretty common theme among groundbreaking ideas.

That's a good question. Unfortunately I think you're asking to compute the Kolmogorov complexity of every interesting concept we have that doesn't yet have an implementation less than n=1000 lines, which is equivalent to the halting problem (modulo unbounded memory). If you could exhaustively list all the interesting algorithms (hard but feasible) you could potentially prove a lower bound for each one's complexity by…

[deleted]

Re: Llm.c – LLM training in simple, pure C/CUDA

#138

Post by matteogrella »

andy99 wrote: post #69

I'd like to think he took the name from my llm.f90 project https://github.com/rbitr/llm.f90 It was originally based off of Karpathy's llama2.c but I renamed it when I added support for other architectures. Probable a coincidence :)

I'm the creator behind https://github.com/nlpodyssey/rwkv.f90. How about joining forces?

Re: Llm.c – LLM training in simple, pure C/CUDA

#139

Post by dsvf »

tverbeure wrote: post #78
jsheard wrote: post #60

Earlier quoted context omitted.

Memory speed is more or less directly proportional to how close the memory is to the processor, with the fastest memory being literally inside the processor (SRAM cache), followed by memory on the same package as the processor (HBM GPUs, Apple M-series), followed by soldered down discrete memory chips (regular GPUs, games consoles), followed by socketed DIMMs in distant last place. There's not really any getting arou…

For data rates, as in bandwidth per IO pin, distance is really only a secondary factor. HBM memory, for example, runs at substantially lower data rates than GDDR, yet it sits right next to the GPU die compared to centimeters for the GDDR. And high-speed serial links run at speeds that are an order of magnitude higher than even the internal register files of a CPU.

[dead]
Post reply on HN