Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch

github.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch

#21

Post by vforno »

novaRom wrote: post #20
vforno wrote: post #19

Earlier quoted context omitted.

Hi, in nanoeuler I use cuBLAS (NVIDIA's super-optimized library) for all matrix multiplications, with the tensor cores in TF32 mode. It's the same thing PyTorch uses underneath, so it's very fast. What I've optimized (and will improve even more) and written by hand are the kernels for the other parts (like FlashAttention, which gave a nice 3x speedup), while I've delegated the large matrices to cuBLAS. Training the 1…

Do you have a guess why your code is so much slower than torch? I didn't look, but there must be no reason to have 2x slower code esp. for a simple grid of FMAs.

Yes, because it has many separate kernels instead of aggressive merges like PyTorch (with Torch Compile). Each pass (norm, matmul, residual, RoPE, etc.) launches its own kernel, which increases launch overhead and memory traffic. CuBLAS helps, but it's not enough to compensate.

Re: Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch

#22

Post by AndReics »

vforno wrote: post #19
AndReics wrote: post #18

Wow that's really cool i'll definitely check it out! have played around with machine learning algorithms built from scratch in c / cuda too, but once i hit the cuda part of it i kinda just left it to the side. i'm curious how did you use CUDA to optimize the matrix multiplications? how optimized is training, does it take much longer then using pytorch?

Hi, in nanoeuler I use cuBLAS (NVIDIA's super-optimized library) for all matrix multiplications, with the tensor cores in TF32 mode. It's the same thing PyTorch uses underneath, so it's very fast. What I've optimized (and will improve even more) and written by hand are the kernels for the other parts (like FlashAttention, which gave a nice 3x speedup), while I've delegated the large matrices to cuBLAS. Training the 1…

i see really cool, where i failed was trying to build my own matrix operations library, it was just too much, but using cuBLAS definitely helps, i'll look into the custom kernels you wrote they seem interesting!

did you build the backprop yourself? it is a really cool project to build and i think you can agree that it teaches you a lot of how LLMS and machine learning in general works.

Re: Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch

#23

Post by vforno »

AndReics wrote: post #22
vforno wrote: post #19

Earlier quoted context omitted.

Hi, in nanoeuler I use cuBLAS (NVIDIA's super-optimized library) for all matrix multiplications, with the tensor cores in TF32 mode. It's the same thing PyTorch uses underneath, so it's very fast. What I've optimized (and will improve even more) and written by hand are the kernels for the other parts (like FlashAttention, which gave a nice 3x speedup), while I've delegated the large matrices to cuBLAS. Training the 1…

i see really cool, where i failed was trying to build my own matrix operations library, it was just too much, but using cuBLAS definitely helps, i'll look into the custom kernels you wrote they seem interesting! did you build the backprop yourself? it is a really cool project to build and i think you can agree that it teaches you a lot of how LLMS and machine learning in general works.

Absolutely yes! With nanoeuler I learned so much by testing every little detail of the project. Every little part you see has been tested and proven several times so that it could be understood and worked.

Re: Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch

#24

Post by gaflo »

dang wrote: post #4
Chu4eeno wrote: post #2

Very weird coding style, did you run astyle --style=python on C code? Also, your LLM left a comment in the cuda source that it is untested, does the cuda stuff work?

> Very weird coding style, did you run astyle --style=python on C code? I'm sure you mean it in a more curious way but this type of comment on a Show HN often comes across as too harshy/snarky/dismissive for what we want here (see https://news.ycombinator.com/showhn.html ).

Consider adding a rule that an author must disclose (in their own words) for what parts and to what extent LLMs have been used to assist their project.

Re: Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch

#25

Post by dang »

gaflo wrote: post #24
dang wrote: post #4

Earlier quoted context omitted.

> Very weird coding style, did you run astyle --style=python on C code? I'm sure you mean it in a more curious way but this type of comment on a Show HN often comes across as too harshy/snarky/dismissive for what we want here (see https://news.ycombinator.com/showhn.html ).

Consider adding a rule that an author must disclose (in their own words) for what parts and to what extent LLMs have been used to assist their project.

I don't think it would help: most people wouldn't know about it, of those who did many wouldn't conform, and there would be no reliable way to enforce it.

Re: Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch

#28

Post by gaflo »

dang wrote: post #25
gaflo wrote: post #24

Earlier quoted context omitted.

Consider adding a rule that an author must disclose (in their own words) for what parts and to what extent LLMs have been used to assist their project.

I don't think it would help: most people wouldn't know about it, of those who did many wouldn't conform, and there would be no reliable way to enforce it.

You're probably right, I guess I'm too optimistic.

Re: Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch

#29

Post by dang »

gaflo wrote: post #28
dang wrote: post #25

Earlier quoted context omitted.

I don't think it would help: most people wouldn't know about it, of those who did many wouldn't conform, and there would be no reliable way to enforce it.

You're probably right, I guess I'm too optimistic.

Sorry for being a downer!
Post reply on HN