Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Luminal – Open-source, search-based GPU compiler

github.com

Post reply on HN
Print view View original

41–50 of 70 posts

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#41

Post by jafioti »

Alifatisk wrote: post #11

So wait, am I understanding this correctly? Instead of applying just predetermined optimization rules or patterns, the compiler formulates the problem as searching through many possible configurations or versions of the code. Each possible version can have different arrangements, tiling sizes, thread block configurations, memory access patterns, and instruction sequences, right? And from my understanding, the “search…

yup! we build a search space by iteratively applying rewrite rules in every possible order (using e-graphs to do this efficiently). the rewrites alter stuff like looping / tiling structures, as well as algebraic rewrites like softmax to online softmax (and then flash attention).

yes optimized kernels for one system will work on other systems with the same hardware. its fine to take a long time compiling if you just compile once and run a lot.

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#42

Post by cedws »

Around the time DeepSeek R2 released there was chatter about how DeepSeek had had an “undocumented” PTX instruction to squeeze as much performance as possible from their hardware. My understanding is that it wasn’t any kind of secret instruction but just a novel way that they put the instruction together.

Would Luminal be capable of rediscovering this trick?

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#43

Post by jafioti »

cedws wrote: post #42

Around the time DeepSeek R2 released there was chatter about how DeepSeek had had an “undocumented” PTX instruction to squeeze as much performance as possible from their hardware. My understanding is that it wasn’t any kind of secret instruction but just a novel way that they put the instruction together. Would Luminal be capable of rediscovering this trick?

hopefully! i dont know the exact trick they used, but the idea is to design the search space such that that trick is discoverable.

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#44

Post by giacomoforte »

Hey, I have been following your project for a while, because I'm kinda interested in progam synthesis. Anyway my question is, how scaleable is the search process itself? Is it a good fit for GPU clusters? I guess benchmarking of candidate kernels takes much longer than generating candidate kernels, or not?

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#45

Post by Lerc »

jafioti wrote: post #16
aleinin wrote: post #13

Cool project! How do you think about targeting hardware-specific ISAs directly? There’s an interesting paper from Citadel ( https://arxiv.org/pdf/1804.06826 ) that highlights inefficiencies in nvcc for the Volta architecture. Do you see Luminal’s search-based paradigm eventually extending beyond outperforming handwritten kernels, towards actually competing with NVIDIA’s compiler optimizations at the PTX level?

yep! currently we're emitting cuda / metal but once the search is better, i want to directly emit ptx / low-level asm on other hardwares.

I don't suppose you have an eye towards verilog in the long term?

I'm curious as to the breadth of possibilities that could be searched. I would imagine something like this could invent flash attention if it cast its net wide enough, but that is a pretty broad net. [Edit: I scrolled back and saw flash attention was explicitly mentioned, cool stuff]

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#46

Post by _0ffh »

jafioti wrote: post #41
Alifatisk wrote: post #11

So wait, am I understanding this correctly? Instead of applying just predetermined optimization rules or patterns, the compiler formulates the problem as searching through many possible configurations or versions of the code. Each possible version can have different arrangements, tiling sizes, thread block configurations, memory access patterns, and instruction sequences, right? And from my understanding, the “search…

yup! we build a search space by iteratively applying rewrite rules in every possible order (using e-graphs to do this efficiently). the rewrites alter stuff like looping / tiling structures, as well as algebraic rewrites like softmax to online softmax (and then flash attention). yes optimized kernels for one system will work on other systems with the same hardware. its fine to take a long time compiling if you just c…

Is/will it be possible to just write a model component with Luminal and then use that as a building block in e.g. Torch or JAX?

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#47

Post by ttoinou »

Is it possible that with all the models you’re testing you’re going to find simple rules to optimize kernels so that we won’t need a meta optimizer in the future ? And just code something straight that applies the most important optimizations. Maybe the current search is always ending up on the same kind of codes in the end

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#48

Post by sitkack »

jakestevens2 wrote: post #20
UncleOxidant wrote: post #19

Earlier quoted context omitted.

How long does this typically take? It sounds time consuming. Also, it seems like this could be similar to doing a GA?

That depends on the model architecture and how it was written since that informs the size of the search space. The typical range is 10 mins to 10 hours. It won't be fast but you only have to do it once and then those optimizations are set for every forward pass.

Do you learn the capabilities of the underlying hardware relative to the kernel src? You should be able to start predicting perf using learned static profiling.

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#49

Post by jafioti »

giacomoforte wrote: post #44

Hey, I have been following your project for a while, because I'm kinda interested in progam synthesis. Anyway my question is, how scaleable is the search process itself? Is it a good fit for GPU clusters? I guess benchmarking of candidate kernels takes much longer than generating candidate kernels, or not?

yep, parallelized profiling across many devices is definitely something i want to add.

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#50

Post by jafioti »

Lerc wrote: post #45
jafioti wrote: post #16

Earlier quoted context omitted.

yep! currently we're emitting cuda / metal but once the search is better, i want to directly emit ptx / low-level asm on other hardwares.

I don't suppose you have an eye towards verilog in the long term? I'm curious as to the breadth of possibilities that could be searched. I would imagine something like this could invent flash attention if it cast its net wide enough, but that is a pretty broad net. [Edit: I scrolled back and saw flash attention was explicitly mentioned, cool stuff]

you suppose correctly ;)
Post reply on HN