Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Luminal – Open-source, search-based GPU compiler

github.com

Post reply on HN
Print view View original

11–20 of 70 posts

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#11

Post by Alifatisk »

So wait, am I understanding this correctly?

Instead of applying just predetermined optimization rules or patterns, the compiler formulates the problem as searching through many possible configurations or versions of the code. Each possible version can have different arrangements, tiling sizes, thread block configurations, memory access patterns, and instruction sequences, right?

And from my understanding, the “search space” is just a collection of all potential versions of the code (kernels) that the compiler can generate from the original input. So for example, the space might include

- Different ways to partition workloads among GPU threads and blocks

- Varying memory access strategies (using shared memory, global memory)

- Various instruction-level optimizations or reordering

- Alternative loop unroll factors or vectorization strategies

The compiler then programmatically produces a large number of candidate kernels by combining different optimizations and configurations. Among these millions of candidates, the compiler tries to find the one that performs best.

In that case, can the compiler print out which gpu configuration works the best for that computer? And will that configuration be applicable to all computers with the same setup?

This is such an interesting technique.

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#12

Post by [deleted] »

Alifatisk wrote: post #11

So wait, am I understanding this correctly? Instead of applying just predetermined optimization rules or patterns, the compiler formulates the problem as searching through many possible configurations or versions of the code. Each possible version can have different arrangements, tiling sizes, thread block configurations, memory access patterns, and instruction sequences, right? And from my understanding, the “search…

[deleted]

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#13

Post by aleinin »

Cool project! How do you think about targeting hardware-specific ISAs directly? There’s an interesting paper from Citadel (https://arxiv.org/pdf/1804.06826) that highlights inefficiencies in nvcc for the Volta architecture. Do you see Luminal’s search-based paradigm eventually extending beyond outperforming handwritten kernels, towards actually competing with NVIDIA’s compiler optimizations at the PTX level?

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#14

Post by jakestevens2 »

Alifatisk wrote: post #11

So wait, am I understanding this correctly? Instead of applying just predetermined optimization rules or patterns, the compiler formulates the problem as searching through many possible configurations or versions of the code. Each possible version can have different arrangements, tiling sizes, thread block configurations, memory access patterns, and instruction sequences, right? And from my understanding, the “search…

Your description is exactly right. We create a search space of all possible kernels and find the best ones based on runtime. The best heuristic is no heuristic.

This obviously creates a combinatorial problem that we mitigate with smarter search.

The kernels are run on the computer the compiler is running on. Since runtime is our gold standard it will search for the best configuration for your hardware target. As long as the setup is mostly the same, the optimizations should carry over, yes.

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#15

Post by jafioti »

diggan wrote: post #10

> Luminal can run Q8 Llama 3 8B on M-series Macbooks at 15-25 tokens per second. The goal is to become the fastest ML framework for any model on any device. Great that some numbers are provided, but in isolation, I'm not sure what they provide. It would be helpful to also share what tok/s you'd get with llama.cpp or something else on the same hardware, so we can actually understand if it's faster or not :) Also inclu…

a lot of the search is still being optimized so we don't match super hand-optimized kernels like llama.cpp has, so we def don't match their tps yet, but i want to make a perf tracking page to see improvements over time and prevent regressions

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#16

Post by jafioti »

aleinin wrote: post #13

Cool project! How do you think about targeting hardware-specific ISAs directly? There’s an interesting paper from Citadel ( https://arxiv.org/pdf/1804.06826 ) that highlights inefficiencies in nvcc for the Volta architecture. Do you see Luminal’s search-based paradigm eventually extending beyond outperforming handwritten kernels, towards actually competing with NVIDIA’s compiler optimizations at the PTX level?

yep! currently we're emitting cuda / metal but once the search is better, i want to directly emit ptx / low-level asm on other hardwares.

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#19

Post by UncleOxidant »

jakestevens2 wrote: post #14
Alifatisk wrote: post #11

So wait, am I understanding this correctly? Instead of applying just predetermined optimization rules or patterns, the compiler formulates the problem as searching through many possible configurations or versions of the code. Each possible version can have different arrangements, tiling sizes, thread block configurations, memory access patterns, and instruction sequences, right? And from my understanding, the “search…

Your description is exactly right. We create a search space of all possible kernels and find the best ones based on runtime. The best heuristic is no heuristic. This obviously creates a combinatorial problem that we mitigate with smarter search. The kernels are run on the computer the compiler is running on. Since runtime is our gold standard it will search for the best configuration for your hardware target. As long…

How long does this typically take? It sounds time consuming. Also, it seems like this could be similar to doing a GA?

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#20

Post by jakestevens2 »

UncleOxidant wrote: post #19
jakestevens2 wrote: post #14

Earlier quoted context omitted.

Your description is exactly right. We create a search space of all possible kernels and find the best ones based on runtime. The best heuristic is no heuristic. This obviously creates a combinatorial problem that we mitigate with smarter search. The kernels are run on the computer the compiler is running on. Since runtime is our gold standard it will search for the best configuration for your hardware target. As long…

How long does this typically take? It sounds time consuming. Also, it seems like this could be similar to doing a GA?

That depends on the model architecture and how it was written since that informs the size of the search space.

The typical range is 10 mins to 10 hours. It won't be fast but you only have to do it once and then those optimizations are set for every forward pass.

Post reply on HN