Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Luminal – Open-source, search-based GPU compiler

github.com

Post reply on HN
Print view View original

51–60 of 70 posts

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#51

Post by jakestevens2 »

sitkack wrote: post #48
jakestevens2 wrote: post #20

Earlier quoted context omitted.

That depends on the model architecture and how it was written since that informs the size of the search space. The typical range is 10 mins to 10 hours. It won't be fast but you only have to do it once and then those optimizations are set for every forward pass.

Do you learn the capabilities of the underlying hardware relative to the kernel src? You should be able to start predicting perf using learned static profiling.

Not today but we will implement memoization of kernels for each hardware backend, yes.

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#52

Post by jakestevens2 »

ttoinou wrote: post #47

Is it possible that with all the models you’re testing you’re going to find simple rules to optimize kernels so that we won’t need a meta optimizer in the future ? And just code something straight that applies the most important optimizations. Maybe the current search is always ending up on the same kind of codes in the end

See my comment on a deeper thread about this. Eventually we will implement static profiling for common kernels so the search doesn't actually have to manually run all of them; many will have a known runtime that we can tie to them.

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#53

Post by almostgotcaught »

jafioti wrote: post #41
Alifatisk wrote: post #11

So wait, am I understanding this correctly? Instead of applying just predetermined optimization rules or patterns, the compiler formulates the problem as searching through many possible configurations or versions of the code. Each possible version can have different arrangements, tiling sizes, thread block configurations, memory access patterns, and instruction sequences, right? And from my understanding, the “search…

yup! we build a search space by iteratively applying rewrite rules in every possible order (using e-graphs to do this efficiently). the rewrites alter stuff like looping / tiling structures, as well as algebraic rewrites like softmax to online softmax (and then flash attention). yes optimized kernels for one system will work on other systems with the same hardware. its fine to take a long time compiling if you just c…

> take a long time compiling

Lol np-hard is still np-hard no matter how you slice it (especially given vague objective functions).

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#54

Post by Reubend »

diggan wrote: post #10

> Luminal can run Q8 Llama 3 8B on M-series Macbooks at 15-25 tokens per second. The goal is to become the fastest ML framework for any model on any device. Great that some numbers are provided, but in isolation, I'm not sure what they provide. It would be helpful to also share what tok/s you'd get with llama.cpp or something else on the same hardware, so we can actually understand if it's faster or not :) Also inclu…

Yeah those numbers look very low to me for something that's supposed to represent a state of the art optimization technique. I think that's lower than other implementations, although it depends on the MacBook.

Nonetheless this project looks very cool, and I hope they can continue improving it to the point where it indeed beats human-led optimizations.

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#57

Post by jafioti »

almostgotcaught wrote: post #53
jafioti wrote: post #41

Earlier quoted context omitted.

yup! we build a search space by iteratively applying rewrite rules in every possible order (using e-graphs to do this efficiently). the rewrites alter stuff like looping / tiling structures, as well as algebraic rewrites like softmax to online softmax (and then flash attention). yes optimized kernels for one system will work on other systems with the same hardware. its fine to take a long time compiling if you just c…

> take a long time compiling Lol np-hard is still np-hard no matter how you slice it (especially given vague objective functions).

np-hard is still solveable with constraints. look at go.

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#58

Post by bojle »

Lerc wrote: post #45
jafioti wrote: post #16

Earlier quoted context omitted.

yep! currently we're emitting cuda / metal but once the search is better, i want to directly emit ptx / low-level asm on other hardwares.

I don't suppose you have an eye towards verilog in the long term? I'm curious as to the breadth of possibilities that could be searched. I would imagine something like this could invent flash attention if it cast its net wide enough, but that is a pretty broad net. [Edit: I scrolled back and saw flash attention was explicitly mentioned, cool stuff]

Equality saturation (something that luminal uses at its core) is a topic for hardware synthesis and verification too. Something like dynamic hardware generation (instead of kernel generation). For example, see this thesis [1] by Samuel Coward of Imperial.

[1] https://samuelcoward.co.uk/assets/pdf/Thesis_Imperial.pdf

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#59

Post by GregarianChild »

jafioti wrote: post #37
GregarianChild wrote: post #36

How is this different from superoptimisation? Also, how do you ensure that newly generated kernels are correct w.r.t. the original naive kernel that you use as specification?

very similar to superoptimisation, but most superoptimisers try to tackle turing-complete code. by just doing a very limited space of computation (linear algebra with 12 primitive ops) the search remains tractable. the search space is designed to remain logically equivalent at all times, by virtue of how its built (applying rewrite rules we know dont change the logical equivalence).

If the search space never leaves the programs that are equivalent to the original specification, that will probably limit the optimisations you can discover. (E.g. if you start out with standard matmul, you will not discover Strassen's algorithm.) This is not a criticism, I'm just trying to understand your algorithm.

Re: Show HN: Luminal – Open-source, search-based GPU compiler

#60

Post by erichocean »

jakestevens2 wrote: post #14
Alifatisk wrote: post #11

So wait, am I understanding this correctly? Instead of applying just predetermined optimization rules or patterns, the compiler formulates the problem as searching through many possible configurations or versions of the code. Each possible version can have different arrangements, tiling sizes, thread block configurations, memory access patterns, and instruction sequences, right? And from my understanding, the “search…

Your description is exactly right. We create a search space of all possible kernels and find the best ones based on runtime. The best heuristic is no heuristic. This obviously creates a combinatorial problem that we mitigate with smarter search. The kernels are run on the computer the compiler is running on. Since runtime is our gold standard it will search for the best configuration for your hardware target. As long…

> that we mitigate with smarter search

aka "a heuristic"

Post reply on HN