Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs

arxiv.org

Post reply on HN
Print view View original

11–16 of 16 posts

Re: CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs

#11

Post by reliabilityguy »

tssge wrote: post #6
rahen wrote: post #2

Strictly speaking, this is very domain-specific and doesn't enable any performance that Triton couldn't already achieve (eliminating global memory round-trips via epilogue fusion is nothing new). The real takeaway is the design shift for LLM-driven codegen rather than handcrafted kernels. LLMs are still bad at low-level hardware optimizations, but really good at high-level composition. Designing compiler abstractions…

>LLMs are still bad at low-level hardware optimizations, but really good at high-level composition. I disagree. While yes they don't have all the architectural quirks of every GPU memorized, they are able to extract such optimizations from ISA docs and online guides. Now with 1M context available on frontier models, they can even fit the whole ISA definition in context (RDNA 3.5 here specifically) and spit out swathe…

> and spit out swathes of optimizations to try.

Without any guarantees of functional correctness.

Re: CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs

#12

Post by HarHarVeryFunny »

tssge wrote: post #6
rahen wrote: post #2

Strictly speaking, this is very domain-specific and doesn't enable any performance that Triton couldn't already achieve (eliminating global memory round-trips via epilogue fusion is nothing new). The real takeaway is the design shift for LLM-driven codegen rather than handcrafted kernels. LLMs are still bad at low-level hardware optimizations, but really good at high-level composition. Designing compiler abstractions…

>LLMs are still bad at low-level hardware optimizations, but really good at high-level composition. I disagree. While yes they don't have all the architectural quirks of every GPU memorized, they are able to extract such optimizations from ISA docs and online guides. Now with 1M context available on frontier models, they can even fit the whole ISA definition in context (RDNA 3.5 here specifically) and spit out swathe…

If you look at Anthropic's recent kernel optimization challenge, and the human leaderboard, humans are soundly beating Claude's best attempt.

I think the reason, as parent suggested, is that LLMs are great at composition (mash-ups/regeneration - this is essentially what they are trained to do), and not so great at innovation. How well they can do relative to a human, on a low level optimization problem, is going to depend on degree of similarity of the problem to things they were trained on and/or have access to.

Re: CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs

#13

Post by boulos »

saagarjha wrote: post #8
tssge wrote: post #6

Earlier quoted context omitted.

>LLMs are still bad at low-level hardware optimizations, but really good at high-level composition. I disagree. While yes they don't have all the architectural quirks of every GPU memorized, they are able to extract such optimizations from ISA docs and online guides. Now with 1M context available on frontier models, they can even fit the whole ISA definition in context (RDNA 3.5 here specifically) and spit out swathe…

The lack of fast GPU kernels written by AI does not lend credence to your theory.

Perhaps you missed work like https://crfm.stanford.edu/2025/05/28/fast-kernels.html ?

Re: CODA: Rewriting Transformer Blocks as GEMM-Epilogue Programs

#16

Post by saagarjha »

boulos wrote: post #13
saagarjha wrote: post #8

Earlier quoted context omitted.

The lack of fast GPU kernels written by AI does not lend credence to your theory.

Perhaps you missed work like https://crfm.stanford.edu/2025/05/28/fast-kernels.html ?

Comparing against torch.compile is not particularly impressive
Post reply on HN