Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compiling LLMs into a MegaKernel: A path to low-latency inference

zhihaojia.medium.com

Post reply on HN
Print view View original

11–20 of 79 posts

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#11

Post by bigcat12345678 »

baq wrote: post #3

Next step - compile straight to verilog so I can buy some LLMs on aliexpress

https://riscv.org/blog/2021/02/hardware-description-language... That was one of the promising ideas before AI & GPUs come to the scene. As CPUs are stagnant, and naturally people want further optimize the middle layers software and hardware.

But I suspect parallel computing in GPU style is going to dominate acclerated computing.

General purpose CPUs are going to stay to become the little brain that orchestrates GPUs.

Ideas of software direct to hardware transition might never be the mainstream.

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#12

Post by kp1197 »

After working pretty closely with vLLM and SGLang over the past few months, this is EXACTLY what I had envisioned what a successor project would look like - analyzing an operation dependency graph and then fusing (or, at a minimum, scheduling tasks smarter). Congrats to the team.

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#13

Post by baq »

bigcat12345678 wrote: post #11
baq wrote: post #3

Next step - compile straight to verilog so I can buy some LLMs on aliexpress

https://riscv.org/blog/2021/02/hardware-description-language... That was one of the promising ideas before AI & GPUs come to the scene. As CPUs are stagnant, and naturally people want further optimize the middle layers software and hardware. But I suspect parallel computing in GPU style is going to dominate acclerated computing. General purpose CPUs are going to stay to become the little brain that orchestrates GPUs.…

I'm thinking more like pseudointellect over serial to attach a $3 esp32 to. Since it's basically tokens in, tokens out, let's just cut the unnecessary parts out. It's like querying the cloud models, except it's your silicon you personally soldered to the esp so nobody will break your home assistant with a system prompt update or a fine tuning run.

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#14

Post by delusional »

scotty79 wrote: post #4

> Traditional LLM systems often rely on sequences of GPU kernel launches and external communication calls, resulting in underutilized hardware. What? Why? This seems like an obvious optimization if it's possible.

In the common case where the processor dispatching those kernel calls is much faster than the kernel calls themselves, you're not likely to see a meaningful increase in throughput.

What you need to do first is get really optimized kernels (since that makes the dispatching relatively more expensive) and THEN this becomes worth doing. People who are really good a writing optimized GPU kernels are just not that easy to get a hold of right now.

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#15

Post by skavi »

Does anyone have an intuition on why this offers significant gains over CUDA Graphs?. The CPU launch cost of a graph is tiny which implies most of the work has been offloaded to the GPU's own scheduler. I'd expect that some I/O marshalling at kernel boundaries could be avoided with megakernels. Maybe some loop fusion? Are there any more interesting optimizations they enable?

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#16

Post by refulgentis »

skavi wrote: post #15

Does anyone have an intuition on why this offers significant gains over CUDA Graphs?. The CPU launch cost of a graph is tiny which implies most of the work has been offloaded to the GPU's own scheduler. I'd expect that some I/O marshalling at kernel boundaries could be avoided with megakernels. Maybe some loop fusion? Are there any more interesting optimizations they enable?

You've hit the nail on the head. The CPU launch cost of a pre-compiled CUDA graph is tiny.

CUDA Graphs are a huge step up from manually launching kernels, but they still treat kernels as monolithic, black-box operations. A megakernel erases the boundaries between those operations.

With CUDA Graphs, as in the example in the article, if you have Matmul -> AllReduce, the AllReduce kernel cannot start until the entire Matmul kernel has finished. The dependency is at the kernel level. With a megakernel, they break these ops into fine-grained "tasks" scheduled across SMs. An AllReduce task that needs data from the first slice of the Matmul can begin as soon as that slice is computed by a few SMs, while other SMs are still working on the rest of the Matmul. This fine-grained software pipelining and compute/communication overlap is simply not possible when the dependency unit is the entire kernel.

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#17

Post by flakiness »

This project is from CMU. Hazy Research at Stanford talked about the megakernel too: https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles

Good to see the competition in this area.

(Edited): Related paper covering the larger "mirage" project, but this doesn't cover the "megakernel" approach: https://arxiv.org/abs/2405.05751

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#18

Post by zhihaojia »

flakiness wrote: post #17

This project is from CMU. Hazy Research at Stanford talked about the megakernel too: https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles Good to see the competition in this area. (Edited): Related paper covering the larger "mirage" project, but this doesn't cover the "megakernel" approach: https://arxiv.org/abs/2405.05751

This is the writer of the blog post. You are right that Stanford's work is a parallel effort. The main difference is that our focus is on compilation: making it easier to generate megakernels automatically.

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#19

Post by zhihaojia »

bytepoet wrote: post #5

This is very cool. I enjoyed going through the writeup and GitHub README. I was wondering if these same optimizations can be brought to bear on training as well, rather than only inference. I guess the challenge here is fusing backward computations with gradient communication. I also saw that this currently does not handle dynamic workloads such as MoE. I recently came across this paper that does exactly this: FlashD…

Thanks for reading the post and github README. Supporting training is definitely feasible but the benefit may not be as significant as low-latency inference since training generally involves much larger kernels, making kernel launch overhead less significant.

Thanks for sharing the FlashDMoE work. Our next step is to support MoE models. Stay tuned!

Post reply on HN