Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Compiling LLMs into a MegaKernel: A path to low-latency inference

zhihaojia.medium.com

Post reply on HN
Print view View original

71–79 of 79 posts

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#71

Post by otabdeveloper4 »

fc417fc802 wrote: post #70
otabdeveloper4 wrote: post #60

Earlier quoted context omitted.

24 gigabytes is more than enough to run a local LLM for a small household or business. This is "gaming PC" territory, not "space heater". I mean people already have PS5's and whatnot in their homes. The hundreds of gigabytes thing exists because the big cloud LLM providers went down the increasing parameter count path. That way is a dead end and we've reached negative returns already. Prompt engineering + finetunes i…

Only if you'll settle for less than state of the art. The best models still tend to be some of the largest ones. Anything that overflows VRAM is going to slow down the response time drastically. "Space heater" is determined by computational horsepower rather than available RAM. How big a context window do you want? Last I checked that was very expensive in terms of RAM and having a large one was highly desirable.

State of the art is achieved by finetuning. Increasing parameter counts is a dead end.

Large contexts are very important but they are cheap compared in terms of RAM compared to the costs of increasing parameter count.

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#72

Post by nravic »

This is super interesting! We do something similar I think by taking a checkpoint after model initialization. I'm curious what you think about our approach, here's some benchmarks: https://docs.cedana.ai/articles/performance-of-cedanas-gpu-i...

We do some on-the-fly optimizations as well (like compiling into CUDA graphs or fusing together calls) which ends up resulting (for some inference engines) faster token throughput too.

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#74

Post by pavelstoev »

zhihaojia wrote: post #48
refibrillator wrote: post #43

Hi author(s), the on-GPU interpreter approach looks like a promising path forward, have you seen this strikingly similar concurrent work? https://news.ycombinator.com/item?id=44111673 I find it curious that fundamentals of the CUDA programming model (eg kernel launches) are being subverted in favor of fine grained task based parallelism that ends up using the hardware more effectively. Makes me wonder if CUDA has bee…

Thanks for the great feedback! Stanford's MegaKernel project tackles a similar challenge but focuses on manual CUDA implementation. While MPK takes a compiler-driven approach—users express their LLMs at the PyTorch level, and MPK automatically compiles them into optimized megakernels. Our goal is to make programming megakernels much more accessible. I completely agree that CUDA can be a limiting factor, especially fo…

Hi Author - thank you very much for the clear and relatively easy-to-understand MPK overview. Could you please also comment on the similarity of your project to Hidet https://pytorch.org/blog/introducing-hidet/

Thank you !

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#75

Post by Gregaros »

Curious if anyone has thoughts on going even further: eschewing soft-ware based inference in favor of a purely ASIC approach to a static LLM. Cost benefits? Software level additional, fine-tuneable layers to allow a degree of improvement and flexibility? We are quickly approaching ‘good enough’ for some tasks—at what point does that mean we’re comfortable locking something in for the ~2-4 year lifespan of a device if there _were_ advantages offered by a hyper-specialized chip?

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#76

Post by Gregaros »

Gregaros wrote: post #75

Curious if anyone has thoughts on going even further: eschewing soft-ware based inference in favor of a purely ASIC approach to a static LLM. Cost benefits? Software level additional, fine-tuneable layers to allow a degree of improvement and flexibility? We are quickly approaching ‘good enough’ for some tasks—at what point does that mean we’re comfortable locking something in for the ~2-4 year lifespan of a device if…

Some further questions:

1. For tasks like autocomplete, keyword routing, or voice transcription, what would the latency and power savings look like on an ASIC vs. even a megakernel GPU setup? Would that justify a fixed-function approach in edge devices or embedded systems?

2. ASICs obviously kill retraining, but could we envision a hybrid setup where a base model is hardwired and a small, soft, learnable module (e.g., LoRA-style residual layers) runs on a general-purpose co-processor?

3. Would the transformer’s fixed topology lend itself to spatial reuse in ASIC design, or is the model’s size (e.g. GPT-3-class) still prohibitive without aggressive weight pruning or quantization?

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#77

Post by jerf »

bigcat12345678 wrote: post #11
baq wrote: post #3

Next step - compile straight to verilog so I can buy some LLMs on aliexpress

https://riscv.org/blog/2021/02/hardware-description-language... That was one of the promising ideas before AI & GPUs come to the scene. As CPUs are stagnant, and naturally people want further optimize the middle layers software and hardware. But I suspect parallel computing in GPU style is going to dominate acclerated computing. General purpose CPUs are going to stay to become the little brain that orchestrates GPUs.…

"General purpose CPUs are going to stay to become the little brain that orchestrates GPUs."

If that was going to happen, it would have happened.

CPUs are genuinely good at what they do, and "what they do" is a lot of tasks that GPUs are actually terrible at. If all we had were GPUs in the world and someone invented a CPU, we'd hail them as a genius. A lot of people seem to think that GPUs are just "better", just ambiently better at everything, but that's lightyears from the truth. They are quite spectacularly terrible at a lot of very common tasks. There's many very good reasons that GPUs are still treated as accelerators for the CPUs and not vice versa.

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#78

Post by lubujackson »

People keep coming up with new metaphors for LLMs to explain their impact and functionality.

Maybe we should think of them like transistors? Right now, we are at the point where we have a room-sized computer than can do multiplication from punch card input.

It is fun to imagine what we could do if we ran, say, 1 million coordinated o3-pro queries at once?

Re: Compiling LLMs into a MegaKernel: A path to low-latency inference

#79

Post by liuliu »

gdiamos wrote: post #64
liuliu wrote: post #10

Earlier quoted context omitted.

It really is not obvious. These launches are asynchronous, and data movement / computation is overlapped properly through CUDA APIs. Even per-kernel launch cost is reduced with the cudagraph introduction. CUDA programming model relies on each kernel to be computationally expensive to make sense, and these are not true for token generation of LLM. And we are talking about network evaluation at higher than 1000 per sec…

It's surprising to me that the field is willing to invest this much in mega-kernels, but not models that generate multiple tokens in parallel...

It is hard to justify tens-of-millions investment in training to just make it faster without any idea how it scores on benchmarks. It is easier to justify keeping the model intact and spend extra millions to make it faster with exotic means (megakernels).

There are some niche research on parallel token generations though as of late...

Post reply on HN