21–30 of 35 posts
This is gonna be a silly question but what does “kernel” mean in this context. I thought it meant like a Linux kernel module but doesn’t seem to be?
If you search CUDA kernel you’d find examples but the term was used in HPC before as well.
This is gonna be a silly question but what does “kernel” mean in this context. I thought it meant like a Linux kernel module but doesn’t seem to be?
They are comparing unoptimized PyTorch inference, something you would never deploy on a device, to a model with custom kernels. Yes, of course the model with custom kernels is faster, whether it's written by a human or an AI. Generally, PyTorch inference is meant to be used during the training process, and when running metrics, not when deploying. When deployed, you should export to ONNX, and then compile the ONNX to…
They are comparing unoptimized PyTorch inference, something you would never deploy on a device, to a model with custom kernels. Yes, of course the model with custom kernels is faster, whether it's written by a human or an AI. Generally, PyTorch inference is meant to be used during the training process, and when running metrics, not when deploying. When deployed, you should export to ONNX, and then compile the ONNX to…
I'm assuming you are talking about https://github.com/onnx/onnx-mlir?
In your experience, how much faster is a "compiled" onnx model vs. using an onnx runtime?
I work on PyTorch and there are many things that make me suspicious about these results. My TL;DR is unless we get a zip file of all the kernels with how they're benchmarked results like this are almost impossible to verify 1. I don't have an M4 but I have an M1 Pro and I tried running the claimed 18x speedup VisionAttention attention example and I get close to identical runtimes. This example has more issues the mai…
A few clarifications:
1. Baselines - We didn't compare to torch.compile because as of PyTorch 2.7, torch.compile doesn't support the MPS backend, and we ran into some issues on many of the problems when using it. GitHub issue: https://github.com/pytorch/pytorch/issues/150121. Once it's supported, it will be the obvious baseline.
2. Methodology - We followed KernelBench’s protocol to establish a baseline on Metal, adding more correctness checks. Warmup and synchronization were done. We recognize the limitations here and are expanding the validation suite.
3. Optimizations - Right now most of the optimizations are fusions, but there is some use of Metal-specific primitives/optimizations. We expect as we make the supervisor more sophisticated, the novelty of the optimized kernels will also increase.
Overall the goal here is to get some % of the benefit of a human expert in kernel engineering, without developer effort. Compiler-based optimizations are great, but hand-tuned implementations are still common for performance-critical models. The hope is that we can automate some of that process.
They are comparing unoptimized PyTorch inference, something you would never deploy on a device, to a model with custom kernels. Yes, of course the model with custom kernels is faster, whether it's written by a human or an AI. Generally, PyTorch inference is meant to be used during the training process, and when running metrics, not when deploying. When deployed, you should export to ONNX, and then compile the ONNX to…
> and then compile the ONNX to the native format of the device. I'm assuming you are talking about https://github.com/onnx/onnx-mlir ? In your experience, how much faster is a "compiled" onnx model vs. using an onnx runtime?
Back in the day TensorFlow had tfdeploy which compiled TensorFlow terms into NumPy matrix operations. Our synthetic tests saw speedups of factor 50.
They are comparing unoptimized PyTorch inference, something you would never deploy on a device, to a model with custom kernels. Yes, of course the model with custom kernels is faster, whether it's written by a human or an AI. Generally, PyTorch inference is meant to be used during the training process, and when running metrics, not when deploying. When deployed, you should export to ONNX, and then compile the ONNX to…
But that’s the thing, I wouldn’t write a custom kernel before AI
I don't do that level of development or operate at that part of the stack but I’m very experienced in software development
AI significantly augments my skillsets in this area
It'd be curious to see how those AI generated kernels compare to kernels generated by https://github.com/tinygrad/tinygrad
GeoHot didn't want to make it only FlashAttention specific, he worked on FlashAttenrion being automatically generated by the optimizer. It's going well
They are comparing unoptimized PyTorch inference, something you would never deploy on a device, to a model with custom kernels. Yes, of course the model with custom kernels is faster, whether it's written by a human or an AI. Generally, PyTorch inference is meant to be used during the training process, and when running metrics, not when deploying. When deployed, you should export to ONNX, and then compile the ONNX to…
> Yes, of course the model with custom kernels is faster, whether it's written by a human or an AI. But that’s the thing, I wouldn’t write a custom kernel before AI I don't do that level of development or operate at that part of the stack but I’m very experienced in software development AI significantly augments my skillsets in this area
> non 100% correctness of kernels wouldn't model not work properly if kernels are even slightly off? wasn't kernels a part of training stack for models? am I missing anything?
The article is referring to GPU compute kernel ( https://en.wikipedia.org/wiki/Compute_kernel ), not the term kernel used in ML/NN/etc.