Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels
1–10 of 35 posts
Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels
#2This is pretty cool.
I initially thought they were writing custom kernels for proprietary models like GPT-5. They aren't - they're using proprietary models to write kernels for a set of ~250 open Pytorch modules.
Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels
#3Are these kernel available ? I'd love to try that !
Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels
#4This is amazing. I wouldn't have thought that AI is this good in niche topics. Very impressive experiment and write up.
Still, I can't help but think we should bet on sth like Mojo instead for the long run.
Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels
#5> non 100% correctness of kernels
wouldn't model not work properly if kernels are even slightly off?
wasn't kernels a part of training stack for models? am I missing anything?
Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels
#6This is amazing. I wouldn't have thought that AI is this good in niche topics. Very impressive experiment and write up. Still, I can't help but think we should bet on sth like Mojo instead for the long run.
Mojo is a terrible language and its main feature (GPU acceleration through Mojo max) is closed source and requires a commercial license to be purchased.
Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels
#7They are comparing unoptimized PyTorch inference, something you would never deploy on a device, to a model with custom kernels.
Yes, of course the model with custom kernels is faster, whether it's written by a human or an AI.
Generally, PyTorch inference is meant to be used during the training process, and when running metrics, not when deploying. When deployed, you should export to ONNX, and then compile the ONNX to the native format of the device.
If you aren't familiar with the pipeline for ML deployment, this is the equivalent of comparing interpreted code to compiled code.
Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels
#8They are comparing unoptimized PyTorch inference, something you would never deploy on a device, to a model with custom kernels. Yes, of course the model with custom kernels is faster, whether it's written by a human or an AI. Generally, PyTorch inference is meant to be used during the training process, and when running metrics, not when deploying. When deployed, you should export to ONNX, and then compile the ONNX to…
[deleted]
Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels
#9> non 100% correctness of kernels wouldn't model not work properly if kernels are even slightly off? wasn't kernels a part of training stack for models? am I missing anything?
The article is referring to GPU compute kernel (https://en.wikipedia.org/wiki/Compute_kernel), not the term kernel used in ML/NN/etc.
Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels
#10It'd be curious to see how those AI generated kernels compare to kernels generated by https://github.com/tinygrad/tinygrad