Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels

gimletlabs.ai

Post reply on HN
Print view View original

31–35 of 35 posts

Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels

#31

Post by rovr138 »

formalsystem wrote: post #19

I work on PyTorch and there are many things that make me suspicious about these results. My TL;DR is unless we get a zip file of all the kernels with how they're benchmarked results like this are almost impossible to verify 1. I don't have an M4 but I have an M1 Pro and I tried running the claimed 18x speedup VisionAttention attention example and I get close to identical runtimes. This example has more issues the mai…

Any insight on the torch.compile issue posted?

Like you said, would be great to test against it.

Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels

#32

Post by rovr138 »

earthnail wrote: post #14
ipsum2 wrote: post #6

Earlier quoted context omitted.

Mojo is a terrible language and its main feature (GPU acceleration through Mojo max) is closed source and requires a commercial license to be purchased.

Why is it a terrible language? Genuinely curious question.

A language where a main feature, Gpu acceleration, is being behind a commercial license is a pretty big issue.

Not on language design, but in the ecosystem and other areas.

Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels

#33

Post by forkeep »

am17an wrote: post #29
yieldcrv wrote: post #27

Earlier quoted context omitted.

> Yes, of course the model with custom kernels is faster, whether it's written by a human or an AI. But that’s the thing, I wouldn’t write a custom kernel before AI I don't do that level of development or operate at that part of the stack but I’m very experienced in software development AI significantly augments my skillsets in this area

The point is those kernels exist already, you can just use them off the shelf. In the case where you're trying to write a production grade kernel without operating at that part of the stack... well good luck with that.

[dead]

Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels

#34

Post by spott »

turbo_wombat wrote: post #7

They are comparing unoptimized PyTorch inference, something you would never deploy on a device, to a model with custom kernels. Yes, of course the model with custom kernels is faster, whether it's written by a human or an AI. Generally, PyTorch inference is meant to be used during the training process, and when running metrics, not when deploying. When deployed, you should export to ONNX, and then compile the ONNX to…

vLLM is a LLM model serving framework written using raw PyTorch.

ONNX doesn’t support a bunch of operations that PyTorch does (it isn’t always possible to convert a PyTorch model to ONNX).

Torchserve runs raw PyTorch.

Generally speaking, PyTorch is pretty well optimized. For Mac it has been historically ignored, so the kernels for MPS were all missing or just bad, but on CUDA and Linux they are pretty good.

Re: Speeding up PyTorch inference on Apple devices with AI-generated Metal kernels

#35

Post by ymsodev »

saagarjha wrote: post #30
ymsodev wrote: post #9

Earlier quoted context omitted.

The article is referring to GPU compute kernel ( https://en.wikipedia.org/wiki/Compute_kernel ), not the term kernel used in ML/NN/etc.

…aren't they the same thing

They're not, but I also misunderstood the original question, they're referring to the correct definition of kernel. I thought they were confusing the GPU kernel with https://en.wikipedia.org/wiki/Kernel_method or https://en.wikipedia.org/wiki/Kernel_(image_processing)
Post reply on HN