Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ExecuTorch: Run PyTorch programs on mobile and edge devices

pytorch.org

Post reply on HN
Print view View original

11–20 of 27 posts

Re: ExecuTorch: Run PyTorch programs on mobile and edge devices

#11

Post by phh »

flakiness wrote: post #6

There is also PyTorch mobile [1]. I'm curious how these two relate. [Edit] And here is the answer [2] > PyTorch Mobile uses TorchScript to allow PyTorch models to run on devices with limited resources. ExecuTorch has a significantly smaller memory size and a dynamic memory footprint resulting in superior performance and portability compared to PyTorch Mobile. Also, ExecuTorch does not rely on TorchScript, and instead…

Yeah it's a bit weird to see the difference.

> Also, ExecuTorch does not rely on TorchScript, and instead leverages PyTorch 2 compiler and export functionality for on-device execution of PyTorch models.

I think this is the major change: AFAIU you could consider ExecuTorch as ""just a rewrite"" of PyTorch mobile using PyTorch 2 compiler.

BUT this is should still be pretty major. It was pretty hard to retarget models to be compatible with TorchScript. But most recent models will want to target PyTorch 2 compiler for the performance boost. Which means they'll automatically get ExecuTorch support.

As someone who got burned several times in trying to port models to TorchScript and run them on Androids, I'm happy they did that.

Re: ExecuTorch: Run PyTorch programs on mobile and edge devices

#12

Post by mlmandude »

shoumikhin wrote: post #10

Hey guys, we are ExecuTorch team, we're excited about launching this! Please ask us anything :)

It looks like executorch is for edge devices (phones / IoT / etc).

I'm currently doing inference on GPUs with libtorch and have a few concerns: (1) It seems like libtorch/torchscript are on a path to getting deprecated and (2) libtorch/torchscript pull in enormously bloated libraries. Should I be looking at executorch? I currently don't see an nvidia backend / integration with tensor rt in https://github.com/pytorch/executorch/tree/main/backends , but seems like it might be possible. Is this something you are thinking about?

Re: ExecuTorch: Run PyTorch programs on mobile and edge devices

#14

Post by BudaDude »

nothrowaways wrote: post #13
shoumikhin wrote: post #10

Hey guys, we are ExecuTorch team, we're excited about launching this! Please ask us anything :)

Nice tool with ugly name

It's one of those names that sound better said aloud than spelled out. Excited to see where this goes though!

Re: ExecuTorch: Run PyTorch programs on mobile and edge devices

#15

Post by iseeyuan »

mlmandude wrote: post #12
shoumikhin wrote: post #10

Hey guys, we are ExecuTorch team, we're excited about launching this! Please ask us anything :)

It looks like executorch is for edge devices (phones / IoT / etc). I'm currently doing inference on GPUs with libtorch and have a few concerns: (1) It seems like libtorch/torchscript are on a path to getting deprecated and (2) libtorch/torchscript pull in enormously bloated libraries. Should I be looking at executorch? I currently don't see an nvidia backend / integration with tensor rt in https://github.com/pytorch/…

Yes ExecuTorch is currently targeted at Edge devices. The runtime is written in C++ with 50KB binary size (without kernels) and should run in most of platforms. You are right that we have not integrated to Nvidia backend yet. Have you tried torch.compile() in PyTorch 2.0? It would do the Nvidia optimization for you without Torchscript. If you have specific binary size or edge specific request, feel free to file issues in https://github.com/pytorch/executorch/issues

Re: ExecuTorch: Run PyTorch programs on mobile and edge devices

#18

Post by modeless »

shoumikhin wrote: post #10

Hey guys, we are ExecuTorch team, we're excited about launching this! Please ask us anything :)

Looks cool! Does the Vulkan backend work on PC? MLC-LLM proves it can work well and it would be cool to have a cross platform, minimal runtime, GPU-agnostic backend for PC too, not just mobile.
Post reply on HN