A PyTorch Approach to ML Infrastructure • Hacker News Board

Re: A PyTorch Approach to ML Infrastructure

#21

Post by cbarrick » Fri, Jun 30, 2023, 1:03 PM UTC

donnygreenberg wrote: post #12

cbarrick wrote: post #11

> Just as PyTorch lets you send a model .to("cuda"), Runhouse enables hardware heterogeneity by letting you send your code (or dataset, environment, pipeline, etc) .to(“cloud_instance”, “on_prem”, “data_store”...), all from inside a Python notebook or script. There’s no need to manually move the code and data around, package into docker containers, or translate into a pipeline DAG. From an SRE perspective, this sound…

It's a great point. The funny thing is that the rest of the world just has dev, QA, and prod staging and canaries, while ML has "the 6 months it takes to translate from notebook to pipeline" or "uploading a new checkpoint/image to the platform." We can just stage properly and release through CD like everyone else does, but not spend 6 months flipping the switch. We built the ability to specify the package for a funct…

Ah, I see. The ability to push to infra is more about the development loop than prod rollouts. Prod can (and should) use CD with a well understood source of truth.

Thanks, I was misunderstanding the purpose of the feature.

Re: A PyTorch Approach to ML Infrastructure

#22

Post by pavelstoev » Fri, Jun 30, 2023, 3:10 PM UTC

donnygreenberg wrote: post #17

pavelstoev wrote: post #14

Have you tired Hidet ? https://pypi.org/project/hidet/

No, I know of CentML but don't deeply know the surface of hardware they compile for. I'm enthusiastic about projects like this and others which integrate with PyTorch 2.0. Flexible compilers make the value of being able to ship your code around to various hardware even more powerful.

Thanks for your comment ! Current focus of optimizations is for Nvidia GPUs but others are in the works. Hidet comes with Hidet.Script which abstracts some of the CUDA struggles and may make the ML optimizations efforts easier to implement. It is still evolving so documentation is limited but here are some examples: https://github.com/hidet-org/hidet/tree/main/python/hidet/gr...

Re: A PyTorch Approach to ML Infrastructure

#23

Post by ipsum2 » Fri, Jun 30, 2023, 10:38 PM UTC

> Please make sure the function does not rely on any local variables, including imports (which should be moved inside the function body)

This seems like a major limitation and pretty antithetical to the PyTorch approach.

Re: A PyTorch Approach to ML Infrastructure

#24

Post by donnygreenberg » Thu, Jul 06, 2023, 2:26 PM UTC

ipsum2 wrote: post #23

> Please make sure the function does not rely on any local variables, including imports (which should be moved inside the function body) This seems like a major limitation and pretty antithetical to the PyTorch approach.

Good catch - this is actually only for functions defined inside interactive Notebook or iPython environments, and we do have an option to bypass it (which serializes the function and state), but you probably don't want it by default. Any function defined inside normal python (even if you're using it within a notebook) doesn't doesn't need imports or variables in scope, you can ship it as is to remote compute. But notebooks are gnarly things, where you might have defined a variable and changed it many times, and finally used it inside a function which you want to run remotely as a service. You probably don't want that variable's state to be shipped over, and PyTorch doesn't do this either. This is why so much of stateful PyTorch is meant to be defined inside new nn.Module classes, to neatly package the state to send over to GPU memory. We offer more flexibility than that, but the more state you ship over, the more likely you are to hit version conflicts between the local and remote environments, which can be really annoying. We practically never run into those issues at Runhouse nowadays because we think we've found a sweet spot.