Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A PyTorch Approach to ML Infrastructure

run.house

Post reply on HN
Print view View original

11–20 of 24 posts

Re: A PyTorch Approach to ML Infrastructure

#11

Post by cbarrick »

> Just as PyTorch lets you send a model .to("cuda"), Runhouse enables hardware heterogeneity by letting you send your code (or dataset, environment, pipeline, etc) .to(“cloud_instance”, “on_prem”, “data_store”...), all from inside a Python notebook or script. There’s no need to manually move the code and data around, package into docker containers, or translate into a pipeline DAG.

From an SRE perspective, this sounds like a nightmare. Controlled releases are really important for reliability. I definitely don't want my devs doing manual rollouts from a notebook.

Re: A PyTorch Approach to ML Infrastructure

#12

Post by donnygreenberg »

cbarrick wrote: post #11

> Just as PyTorch lets you send a model .to("cuda"), Runhouse enables hardware heterogeneity by letting you send your code (or dataset, environment, pipeline, etc) .to(“cloud_instance”, “on_prem”, “data_store”...), all from inside a Python notebook or script. There’s no need to manually move the code and data around, package into docker containers, or translate into a pipeline DAG. From an SRE perspective, this sound…

It's a great point. The funny thing is that the rest of the world just has dev, QA, and prod staging and canaries, while ML has "the 6 months it takes to translate from notebook to pipeline" or "uploading a new checkpoint/image to the platform." We can just stage properly and release through CD like everyone else does, but not spend 6 months flipping the switch. We built the ability to specify the package for a function as a git repo and a particular revision to enable this, and hopefully it means more people rely on version control as the source of truth in prod and not the most recently uploaded model checkpoint to Sagemaker. During experimentation though, it really is frustrating that many systems only allow you run what you've committed.

We've also built a basic permissioning system to control who can actually overwrite the saved version of a resource, so there are no accidents. E.g. if the prod inference blob is saved at "mikes_pizza/nlp/bert/bert_prod", you can set it so only x accounts can overwrite that metadata to point to a new model. Ideally we just inherit existing RBAC groups sometime soon.

Does that make sense? Curious if you had something else in mind as far as the danger.

Re: A PyTorch Approach to ML Infrastructure

#15

Post by extr »

donnygreenberg wrote: post #9
extr wrote: post #6

Very interesting. I just worked to implement a baby version of this kind of system at work. Similar to this project, our basic use case was allowing researchers to quickly/easily execute their arbitrary R&D code on cloud resources. It's difficult to know in advance what they might be doing, and we wanted to avoid a situation where they are pushing a docker container or submitting a file every time they change somethi…

Hi! That's awesome to hear, and very aligned with the devx we're going for. How was your system received? In fact we totally agree and are not cloudpickling the function because of the package minor version issues. We sync over the code to the destination environment and the server imports it fresh, which is much more robust. The one piece of code that cloudpickles functions is a trap door for certain weird situation…

Our system was super well received minus the stability issues. I think the UX of being able to "ship" code like this is a big leap over the alternatives - it actually gives me a lot of confidence in the approach seeing that someone else had a similar thought.

Very interesting about the implementation. I admittedly did not read that closely and clearly did not grok the what the actual hot path was there, will check it out more. May have to borrow your approach or perhaps just adopt this wholesale :) Regardless, super cool project, will be following.

Re: A PyTorch Approach to ML Infrastructure

#16

Post by donnygreenberg »

extr wrote: post #15
donnygreenberg wrote: post #9

Earlier quoted context omitted.

Hi! That's awesome to hear, and very aligned with the devx we're going for. How was your system received? In fact we totally agree and are not cloudpickling the function because of the package minor version issues. We sync over the code to the destination environment and the server imports it fresh, which is much more robust. The one piece of code that cloudpickles functions is a trap door for certain weird situation…

Our system was super well received minus the stability issues. I think the UX of being able to "ship" code like this is a big leap over the alternatives - it actually gives me a lot of confidence in the approach seeing that someone else had a similar thought. Very interesting about the implementation. I admittedly did not read that closely and clearly did not grok the what the actual hot path was there, will check it…

Excellent! Don't hesitate to reach out (donny at run dot house) if you want to chat about adopting our approach or using Runhouse.

Re: A PyTorch Approach to ML Infrastructure

#17

Post by donnygreenberg »

pavelstoev wrote: post #14

Have you tired Hidet ? https://pypi.org/project/hidet/

No, I know of CentML but don't deeply know the surface of hardware they compile for. I'm enthusiastic about projects like this and others which integrate with PyTorch 2.0. Flexible compilers make the value of being able to ship your code around to various hardware even more powerful.

Re: A PyTorch Approach to ML Infrastructure

#18

Post by donnygreenberg »

chenzhekl wrote: post #13

How do you compare Runhouse with Ray which also simplifies distributed computing?

Good question. We actually use Ray to handle a bunch of the scheduling within the compute, but largely see our role as outside the compute. Meaning, Ray provides a powerful DSL for distributed compute, while we are aggressively DSL-free so users can ship Ray code, PyTorch Distributed, Accelerate, Horovod, etc. to their hardware through Runhouse. We're more focused on connecting disparate compute and storage and making them multiplayer (but largely see the cluster as an opaque unit of compute) while they're more focused on enabling distribution inside the cluster, if that makes sense.

Re: A PyTorch Approach to ML Infrastructure

#20

Post by donnygreenberg »

m_ke wrote: post #19

Since people are suggesting alternatives, I'd like to shoutout skypilot: https://github.com/skypilot-org/skypilot EDIT: looks like this actually uses it under the hood: https://github.com/run-house/runhouse/blob/main/requirements...

Yes, we work pretty closely with them and they're lovely. Everyone should try SkyPilot.
Post reply on HN