Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kubeflow – Machine Learning Toolkit for Kubernetes

github.com

Post reply on HN
Print view View original

11–20 of 35 posts

Re: Kubeflow – Machine Learning Toolkit for Kubernetes

#11

Post by TheIronYuppie »

nikolay wrote: post #6
mountainerd wrote: post #5

Pretty interesting. I'm guessing this is something that Google uses internally for their Kubernetes workflows.

I am at KubeCon 2017 in Austin, TX and, yeah, based on the presentation, it looks like an internal tool they just opened to the public with some bold goals.

Hi! It was actually designed from the start to be an extension of GitHub.com/tensor flow/k8s and then it took on larger goals = Making an entire ML stack (of any ML framework) easy to use, portable and composable on k8s.

Disclosure: I work at Google on Kubeflow

Re: Kubeflow – Machine Learning Toolkit for Kubernetes

#12

Post by TheIronYuppie »

smooc wrote: post #2

This looks pretty cool. Is it dependent on Google’s kubernetes or can it be run on Openshift or DC/OS as well?

Yep, anywhere k8s runs! (Or that's the idea anyway). If it doesn't, please file a bug!

Disclosure: I work at Google on Kubeflow

Re: Kubeflow – Machine Learning Toolkit for Kubernetes

#13

Post by TheIronYuppie »

sailingparrot wrote: post #9
smooc wrote: post #2

This looks pretty cool. Is it dependent on Google’s kubernetes or can it be run on Openshift or DC/OS as well?

The controller part is a Custom Resource Definition. It can run on any cloud. However, to benefit from GPUs you need to configure the controller correctly. Default configuration are there for GCP and Azure, but you would need to do that manually for other cloud (not that it is very hard)

Correct, anytime you bleed through hardware, it requires some setup, sadly

Disclosure: I work at Google on Kubeflow

Re: Kubeflow – Machine Learning Toolkit for Kubernetes

#14

Post by TheIronYuppie »

ericand wrote: post #4
smooc wrote: post #2

This looks pretty cool. Is it dependent on Google’s kubernetes or can it be run on Openshift or DC/OS as well?

It says it runs in "in any environment in which Kubernetes runs." So as long as you are asking if it runs on Openshift's Kubernetes, than yes.

Absolutely! Redhat are already contributing :)

Disclosure: I work at Google on Kubeflow

Re: Kubeflow – Machine Learning Toolkit for Kubernetes

#15

Post by pacala »

TheIronYuppie wrote: post #7

Hi! I’m David Aronchick, PM on Kubeflow, I’m happy to answer any questions! I was one of the early PMs on Kubernetes, and we very much want to make this a community project, so please join us in thinking about what’s next! - GH: https://GitHub.com/Google/Kubeflow - kubeflow-Discuss: https://groups.google.com/forum/m/#!forum/kubeflow-discuss NOTE: The name "flow" does not refer directly to TensorFlow; if anything it's…

Great to hear this not tied to TensorFlow! How would one use a different DL platform, say PyTorch or DyNet?

Re: Kubeflow – Machine Learning Toolkit for Kubernetes

#16

Post by ethbro »

TheIronYuppie wrote: post #7

Hi! I’m David Aronchick, PM on Kubeflow, I’m happy to answer any questions! I was one of the early PMs on Kubernetes, and we very much want to make this a community project, so please join us in thinking about what’s next! - GH: https://GitHub.com/Google/Kubeflow - kubeflow-Discuss: https://groups.google.com/forum/m/#!forum/kubeflow-discuss NOTE: The name "flow" does not refer directly to TensorFlow; if anything it's…

Kudos on the release, and thanks for your and the team's work!

Any thoughts on this vs managed ml-engine? Cost-aside, seems like this nibbles on the smaller scale "but ml tooling is too hard" use cases?

Re: Kubeflow – Machine Learning Toolkit for Kubernetes

#17

Post by TheIronYuppie »

pacala wrote: post #15
TheIronYuppie wrote: post #7

Hi! I’m David Aronchick, PM on Kubeflow, I’m happy to answer any questions! I was one of the early PMs on Kubernetes, and we very much want to make this a community project, so please join us in thinking about what’s next! - GH: https://GitHub.com/Google/Kubeflow - kubeflow-Discuss: https://groups.google.com/forum/m/#!forum/kubeflow-discuss NOTE: The name "flow" does not refer directly to TensorFlow; if anything it's…

Great to hear this not tied to TensorFlow! How would one use a different DL platform, say PyTorch or DyNet?

The steps would basically be:

- Containerize the DL platform

- Create a k8s manifest (similar to our CRD if necessary)

- Create a service endpoint

- integrate all that into the JH deployment

This is less hard than it sounds, but we'd love help! We only started with TF because that's what we know.

Disclosure: I work at Google on Kubeflow

Re: Kubeflow – Machine Learning Toolkit for Kubernetes

#18

Post by TheIronYuppie »

ethbro wrote: post #16
TheIronYuppie wrote: post #7

Hi! I’m David Aronchick, PM on Kubeflow, I’m happy to answer any questions! I was one of the early PMs on Kubernetes, and we very much want to make this a community project, so please join us in thinking about what’s next! - GH: https://GitHub.com/Google/Kubeflow - kubeflow-Discuss: https://groups.google.com/forum/m/#!forum/kubeflow-discuss NOTE: The name "flow" does not refer directly to TensorFlow; if anything it's…

Kudos on the release, and thanks for your and the team's work! Any thoughts on this vs managed ml-engine? Cost-aside, seems like this nibbles on the smaller scale "but ml tooling is too hard" use cases?

Thank you! We still love Google Cloud ML engine - it's perfect for those who want to run in the cloud and want a layer of abstraction. This is for people who want portable stacks and a bit more control; and/or want to use their Kubernetes deployments (particularly on-premise or multi-purpose).

Does that help?

Disclosure: I work at Googlr on Kubeflow

Re: Kubeflow – Machine Learning Toolkit for Kubernetes

#19

Post by quadrature »

TheIronYuppie wrote: post #10
mountainerd wrote: post #5

Pretty interesting. I'm guessing this is something that Google uses internally for their Kubernetes workflows.

Its very close to how we think about ML internally, but not what we use. Your best bet to read that is look at the TFX paper[1] which describes our internal thoughts in great detail. (Though Kubeflow is not designed to be an externalization of TFX, we're very much working in collaboration with that team) [1] http://www.kdd.org/kdd2017/papers/view/tfx-a-tensorflow-base... Disclosure: I work at Google on Kubeflow

Does google intend on open sourcing TFX ?. I only ask because we're building a lot of the same infrastructure.

Re: Kubeflow – Machine Learning Toolkit for Kubernetes

#20

Post by ecnahc515 »

TheIronYuppie wrote: post #18
ethbro wrote: post #16

Earlier quoted context omitted.

Kudos on the release, and thanks for your and the team's work! Any thoughts on this vs managed ml-engine? Cost-aside, seems like this nibbles on the smaller scale "but ml tooling is too hard" use cases?

Thank you! We still love Google Cloud ML engine - it's perfect for those who want to run in the cloud and want a layer of abstraction. This is for people who want portable stacks and a bit more control; and/or want to use their Kubernetes deployments (particularly on-premise or multi-purpose). Does that help? Disclosure: I work at Googlr on Kubeflow

Do you foresee some kind of integration between Google Cloud ML engine and Tensorflow on k8s in the future?
Post reply on HN