Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPUs as a service with Kubernetes Engine are now generally available

cloudplatform.googleblog.com

Post reply on HN
Print view View original

11–20 of 24 posts

Re: GPUs as a service with Kubernetes Engine are now generally available

#11

Post by puzzle »

fpgaminer wrote: post #6

On a related note, last week I took a dive into Kubernetes on gcloud for a personal project and came out with some interesting knowledge. First off, this was for a _small_ personal project. Something that I originally intended to run on an f1-micro. I decided to check out Kubernetes mostly to learn, but also to see if it could offer a more maintainable setup (typically I just write a mess of provisioning shell script…

Were the system pods using all that memory or just reserving it? It's not straightforward to scale them, because the node might run just your tiny Rust server or 20 high traffic web apps. You don't want the log agent to keel over just because of the latter. GKE and many other Kubernetes deployments use something called addon-resizer to determine CPU and RAM given to cluster services. The problem is that, typically, it scales based on node count and the settings are usually conservative on the lower end, i.e. your case of a single node. I think it assumes clusters are all at least 10/15 nodes. On a test cluster, I see the metrics server using only 16MB of RAM, but it requests 104MB. Ironically, the autoscaling nanny in the same pod uses another 8MB.

This is a known issue that is not easy to solve in the general case. I think Tim Hockin ran a conversation about how to autoscale on the very low end at last year's Kubecon, with people like you in mind. The other use case he brought up is how to set up services in a Minikube cluster that might be running in a 2GB VM.

Re: GPUs as a service with Kubernetes Engine are now generally available

#12

Post by w1nk »

minimaxir wrote: post #2

With the new discounts on preemptible GPUs ( https://cloudplatform.googleblog.com/2018/06/Introducing-imp... ), the economics of quickly spinning up a fleet of GPUs with Kubernetes for a quick parallelizable ML task become very interesting . (assuming that Google allows enough GPU quota for a fleet of GPUs for nonenterprise users anyways) What I want to use Kubernetes + instant-GPU-fleet for deep learning hyperparame…

Out of curiosity, why are you grid searching if you have access to google's infrastructure (vizier, etc).

Re: GPUs as a service with Kubernetes Engine are now generally available

#13

Post by kozikow »

We have been using GPUs with GKE for a while. At some point, we used 20+ GPUs in an production workflow without any problems.

Everything generally works well, maybe except the initial phase when some containers won't port well from nvidia-docker-compose due to problems with Cuda libraries. Ideally, you need to match the version of Cuda everywhere.

My dev setup for quick experimentation with GPU docker container on GKE: https://tensorflight.blog/2018/02/23/dev-environment-for-gke... .

Re: GPUs as a service with Kubernetes Engine are now generally available

#14

Post by minimaxir »

w1nk wrote: post #12
minimaxir wrote: post #2

With the new discounts on preemptible GPUs ( https://cloudplatform.googleblog.com/2018/06/Introducing-imp... ), the economics of quickly spinning up a fleet of GPUs with Kubernetes for a quick parallelizable ML task become very interesting . (assuming that Google allows enough GPU quota for a fleet of GPUs for nonenterprise users anyways) What I want to use Kubernetes + instant-GPU-fleet for deep learning hyperparame…

Out of curiosity, why are you grid searching if you have access to google's infrastructure (vizier, etc).

You mean Cloud ML Engine? (https://cloud.google.com/ml-engine/)

Cloud ML Engine is a bit opaque in terms of price efficiency (how powerful is a "training unit"?) but I suppose that's another option.

Re: GPUs as a service with Kubernetes Engine are now generally available

#15

Post by w1nk »

minimaxir wrote: post #14
w1nk wrote: post #12

Earlier quoted context omitted.

Out of curiosity, why are you grid searching if you have access to google's infrastructure (vizier, etc).

You mean Cloud ML Engine? ( https://cloud.google.com/ml-engine/ ) Cloud ML Engine is a bit opaque in terms of price efficiency (how powerful is a "training unit"?) but I suppose that's another option.

Hm, actually sorry I assumed they had made vizier available as a blackbox service (https://static.googleusercontent.com/media/research.google.c...). If they haven't, you can save money/time by employing better than grid search. There are some re implementations of vizier available, I'll see if I can make ours too (email in profile). I do agree with that billing can be..complex ;)

Re: GPUs as a service with Kubernetes Engine are now generally available

#16

Post by minimaxir »

w1nk wrote: post #15
minimaxir wrote: post #14

Earlier quoted context omitted.

You mean Cloud ML Engine? ( https://cloud.google.com/ml-engine/ ) Cloud ML Engine is a bit opaque in terms of price efficiency (how powerful is a "training unit"?) but I suppose that's another option.

Hm, actually sorry I assumed they had made vizier available as a blackbox service ( https://static.googleusercontent.com/media/research.google.c... ). If they haven't, you can save money/time by employing better than grid search. There are some re implementations of vizier available, I'll see if I can make ours too (email in profile). I do agree with that billing can be..complex ;)

From the paper synopsis, it's implemented as a part of HyperTune. (https://cloud.google.com/ml-engine/docs/tensorflow/using-hyp...)

Re: GPUs as a service with Kubernetes Engine are now generally available

#17

Post by jamesblonde »

minimaxir wrote: post #2

With the new discounts on preemptible GPUs ( https://cloudplatform.googleblog.com/2018/06/Introducing-imp... ), the economics of quickly spinning up a fleet of GPUs with Kubernetes for a quick parallelizable ML task become very interesting . (assuming that Google allows enough GPU quota for a fleet of GPUs for nonenterprise users anyways) What I want to use Kubernetes + instant-GPU-fleet for deep learning hyperparame…

If you are adventurous enough to try a more user-friendly platform for Distributed TensorFlow (Horovod, grid-search using Spark/TensorFlow), have a look at http://www.hops.io (watch video on front page). In Hops, you don't program infrastructure (no YAML file, no Dockerfiles). You can run with 100s of GPUs for grid search using this python code:

  def train(param1, param2, ..):
    # tf code here (import any libraries here)

  dict = { 'lr': [0.001, 0.0001, 0.00001], 'dropout': [0.4, 0.6, 0.8]}
  experiment.launch(spark, train, dict)
The reason this works on clusters is that there is a persistent conda environment installed on all hosts in the cluster for every 'project' (think Github project). The above function 'train' is a pyspark mapper and the python libraries needed in it are found in the local conda environment. You install conda libraries by search/click in a UI, not by writing a Dockerfile. You create your cluster by selecting how many GPUs/memory you need in a UI, not by writing a YML file.

Diclosure: i work on this project.

Re: GPUs as a service with Kubernetes Engine are now generally available

#18

Post by jfoutz »

brianwawok wrote: post #8
fpgaminer wrote: post #6

On a related note, last week I took a dive into Kubernetes on gcloud for a personal project and came out with some interesting knowledge. First off, this was for a _small_ personal project. Something that I originally intended to run on an f1-micro. I decided to check out Kubernetes mostly to learn, but also to see if it could offer a more maintainable setup (typically I just write a mess of provisioning shell script…

Sorry, this is not a use case they should worry about. Running a $4 server to play with it. You can use minikube for that. Many of us really really really love GKE and get a lot of use out of it. And yes, I run it myself on about 1h of work a week (no dedicated devops) for my entire startup.

Not sure if they're still doing it, but it was trivial to get a $300 credit to spin up enough hosts to make sense.

the great Kelsey Hightower's tutorials take advantage of the credits. https://github.com/kelseyhightower/kubernetes-the-hard-way/b...

Re: GPUs as a service with Kubernetes Engine are now generally available

#19

Post by ethanwillis »

erikb wrote: post #5

How about making vanilla k8s usable on-premise first...

I have a lot of bare metal servers in a rack at my home.. Tried getting kube running well last night actually.. Well if you're using Ubuntu server 18.04 GOOD LUCK. There's plenty of issues in the various repositories around kube. I eventually just tried using Canonical's conjure-up tool to install kube.

You'd think that Canonical's tools would work on their LTS right? wrong.. absolutely wrong. It couldn't even do an openstack install either. Absolutely abysmal.

Re: GPUs as a service with Kubernetes Engine are now generally available

#20

Post by fpgaminer »

wstrange wrote: post #10
fpgaminer wrote: post #6

On a related note, last week I took a dive into Kubernetes on gcloud for a personal project and came out with some interesting knowledge. First off, this was for a _small_ personal project. Something that I originally intended to run on an f1-micro. I decided to check out Kubernetes mostly to learn, but also to see if it could offer a more maintainable setup (typically I just write a mess of provisioning shell script…

A PaaS solution (like App Engine) would be far more appropriate for what you want to do. Unless your goal is to learn Kubernetes - in which case minikube works just fine. W.r.t. to the GCP console and tools - I guess it's a preference thing - but I vastly prefer them over the AWS tools. They work fine for me. I like the feature in the GUI where it shows you the equivalent gcloud command line.

I love App Engine and have used it in the past, but it: 1) is quite old and receives precious little love from Google; 2) isn't great if you want to use a relational database; 3) not a great option for applications that require iron clad security.

To be clear on #1, App Engine has been nothing but reliable for me. Yet it receives few updates; for example, only supporting Python 2.7...

#2: It works great with Datastore, but for SQL you have to use a separate instance or Cloud SQL; either will cost additional money and maintenance. And, last I checked, Postgres was a no-go for App Engine.

#3: It can be hard to secure App Engine apps properly. User data leaking in the logs, for example. And I've encountered a few bugs that lead me to distrust the runtime they use. (I reported the bugs, but still). Where security is of the utmost importance, I have to opt for my own stack.

[This is all Standard Environment. Flexible is brutally expensive.]

Post reply on HN