GPUs as a service with Kubernetes Engine are now generally available • Hacker News Board

Re: GPUs as a service with Kubernetes Engine are now generally available

#21

Post by fpgaminer » Tue, Jun 19, 2018, 10:33 PM UTC

puzzle wrote: post #11

fpgaminer wrote: post #6

On a related note, last week I took a dive into Kubernetes on gcloud for a personal project and came out with some interesting knowledge. First off, this was for a _small_ personal project. Something that I originally intended to run on an f1-micro. I decided to check out Kubernetes mostly to learn, but also to see if it could offer a more maintainable setup (typically I just write a mess of provisioning shell script…

Were the system pods using all that memory or just reserving it? It's not straightforward to scale them, because the node might run just your tiny Rust server or 20 high traffic web apps. You don't want the log agent to keel over just because of the latter. GKE and many other Kubernetes deployments use something called addon-resizer to determine CPU and RAM given to cluster services. The problem is that, typically, i…

> Were the system pods using all that memory or just reserving it?

700MB was the sum of all the requested minimum RAM for all those service pods. So yeah, you're probably right that they're ceilings of sorts. Still it's a bit crazy to see a logging service, who's job is merely to haul logs off to a different server, requesting 200MB.

I'm also bewildered by Container Optimized OS's memory consumption. IIRC it was 500MB+ bare; doing nothing. As reported by top. I forget which, but I stood up either Debian Stretch or Ubuntu 18.04 and it was only ~200MB with Docker installed.

Re: GPUs as a service with Kubernetes Engine are now generally available

#22

Post by puzzle » Tue, Jun 19, 2018, 11:06 PM UTC

fpgaminer wrote: post #21

puzzle wrote: post #11

Earlier quoted context omitted.

Were the system pods using all that memory or just reserving it? It's not straightforward to scale them, because the node might run just your tiny Rust server or 20 high traffic web apps. You don't want the log agent to keel over just because of the latter. GKE and many other Kubernetes deployments use something called addon-resizer to determine CPU and RAM given to cluster services. The problem is that, typically, i…

> Were the system pods using all that memory or just reserving it? 700MB was the sum of all the requested minimum RAM for all those service pods. So yeah, you're probably right that they're ceilings of sorts. Still it's a bit crazy to see a logging service, who's job is merely to haul logs off to a different server, requesting 200MB. I'm also bewildered by Container Optimized OS's memory consumption. IIRC it was 500M…

The logging service numbers are easily explained: the remote server might have transient failures, so the forwarder will cache stuff in memory (the alternative is to just stop reading the logs, but then you risk losing messages if the pod dies in the meantime). You complained about Go, but it doesn't help that the fluentd agent is written in Ruby. There's a new Go rewrite of it, but I don't think GKE or others use it yet.

Was COS a standalone GCE instance or GKE? In the latter case, memory will be used by the usual suspects: fluentd, kubelet, kube-proxy, docker, node-problem-detector. For both, there are also a few Google daemons in Python (ugh).

Re: GPUs as a service with Kubernetes Engine are now generally available

#23

Post by erikb » Wed, Jun 20, 2018, 8:14 AM UTC

ethanwillis wrote: post #19

erikb wrote: post #5

How about making vanilla k8s usable on-premise first...

I have a lot of bare metal servers in a rack at my home.. Tried getting kube running well last night actually.. Well if you're using Ubuntu server 18.04 GOOD LUCK. There's plenty of issues in the various repositories around kube. I eventually just tried using Canonical's conjure-up tool to install kube. You'd think that Canonical's tools would work on their LTS right? wrong.. absolutely wrong. It couldn't even do an…

Yep and there are altogether problems that come up after you got it running. E.g. you probably want to make use of PVCs, but there is no stable dynamic-storage provider yet. Google isn't even working on it as far as github is showing.

Re: GPUs as a service with Kubernetes Engine are now generally available

#24

Post by thesandlord » Wed, Jun 20, 2018, 10:33 PM UTC

fpgaminer wrote: post #20

wstrange wrote: post #10

Earlier quoted context omitted.

A PaaS solution (like App Engine) would be far more appropriate for what you want to do. Unless your goal is to learn Kubernetes - in which case minikube works just fine. W.r.t. to the GCP console and tools - I guess it's a preference thing - but I vastly prefer them over the AWS tools. They work fine for me. I like the feature in the GUI where it shows you the equivalent gcloud command line.

I love App Engine and have used it in the past, but it: 1) is quite old and receives precious little love from Google; 2) isn't great if you want to use a relational database; 3) not a great option for applications that require iron clad security. To be clear on #1, App Engine has been nothing but reliable for me. Yet it receives few updates; for example, only supporting Python 2.7... #2: It works great with Datastor…

1) Node.js was just released. Python 3 is in alpha. Definitely check it out! There is a brand new sandbox that allows us to ship new runtimes much faster now.

2) True, but you would need this no matter where you are running right? Also, I think the new sandbox runtimes (Java8, Node.js, Python3) should support Postgres.

3) Leaking data in the logs can happen with any app though? What in particular are you seeing? WRT sandbox, the new sandbox should be a lot more robust as well.

I'd definitely give App Engine another shot, maybe in the near future. We are definitely investing a lot on it, I'm sorry it hasn't felt that way for a while.

(I work for GCP)