Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Putting GPUs to work with Kubernetes

medium.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Putting GPUs to work with Kubernetes

#41

Post by puzzle »

samnco wrote: post #39
puzzle wrote: post #32

Earlier quoted context omitted.

What is the requirement for privileged containers? The post never explains it.

privileged containers are required for the GPU to be shared with the containers. By default, the bundle come with a "auto" tag, which will activate privileged containers just when GPUs are detected. You can enforce "false" to remove that, but then you won't be able to run GPU workloads. Or you can enforce "yes" and have them activated all the time. Does that answer the question? Not sure if I understood it right.

The Kubernetes docs don't say anything about having to use privileged containers for GPU support. Privileged containers are given tens of Linux capabilities; which of those are actually needed in your setup? Or, conversely, which specific step would fail for an unprivileged container?

Just because I want to use a GPU shouldn't require the power to change the clock, switch UIDs, chown files, mess with logs, reboot the machine, etc.

Re: Putting GPUs to work with Kubernetes

#42

Post by aquadrop »

ferrantim wrote: post #37
aquadrop wrote: post #19

Earlier quoted context omitted.

Thanks for the answer. How easy will it be to transfer this cluster to another set of servers (with data copy)? Like, stop the service for several minutes, push the button "Transfer" and start service on new servers after that. As I understand you'll need rook for something like that?

You can use Portworx[1] to transfer data between servers and datacenters. It has a native kubernetes driver, is production ready being used by GE and Lufthansa airlines among others and does sync and async replication of data between environments. (disclosure I work for Portworx). [1] https://portworx.com/

Yes, looks interesting, but seems enterprisey and not open source?

Re: Putting GPUs to work with Kubernetes

#43

Post by marcoceppi »

puzzle wrote: post #41
samnco wrote: post #39

Earlier quoted context omitted.

privileged containers are required for the GPU to be shared with the containers. By default, the bundle come with a "auto" tag, which will activate privileged containers just when GPUs are detected. You can enforce "false" to remove that, but then you won't be able to run GPU workloads. Or you can enforce "yes" and have them activated all the time. Does that answer the question? Not sure if I understood it right.

The Kubernetes docs don't say anything about having to use privileged containers for GPU support. Privileged containers are given tens of Linux capabilities; which of those are actually needed in your setup? Or, conversely, which specific step would fail for an unprivileged container? Just because I want to use a GPU shouldn't require the power to change the clock, switch UIDs, chown files, mess with logs, reboot the…

Since the GPU libraries are hosted on the node, privileged flag is typically required to make that possible. I'm sure there will be improvements to not require privileged, but today it's mostly a requirement to get anything useful out of containers tapping into GPU.

That said, if you set the allow-privileged flag to false GPU drivers will still be installed but you may not be able to make use of the cuda cores

Re: Putting GPUs to work with Kubernetes

#44

Post by yebyen »

aquadrop wrote: post #40
yebyen wrote: post #34

Earlier quoted context omitted.

So, your thought process has apparently led you quickly to the same conundrum that I have found myself in, after experimenting with this stuff for several generations. You might be interested in Deis v1 PAAS as a historical reference. Deis is a company that specializes in Kubernetes (was just bought by Microsoft). They have been in the container orchestration game since before Kubernetes was a kid (and even before co…

Thanks, I know that distributed storage is hard, that's why I would be ok if K8s could just work with something like docker compose volumes on local storage and copy them between servers if needed.

It would be cool if Kubernetes had a native distributed filesystem. I don't read the future roadmaps but I wouldn't be too surprised to see it coming in a future release.

The first thing that anyone doing serious deployments needs is an image registry. For that to be HA hosted on a cluster, you need some kind of distributed filesystem.

But those PD/EBS solutions are pretty compelling and they're not going away.

Re: Putting GPUs to work with Kubernetes

#45

Post by puzzle »

marcoceppi wrote: post #43
puzzle wrote: post #41

Earlier quoted context omitted.

The Kubernetes docs don't say anything about having to use privileged containers for GPU support. Privileged containers are given tens of Linux capabilities; which of those are actually needed in your setup? Or, conversely, which specific step would fail for an unprivileged container? Just because I want to use a GPU shouldn't require the power to change the clock, switch UIDs, chown files, mess with logs, reboot the…

Since the GPU libraries are hosted on the node, privileged flag is typically required to make that possible. I'm sure there will be improvements to not require privileged, but today it's mostly a requirement to get anything useful out of containers tapping into GPU. That said, if you set the allow-privileged flag to false GPU drivers will still be installed but you may not be able to make use of the cuda cores

That's weird, because all the times I tried the experimental support, it didn't need privileged containers. From the YAML files, it looks like it's using hostPath directories, but those don't require special privileges, unless you need to write to them:

https://kubernetes.io/docs/concepts/storage/volumes/#hostpat...

I suspect that there is a bug somewhere.

Re: Putting GPUs to work with Kubernetes

#46

Post by ferrantim »

tokenizerrr wrote: post #20
samnco wrote: post #15

Earlier quoted context omitted.

You have several options: * Run Ceph in separate nodes and connect it to the cluster. With Juju, you can do that from the bundle, as Ceph is also a supported workloads. This gives you scale for storage * Run Ceph within the cluster with a Helm chart. We see that for openstack-helm for example. Also gives you scale, but the lack of device discovery makes it less interesting in my opinion * Run an NFS server, plain eas…

So Ceph is the preferred storage provider? I've noticed there is a huge list, including GlusterFS. Do you have experience with any of the other ones?

Ceph is not a good solution at all for databases. Your performance is going to be terrible.

Ceph is object-based SDS solution designed to take servers with local drives and create a SAN out of them. In order to do this, they take each LUN (Ceph volume) and scatter the data across all nodes in the cluster. They do not assume that applications will run on these servers themselves... they assume compute is elsewhere, like a traditional SAN. The goal is to replace a SAN with servers, not create a converged platform. Also, Ceph was designed during an age where an Intel server did NOT have tier-1 capacity (8 - 20 TB), which is why they shard a volume across so many servers.

This causes a problem for modern applications like Cassandra, Mongo, Kafka etc, where they like to scaleout themselves and want a converged system, where data is not scattered, but on the node where an instance of that cluster runs. Ceph also disrupts (undo) the HA capabilities that these scaleout applications have (For example, a Cassandra instances data will not be on a node on which it thinks it is).

Re: Putting GPUs to work with Kubernetes

#47

Post by puzzle »

puzzle wrote: post #45
marcoceppi wrote: post #43

Earlier quoted context omitted.

Since the GPU libraries are hosted on the node, privileged flag is typically required to make that possible. I'm sure there will be improvements to not require privileged, but today it's mostly a requirement to get anything useful out of containers tapping into GPU. That said, if you set the allow-privileged flag to false GPU drivers will still be installed but you may not be able to make use of the cuda cores

That's weird, because all the times I tried the experimental support, it didn't need privileged containers. From the YAML files, it looks like it's using hostPath directories, but those don't require special privileges, unless you need to write to them: https://kubernetes.io/docs/concepts/storage/volumes/#hostpat... I suspect that there is a bug somewhere.

Ah, wait:

https://github.com/madeden/blogposts/blob/master/k8s-gpu-clo...

You don't need to mount the /dev entries into the container at all. The experimental support creates them automatically for you when you are using GPU resources. Perhaps it's device nodes, not the libraries that required privileges?

Re: Putting GPUs to work with Kubernetes

#48

Post by samnco »

puzzle wrote: post #47
puzzle wrote: post #45

Earlier quoted context omitted.

That's weird, because all the times I tried the experimental support, it didn't need privileged containers. From the YAML files, it looks like it's using hostPath directories, but those don't require special privileges, unless you need to write to them: https://kubernetes.io/docs/concepts/storage/volumes/#hostpat... I suspect that there is a bug somewhere.

Ah, wait: https://github.com/madeden/blogposts/blob/master/k8s-gpu-clo... You don't need to mount the /dev entries into the container at all. The experimental support creates them automatically for you when you are using GPU resources. Perhaps it's device nodes, not the libraries that required privileges?

Aaah that is interesting. Let me dive into this later today and test my charts without that. It would actually make my life way easier for charting. I got that from a very early stage work and never questioned it again (the /dev stuff). Thanks for pointing that out.

Re: Putting GPUs to work with Kubernetes

#49

Post by ferrantim »

aquadrop wrote: post #42
ferrantim wrote: post #37

Earlier quoted context omitted.

You can use Portworx[1] to transfer data between servers and datacenters. It has a native kubernetes driver, is production ready being used by GE and Lufthansa airlines among others and does sync and async replication of data between environments. (disclosure I work for Portworx). [1] https://portworx.com/

Yes, looks interesting, but seems enterprisey and not open source?

Correct, it is not opensource but you can use it for free on up to 3 nodes. Not sure what "enterprisey" means but to me that sounds like it works well for large deployments and that is true. We have customers running us with Mesos on up to 1000 nodes and K8s on hundreds of nodes.

Re: Putting GPUs to work with Kubernetes

#50

Post by aquadrop »

ferrantim wrote: post #49
aquadrop wrote: post #42

Earlier quoted context omitted.

Yes, looks interesting, but seems enterprisey and not open source?

Correct, it is not opensource but you can use it for free on up to 3 nodes. Not sure what "enterprisey" means but to me that sounds like it works well for large deployments and that is true. We have customers running us with Mesos on up to 1000 nodes and K8s on hundreds of nodes.

"Request a demo" and no pricing is what I mean with "enterprisey".
Post reply on HN