Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Putting GPUs to work with Kubernetes

medium.com

Post reply on HN
Print view View original

51–54 of 54 posts

Re: Putting GPUs to work with Kubernetes

#51

Post by tokenizerrr »

ferrantim wrote: post #46
tokenizerrr wrote: post #20

Earlier quoted context omitted.

So Ceph is the preferred storage provider? I've noticed there is a huge list, including GlusterFS. Do you have experience with any of the other ones?

Ceph is not a good solution at all for databases. Your performance is going to be terrible. Ceph is object-based SDS solution designed to take servers with local drives and create a SAN out of them. In order to do this, they take each LUN (Ceph volume) and scatter the data across all nodes in the cluster. They do not assume that applications will run on these servers themselves... they assume compute is elsewhere, li…

Do you happen to have any suggestions for alternatives to look into?

Re: Putting GPUs to work with Kubernetes

#52

Post by samnco »

puzzle wrote: post #47
puzzle wrote: post #45

Earlier quoted context omitted.

That's weird, because all the times I tried the experimental support, it didn't need privileged containers. From the YAML files, it looks like it's using hostPath directories, but those don't require special privileges, unless you need to write to them: https://kubernetes.io/docs/concepts/storage/volumes/#hostpat... I suspect that there is a bug somewhere.

Ah, wait: https://github.com/madeden/blogposts/blob/master/k8s-gpu-clo... You don't need to mount the /dev entries into the container at all. The experimental support creates them automatically for you when you are using GPU resources. Perhaps it's device nodes, not the libraries that required privileges?

Hello,

OK I gave it a try and you are absolutely right. For the nvidia-smi, I could run it the /dev/nvidia0, which is cool.

I was also able to run it unprivileged. I guess my mistake was to believe the example from the docs and not test without.

Thanks for sharing that, I'll update my charts and the post accordingly.

Re: Putting GPUs to work with Kubernetes

#53

Post by puzzle »

samnco wrote: post #52
puzzle wrote: post #47

Earlier quoted context omitted.

Ah, wait: https://github.com/madeden/blogposts/blob/master/k8s-gpu-clo... You don't need to mount the /dev entries into the container at all. The experimental support creates them automatically for you when you are using GPU resources. Perhaps it's device nodes, not the libraries that required privileges?

Hello, OK I gave it a try and you are absolutely right. For the nvidia-smi, I could run it the /dev/nvidia0, which is cool. I was also able to run it unprivileged. I guess my mistake was to believe the example from the docs and not test without. Thanks for sharing that, I'll update my charts and the post accordingly.

Awesome! Happy to hear that more containers will run without unneeded privileges.

Re: Putting GPUs to work with Kubernetes

#54

Post by ferrantim »

tokenizerrr wrote: post #51
ferrantim wrote: post #46

Earlier quoted context omitted.

Ceph is not a good solution at all for databases. Your performance is going to be terrible. Ceph is object-based SDS solution designed to take servers with local drives and create a SAN out of them. In order to do this, they take each LUN (Ceph volume) and scatter the data across all nodes in the cluster. They do not assume that applications will run on these servers themselves... they assume compute is elsewhere, li…

Do you happen to have any suggestions for alternatives to look into?

You could look at Portworx (disclosure, I work there so am biased, but you can test it yourself for free)
Post reply on HN