"1060GTX at home but on consumer grade Intel NUC"
A bit OT, but I'd like to see how this works...
Ah, very cool - https://www.youtube.com/watch?v=wyY-lTmgb8c
Putting GPUs to work with Kubernetes
21–30 of 54 posts
Re: Putting GPUs to work with Kubernetes
#22Earlier quoted context omitted.
You have several options: * Run Ceph in separate nodes and connect it to the cluster. With Juju, you can do that from the bundle, as Ceph is also a supported workloads. This gives you scale for storage * Run Ceph within the cluster with a Helm chart. We see that for openstack-helm for example. Also gives you scale, but the lack of device discovery makes it less interesting in my opinion * Run an NFS server, plain eas…
So Ceph is the preferred storage provider? I've noticed there is a huge list, including GlusterFS. Do you have experience with any of the other ones?
Ah you are right, I forgot about glusterfs. My bad.
Canonical at this stage only supports Ceph commercially, but it doesn't mean GlusterFS is not a good option. I haven't tried it myself, so can't tell.
Anyone?
Re: Putting GPUs to work with Kubernetes
#23Earlier quoted context omitted.
You have several options for this. If it is non HA, then you can pin a RC to a specific node, and use hostpath storage. if the container fails, it will always respawn on the same node, maximizing uptime and also having max capacity from your local SSD. Alternatively, you can also run rook, which is backed by Ceph, and use affinity to make sure that your pods are very close to storage, and gain back some of the speed.…
Thanks for the answer. How easy will it be to transfer this cluster to another set of servers (with data copy)? Like, stop the service for several minutes, push the button "Transfer" and start service on new servers after that. As I understand you'll need rook for something like that?
Typically, you would have a set of "helm charts" (packages) for your application(s).
So deploying, without data, would be something like a sequence of "helm install app-appId --values /path/to/config/for/environment.yaml"
At this stage, you have no data. Normally, if you app depends on data, the pods will keep failing until data has been moved and is available (at which point they stabilize in an equilibrium state).
If you run Ceph in both environments, then you may want to use Ceph replication. If you use another storage layer, then it's also up to you to make sure the data is moved around.
rook should have support for that use case. However, it is still alpha as per the GitHub readme, so use at your own risk. However, you may want to consider the data replication problem outside of the scope of k8s.
If the 2 sets of clusters are physically close to each other, you may want to just point the new apps to the old data and pull the switch from the first one.
Another option would be to run another beta feature in K8s called Federation, which allows to manage several cluster via a single control plane.
Sorry for the long comment, your question has a broad scope, and it's hard to answer without diving into details.
Re: Putting GPUs to work with Kubernetes
#24"1060GTX at home but on consumer grade Intel NUC" A bit OT, but I'd like to see how this works... Ah, very cool - https://www.youtube.com/watch?v=wyY-lTmgb8c
Actually, it was a fun DIY project I did a while ago. You can read about it here: https://hackernoon.com/installing-a-diy-bare-metal-gpu-clust...
It works, but the GPUs aren't very stable at 4x vs. a normal 16x.
Re: Putting GPUs to work with Kubernetes
#25"1060GTX at home but on consumer grade Intel NUC" A bit OT, but I'd like to see how this works... Ah, very cool - https://www.youtube.com/watch?v=wyY-lTmgb8c
Actually, it was a fun DIY project I did a while ago. You can read about it here: https://hackernoon.com/installing-a-diy-bare-metal-gpu-clust... It works, but the GPUs aren't very stable at 4x vs. a normal 16x.
That's one problem, another is the size of the powersupply. And maybe that's the only problem, I don't see why a GPU would become unstable when using fewer lanes, all it should do is get slower.
Re: Putting GPUs to work with Kubernetes
#26Earlier quoted context omitted.
Actually, it was a fun DIY project I did a while ago. You can read about it here: https://hackernoon.com/installing-a-diy-bare-metal-gpu-clust... It works, but the GPUs aren't very stable at 4x vs. a normal 16x.
That's one problem, another is the size of the powersupply. And maybe that's the only problem, I don't see why a GPU would become unstable when using fewer lanes, all it should do is get slower.
I don't know. Maybe the make of the extenders isn't very good, I saw other people with similar issues.
The PSU is the Corsair AX1500i (1500W), with 10x lines for GPUs. It's robust on paper, didn't have any problem with just 4 plugged in.
But I must say... The T630 are very noisy compared to these, but so much more powerful #NotGoingBack
Re: Putting GPUs to work with Kubernetes
#27Earlier quoted context omitted.
You have several options: * Run Ceph in separate nodes and connect it to the cluster. With Juju, you can do that from the bundle, as Ceph is also a supported workloads. This gives you scale for storage * Run Ceph within the cluster with a Helm chart. We see that for openstack-helm for example. Also gives you scale, but the lack of device discovery makes it less interesting in my opinion * Run an NFS server, plain eas…
So Ceph is the preferred storage provider? I've noticed there is a huge list, including GlusterFS. Do you have experience with any of the other ones?
Gluster is a good alternative, Ceph can be prickly if you don't want a block device and instead just need a filesystem. CephFS fills sort of the same role, but does so on top of Ceph rbd.
Since GlusterFS /can use/ NFSv4 as a client, it should work with the stuff @samco_23 uses
Re: Putting GPUs to work with Kubernetes
#28Earlier quoted context omitted.
Thanks for the answer. How easy will it be to transfer this cluster to another set of servers (with data copy)? Like, stop the service for several minutes, push the button "Transfer" and start service on new servers after that. As I understand you'll need rook for something like that?
Typically, you would have a set of "helm charts" (packages) for your application(s). So deploying, without data, would be something like a sequence of "helm install app-appId --values /path/to/config/for/environment.yaml" At this stage, you have no data. Normally, if you app depends on data, the pods will keep failing until data has been moved and is available (at which point they stabilize in an equilibrium state).…
Thanks for your explanation.
Looks like k8s is designed by Google with Google in mind - where data is somewhere over network and main problem is to manage code. I just thought it could be perfect solution to completely abstract and isolate such small services and move them around on local hardware with one push of a button, but it seems I'll have to think about external ways to manage data replication.
Re: Putting GPUs to work with Kubernetes
#29Earlier quoted context omitted.
That's one problem, another is the size of the powersupply. And maybe that's the only problem, I don't see why a GPU would become unstable when using fewer lanes, all it should do is get slower.
I don't know. Maybe the make of the extenders isn't very good, I saw other people with similar issues. The PSU is the Corsair AX1500i (1500W), with 10x lines for GPUs. It's robust on paper, didn't have any problem with just 4 plugged in. But I must say... The T630 are very noisy compared to these, but so much more powerful #NotGoingBack
I just bought a GTX1080ti + a similar corsair as an upgrade for my 3 year old Dell, it works like a charm.
If you have a PSU that big then that probably isn't the problem. I thought you might be using the PSU that comes with those extender boxes and they usually are very puny (250 W or so).
Do you use it for gaming or for CUDA?
Do you run the 4 GPUs in the extender?
Re: Putting GPUs to work with Kubernetes
#30Earlier quoted context omitted.
I don't know. Maybe the make of the extenders isn't very good, I saw other people with similar issues. The PSU is the Corsair AX1500i (1500W), with 10x lines for GPUs. It's robust on paper, didn't have any problem with just 4 plugged in. But I must say... The T630 are very noisy compared to these, but so much more powerful #NotGoingBack
I just bought a GTX1080ti + a similar corsair as an upgrade for my 3 year old Dell, it works like a charm. If you have a PSU that big then that probably isn't the problem. I thought you might be using the PSU that comes with those extender boxes and they usually are very puny (250 W or so). Do you use it for gaming or for CUDA? Do you run the 4 GPUs in the extender?
Yes, each GPUs has a 4x -> 16x and a 4x-4x extender, in addition to the m.2 -> PCI-e 4x adapter.
So many potential failure points in there. The sole use case is CUDA. Essentially I wanted a portable cluster with GPUs and that did the work for a couple of month. Now it's getting more serious so the switch to T630 makes sense, and I repurposed the NUCs into the control plane of the K8s cluster.