Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Kubernetes to 2,500 Nodes

blog.openai.com

Post reply on HN
Print view View original

31–34 of 34 posts

Re: Scaling Kubernetes to 2,500 Nodes

#31

Post by jo909 »

drewrobb wrote: post #27

I'm surprised that the scaling story of k8s/(+etcd?) is still so far behind mesos/zk. There have been mesos clusters at over 10k Nodes for several years now. I have never personally needed more than a few hundred mesos agents, but these have been added without any noticeable impact on our extremely modestly provisioned (and multi purpose) zk cluster or any other components. Has anyone used both systems and can speak…

A lot of the issues were kind of "external" and while worth thinking about for every deployment, not really something the k8s project can do much about other than warn in the documentation.

  - disk latency
  - monitoring queries
  - homemade autoscaler killing all etcd nodes
  - custom scheduling policy moving many kubedns processes to the same node
  - unusually large docker images
  - "sharing" gcr.io request quotas because of Azure NAT IPs
That's not to say that Mesos is not indeed scaling better or easier. I don't know enough about Mesos.

Re: Scaling Kubernetes to 2,500 Nodes

#32

Post by eggie5 »

frakkingcylons wrote: post #23
thesandlord wrote: post #14

Earlier quoted context omitted.

According to the article, they are using NC24 VMS, which have 4 K80s attached. So yes, I would assume they are using GPUs. Check out https://github.com/google/kubeflow if you are interested in doing the same. (Disclaimer: I work for GCP doing K8s stuff, I know GKE clusters support GPUs and Kubeflow, not 100% sure if AKS supports it or if you need to set up your own cluster like OpenAI did.)

I have a somewhat off-topic question as a complete TensorFlow beginner and it seems like you'd be in the know: If I want to train a TF model distributed over many machines in GCP, it seems like I could use Cloud ML Engine or deploy Kubeflow to a K8s cluster running in GKE and train it there. What should I consider when choosing between these two options? Is there another option I should consider?

1. Well deploying a k8s cluster is a huge engineering challenge. 2. And doing distributed training in general is an engineering challenge. 3. And combine the two for an even larger engineering challenge of doing distributed training on k8s using GPUs.

ML Engine is an order of magnitude easier. You just have to do step 2 and setup your model to employ multiple GPUs.

Re: Scaling Kubernetes to 2,500 Nodes

#33

Post by sailingparrot »

eggie5 wrote: post #30
raccoonone wrote: post #19

Earlier quoted context omitted.

Yep, we've been using GPUs for quite a while (even before the alpha support in Kube), both the K80s in Azure and some Pascals in our own clusters. With the support in Kube now it's quite seamless.

That's some groundbreaking work: GPUs in K8s. By Kube you mean the KubeFlow project?

Late reply, but Kube meant Kubernetes not Kubeflow. Alpha GPU support landed in 1.6 if my memory serves me right. Before that you had to do a bunch of stuff manually to make GPU work, mostly around scheduling etc. Since 1.6, Kubernetes will automatically detect the GPUs on your node and thus correctly assign the workloads where they fit. Kubeflow is an abstraction layer on top of that, that helps a lot when you want to do things such as distributed TensorFlow training. It also helps a bit for simpler jobs by (almost) removing the need to manually mount the NVIDIA drivers from the host into the container for example.

Re: Scaling Kubernetes to 2,500 Nodes

#34

Post by henningpeters »

frakkingcylons wrote: post #23
thesandlord wrote: post #14

Earlier quoted context omitted.

According to the article, they are using NC24 VMS, which have 4 K80s attached. So yes, I would assume they are using GPUs. Check out https://github.com/google/kubeflow if you are interested in doing the same. (Disclaimer: I work for GCP doing K8s stuff, I know GKE clusters support GPUs and Kubeflow, not 100% sure if AKS supports it or if you need to set up your own cluster like OpenAI did.)

I have a somewhat off-topic question as a complete TensorFlow beginner and it seems like you'd be in the know: If I want to train a TF model distributed over many machines in GCP, it seems like I could use Cloud ML Engine or deploy Kubeflow to a K8s cluster running in GKE and train it there. What should I consider when choosing between these two options? Is there another option I should consider?

RiseML provides a higher-level abstraction than Kubeflow that is more similar to Google Cloud ML. I would love to get your feedback on our solution: https://riseml.com

Btw: we are currently preparing an open-source release

Disclaimer: I am co-founder at RiseML

Post reply on HN