Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Kubernetes to 2,500 Nodes

blog.openai.com

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Scaling Kubernetes to 2,500 Nodes

#21

Post by scarface74 »

merb wrote: post #20

what I find amazing about k8s is that it's one of the first solution that is relativly simple for a small cluster (HA, while schedule stuff on the masters), but can scale amazingly well even for a big cluster. you can start with 3 nodes with like 8gb per machine (or less, I guess even 2gb is feasible if you only want to use like 1-1,5gb of memory per machine). (non ha can of course be smaller)

The Nomad executable is self contained and less than 15MB if I remember correctly. It can be used with Docker containers, shell scripts, or just raw executables.

https://www.hashicorp.com/c1m.html

It was dead simple to install and use compared to my brief experience with k8s.

Re: Scaling Kubernetes to 2,500 Nodes

#22

Post by myrandomcomment »

Ran into the ARP scale issues when trying to put 1000 containers on a system for scale testing over year ago. strace helped figure out where the issues was and what settings to change. I guess I should have sent an email to the mailing list. At that time if you searched for scaling to 1000 docker contains was a failed search, as it was "hey here is how I scaled to 1000 containers over X numbers of nodes". No one was crazy enough to try to get 1000 on a single machine.

Re: Scaling Kubernetes to 2,500 Nodes

#23

Post by frakkingcylons »

thesandlord wrote: post #14
eggie5 wrote: post #12

Does OpenAI train w/ GPUs on k8s clusters?

According to the article, they are using NC24 VMS, which have 4 K80s attached. So yes, I would assume they are using GPUs. Check out https://github.com/google/kubeflow if you are interested in doing the same. (Disclaimer: I work for GCP doing K8s stuff, I know GKE clusters support GPUs and Kubeflow, not 100% sure if AKS supports it or if you need to set up your own cluster like OpenAI did.)

I have a somewhat off-topic question as a complete TensorFlow beginner and it seems like you'd be in the know:

If I want to train a TF model distributed over many machines in GCP, it seems like I could use Cloud ML Engine or deploy Kubeflow to a K8s cluster running in GKE and train it there.

What should I consider when choosing between these two options? Is there another option I should consider?

Re: Scaling Kubernetes to 2,500 Nodes

#24

Post by emmelaich »

roscoebeezie wrote: post #9

As a person who doesn’t understand containers, where do I go to learn the basics?

The defining characteristic of containers are constrained processes (constrained by one or more of memory, cpu, access controls..); isolated namespaces (your own tmp, root, users, process list) plus some (possibly limited) additional isolation from the rest of the hosts files and hardware.

Docker adds to this 1. a packaging that lets you define what goes into a container (Dockerfile) and format (docker image) - which can be downloaded, extracted, manipulated and uploaded. 2. a way to stop (freeze) and start (thaw) a container 3. tools for controlling network capabilities within a container and between the host and container or other containers on other hosts.

Those are the essentials.

Kubernetes (and other tools) expand on this in terms of orchestration -- especially the internetworking aspect but also failover and load balancing.

Re: Scaling Kubernetes to 2,500 Nodes

#25

Post by rarudduck »

brianwawok wrote: post #6
bdburns wrote: post #4

Earlier quoted context omitted.

VMs in all clouds are always tied to specific machines. If that machine fails unexpectedly then those VMs will restart. If it is a controlled reboot (e.g. host update) then they may not restart...

Well, at least in Google Cloud for planned updates you can get your VM host migrated and not lose a Node due to a planned maintenance. I am not aware if Azure supports this, but my guess is they do not.

They (Azure) do for most operations.

https://docs.microsoft.com/en-us/azure/virtual-machines/wind...

I've found actual reboots to be rare - the exception being the recent Spectre / Meltdown patching.

Re: Scaling Kubernetes to 2,500 Nodes

#26

Post by oso2k »

roscoebeezie wrote: post #9

As a person who doesn’t understand containers, where do I go to learn the basics?

DISCLAIMER: I'm a Consulting Architect at Red Hat in our Container and PaaS Practice Group. An expert, knowledge building, & community building/supporting group. A Community of Practice, we call it.

We (Red Hat) make the following references (beyond what our training & docs provide) available to our consultants, customers, and world at-large. BTW, if something isn't clear, is wrong, or you want to discuss a point, reach out to us on GitHub. Just about all of our products, software, and documentation are up on GitHub.

I'd also recommend playing with Minishift or Minikube. Great way to put a quick sandbox on your laptop.

The source GitHub Repo: https://github.com/redhat-cop/openshift-playbooks

Building Blocks of OpenShift (& Kubernetes): http://v1.uncontained.io/playbooks/fundamentals/building_blo...

Docker Fundamentals Reference: http://v1.uncontained.io/playbooks/fundamentals/docker_refer...

Minishift: https://docs.openshift.org/latest/minishift/getting-started/...

Re: Scaling Kubernetes to 2,500 Nodes

#27

Post by drewrobb »

I'm surprised that the scaling story of k8s/(+etcd?) is still so far behind mesos/zk. There have been mesos clusters at over 10k Nodes for several years now.

I have never personally needed more than a few hundred mesos agents, but these have been added without any noticeable impact on our extremely modestly provisioned (and multi purpose) zk cluster or any other components.

Has anyone used both systems and can speak to any advantages of k8s for these types of workloads?

Also is anyone using some kind of torrent approach as a more reasonable solution to avoid network bottlenecks when distributing big docker images to a large number of nodes?

Re: Scaling Kubernetes to 2,500 Nodes

#29

Post by justingood »

EDevil wrote: post #28

Isn’t it a problem to have etcd store its state on a non persistent volume? How do they recover it after a restart? I suppose it's not a manual process.

The replacement machine will start pulling its data from the remaining nodes when it joins the cluster. However, it's recommended to migrate the failed node's data first if it's greater than 50MB: https://github.com/coreos/etcd/blob/master/Documentation/op-...

Re: Scaling Kubernetes to 2,500 Nodes

#30

Post by eggie5 »

raccoonone wrote: post #19
eggie5 wrote: post #12

Does OpenAI train w/ GPUs on k8s clusters?

Yep, we've been using GPUs for quite a while (even before the alpha support in Kube), both the K80s in Azure and some Pascals in our own clusters. With the support in Kube now it's quite seamless.

That's some groundbreaking work: GPUs in K8s. By Kube you mean the KubeFlow project?
Post reply on HN