Scaling Kubernetes to 2,500 Nodes • Hacker News Board

Re: Scaling Kubernetes to 2,500 Nodes

#11

Post by btmiller » Thu, Jan 18, 2018, 9:23 PM UTC

roscoebeezie wrote: post #9

As a person who doesn’t understand containers, where do I go to learn the basics?

At the fundamental layer, start with cgroups and namespaces. If you're looking for practical knowledge, start with making and running a Docker image - usually a basic "hello world" web server (nginx) is a good one to start on :)

Re: Scaling Kubernetes to 2,500 Nodes

#12

Post by eggie5 » Thu, Jan 18, 2018, 9:25 PM UTC

Does OpenAI train w/ GPUs on k8s clusters?

Re: Scaling Kubernetes to 2,500 Nodes

#13

Post by djb_hackernews » Thu, Jan 18, 2018, 9:37 PM UTC

350TB of memory, and 50,000 cores, nice.

ARP caching seems to be a common issue in cloud environments. AWS recommends turning it off and does so itself in their Amazon Linux distro.

Re: Scaling Kubernetes to 2,500 Nodes

#14

Post by thesandlord » Thu, Jan 18, 2018, 9:43 PM UTC

eggie5 wrote: post #12

Does OpenAI train w/ GPUs on k8s clusters?

According to the article, they are using NC24 VMS, which have 4 K80s attached. So yes, I would assume they are using GPUs.

Check out https://github.com/google/kubeflow if you are interested in doing the same.

(Disclaimer: I work for GCP doing K8s stuff, I know GKE clusters support GPUs and Kubeflow, not 100% sure if AKS supports it or if you need to set up your own cluster like OpenAI did.)

Re: Scaling Kubernetes to 2,500 Nodes

#15

Post by krat0sprakhar » Thu, Jan 18, 2018, 9:48 PM UTC

roscoebeezie wrote: post #9

As a person who doesn’t understand containers, where do I go to learn the basics?

Here's a Docker tutorial I wrote targeted at beginners: https://docker-curriculum.com/

Re: Scaling Kubernetes to 2,500 Nodes

#16

Post by DannyB2 » Thu, Jan 18, 2018, 10:09 PM UTC

roscoebeezie wrote: post #9

As a person who doesn’t understand containers, where do I go to learn the basics?

Once you understand the basics of Docker, which is containers, then look at Kubernetes. One place you might start is with spinning up a single VM with RancherOS.

Re: Scaling Kubernetes to 2,500 Nodes

#17

Post by nickjj » Thu, Jan 18, 2018, 10:13 PM UTC

roscoebeezie wrote: post #9

As a person who doesn’t understand containers, where do I go to learn the basics?

If you want to avoid doing a bunch of research on your own I've put together an up to date self paced video course at https://diveintodocker.com/.

It covers everything from "What is Docker?" to learning how to apply it to your own projects. There's a tiny bit of theory, followed by lots of guided labs and examples.

In case you're curious, I've been using Docker in development and production since 2014 and am also a Docker Captain (TL;DR is Docker reached out to me to join their team as a trusted content provider).

Re: Scaling Kubernetes to 2,500 Nodes

#18

Post by twakefield » Thu, Jan 18, 2018, 10:17 PM UTC

roscoebeezie wrote: post #9

As a person who doesn’t understand containers, where do I go to learn the basics?

We have Docker 101 [0] and Kubernetes 101 [1] tutorials on our github repo which may be helpful, assuming you have an understanding of Linux. Or checkout the resources on Docker's website [2].

[0] https://github.com/gravitational/workshop/blob/master/docker...

[1] https://github.com/gravitational/workshop/blob/master/k8s101...

[2] https://www.docker.com/what-docker

Re: Scaling Kubernetes to 2,500 Nodes

#19

Post by raccoonone » Thu, Jan 18, 2018, 10:30 PM UTC

eggie5 wrote: post #12

Does OpenAI train w/ GPUs on k8s clusters?

Yep, we've been using GPUs for quite a while (even before the alpha support in Kube), both the K80s in Azure and some Pascals in our own clusters. With the support in Kube now it's quite seamless.

Re: Scaling Kubernetes to 2,500 Nodes

#20

Post by merb » Thu, Jan 18, 2018, 11:07 PM UTC

what I find amazing about k8s is that it's one of the first solution that is relativly simple for a small cluster (HA, while schedule stuff on the masters), but can scale amazingly well even for a big cluster. you can start with 3 nodes with like 8gb per machine (or less, I guess even 2gb is feasible if you only want to use like 1-1,5gb of memory per machine). (non ha can of course be smaller)