Scaling Kubernetes to 2,500 Nodes
blog.openai.com
Scaling Kubernetes to 2,500 Nodes
1–10 of 34 posts
Re: Scaling Kubernetes to 2,500 Nodes
#2(Azure containers lead here)
Awesome to see OpenAI scale Kubernetes on Azure!
Re: Scaling Kubernetes to 2,500 Nodes
#3(Azure containers lead here) Awesome to see OpenAI scale Kubernetes on Azure!
(in Azure) Are VMs still tied to a specific machine and thus if the machine goes down the VMs need be restarted?
Re: Scaling Kubernetes to 2,500 Nodes
#4(Azure containers lead here) Awesome to see OpenAI scale Kubernetes on Azure!
(in Azure) Are VMs still tied to a specific machine and thus if the machine goes down the VMs need be restarted?
VMs in all clouds are always tied to specific machines. If that machine fails unexpectedly then those VMs will restart. If it is a controlled reboot (e.g. host update) then they may not restart...
Re: Scaling Kubernetes to 2,500 Nodes
#5Earlier quoted context omitted.
(in Azure) Are VMs still tied to a specific machine and thus if the machine goes down the VMs need be restarted?
VMs in all clouds are always tied to specific machines. If that machine fails unexpectedly then those VMs will restart. If it is a controlled reboot (e.g. host update) then they may not restart...
On Google's cloud, at least, VMs don't get killed for planned hypervisor downtime, which is a potential differentiator for them. Of course, this should matter a lot less for workloads that have been explicitly designed run on Kubernetes...
Re: Scaling Kubernetes to 2,500 Nodes
#6Earlier quoted context omitted.
(in Azure) Are VMs still tied to a specific machine and thus if the machine goes down the VMs need be restarted?
VMs in all clouds are always tied to specific machines. If that machine fails unexpectedly then those VMs will restart. If it is a controlled reboot (e.g. host update) then they may not restart...
Well, at least in Google Cloud for planned updates you can get your VM host migrated and not lose a Node due to a planned maintenance. I am not aware if Azure supports this, but my guess is they do not.
Re: Scaling Kubernetes to 2,500 Nodes
#7This is a really fantastic set of general "how to tune kubernetes and the various components for large clusters". Thanks for writing this up!
Re: Scaling Kubernetes to 2,500 Nodes
#8(Azure containers lead here) Awesome to see OpenAI scale Kubernetes on Azure!
(in Azure) Are VMs still tied to a specific machine and thus if the machine goes down the VMs need be restarted?
No. I work exclusively with scaling Linux workloads in Azure and local failover happens fairly regularly without the user ever seeing any indication. The hypervisor has its own cache which is tied more to your storage acct & runtime data-disks than the VM itself. This is true even with the lowest cost storage option: LRS standard blob-storage.
Howwwwever, LRS storage does not save you if the whole datacenter goes down, or during scheduled maintenance (when the whole datacenter is down.) For that, you'll need ZRS (which does failover to a co-lo in the event of the primary datacenter going down) or GRS (for which you can configure/test your failover options.)
Also, Microsoft's strength is in their PaaS services, like app-service or Azure-Functions. Those usually have CosmosDB on the backend, which is pretty much the best failover/DB-availability server-software on the market in my opinion.
Re: Scaling Kubernetes to 2,500 Nodes
#9As a person who doesn’t understand containers, where do I go to learn the basics?
Re: Scaling Kubernetes to 2,500 Nodes
#10As a person who doesn’t understand containers, where do I go to learn the basics?
Docker has a good overview in my opinion: