Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

KWOK: Kubernetes WithOut Kubelet

kubernetes.dev

Post reply on HN
Print view View original

41–50 of 50 posts

Re: KWOK: Kubernetes WithOut Kubelet

#41

Post by simianpirate »

lmeyerov wrote: post #33

Slight tangent inspired by the premise, but why do EKS etc cap the number of pods and other resources on a node? We try to enable our customers to start on 1 node and scale up at their discretion, but found we had to into all sorts of contortions to make that work. Likewise, k8s is resource hungry. We can run the docker compose version on a much smaller laptop wrt CPU & RAM.

Kubernetes has a default (and considered best practice) limit of 110 pods per node. Are you seeing them cap it lower than that? I've never checked this personally on EKS, TBH.

Also, what do you mean by capping resources? In my experience with EKS, I haven't had any issues fully utilizing a nodes resources, even in EKS.

Re: KWOK: Kubernetes WithOut Kubelet

#42

Post by mdaniel »

quantumcd wrote: post #35
lmeyerov wrote: post #33

Slight tangent inspired by the premise, but why do EKS etc cap the number of pods and other resources on a node? We try to enable our customers to start on 1 node and scale up at their discretion, but found we had to into all sorts of contortions to make that work. Likewise, k8s is resource hungry. We can run the docker compose version on a much smaller laptop wrt CPU & RAM.

It's been a while since I've used EKS, but I seem to recall it's primarily based on the maximum ENIs the instance type supports.

I believe you're correct, although pedantically that would only apply if one is using their vpc-cni https://github.com/aws/amazon-vpc-cni-k8s#readme> and not with a competing CNI. Kubelet offers a configurable for the number of Pods per Node https://github.com/kubernetes/kubelet/blob/v0.26.2/config/v1...> which defaults to 110 for what I would presume is CIDR or pid cgroups reasons and thus is unlikely to differ by instance size as the ENI limit you mention does (IIRC)

Re: KWOK: Kubernetes WithOut Kubelet

#43

Post by preisschild »

MuffinFlavored wrote: post #36
debarshri wrote: post #32

One of the use cases mentioned here is testing. One should also look at kind [1]. It is light weight, a really good candidate to setup dev environments. [1] https://kind.sigs.k8s.io/

I wonder how big of an effort it would be for Apple to take some engineers and make Mac OS support kernel namespaces/cgroups that would bridge the gap between all of these cool container features Linux has that requires a virtual machine to recreate on Mac.

Would this solve anything? Most containers use a linux userspace, which requires an actual linux kernel.

Your idea would more similar to Flatpak (containerized GUI apps).

Re: KWOK: Kubernetes WithOut Kubelet

#44

Post by remram »

moondev wrote: post #40
nilptr wrote: post #39

Earlier quoted context omitted.

Wait really? How does Docker macOS work on M1/M2? Or does it not?

Docker Desktop runs dockerd in a Linux VM with Apple's hypervisor framework. You can also run containers in a Linux VM with Parallels or VMware Fusion hypervisors, QEMU on macOS also uses HVF not KVM. But you can't run VMs inside those VMs as it stands today. This works fine on Intel Macs which means you can't experiment and use KVM - one of the killer features of Linux and things like https://kubevirt.io/ and firecr…

It seems to me that the right fix is for Docker Desktop to support M1. Docker, kubelet, the k8s control plane, and everything else has supported ARM for ages. There is no need for that extra VM and therefore no blocker on nested virtualization.

Re: KWOK: Kubernetes WithOut Kubelet

#45

Post by freshmatrix »

lmeyerov wrote: post #33

Slight tangent inspired by the premise, but why do EKS etc cap the number of pods and other resources on a node? We try to enable our customers to start on 1 node and scale up at their discretion, but found we had to into all sorts of contortions to make that work. Likewise, k8s is resource hungry. We can run the docker compose version on a much smaller laptop wrt CPU & RAM.

# of pods are essentially capped by the worker node choice.

below excerpt from: https://github.com/awslabs/amazon-eks-ami/blob/master/files/...

  # Mapping is calculated from AWS EC2 API using the following formula:
  # * First IP on each ENI is not used for pods
  # * +2 for the pods that use host-networking (AWS CNI and kube-proxy)
  #
  #   # of ENI * (# of IPv4 per ENI - 1) + 2
  #
  # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI
doc on EC2 instance types ("Amazon EKS recommended maximum pods for each Amazon EC2 instance type") https://docs.aws.amazon.com/eks/latest/userguide/choosing-in...

Re: KWOK: Kubernetes WithOut Kubelet

#46

Post by sofixa »

remram wrote: post #44
moondev wrote: post #40

Earlier quoted context omitted.

Docker Desktop runs dockerd in a Linux VM with Apple's hypervisor framework. You can also run containers in a Linux VM with Parallels or VMware Fusion hypervisors, QEMU on macOS also uses HVF not KVM. But you can't run VMs inside those VMs as it stands today. This works fine on Intel Macs which means you can't experiment and use KVM - one of the killer features of Linux and things like https://kubevirt.io/ and firecr…

It seems to me that the right fix is for Docker Desktop to support M1. Docker, kubelet, the k8s control plane, and everything else has supported ARM for ages. There is no need for that extra VM and therefore no blocker on nested virtualization.

It's not an ARM problem, it's a kernel/OS one. Same as Windows, macOS simply doesn't have what it takes (namespaces, cgroups, etc.) to run Docker/Linux containers natively, therefore an intermediary Linux VM is needed.

Re: KWOK: Kubernetes WithOut Kubelet

#47

Post by mac-chaffee »

lmeyerov wrote: post #33

Slight tangent inspired by the premise, but why do EKS etc cap the number of pods and other resources on a node? We try to enable our customers to start on 1 node and scale up at their discretion, but found we had to into all sorts of contortions to make that work. Likewise, k8s is resource hungry. We can run the docker compose version on a much smaller laptop wrt CPU & RAM.

Kubernetes is only tested against 110 pods per node: https://kubernetes.io/docs/setup/best-practices/cluster-larg...

In my testing, it's a limitation in either the container runtime or kubelet in processing all the events that flow from pods. And since the container networking and container storage interfaces aren't part of Kubernetes, there are likely scaling issues in those pieces of software as well.

Re: KWOK: Kubernetes WithOut Kubelet

#48

Post by remram »

sofixa wrote: post #46
remram wrote: post #44

Earlier quoted context omitted.

It seems to me that the right fix is for Docker Desktop to support M1. Docker, kubelet, the k8s control plane, and everything else has supported ARM for ages. There is no need for that extra VM and therefore no blocker on nested virtualization.

It's not an ARM problem, it's a kernel/OS one. Same as Windows, macOS simply doesn't have what it takes (namespaces, cgroups, etc.) to run Docker/Linux containers natively, therefore an intermediary Linux VM is needed.

Doh, I forgot about that. You're right, the VM is needed anyway.

Re: KWOK: Kubernetes WithOut Kubelet

#49

Post by MuffinFlavored »

moondev wrote: post #40
nilptr wrote: post #39

Earlier quoted context omitted.

Wait really? How does Docker macOS work on M1/M2? Or does it not?

Docker Desktop runs dockerd in a Linux VM with Apple's hypervisor framework. You can also run containers in a Linux VM with Parallels or VMware Fusion hypervisors, QEMU on macOS also uses HVF not KVM. But you can't run VMs inside those VMs as it stands today. This works fine on Intel Macs which means you can't experiment and use KVM - one of the killer features of Linux and things like https://kubevirt.io/ and firecr…

> Docker Desktop runs dockerd in a Linux VM with Apple's hypervisor framework

Which in my experience uses at a minimum 4gb of RAM just sitting idle with no containers running

Re: KWOK: Kubernetes WithOut Kubelet

#50

Post by moondev »

sofixa wrote: post #46
remram wrote: post #44

Earlier quoted context omitted.

It seems to me that the right fix is for Docker Desktop to support M1. Docker, kubelet, the k8s control plane, and everything else has supported ARM for ages. There is no need for that extra VM and therefore no blocker on nested virtualization.

It's not an ARM problem, it's a kernel/OS one. Same as Windows, macOS simply doesn't have what it takes (namespaces, cgroups, etc.) to run Docker/Linux containers natively, therefore an intermediary Linux VM is needed.

Agreed, but considering that kubernetes now supports joining windows workers to run windows containers, as well as integrated support for dockerd inside wsl2... That leaves macOS as honestly the worst platform for any kind of container related work.
Post reply on HN