Seems that metrics providing visibility into the "network connectivity was flaky", like looking at response times (particularly 95/99 percentile) and digging into the pod, which gives you the node, would have isolated the problem pretty quickly to a single node. If a problem is isolated to a node, first thing to look at would be node logs. Would that pattern not have worked in this case?
Debugging a TCP socket leak in a Kubernetes cluster
11–12 of 12 posts
Re: Debugging a TCP socket leak in a Kubernetes cluster
#12Checking the logs should be on everyone mind when dealing with issues.
Indeed! And when the logs show nothing being wrong? All k8s components were reporting that everything is fine.
Because not all problem are userspace related, it's very common on Linux to check for kernel logs. Especially when you know how Kubernetes deals with networking ( using iptables ect ... )