Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Kubernetes Probes Work

ngrok.com

Post reply on HN
Print view View original

11–20 of 26 posts

Re: How Kubernetes Probes Work

#11

Post by cmckn »

stackskipton wrote: post #6

SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting? Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away. Maybe TCP connections are in…

> what's the problem with restarting?

Exponential backoff can delay recovery up to kubelet’s maxContainerRestartPeriod (default 5m).

Re: How Kubernetes Probes Work

#12

Post by javier2 »

stackskipton wrote: post #6

SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting? Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away. Maybe TCP connections are in…

cascading failures on upstream services. then you get 20 different services failing instead of the single one.

Re: How Kubernetes Probes Work

#13

Post by jaggederest »

arccy wrote: post #7
stackskipton wrote: post #6

SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting? Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away. Maybe TCP connections are in…

Thundering herd / cascading outages. You take out a large enough portion of your fleet, and the remaining load overloads your remaining nodes one by one as they restart, so you can never have enough healthy nodes.

That's a problem for circuitbreakers on these kinds of actions, not lying on health checks.

Something like healthcheck fails -> restart -> healthcheck fails -> restart -> healthcheck fails -> circuit breaker trip, alarm raised, give up until manual intervention or X minutes have passed

Re: How Kubernetes Probes Work

#15

Post by peterabbitcook »

stackskipton wrote: post #6

SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting? Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away. Maybe TCP connections are in…

What are your feelings about using initContainers and wait-for-it to skirt the thundering-herd problem?

Re: How Kubernetes Probes Work

#16

Post by connicpu »

stackskipton wrote: post #6

SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting? Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away. Maybe TCP connections are in…

The better solution is to not have too many critical upstream services :)

Re: How Kubernetes Probes Work

#17

Post by deathanatos »

jaggederest wrote: post #13
arccy wrote: post #7

Earlier quoted context omitted.

Thundering herd / cascading outages. You take out a large enough portion of your fleet, and the remaining load overloads your remaining nodes one by one as they restart, so you can never have enough healthy nodes.

That's a problem for circuitbreakers on these kinds of actions, not lying on health checks. Something like healthcheck fails -> restart -> healthcheck fails -> restart -> healthcheck fails -> circuit breaker trip, alarm raised, give up until manual intervention or X minutes have passed

That circuitbreaker exists, by default. It is "CrashloopBackoff", here, and TFA covers it. (& it's an "until X minutes have passed" kind, by default.)

Re: How Kubernetes Probes Work

#18

Post by figmert »

stackskipton wrote: post #6

SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting? Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away. Maybe TCP connections are in…

My favourite: misconfigured Linkerd setup that causes CA certs to rotate every month :) Definitely worth restarting on that

Re: How Kubernetes Probes Work

#19

Post by dilyevsky »

deathanatos wrote: post #17
jaggederest wrote: post #13

Earlier quoted context omitted.

That's a problem for circuitbreakers on these kinds of actions, not lying on health checks. Something like healthcheck fails -> restart -> healthcheck fails -> restart -> healthcheck fails -> circuit breaker trip, alarm raised, give up until manual intervention or X minutes have passed

That circuitbreaker exists, by default. It is "CrashloopBackoff", here, and TFA covers it. (& it's an "until X minutes have passed" kind, by default.)

backoff is only applied to individual pods/containers not across pods. the point is at scale it's easy to get into a situation where it's not possible to recover without (usually manual) full service drain

Re: How Kubernetes Probes Work

#20

Post by solatic »

erulabs wrote: post #8
arccy wrote: post #7

Earlier quoted context omitted.

Thundering herd / cascading outages. You take out a large enough portion of your fleet, and the remaining load overloads your remaining nodes one by one as they restart, so you can never have enough healthy nodes.

SRE team debates correctness versus availability for the 540th time this year You're both correct, of course!

You'd be surprised how many engineering leaders don't understand the CAP theorem and will fail engineers on interviews for picking the one they don't agree with instead of communicating their expectations clearly (dodged a bullet on that one ...)
Post reply on HN