How Kubernetes Probes Work
ngrok.com
How Kubernetes Probes Work
1–10 of 26 posts
Re: How Kubernetes Probes Work
#2This does not state anything new, but explains it so much well than the kubernetes documentation.
Re: How Kubernetes Probes Work
#3This does not state anything new, but explains it so much well than the kubernetes documentation.
Sometimes the k8s docs remind me of google's documentation
Re: How Kubernetes Probes Work
#4Re: How Kubernetes Probes Work
#5This does not state anything new, but explains it so much well than the kubernetes documentation.
Sam has a real way with words when it comes to educational content.
Though he did find a legit Kubernetes bug while writing the post, so technically there was at least one new thing :)
Re: How Kubernetes Probes Work
#6SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting?
Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away.
Maybe TCP connections are in stuck weird state, resetting the process generally helps with that.
Maybe someone gave you bad ENV VARs and you cannot connect to database, by refusing to progress the rollout, no outage generated.
So yea, if you are not ready to do work including critical upstream dependencies, don't lie to system and say you are.
Re: How Kubernetes Probes Work
#7SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting? Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away. Maybe TCP connections are in…
Thundering herd / cascading outages. You take out a large enough portion of your fleet, and the remaining load overloads your remaining nodes one by one as they restart, so you can never have enough healthy nodes.
Re: How Kubernetes Probes Work
#8SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting? Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away. Maybe TCP connections are in…
Thundering herd / cascading outages. You take out a large enough portion of your fleet, and the remaining load overloads your remaining nodes one by one as they restart, so you can never have enough healthy nodes.
SRE team debates correctness versus availability for the 540th time this year
You're both correct, of course!
Re: How Kubernetes Probes Work
#9Earlier quoted context omitted.
Thundering herd / cascading outages. You take out a large enough portion of your fleet, and the remaining load overloads your remaining nodes one by one as they restart, so you can never have enough healthy nodes.
SRE team debates correctness versus availability for the 540th time this year You're both correct, of course!
What this guy said :point_up:
My personal take-away is this: whatever you choose, make sure it's consistent across services (not serviceA behaves like X and serviceB like Y) and make sure eng teams know _how_ these are configured and what can go wrong. They'll figure out the rest.
Re: How Kubernetes Probes Work
#10SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting? Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away. Maybe TCP connections are in…
1. was already mentioned in sibling - cascade failures
2. you'll have massive number of restarts for various flake reasons and missing things that got papered over with restarts until you hit 1 and everything is broken. another popular version of this is "just restart when memory leaks too much"