I need to know how to animate things like this for internal documentation.
The author re-implemented a bunch of k8s logic in a typescript library just for these animations: https://github.com/ngrok/webernetes
How Kubernetes Probes Work
21–26 of 26 posts
Re: How Kubernetes Probes Work
#22SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting? Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away. Maybe TCP connections are in…
Unlike sibling commenters who just read about "thundering herd" problems on the Internet, as someone who spent significant time in SRE roles, I agree with you as a matter of what the default approach should be. If you have a small cluster and no more than a handful of services... what thundering herd problem is there supposed to be, exactly, with so few "cattle" in the "herd"? Meanwhile, there are serious benefits, as you describe.
Large clusters with dozens of services and traces that go several services deep, with each service owned by a different team, are a whole 'nother ballgame, especially when overall production uptime is owned by an SRE team and not by the developer teams who wrote each of those services. And even in this scenario, you're not necessarily wrong; the risk attached to the cascading failure is domain-specific and may be acceptable.
Re: How Kubernetes Probes Work
#23Earlier quoted context omitted.
That circuitbreaker exists, by default. It is "CrashloopBackoff", here, and TFA covers it. (& it's an "until X minutes have passed" kind, by default.)
backoff is only applied to individual pods/containers not across pods. the point is at scale it's easy to get into a situation where it's not possible to recover without (usually manual) full service drain
Yeah that's why I have a manual intervention breaker that goes across all the pods/nodes etc. CrashLoopBackoff is great for selfhealing but when things go really pear shaped you want something that catches the global state. Saw it activate during an AWS outage one time where new nodes were unhealthy on start, for example.
Re: How Kubernetes Probes Work
#24SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting? Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away. Maybe TCP connections are in…
I think the best approach is to have an endpoint which returns the current state without probing the database when you check the endpoint.
Like you have a threat which does all your checks for connectivity write it into memory and the api endpoint checks the last values written.
Then you can respond to different upstream issues differently: Your DB says 'permission denied' due to broken auth data? Your Pod is not ready.
Your backend understand what a db maintenance mode is and should still return "Maintenance", your pod stays ready.
It can be more nuanced.
But the problem with restarting is, as he stated in the blog: What if now EVERY pod restarts in parallel.
Re: How Kubernetes Probes Work
#25I need to know how to animate things like this for internal documentation.
The author re-implemented a bunch of k8s logic in a typescript library just for these animations: https://github.com/ngrok/webernetes
I ported Kubernetes to the browser https://ngrok.com/blog/i-ported-kubernetes-to-the-browser
Re: How Kubernetes Probes Work
#26SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting? Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away. Maybe TCP connections are in…
I think it really depends on how much control you have over what you're deploying. In one school regularly restarting something with a memory leak that you can't fix is the correct solution.
On the other hand that's just putting a bandaid on the real issue, and if you can you should just fix the memory leak instead of just covering it up. There's also short vs long term solutions to consider. But saying there's a one size fits all solution is kinda silly.