We built a self-healing system to survive a concurrency bug at Netflix
pushtoprod.substack.com
We built a self-healing system to survive a concurrency bug at Netflix
1–10 of 184 posts
Re: We built a self-healing system to survive a concurrency bug at Netflix
#2[deleted]
Re: We built a self-healing system to survive a concurrency bug at Netflix
#3One of the things I am greatful for kubernetes and the killing of pods.
Had a similar problem but memory wise with a pesky memory leak, and the short term solution was to do nothing as instances would to do nothing.
Re: We built a self-healing system to survive a concurrency bug at Netflix
#4One of the things I am greatful for kubernetes and the killing of pods. Had a similar problem but memory wise with a pesky memory leak, and the short term solution was to do nothing as instances would to do nothing.
During one of my past gigs, this exact feature hid a huge memory leak, in old code, always running on k8s which we found out only when we moved some instances to bare metal.
Re: We built a self-healing system to survive a concurrency bug at Netflix
#5[deleted]
Re: We built a self-healing system to survive a concurrency bug at Netflix
#6This was a nice short read. A simple (temporary) solution, yet a clever one.
How was he managing the instances? Was he using kubernetes, or did he write some script to manage the auto terminating of the instances?
It would also be nice to know why:
1. Killing was quicker than restarting. Perhaps because of the business logic built into the java application?
2. Killing was safe. How was the system architectured so that the requests weren't dropped altogether.
EDIT: formatting
Re: We built a self-healing system to survive a concurrency bug at Netflix
#7That was a bit underwhelming compared to what the headline set my expectations up for, but definitely a good idea and neat solution.
Re: We built a self-healing system to survive a concurrency bug at Netflix
#8This was a nice short read. A simple (temporary) solution, yet a clever one. How was he managing the instances? Was he using kubernetes, or did he write some script to manage the auto terminating of the instances? It would also be nice to know why: 1. Killing was quicker than restarting. Perhaps because of the business logic built into the java application? 2. Killing was safe. How was the system architectured so tha…
[deleted]
Re: We built a self-healing system to survive a concurrency bug at Netflix
#9This was a nice short read. A simple (temporary) solution, yet a clever one. How was he managing the instances? Was he using kubernetes, or did he write some script to manage the auto terminating of the instances? It would also be nice to know why: 1. Killing was quicker than restarting. Perhaps because of the business logic built into the java application? 2. Killing was safe. How was the system architectured so tha…
The author mentions 2011 as the time they switched from REST to RPC-ish APIs, and this issue was related to that migration.
Kubernetes launched in 2014, if memory serves, and it took a bit before widespread adoption, so I’m guessing this was some internal solution.
This was a great read, and harkens back to the days of managing 1000s of cores on bare metal!
Re: We built a self-healing system to survive a concurrency bug at Netflix
#10Interesting read, the fix seems to be straightforward, but I'd have a few more questions if I was trying to do something similar.
Is software deployed regularly on this cluster? Does that deployment happen faster than the rate at which they were losing CPUs? Why not just periodically force a deployment, given it's a repeated process that probably already happens frequently.
What happens to the clients trying to connect to the stuck instances? Did they just get stuck/timeout? Would it have been better to have more targeted terminations/full terminations instead?