Netflix Simplified Batch Compute with Kueue
netflixtechblog.com
Netflix Simplified Batch Compute with Kueue
1–10 of 14 posts
Re: Netflix Simplified Batch Compute with Kueue
#2Anyone know if Netflix does anything for the k8s storage layer? I imagine they are at the scale where etcd starts to go kaboom? Or maybe they have enough cells where that isn’t a problem?
Given Amazon and Google have their own secret sauce for replacing etcd, I am wondering if Netflix does anything special?
Re: Netflix Simplified Batch Compute with Kueue
#3Anyone know if Netflix does anything for the k8s storage layer? I imagine they are at the scale where etcd starts to go kaboom? Or maybe they have enough cells where that isn’t a problem? Given Amazon and Google have their own secret sauce for replacing etcd, I am wondering if Netflix does anything special?
It's possible they are using kine: https://github.com/k3s-io/kine
Re: Netflix Simplified Batch Compute with Kueue
#4Anyone know if Netflix does anything for the k8s storage layer? I imagine they are at the scale where etcd starts to go kaboom? Or maybe they have enough cells where that isn’t a problem? Given Amazon and Google have their own secret sauce for replacing etcd, I am wondering if Netflix does anything special?
This runs on AWS managed EKS these days, this talk goes into more detail about Netflix's special sauce around the k8s control plane: https://www.youtube.com/watch?v=vaTOiXR2KSM
Netflix actually has much fewer cells than you'd expect btw, their special sauce IMO is federation and using a small subset of k8s APIs.
Re: Netflix Simplified Batch Compute with Kueue
#5Congrats, this is awesome!
Re: Netflix Simplified Batch Compute with Kueue
#6It's refreshing to see a tech article that isn't about AI. It feels like 5 years ago.
Re: Netflix Simplified Batch Compute with Kueue
#7I see Netflix pumping out tech articles but can't help but notice how much worse the UI experience is getting. Video erroring out, general slowness etc.
Did they just give up?
Re: Netflix Simplified Batch Compute with Kueue
#8It certainly feels like Netflix is now a k8s shop.
And it probably only a matter of time until they start repatriating workloads to optimize for costs. Then the world will sit up and notice.
Re: Netflix Simplified Batch Compute with Kueue
#9It certainly feels like Netflix is now a k8s shop. And it probably only a matter of time until they start repatriating workloads to optimize for costs. Then the world will sit up and notice.
I don’t get what you’re implying. What is repatriating; You think they will move their workloads to on-prem?
Is there something different about the world that changed the trade-off calculus for cloud vs on-prem from how it was in the last 15 years compared to now?
(I’m as anti-cloud-overspend as the next guy on hn btw. Just trying to make sense of your comment’s worldview.)
Re: Netflix Simplified Batch Compute with Kueue
#10Anyone know if Netflix does anything for the k8s storage layer? I imagine they are at the scale where etcd starts to go kaboom? Or maybe they have enough cells where that isn’t a problem? Given Amazon and Google have their own secret sauce for replacing etcd, I am wondering if Netflix does anything special?
This runs on AWS managed EKS these days, this talk goes into more detail about Netflix's special sauce around the k8s control plane: https://www.youtube.com/watch?v=vaTOiXR2KSM Netflix actually has much fewer cells than you'd expect btw, their special sauce IMO is federation and using a small subset of k8s APIs.
I am surprised a company at that scale is running on managed EKS, maybe I underestimate how large the clusters are.