key take away that isn't mentioned in the article:
- Put resource limits on your pods.
On Infrastructure at Scale: A Cascading Failure of Distributed Systems
11–20 of 29 posts
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#12I wonder if Target stores their customers cardholder PIN numbers still [1].
[1] https://www.theguardian.com/world/2013/dec/27/target-hackers...
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#13The only real way to deal with this is to test distributed systems.
Doing so is hard, but only way to reliably know a system behaves given unpredictable failures.
So learn up:
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#14Interesting. Is there a benefit to running Kubernetes in Openstack in their case vs running it on bare metal?
Storage and network stability is much higher if you run it on top of Openstack. k8s on baremetal is a pain in the butt. Disadvantage is that you have even more layers of abstraction.
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#15key take away that isn't mentioned in the article: - Put resource limits on your pods.
What’s your reasoning?
From my experience, a memory limit, if hit, will cause periodic OOM and restart; and a CPU limit will cause slowing-down of the processes.
The tale already mentions restarting and slowing-down; it’s not clear to me that adding more would have improved anything.
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#16key take away that isn't mentioned in the article: - Put resource limits on your pods.
What’s your reasoning? From my experience, a memory limit, if hit, will cause periodic OOM and restart; and a CPU limit will cause slowing-down of the processes. The tale already mentions restarting and slowing-down; it’s not clear to me that adding more would have improved anything.
The issue was the Docker daemon was affected due to poor resource configuration. It should run at a higher priority and with guaranteed resources so it can't be impacted by user mode tasks.
k8s has been slow to catch up in this area but finally has priorities and preemption. That said Docker normally doesn't run in a pod so it's also a matter of setting the node total allocatable CPU to something reasonable and ensuring all pods spawned by kubelet are nested under a namespace that has a lower priority than the system tasks (kubelet itself, Docker if you use it etc).
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#17Earlier quoted context omitted.
What’s your reasoning? From my experience, a memory limit, if hit, will cause periodic OOM and restart; and a CPU limit will cause slowing-down of the processes. The tale already mentions restarting and slowing-down; it’s not clear to me that adding more would have improved anything.
The issue was the Docker daemon was affected due to poor resource configuration. It should run at a higher priority and with guaranteed resources so it can't be impacted by user mode tasks. k8s has been slow to catch up in this area but finally has priorities and preemption. That said Docker normally doesn't run in a pod so it's also a matter of setting the node total allocatable CPU to something reasonable and ensur…
And it could be easily done by adjusting Docker's systemd service unit.
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#18Earlier quoted context omitted.
What’s your reasoning? From my experience, a memory limit, if hit, will cause periodic OOM and restart; and a CPU limit will cause slowing-down of the processes. The tale already mentions restarting and slowing-down; it’s not clear to me that adding more would have improved anything.
The issue was the Docker daemon was affected due to poor resource configuration. It should run at a higher priority and with guaranteed resources so it can't be impacted by user mode tasks. k8s has been slow to catch up in this area but finally has priorities and preemption. That said Docker normally doesn't run in a pod so it's also a matter of setting the node total allocatable CPU to something reasonable and ensur…
yeah after reading that I came to the same conclusion and tought that he mostly blames it on cluster size, which is just wrong.
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#19This was a fascinating read.
> I’ve had concerns about the sidecars in the past, however after this event I am convinced that having each workload ship with their own logging and metric sidecars is the right thing. If these had been a shared resource to the cluster, the problem almost certainly would have been exacerbated and much harder to fix.
While I'm all for the sidecars model, I'm not sure that I agree with this...
If logging/metrics ingestion was a shared resource that applications were calling out to directly (basically a shim for kafka 99% of the time), it seems they could have built in a layer to dump writes to disk in the case kafka was unreachable. Even if they didn't and the shared service just started failing, that's a single (likely highly scaled) service failing (or best case 503ing and dropping logging+metrics where necessary) not service downtime/the cascading failures. Maybe I'm misunderstanding something about this quote.
Maybe it's just me, but workloads going down due to a sidecar that handles logging/metrics doesn't sound like the right prioritization to me.
Also, like other people have said, resource limits & using the kubernetes compute reservation systems[0] also seems like a key bit that was missing from this.
[0]: https://kubernetes.io/docs/tasks/administer-cluster/reserve-...
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#20The authors conclusions seem questionable. Ryan Belgrave's comment on them is worth reading.
https://medium.com/@rmb938/thanks-dan-for-the-great-post-1ef...