Complex systems, even when think you designed to avoid complexity, are unpredictable when shit happens. Any system which is hard to completely understand is likely to fail in unforeseen ways, and possibly be difficult to recover from or to know how to change to keep it from happening again. Sadly I see these types of failures all the time where I work, though not quite the same scale. One time a simple DNS update brought down our operations worldwide, even in places thought not to be connected at all.
On Infrastructure at Scale: A Cascading Failure of Distributed Systems
21–29 of 29 posts
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#22Some questions I had after reading it: * Would this have been less painful with Netflix-esque patterns? Circuit breakers, throttling and the like? Failing hard on purpose is often easier to diagnose than failing fuzzily without clear cause. * Would it be acceptable to maintain spare capacity? For a prod cluster on the main money path, I think everyone thinks yes. Who holds the decision bit for dev, which is a second-…
The risk with having a reset mentality, "after all it's only dev" (edit: not intending to imply you said this), is that the opportunity to learn is lost. If they took the reset tack, then this occurred in production and they didn't have that option available, choosing the reset path would suddenly be seen in hindsight as a lost opportunity.
I hadn't thought of that. I still think it would be helpful in prod, though.
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#23For those that are not too versed in distributed systems, kubernetes or docker I highly suggest you read the comments on the post. I am going to promot myself a bit here and link directly to my comment https://medium.com/@rmb938/thanks-dan-for-the-great-post-1ef...
Sadly a large environment was brought down due to not following best practices. This event probably could have been prevent or at least have very little impact if best practices were followed.
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#24I wonder if Target stores their customers cardholder PIN numbers still [1]. [1] https://www.theguardian.com/world/2013/dec/27/target-hackers...
According to your own linked article, target does not store the PIN numbers.
What happens is, the PIN pad encrypts the PIN code and other payment info, using a key known only to the card issuer (i.e. VISA). This encrypted data then finds it way to the card issuer (e.g. Visa) for verification via one of a few possible paths, either PSTN dialup, or more common these days, over the internet. In the Target incident, hackers grabbed this data as it was being transferred over the LAN.
(For completeness, there are many more organisations this encrypted data passes through between the merchant and card issuer, but nobody but the card issuer can decrypt it)
Also, it's incredibly off topic for the post. I'll bet that's why your getting all the downvotes.
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#25Interesting. Is there a benefit to running Kubernetes in Openstack in their case vs running it on bare metal?
Storage and network stability is much higher if you run it on top of Openstack. k8s on baremetal is a pain in the butt. Disadvantage is that you have even more layers of abstraction.
I understand storage, but could you elaborate on why there would be higher network stability with Openstack?
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#26key take away that isn't mentioned in the article: - Put resource limits on your pods.
What’s your reasoning? From my experience, a memory limit, if hit, will cause periodic OOM and restart; and a CPU limit will cause slowing-down of the processes. The tale already mentions restarting and slowing-down; it’s not clear to me that adding more would have improved anything.
You want to signal early that a node is unable to do what is requested of it.
Its the same with unbounded queues, you need to put in limits so that alarms are triggered much earlier.
infinitely spinning up new things is generally bad. Unless you are responding to an external signal. Minimum deploy times again are useful. There is a financial cost (on the cloud, not on real tin) to short term instances. So you need to have them on as long as possible.
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#27Earlier quoted context omitted.
What’s your reasoning? From my experience, a memory limit, if hit, will cause periodic OOM and restart; and a CPU limit will cause slowing-down of the processes. The tale already mentions restarting and slowing-down; it’s not clear to me that adding more would have improved anything.
You want to signal early that a node is unable to do what is requested of it. Its the same with unbounded queues, you need to put in limits so that alarms are triggered much earlier. infinitely spinning up new things is generally bad. Unless you are responding to an external signal. Minimum deploy times again are useful. There is a financial cost (on the cloud, not on real tin) to short term instances. So you need to…
Sounds like a Kubernetes resource request, rather than a limit.
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#28Earlier quoted context omitted.
What’s your reasoning? From my experience, a memory limit, if hit, will cause periodic OOM and restart; and a CPU limit will cause slowing-down of the processes. The tale already mentions restarting and slowing-down; it’s not clear to me that adding more would have improved anything.
The issue was the Docker daemon was affected due to poor resource configuration. It should run at a higher priority and with guaranteed resources so it can't be impacted by user mode tasks. k8s has been slow to catch up in this area but finally has priorities and preemption. That said Docker normally doesn't run in a pod so it's also a matter of setting the node total allocatable CPU to something reasonable and ensur…
Kubernetes has a “-system-reserved” flag for that purpose. Limits are something else entirely.
Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems
#29key take away that isn't mentioned in the article: - Put resource limits on your pods.
What’s your reasoning? From my experience, a memory limit, if hit, will cause periodic OOM and restart; and a CPU limit will cause slowing-down of the processes. The tale already mentions restarting and slowing-down; it’s not clear to me that adding more would have improved anything.
if you've got an app that constantly uses more memory then it's either a memory leak or poor tuning which results in that.
An app without a CPU limit can completely consume the hosts CPU and take down everything else on that node. If you've got a very large instance with a lot of apps, a bunch of them are gonna experience a poor QoS.
Their tale suggests their sidecar pods/containers (which likely didn't have limits) made their production issues worse. a CPU limit on those pods would have limited the damage those pods caused to the rest of their infra.