Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On Infrastructure at Scale: A Cascading Failure of Distributed Systems

medium.com

Post reply on HN
Print view View original

1–10 of 29 posts

Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems

#3

Post by tomnipotent »

> This problem did not affect the TAP production environment in the same way due to it simply being a much smaller environment

Is it common for a dev environment to be larger than production...? Maybe paints a picture about Target's ratio of software developers to traffic/load?

Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems

#5

Post by throwawayjava »

tomnipotent wrote: post #3

> This problem did not affect the TAP production environment in the same way due to it simply being a much smaller environment Is it common for a dev environment to be larger than production...? Maybe paints a picture about Target's ratio of software developers to traffic/load?

He's not saying that the total dev environment is bigger than the total prod environment. He's saying that this one cluster of dev machines is much bigger than other clusters of machines (dev or prod). that could be true while simultaneously having many more prod machines than dev machines.

The author explains: however there is one cluster that was built significantly larger than the others and hosted around 2,000 development environment workloads. (This was an artifact of the early days of running “smaller clusters, more of them”, when we were trying to figure out the right size of “smaller”.)

Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems

#6

Post by tomnipotent »

throwawayjava wrote: post #5
tomnipotent wrote: post #3

> This problem did not affect the TAP production environment in the same way due to it simply being a much smaller environment Is it common for a dev environment to be larger than production...? Maybe paints a picture about Target's ratio of software developers to traffic/load?

He's not saying that the total dev environment is bigger than the total prod environment. He's saying that this one cluster of dev machines is much bigger than other clusters of machines (dev or prod). that could be true while simultaneously having many more prod machines than dev machines. The author explains: however there is one cluster that was built significantly larger than the others and hosted around 2,000 de…

Thanks, somehow missed that.

Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems

#7

Post by jacques_chester »

Some questions I had after reading it:

* Would this have been less painful with Netflix-esque patterns? Circuit breakers, throttling and the like? Failing hard on purpose is often easier to diagnose than failing fuzzily without clear cause.

* Would it be acceptable to maintain spare capacity? For a prod cluster on the main money path, I think everyone thinks yes. Who holds the decision bit for dev, which is a second-order source of money?

* Can I avoid the second act by having a clean way to (1) reset the entire system and (2) bring it up in an orderly fashion, as quickly as possible? Not every app needs to be relaunched simultaneously, there's presumably _some_ order of preference. Once the cluster state began to oscillate it doesn't seem like there were any easy options other than resetting it.

Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems

#8

Post by kronin »

tomnipotent wrote: post #3

> This problem did not affect the TAP production environment in the same way due to it simply being a much smaller environment Is it common for a dev environment to be larger than production...? Maybe paints a picture about Target's ratio of software developers to traffic/load?

Another possibility is that they deploy feature branches in dev, so for a given service there are more versions running simultaneously.

Re: On Infrastructure at Scale: A Cascading Failure of Distributed Systems

#9

Post by kronin »

jacques_chester wrote: post #7

Some questions I had after reading it: * Would this have been less painful with Netflix-esque patterns? Circuit breakers, throttling and the like? Failing hard on purpose is often easier to diagnose than failing fuzzily without clear cause. * Would it be acceptable to maintain spare capacity? For a prod cluster on the main money path, I think everyone thinks yes. Who holds the decision bit for dev, which is a second-…

The risk with having a reset mentality, "after all it's only dev" (edit: not intending to imply you said this), is that the opportunity to learn is lost. If they took the reset tack, then this occurred in production and they didn't have that option available, choosing the reset path would suddenly be seen in hindsight as a lost opportunity.
Post reply on HN