I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…
The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."
Knightmare: A DevOps Cautionary Tale (2014)
101–110 of 294 posts
Re: Knightmare: A DevOps Cautionary Tale (2014)
#102Re: Knightmare: A DevOps Cautionary Tale (2014)
#103I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…
The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."
Re: Knightmare: A DevOps Cautionary Tale (2014)
#104They were missing any kind of risk mitigation steps, in their deployment practice.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#105Earlier quoted context omitted.
The thing I was surprised about is that they survived! After all that they still got a $400 million cash bailout!
It wasn't a bailout, it was investment money. Bailout could imply government throwing a lifeguard
Almost all bailouts are investments, whether it's a government or private bailout.
The investments are questionable sometimes, but they're still investments.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#106Re: Knightmare: A DevOps Cautionary Tale (2014)
#107Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should have never been thought of or approved.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#108Was this Knight Capital?
> Knight Capital Group
Yep. Practically the canonical case study in deployment errors.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#109Earlier quoted context omitted.
It wasn't a bailout, it was investment money. Bailout could imply government throwing a lifeguard
> It wasn't a bailout, it was investment money. Almost all bailouts are investments, whether it's a government or private bailout. The investments are questionable sometimes, but they're still investments.
If the answer is nothing and no, then it’s a bailout philosophically. If the existing investors get diluted then they’re in part paying for the new capital injection.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#110Earlier quoted context omitted.
I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t look right.
But how would you even start to define something as stochastic as trading activity as “not looking right”?