I'm not sure the conclusion of the post is the "One and Only Answer" because a fully automated deploy process has another risk that has bitten both AWS and Google at some point: fully automatically taking down huge amounts of instances.
Knightmare: A DevOps Cautionary Tale (2014)
11–20 of 34 posts
Re: Knightmare: A DevOps Cautionary Tale (2014)
#12I'm not sure the conclusion of the post is the "One and Only Answer" because a fully automated deploy process has another risk that has bitten both AWS and Google at some point: fully automatically taking down huge amounts of instances.
A lot of times those issues have been "fully automated (but with human inputs)" or "fully automated with no guardrails"
Re: Knightmare: A DevOps Cautionary Tale (2014)
#13Re: Knightmare: A DevOps Cautionary Tale (2014)
#14Anyway, the guy told me that they had multiple big red physical kill switches so that they could immediately turn things off if shit ever hit the fan with their systems.
If you have ever spent time in Michigan you'll notice that the manufacturer test vehicles have a big ass red button on the dashboard to kill the vehicle in case something goes wrong.
I cannot imagine doing anything remotely close to this sort of thing without a big ass red kill switch on my desk.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#15Previous discussions on HN: 2014: https://news.ycombinator.com/item?id=7652036 2015: https://news.ycombinator.com/item?id=8994701
> The post is quite poor and suffer a lot from hindsight bias.
> Following article is so much better
> https://www.kitchensoap.com/2013/10/29/counterfactuals-knight-capital/Re: Knightmare: A DevOps Cautionary Tale (2014)
#16Previous discussions on HN: 2014: https://news.ycombinator.com/item?id=7652036 2015: https://news.ycombinator.com/item?id=8994701
Thanks. This comment from 2015 by `ooOOoo` is also worth highlighting. > The post is quite poor and suffer a lot from hindsight bias. > Following article is so much better > https://www.kitchensoap.com/2013/10/29/counterfactuals-knight-capital/
Re: Knightmare: A DevOps Cautionary Tale (2014)
#17Previous discussions on HN: 2014: https://news.ycombinator.com/item?id=7652036 2015: https://news.ycombinator.com/item?id=8994701
Thanks - the top comment from vijucat in the 2015 discussion is anxiety inducing. “ - Ctrl-r for reverse-search through history - typing 'ps' to find the process status utility (of course) - pressing Enter,....and realizing that Ctrl-r actually found 'stopserver.sh' in history instead. (There's a ps inside stoPServer.sh)”
Re: Knightmare: A DevOps Cautionary Tale (2014)
#18I'm not sure the conclusion of the post is the "One and Only Answer" because a fully automated deploy process has another risk that has bitten both AWS and Google at some point: fully automatically taking down huge amounts of instances.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#19Amusing personal anecdote -- the Knight debacle caused the market in general to tumble. The week before a coworker of mine -- sure of a market drop but for other reasons -- had bought a raft or puts on the S&P 500. When I saw looking glum at work after the Knight news broke, I asked him what was wrong didn't you make a ton? Yeah, he said, but I can't get out cause my account's with Knight.
Re: Knightmare: A DevOps Cautionary Tale (2014)
#20Amusing personal anecdote -- the Knight debacle caused the market in general to tumble. The week before a coworker of mine -- sure of a market drop but for other reasons -- had bought a raft or puts on the S&P 500. When I saw looking glum at work after the Knight news broke, I asked him what was wrong didn't you make a ton? Yeah, he said, but I can't get out cause my account's with Knight.
Cool story, didn't happen. There were no retail trading accounts at Knight. In fact, there was no outside money of any kind. The S&P500 fell about 0.75% on the day in question: a non-trivial decline, but not really remarkable. It was up about 0.4% on the week. Also, this is incredibly not how the OCC deals with members in default.
The article states “The NYSE was planning to launch a new Retail Liquidity Program (a program meant to provide improved pricing to retail investors through retail brokers, like Knight)”
This pretty strongly implies Knight was a retail broker.
I assume I’m missing something- can you clarify?