Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

11–20 of 34 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#11
post #4

I'm not sure the conclusion of the post is the "One and Only Answer" because a fully automated deploy process has another risk that has bitten both AWS and Google at some point: fully automatically taking down huge amounts of instances.

A lot of times those issues have been "fully automated (but with human inputs)" or "fully automated with no guardrails"

Re: Knightmare: A DevOps Cautionary Tale (2014)

#12
post #4

I'm not sure the conclusion of the post is the "One and Only Answer" because a fully automated deploy process has another risk that has bitten both AWS and Google at some point: fully automatically taking down huge amounts of instances.

A lot of times those issues have been "fully automated (but with human inputs)" or "fully automated with no guardrails"

This seems to cover all the cases. Either there are guardrails (as human inputs), or there aren't. Unless I'm missing a middle ground here?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#13
Amusing personal anecdote -- the Knight debacle caused the market in general to tumble. The week before a coworker of mine -- sure of a market drop but for other reasons -- had bought a raft or puts on the S&P 500. When I saw looking glum at work after the Knight news broke, I asked him what was wrong didn't you make a ton? Yeah, he said, but I can't get out cause my account's with Knight.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#14
Back when I used to smoke I would ocassionally hang out with this guy from an investment bank that traded on the Japanese exchange. They had really cool working hours (started a lot later in the day) because we were based in Hawaii which is a few hours behind Japan.

Anyway, the guy told me that they had multiple big red physical kill switches so that they could immediately turn things off if shit ever hit the fan with their systems.

If you have ever spent time in Michigan you'll notice that the manufacturer test vehicles have a big ass red button on the dashboard to kill the vehicle in case something goes wrong.

I cannot imagine doing anything remotely close to this sort of thing without a big ass red kill switch on my desk.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#15

Previous discussions on HN: 2014: https://news.ycombinator.com/item?id=7652036 2015: https://news.ycombinator.com/item?id=8994701

Thanks. This comment from 2015 by `ooOOoo` is also worth highlighting.

   > The post is quite poor and suffer a lot from hindsight bias.
   > Following article is so much better
   > https://www.kitchensoap.com/2013/10/29/counterfactuals-knight-capital/

Re: Knightmare: A DevOps Cautionary Tale (2014)

#16

Previous discussions on HN: 2014: https://news.ycombinator.com/item?id=7652036 2015: https://news.ycombinator.com/item?id=8994701

Thanks. This comment from 2015 by `ooOOoo` is also worth highlighting. > The post is quite poor and suffer a lot from hindsight bias. > Following article is so much better > https://www.kitchensoap.com/2013/10/29/counterfactuals-knight-capital/

FYI, this doesn’t come through as expected: the link isn’t clickable and the text isn’t readable. Please avoid using code formatting for quotes; > * ... * is a readable alternative. https://i.imgur.com/YvMA1uV.png

Re: Knightmare: A DevOps Cautionary Tale (2014)

#17

Previous discussions on HN: 2014: https://news.ycombinator.com/item?id=7652036 2015: https://news.ycombinator.com/item?id=8994701

Thanks - the top comment from vijucat in the 2015 discussion is anxiety inducing. “ - Ctrl-r for reverse-search through history - typing 'ps' to find the process status utility (of course) - pressing Enter,....and realizing that Ctrl-r actually found 'stopserver.sh' in history instead. (There's a ps inside stoPServer.sh)”

Something similar happened in production at work last year, fortunately not in my team.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#18
post #4

I'm not sure the conclusion of the post is the "One and Only Answer" because a fully automated deploy process has another risk that has bitten both AWS and Google at some point: fully automatically taking down huge amounts of instances.

Not to mention your deployment code itself can be buggy and is very difficult to write tests for. I actually got bitten by this recently. An automated deployment that I wrote years ago had an edge case race condition that could cause multiple deployments running from containers on the same docker host to collide, where the package from the first deployment would be pushed to the target of the second deployment. That deployment worked reliably for years, until one day it didn't. It was... a very stressful day.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#19
post #13

Amusing personal anecdote -- the Knight debacle caused the market in general to tumble. The week before a coworker of mine -- sure of a market drop but for other reasons -- had bought a raft or puts on the S&P 500. When I saw looking glum at work after the Knight news broke, I asked him what was wrong didn't you make a ton? Yeah, he said, but I can't get out cause my account's with Knight.

Cool story, didn't happen. There were no retail trading accounts at Knight. In fact, there was no outside money of any kind. The S&P500 fell about 0.75% on the day in question: a non-trivial decline, but not really remarkable. It was up about 0.4% on the week. Also, this is incredibly not how the OCC deals with members in default.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#20
post #13

Amusing personal anecdote -- the Knight debacle caused the market in general to tumble. The week before a coworker of mine -- sure of a market drop but for other reasons -- had bought a raft or puts on the S&P 500. When I saw looking glum at work after the Knight news broke, I asked him what was wrong didn't you make a ton? Yeah, he said, but I can't get out cause my account's with Knight.

Cool story, didn't happen. There were no retail trading accounts at Knight. In fact, there was no outside money of any kind. The S&P500 fell about 0.75% on the day in question: a non-trivial decline, but not really remarkable. It was up about 0.4% on the week. Also, this is incredibly not how the OCC deals with members in default.

>There were no retail trading accounts at Knight.

The article states “The NYSE was planning to launch a new Retail Liquidity Program (a program meant to provide improved pricing to retail investors through retail brokers, like Knight)”

This pretty strongly implies Knight was a retail broker.

I assume I’m missing something- can you clarify?

Post reply on HN