Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

21–30 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#21

Post by notnmeyer »

hyperhopper wrote: post #15

Yes, the deployment practices were bad, but they still would have had an issue even with proper practices. The real issue was re-using an old flag. That should have never been thought of or approved.

this is what stood out to me reading the story. i wonder if there was a reason why they opted for this, however half-baked.

it reads less to me like a case for devops as it does a case for better practices at every stage of development. how arrogant or willfully ignorant do you have to be to operate like this considering what’s at stake?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#22

Post by gumballindie »

> Had Knight implemented an automated deployment system – complete with configuration, deployment and test automation – the error that cause the Knightmare would have been avoided.

Would it have been avoided though? Configuration, deployment and test automation mean nothing if they don't do what they are supposed to do. Regardless of how many tests you have, if you don't test for the right stuff it's all useless.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#23

Post by Nevermark »

loa_in_ wrote: post #19
tomp wrote: post #14

Ah, Knight Capital. The warning story for every quant trader / engineer. This is what people don't realize when they say HFT (high frequency trading) is risk-free, leeching off people, etc. You make a million every day with very little volatility (the traditional way of quantifying "risk" in finance) but one little mistake, and you're gone. The technical term is "picking up pennies in front of a steamroller (train)".…

I don't see how anything here goes against it leeching off people

Leeching implies someone has found a way to skim value from you without providing value.

Someone taking on loads of risk to carry out your commands efficiently is providing value.

You can argue whether they are doing so competently or not, or whether they are pricing optimally or not, but they are not just ”takers” or “leeches”.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#24

Post by 40yearoldman »

lol. No. Deployments were not the issue. At any given time an automated deployment system could have had a mistake introduced that resulted in bad code being sent to the system. It does not matter if it was old or new code. Any code could have had this bug.

What the issue was, and it’s one that I see often. Firstly no vision into the system. Not even a dash board showing the softwares running version. How often i see people ship software without a banner posting its version and or an endpoint that simply reports the version.

Secondly no god damn kill switch. You are working with money!! Shutting down has to be an option.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#26

Post by dmurray »

lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

> The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way.

The blame completely lies with the risk management team.

The market knew there was a terrible problem, Knight knew there was a problem, yet it took 45 minutes of trying various hotfixes before they ceased trading. Either because they didn't have a kill switch, or because no one was empowered to pull the kill switch because of the opportunity cost (perhaps pulling the switch at the wrong time costs $500k in opportunity).

I worked for a competitor to Knight at the time, and we deployed terrible bugs to production all the time, and during post mortems we couldn't fathom the same thing happening to us. A dozen automated systems would have kicked in to stop individual trades, and any senior trader or operations person could have got a kill switch pulled with 60 seconds of dialogue, and not feared the repercussions. Actually, we made way less of Knight's $400m than we could have because our risk systems kept shutting strategies down because what was happening was "too good to be true".

Re: Knightmare: A DevOps Cautionary Tale (2014)

#27

Post by valdiorn »

Literally everyone in quant finance knows about knight capital. It even has its own phrase; "pulling a knight capital" (meaning; cutting corners on mission critical systems, even ones that can bankrupt the company in an instant, and experiencing the consequences)

Re: Knightmare: A DevOps Cautionary Tale (2014)

#28

Post by SoftTalker »

ledauphin wrote: post #18
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

Or at least not by a developer who has made that sort of mistake in the past.

I don't know what software engineering programs teach these days, but in the 1980s there was very little inclusion of case studies of things that went wrong. This was unlike the courses in the business school (my undergrad was CS major + business minor) nor I would presume what real engineering disciplines teach.

My first exposure to a fuckup in production was a fuckup in production on my first job.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#29

Post by 40yearoldman »

40yearoldman wrote: post #24

lol. No. Deployments were not the issue. At any given time an automated deployment system could have had a mistake introduced that resulted in bad code being sent to the system. It does not matter if it was old or new code. Any code could have had this bug. What the issue was, and it’s one that I see often. Firstly no vision into the system. Not even a dash board showing the softwares running version. How often i see…

Oh god. I just realized this is a PM. A plight on software engineering. People who play technical, and “take the requirements from the customer to the engineer”. What’s worse is when they play engineer too.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#30

Post by lopkeny12ko »

ledauphin wrote: post #18
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

The blame here may indeed lie with whoever decided that reusing an old flag was a good idea. As anyone who has been in software development for any time can attest, this decision was not necessarily - and perhaps not even likely - made by a "developer."

I doubt any manager or VP cares or knows enough about the technical details of the code to dictate the name that should be used for a feature flag, of all things.
Post reply on HN