Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

121–130 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#121

Post by dang »

Related:

Knightmare: A DevOps Cautionary Tale (2014) - https://news.ycombinator.com/item?id=22250847 - Feb 2020 (33 comments)

Knightmare: A DevOps Cautionary Tale (2014) - https://news.ycombinator.com/item?id=8994701 - Feb 2015 (85 comments)

Knightmare: A DevOps Cautionary Tale - https://news.ycombinator.com/item?id=7652036 - April 2014 (60 comments)

Also:

The $440M software error at Knight Capital (2019) - https://news.ycombinator.com/item?id=31239033 - May 2022 (172 comments)

Bugs in trading software cost Knight Capital $440M - https://news.ycombinator.com/item?id=4329495 - Aug 2012 (1 comment)

Knight Capital Says Trading Glitch Cost It $440 Million - https://news.ycombinator.com/item?id=4329101 - Aug 2012 (90 comments)

Others?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#122

Post by salawat »

callalex wrote: post #70
daft_pink wrote: post #39

I'm so glad I don't write code that automatically routes millions of dollars with no human intervention. It's like writing code that flies a jumbo jet. Who wants that kind of responsibility.

It’s fine to have that kind of responsibility, but it has to actually be your responsibility. Which means you have to be empowered to say “no, we aren’t shipping this until XYZ is fixed” even if XYZ will take another two years to build and the boss wants to ship tomorrow.

Yep. Until the capacity to say unoverridably "No" materializes, there's a lot of code I refuse to have responsibility for delegated to me.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#123

Post by rcpt »

lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

It seems like the kind of thing that would be canaried which is the kind of thing that you'd typically build alongside automated deployment

Re: Knightmare: A DevOps Cautionary Tale (2014)

#124

Post by [deleted] »

xyst wrote: post #76

Having worked in some Fortune 500 financial firms and low rent “fintech” upstarts, I am not surprised this happened. Decades of bandaid fixes, years of rotating out different consultants/contractors, and software rot. Plus years of emphasizing mid level management over software quality. As other have mentioned, I don’t think “automation of deployment” would have prevented this company’s inevitable downfall. If it was…

[deleted]

Re: Knightmare: A DevOps Cautionary Tale (2014)

#125

Post by nly »

xcv123 wrote: post #99
jonplackett wrote: post #85

Earlier quoted context omitted.

The thing I was surprised about is that they survived! After all that they still got a $400 million cash bailout!

They were probably worth much more than $400M before the failure so it was a good investment opportunity. They would have been a money printing machine aside from this one major fuckup.

Their IP (proprietary trading algos) etc were probably worth a lot at the time.

These days probably not so. I wouldn't imagine there are any market makers left in NYSE and NASDAQ who aren't deploying FPGAs to gain a speed edge.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#126

Post by hammeiam »

shric wrote: post #79
hgomersall wrote: post #41

Earlier quoted context omitted.

I'm so glad I'm not wasting my life working in finance.

I've worked in various small to medium IT companies, a FAANG and another fortune 500 tech company. 6 months ago I moved to a proprietary trading company/market maker and it's the most interesting and satisfying place I've worked so far. I hope to continue to "waste my life" for many years to come.

May I ask which one, and what your process was that led you to them?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#127

Post by wruza »

daft_pink wrote: post #39

I'm so glad I don't write code that automatically routes millions of dollars with no human intervention. It's like writing code that flies a jumbo jet. Who wants that kind of responsibility.

It feels anxiety inducing at first, but if you have good controls and monitoring in place, it becomes daily routine. You basically address the points you naturally have and the more reasonably anxious you are, the better for the business. From my experience with finance, I’d wager that problem at Knight was 10% tech issues, 90% CTO-ish person feeling ballsy. In general, not exactly that day or week.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#129

Post by thrashh »

lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

I agree. It doesn’t matter if you give an inexperienced person a hammer or a saw — they’ll still screw it up.

My biggest pet peeve is they NO ONE ever does failure modeling.

I swear everyone builds things assuming it will work perfectly. Then when you mention if one part fails, it will completely bring down everything, they’ll say that it’s a 1 in a million chance. Yeah, the problem isn’t that it’s unlikely, it’s that when it does happen, you’ve perfectly designed your system to destroy itself.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#130

Post by Jorge1o1 »

distortionfield wrote: post #91
amluto wrote: post #43

Earlier quoted context omitted.

I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t look right.

But how would you even start to define something as stochastic as trading activity as “not looking right”?

I’ve had to fill out forms for new algorithms / quant strategies with questions like:

- how many orders per minute do you expect to create?

- how many orders per minute do you expect to cancel/amend?

- what’s your max per-ticker position?

- what’s your max strategy-level GMV/NMV?

Etc.

Any one of those questions can be used to set up killswitches.

[edited for formatting]

Post reply on HN