Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Knightmare: A DevOps Cautionary Tale (2014)

dougseven.com

Post reply on HN
Print view View original

171–180 of 294 posts

Re: Knightmare: A DevOps Cautionary Tale (2014)

#171

Post by TheAlchemist »

thrashh wrote: post #129
lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

I agree. It doesn’t matter if you give an inexperienced person a hammer or a saw — they’ll still screw it up. My biggest pet peeve is they NO ONE ever does failure modeling. I swear everyone builds things assuming it will work perfectly. Then when you mention if one part fails, it will completely bring down everything, they’ll say that it’s a 1 in a million chance. Yeah, the problem isn’t that it’s unlikely, it’s tha…

No one, is quite a bold assumption !

It's actually quite routine stuff now in finance at least - to perform some kind of 'fire test' on a regular basis - you shut down some components during the day, and switch to backups solutions, to test everything works smoothly.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#172

Post by pxmpxm »

hgomersall wrote: post #41
daft_pink wrote: post #39

I'm so glad I don't write code that automatically routes millions of dollars with no human intervention. It's like writing code that flies a jumbo jet. Who wants that kind of responsibility.

I'm so glad I'm not wasting my life working in finance.

Actually it's one of the few truly intellectually-pure endeavors. Everything else is the same pursuit with extra steps:

Make a trading strategy to make money

vs

Make a cutting edge machine learning classifier to back out latent meaning in search queries to produce better search results to drive more traffic to google to sell ads to make money

Re: Knightmare: A DevOps Cautionary Tale (2014)

#173

Post by pxmpxm »

UncleMeat wrote: post #34
Nevermark wrote: post #23

Earlier quoted context omitted.

Leeching implies someone has found a way to skim value from you without providing value. Someone taking on loads of risk to carry out your commands efficiently is providing value. You can argue whether they are doing so competently or not, or whether they are pricing optimally or not, but they are not just ”takers” or “leeches”.

How are they taking on loads of risk? Risk has a particular meaning in investment and "well, a bug can blow up my company" isn't part of that meaning. Simply creating risky (in the colloquial meaning) things is not itself a reason to deserve money.

> How are they taking on loads of risk?

This entire story is about a trading firm that lost 400m trying to provide market liquidity. Which part of the loads of risk isn't clear in this context?

Re: Knightmare: A DevOps Cautionary Tale (2014)

#175

Post by harerazer »

SoftTalker wrote: post #38
thorum wrote: post #13

Honestly seems like the market itself should have safeguards against this kind of thing.

The safeguard is "you go bankrupt if you fuck up" Imagine there was some way for a trading company to execute billions of dollars of trades and they say "ooops, sorry, that was all a mistake" can you not see how that would be abused? Now, the story also says that within a minute of the market opening, the experienced traders knew something was wrong. Do they bear any culpability for jumping on those trades, making th…

This isn’t really correct. Typically exchanges have safety parameters which market makers can set according to how they wish to trade, and if you exceed those your orders will no longer be accepted and existing orders may also be pulled.

Obviously there are false positives occasionally and there is typically communication between the exchange and the market maker to ensure those don’t reoccur.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#176

Post by shric »

daft_pink wrote: post #39

I'm so glad I don't write code that automatically routes millions of dollars with no human intervention. It's like writing code that flies a jumbo jet. Who wants that kind of responsibility.

I don't know if it's like this at every company, but typically there are plenty of humans keeping a close eye on what's going on whenever the software is placing orders on an exchange.

I suspect we can thank this incident in part.

Re: Knightmare: A DevOps Cautionary Tale (2014)

#177

Post by lolinder »

> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point

This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get resurrected, no rogue server.

Maybe Knight Capital wasn't using version control and held onto this code "just in case", but I've seen this same resistance to deleting code in programmers working in repos that are completely under VCS, and it's flabbergasting. If you need it again, you can always bring it back from version control. If you need it again but forget it's there, you'd do the same with the dead code path. Leaving it in the source tree is pure liability.

EDIT: Kevlin Henney gave an excellent talk at GOTO about software reliability and he touches on this, using Knight Capital as the example—he actually cites this very blog post [0]. The whole talk is excellent, but I've linked the three minutes where he talks about Knight Capital.

> The problem is there is no code that is truly dead. It turns out all you need to do is make a small assumption, a change of an assumption and then suddenly it's no longer dead, it's zombie code. It has come back to life and the zombie apocalypse costs money.

[0] https://youtu.be/IiGXq3yY70o?si=hZ9HB2dlfj0vHvNK&t=463

Re: Knightmare: A DevOps Cautionary Tale (2014)

#178

Post by emmelaich »

lopkeny12ko wrote: post #17

I'm not sure how automated deployments would have solved this problem. In fact, if anything, it would have magnified the impact and fallout of the problem. Substitute "a developer forgot to upload the code to one of the servers" for "the deployment agent errored while downloading the new binary/code onto the server and a bug in the agent prevented the error from being surfaced." Now you have the same failure mode, an…

Technically, a flag re-use was the most impactful error, code wise.

A flag of such importance should not be just on/off; the ON should require a positive response / receipt containing the name and version of the code being turned on.

[edit - don't mean for each trade, I mean validation on startup]

Re: Knightmare: A DevOps Cautionary Tale (2014)

#179

Post by roughly »

This is the Ur “devops fuckup” tale - I’ve told this to junior engineers who’ve bodged a deploy to make them feel better. I’ve been in this field for 20 years, and I can’t imagine I’ll ever have a day as bad as the engineers who got bit by this fuckup.
Post reply on HN