191–200 of 294 posts
Earlier quoted context omitted.
I would argue the real issue was the lack of an automated system (or multiple automated systems) that would hit the kill switch if the trading activity didn’t look right.
But how would you even start to define something as stochastic as trading activity as “not looking right”?
Earlier quoted context omitted.
How are they taking on loads of risk? Risk has a particular meaning in investment and "well, a bug can blow up my company" isn't part of that meaning. Simply creating risky (in the colloquial meaning) things is not itself a reason to deserve money.
When you make markets you are literally paid the spread to assume the risk of holding the position.
Earlier quoted context omitted.
> The blame here lies squarely with the developers--the code was written in a non-backwards-compatible way. The blame completely lies with the risk management team. The market knew there was a terrible problem, Knight knew there was a problem, yet it took 45 minutes of trying various hotfixes before they ceased trading. Either because they didn't have a kill switch, or because no one was empowered to pull the kill sw…
They were asleep at the wheel, not unlike all the random brokerages that blew up when swiss central bank pulled the CHF peg in 2015. This is a culture problem - as soon as you load up your trading firm with a bunch of software industry hires, you end up with jiras and change management workflows instead of people on deck that have context for what they're doing. That's the only way to explain reverse scalping for 45…
I feel like the first thing I would build into any automated trading system is a kill switch? then every single diff or pull request I add would have some sort of automated testing to ensure the kill switch still works. Also I'd manually flip it on/off once a day to make sure it works for real. That seems like the single most important thing to build and make sure works. Or is the system too complex for something lik…
> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…
An update to simply "remove old code" might be difficult if someone sees any change as creating a risk of something going wrong. And to be fair, any change is a risk, but so is leaving old code around.
At least now we have this case to point to as a clear example of the risk.
> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…
What completely baffles me everytime I come across their story is that they repurposed an existing flag instead of making a new one! Why?
> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…
I think a lot of developers only know the basics of git. They can check in changes, they can look at history with git log, and maybe they know how to use git blame.
They often don't know how to filter git history. They often don't know about the git pickaxe, or about exclude patterns, and don't even think to question if you can do something like "git log -G'int.*foo\(' -- ':(exclude)directory'" to search for 'foo' in the git log, excluding some directory.
They know how to "grep" within the existing code tree though, so they know if it's not deleted they can find it again with the right grep. If it's deleted, they might not know how to find it in git history.
I sympathize with this to a degree actually. Code in the git log is invisible to a lot of tooling, so for example it won't show up in autocomplete if your text editor might have otherwise suggested it, it won't show up in your library documentation, etc.
If you truly think the code will be used again, I think it's at least defensible to leave it in the tree so that it comes along for refactors, and ends up being found when it's needed.
For cases like Knight capital, where it's obviously never going to be useful again, it's not defensible of course.
"The company agreed to be acquired by Getco LLC in December 2012 after an August 2012 trading error lost $460 million. The merger was completed in July 2013, forming KCG Holdings.
...On April 20, 2017, KCG announced that it had agreed to be acquired by Virtu Financial for $20 per share in cash in a deal valued at approximately $1.4 billion."
> why code that had been dead for 8-years was still present in the code base is a mystery, but that’s not the point This seems to be exactly the point! For 8 years they left unused code in place, seemingly only bothering to remove it because they wanted to repurpose a flag. If they'd done the right thing 8 years prior and removed code they weren't using, this story plays out very differently. No ancient routines get…
> I've seen this same resistance to deleting code in programmers working in repos that are completely under VCS, and it's flabbergasting I think a lot of developers only know the basics of git. They can check in changes, they can look at history with git log, and maybe they know how to use git blame. They often don't know how to filter git history. They often don't know about the git pickaxe, or about exclude pattern…
You don't want these to be visible to autocomplete, because they're outdated and would need major modification to be correct again. If you do need to resurrect them, they're trivial to find in the git history—just search for the commit named "remove foo"—and they should pass through code review as if they were brand new code, because a lot of stuff will have changed around them in the intervening time.
I worked in fintech for a few years. I'll never again work on software that's responsible for trading, you could offer $1M/year and I wouldn't take it. By far the most stress I've ever experienced at a job.