The $440M software error at Knight Capital (2019)
henricodolfing.com
The $440M software error at Knight Capital (2019)
1–10 of 177 posts
Re: The $440M software error at Knight Capital (2019)
#2The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012.
Some of the issues mentioned include:
- Keeping synthetic test data generation as part of a production build.
- Keeping dead code for years.
- Re-purposing a feature flag.
- Refactoring without regression tests.
- Manual deployments without peer reviews. They forgot to update one of their servers with the new code.
- Automated alerts sent via email were ignored.
- Rolled back to a version of the code running on the server they forgot to update, making things worse.
- Rushing out a release without proper software engineering hygiene.
For those here who are in HFT circles, have things improved after the Knight Capital Group debacle?
edit: formatting
Re: The $440M software error at Knight Capital (2019)
#3The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…
Hard to speak for HFT in general. Like in software, different firms have different levels of hygiene. About half of your bullet points were true of my previous employer, at my time of leaving.
Re: The $440M software error at Knight Capital (2019)
#4Back in the day $440M loss due to coding error was a landmark warning case. How could this happen??
In 2021 alone something like $10B was lost due to bugs in defi land.
Something about the worst possible thing could happen tends to happen eventually and it gets worse every passing year.
Re: The $440M software error at Knight Capital (2019)
#5The incident happened after a technician forgot to copy the new Retail Liquidity Program (RLP) code to one of the eight SMARS computer servers, which was Knight's automated routing system for equity orders. RLP code repurposed a flag that was formerly used to activate an old function known as 'Power Peg'. Power Peg was designed to move stock prices higher and lower in order to verify the behavior of trading algorithms in a controlled environment. Therefore, orders sent with the repurposed flag to the eighth server triggered the defective Power Peg code still present on that server [1]
> Power Peg was designed to move stock prices higher and lower in order to verify the behavior of trading algorithms in a controlled environment.
This is insane. Make one wonder, what is or isn't actually being deployed in prod in 2022.
[1] https://en.wikipedia.org/wiki/Knight_Capital_Group#2012_stoc...
Re: The $440M software error at Knight Capital (2019)
#6Interesting. Five years prior, this story was posted on this blog: https://dougseven.com/2014/04/17/knightmare-a-devops-caution...
Re: The $440M software error at Knight Capital (2019)
#7The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…
Some of this is unforgivable, but reflecting on it I also realized that software engineering at quant firms has an almost impossible mandate. You want something akin to the extreme rigor of mission critical software (airplanes, cars, NASA, etc), while also remaining nimble enough to modify strategies as market conditions rapidly evolve.
Re: The $440M software error at Knight Capital (2019)
#8The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…
Some of this is unforgivable, but reflecting on it I also realized that software engineering at quant firms has an almost impossible mandate. You want something akin to the extreme rigor of mission critical software (airplanes, cars, NASA, etc), while also remaining nimble enough to modify strategies as market conditions rapidly evolve.
Same is true for blockchain smartcontracts, which have similar catastrophic consequences.
Re: The $440M software error at Knight Capital (2019)
#9The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…
Some of this is unforgivable, but reflecting on it I also realized that software engineering at quant firms has an almost impossible mandate. You want something akin to the extreme rigor of mission critical software (airplanes, cars, NASA, etc), while also remaining nimble enough to modify strategies as market conditions rapidly evolve.
Why unforgivable? It's only numbers in an account. No one died.
Re: The $440M software error at Knight Capital (2019)
#10The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…
Some of this is unforgivable, but reflecting on it I also realized that software engineering at quant firms has an almost impossible mandate. You want something akin to the extreme rigor of mission critical software (airplanes, cars, NASA, etc), while also remaining nimble enough to modify strategies as market conditions rapidly evolve.
It is challenging, although, with financial markets, it seems like it would be simpler to have some automatic anomaly detection mechanism to unplug or slow things down to prevent further damage.