Live data from Hacker News

The $440M software error at Knight Capital (2019)

henricodolfing.com

21–30 of 177 posts

Re: The $440M software error at Knight Capital (2019)

#21

Back in the day $440M loss due to coding error was a landmark warning case. How could this happen?? In 2021 alone something like $10B was lost due to bugs in defi land. Something about the worst possible thing could happen tends to happen eventually and it gets worse every passing year.

Was just about to comment along these lines. If I read about this a few years ago I would be shocked. Now after seeing so many flubs in the crypto space, my reaction is just 'meh'

Re: The $440M software error at Knight Capital (2019)

#22
post #2

The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…

I worked in a lot of front office groups in investment banking. The short spell I did in HFT had great software development and DevOps practices.

Re: The $440M software error at Knight Capital (2019)

#23

Back in the day $440M loss due to coding error was a landmark warning case. How could this happen?? In 2021 alone something like $10B was lost due to bugs in defi land. Something about the worst possible thing could happen tends to happen eventually and it gets worse every passing year.

I actually always think of the Knight case and similar ones when people see a DeFi organization have an issue and extrapolate that to an issue with the entire DeFi concept.

Its so obvious that those people have no clue whats going on in the markets they respect. Truth be told, many of them dont like markets at all. So its just a lack of exposure and compounded ignorance.

Re: The $440M software error at Knight Capital (2019)

#24

Back in the day $440M loss due to coding error was a landmark warning case. How could this happen?? In 2021 alone something like $10B was lost due to bugs in defi land. Something about the worst possible thing could happen tends to happen eventually and it gets worse every passing year.

I actually always think of the Knight case and similar ones when people see a DeFi organization have an issue and extrapolate that to an issue with the entire DeFi concept. Its so obvious that those people have no clue whats going on in the markets they respect. Truth be told, many of them dont like markets at all. So its just a lack of exposure and compounded ignorance.

Many traditional finance issues are fixable though, there are many more errors which don’t become big stories because they are reasonably reversed as only minor inconveniences.

Re: The $440M software error at Knight Capital (2019)

#25
post #2

The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…

This is all basic stuff I look to set up in every team, and it's crazy given how these firms work directly with tons of money that they don't have an even higher standard. Guess I wasn't wrong turning down these roles.

Re: The $440M software error at Knight Capital (2019)

#27

at the end .. its just money going from one account to another right? Its not like some physical thing that has perished and cant be brought back. Why is it difficult to reverse the transactions?

Because those transactions cause other transactions, which cause others, and so on and so forth. You'd have to reset the market for the day.

Imagine how pissed you'd be if you made money off Knight's mistake and it all just disappeared the next day.

Re: The $440M software error at Knight Capital (2019)

#29
post #11
post #2

The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…

I worked in algo trading for years, eventually got out because quite frankly the level of risk I was carrying on my shoulders everyday for what I was being paid were just way out of whack, I at least personally never got the huge pay days that people talked about until after I left finance for more pure tech. Interestingly, I worked at Knight and my team pioneered trying to blow up the firm, but that was in 2004, and…

I appreciate your comment about pay. Recruiters will often tell me "it's finance so of course the pay will be substantial." Then when we get to talking numbers they're like "300k a year". Oh, you mean the going rate at a FAANG? And I have to move to New York or Chicago, work more hours, and actively work for people who I know are taking home paychecks with 7+ zeroes on them? Come on. Sometimes it's 400 plus bonus or whatever, which is based on fund performance and yada yada. But it feels way off. I had heard so much about the staggering paydays at these places but it seems you need an ML PHD or some trading chops to be part of that.

Re: The $440M software error at Knight Capital (2019)

#30
post #2

The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…

I used to work in HFT. I have seen highly variable practices in this case, including a "mini-knight" incident in the single-digit millions due to tech debt and poor test coverage. However, the most useful change that has resulted from the KCG debacle was adding several layers of kill switches, a dedicated ops team to watch trading and flip the kill switches, and embracing devops automation.

There is a much more serious focus on having a defense in depth, and making sure that problems like this are noticed before they become an issue. Rollbacks are no longer the first action when something goes wrong: the kill switch comes first.

Dead code, tech debt, repurposed flags, and spotty test coverage are everywhere still.

Post reply on HN