Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The $440M software error at Knight Capital (2019)

henricodolfing.com

Post reply on HN
Print view View original

31–40 of 177 posts

Re: The $440M software error at Knight Capital (2019)

#31

Post by vmception »

colechristensen wrote: post #24
vmception wrote: post #23

Earlier quoted context omitted.

I actually always think of the Knight case and similar ones when people see a DeFi organization have an issue and extrapolate that to an issue with the entire DeFi concept. Its so obvious that those people have no clue whats going on in the markets they respect. Truth be told, many of them dont like markets at all. So its just a lack of exposure and compounded ignorance.

Many traditional finance issues are fixable though, there are many more errors which don’t become big stories because they are reasonably reversed as only minor inconveniences.

like how Credit Suisse is going to reverse their Bill Hwang losses? I guess in this conversation we can't distinguish from irreversible asset value and liquidity issue to misdirected transactions that inherit partial reversibility.

similarly, maybe you/they just don't see the headlines of thwarted attacks in DeFi that work specifically due to design considerations.

I'll take the permission to fail. The rapid iteration creates some really fascinating systems in very short time periods, for me. One project implodes, 100 (or 1000) more harden, bigger money comes in creating more assurances for users like easier recovery and compensation paths, all while continuing to rapidly iterate.

Re: The $440M software error at Knight Capital (2019)

#32

Post by strgcmc »

ceejayoz wrote: post #27
robofanatic wrote: post #26

at the end .. its just money going from one account to another right? Its not like some physical thing that has perished and cant be brought back. Why is it difficult to reverse the transactions?

Because those transactions cause other transactions, which cause others, and so on and so forth. You'd have to reset the market for the day. Imagine how pissed you'd be if you made money off Knight's mistake and it all just disappeared the next day.

Except, well, cancelling transactions obviously does happen, sometimes: https://www.reuters.com/business/lme-suspends-nickel-trading...

Knight was probably too messy to rollback cleanly, but that just means it's a matter of cost/complexity/politics... if you're a big enough player, then the exchange will do you favors, like in the LME case.

Free markets, lol

Re: The $440M software error at Knight Capital (2019)

#33

Post by posterboy »

rebelos wrote: post #7
bfm wrote: post #2

The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…

Some of this is unforgivable, but reflecting on it I also realized that software engineering at quant firms has an almost impossible mandate. You want something akin to the extreme rigor of mission critical software (airplanes, cars, NASA, etc), while also remaining nimble enough to modify strategies as market conditions rapidly evolve.

That's a weird statement.

The extreme rigor on the one hand seems to require a value judgement of the real benefits to HTF that I'm not willing to make. The remaining nimble'ity, on the other hand, is an odd word to use over agility or old fashioned responsibility. The benefit is proportional to it, but not exclusively.

The rapidly evolving market conditions concern regular trade too. Swift reactions are expected in any other systems application. "almost impossible" is a weasel word. It's almost impossible to win except for the last man standing, is that it? And there's no practical upper limit to nimble'y, though conservative estimates indicate that less work is more.

What's missing is the perverse incentives, corrupt policies, sociopathic leadership, ...

Re: The $440M software error at Knight Capital (2019)

#34

Post by rubyskills »

ceejayoz wrote: post #27
robofanatic wrote: post #26

at the end .. its just money going from one account to another right? Its not like some physical thing that has perished and cant be brought back. Why is it difficult to reverse the transactions?

Because those transactions cause other transactions, which cause others, and so on and so forth. You'd have to reset the market for the day. Imagine how pissed you'd be if you made money off Knight's mistake and it all just disappeared the next day.

Exactly this. If you're a market maker, likely your trades impact your own trades too. As you accumulate a position, your average price is going up with it. Trades should not just roll back because one large hedge fund screwed up. Imagine being a retail trader with that expectation. Would be nice!

Re: The $440M software error at Knight Capital (2019)

#35

Post by rubyskills »

strgcmc wrote: post #32
ceejayoz wrote: post #27

Earlier quoted context omitted.

Because those transactions cause other transactions, which cause others, and so on and so forth. You'd have to reset the market for the day. Imagine how pissed you'd be if you made money off Knight's mistake and it all just disappeared the next day.

Except, well, cancelling transactions obviously does happen, sometimes: https://www.reuters.com/business/lme-suspends-nickel-trading... Knight was probably too messy to rollback cleanly, but that just means it's a matter of cost/complexity/politics... if you're a big enough player, then the exchange will do you favors, like in the LME case. Free markets, lol

This is much easier to do in a centralized futures market. I can't imagine a rollback in stocks being easy or possible.

Re: The $440M software error at Knight Capital (2019)

#36

Post by commandlinefan »

pclmulqdq wrote: post #30
bfm wrote: post #2

The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…

I used to work in HFT. I have seen highly variable practices in this case, including a "mini-knight" incident in the single-digit millions due to tech debt and poor test coverage. However, the most useful change that has resulted from the KCG debacle was adding several layers of kill switches, a dedicated ops team to watch trading and flip the kill switches, and embracing devops automation. There is a much more serio…

> poor test coverage

Yet you don't have to hang around here long to be told that "Unit Testing is Overrated": https://tyrrrz.me/blog/unit-testing-is-overrated

Re: The $440M software error at Knight Capital (2019)

#37

Post by anamax »

robofanatic wrote: post #26

at the end .. its just money going from one account to another right? Its not like some physical thing that has perished and cant be brought back. Why is it difficult to reverse the transactions?

> Why is it difficult to reverse the transactions?

Why should the transactions be reversed?

If things had gone according to plan, Knight would have made several million dollars that day, some likely because of a mistake by someone else or an unavoidable circumstance, just like it did on other days.

Those other people weren't made whole, so why should Knight be any different?

Re: The $440M software error at Knight Capital (2019)

#38

Post by caffeine »

fullsend wrote: post #29
kevstev wrote: post #11

Earlier quoted context omitted.

I worked in algo trading for years, eventually got out because quite frankly the level of risk I was carrying on my shoulders everyday for what I was being paid were just way out of whack, I at least personally never got the huge pay days that people talked about until after I left finance for more pure tech. Interestingly, I worked at Knight and my team pioneered trying to blow up the firm, but that was in 2004, and…

I appreciate your comment about pay. Recruiters will often tell me "it's finance so of course the pay will be substantial." Then when we get to talking numbers they're like "300k a year". Oh, you mean the going rate at a FAANG? And I have to move to New York or Chicago, work more hours, and actively work for people who I know are taking home paychecks with 7+ zeroes on them? Come on. Sometimes it's 400 plus bonus or…

The attitude that finance pays more is a leftover from a previous era. 10-15 years ago it was true: the profits from HFT were so also way, way bigger and split up amongst a much smaller group of firms.

Now those firms are all in a completely competitive industry squeezing each other for basis points.

Meanwhile the definition of a FAANG is that it has an effective monopoly, and these companies are taking in way more money than the HFT industry. (Netflix is losing its monopoly but we can’t really drop N from the acronym without a replacement..)

Re: The $440M software error at Knight Capital (2019)

#39

Post by gjs278 »

fullsend wrote: post #29
kevstev wrote: post #11

Earlier quoted context omitted.

I worked in algo trading for years, eventually got out because quite frankly the level of risk I was carrying on my shoulders everyday for what I was being paid were just way out of whack, I at least personally never got the huge pay days that people talked about until after I left finance for more pure tech. Interestingly, I worked at Knight and my team pioneered trying to blow up the firm, but that was in 2004, and…

I appreciate your comment about pay. Recruiters will often tell me "it's finance so of course the pay will be substantial." Then when we get to talking numbers they're like "300k a year". Oh, you mean the going rate at a FAANG? And I have to move to New York or Chicago, work more hours, and actively work for people who I know are taking home paychecks with 7+ zeroes on them? Come on. Sometimes it's 400 plus bonus or…

No post body was provided.

Re: The $440M software error at Knight Capital (2019)

#40

Post by bfm »

user3939382 wrote: post #18

Here’s a 225 million dollar oopsie from 2005 https://www.foxnews.com/story/typing-error-causes-225m-loss-...

Today there was a 300B oopsie in Europe caused by a Citibank "glitch" bloomberg.com/news/articles/2022-05-02/citi-s-london-trading-desk-behind-rare-european-flash-crash
Post reply on HN