Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The $440M software error at Knight Capital (2019)

henricodolfing.com

Post reply on HN
Print view View original

71–80 of 177 posts

Re: The $440M software error at Knight Capital (2019)

#71

Post by sokoloff »

bob1029 wrote: post #52
bfm wrote: post #2

The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…

Repurposing feature flags is some kind of next dimension horror for me. We've got quite a few of these to deal with, and if someone started changing what they mean we'd be fucked super fast. Simply suggesting that we alter the meaning of an existing FF would result in the resignation of a non-zero number of project managers on my team. Rolling back code is another thing I have no tolerance for anymore. The only optio…

You’re likely using human-readable names for the flags and shipping a multi-MB payload of JS and JSON. An HFT firm is likely bit-packing flags so they can send an 8 byte payload rather than 10 and might be using an FPGA hanging right off the PHY to figure out “is this message even interesting to me?”

Your feature flag might be “SHOW_STRIKETHROUGH_PRICING_ON_CROSSSELL_OFFERS”; theirs is a bit mask macro to pick off the 5th bit from the 7th byte. (Why do they care? Because if they allow themselves to get fat and slow, a competitor will take the money.)

Roll-forward only, same day SLA is probably right for your business, but isn’t for a company that could have their systems dusting off $1M every handful of seconds that bad code is running.

Different business problems call for different technical approaches. You should no more adopt theirs than they should yours.

Re: The $440M software error at Knight Capital (2019)

#72

Post by hn_go_brrrrr »

bob1029 wrote: post #52
bfm wrote: post #2

The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…

Repurposing feature flags is some kind of next dimension horror for me. We've got quite a few of these to deal with, and if someone started changing what they mean we'd be fucked super fast. Simply suggesting that we alter the meaning of an existing FF would result in the resignation of a non-zero number of project managers on my team. Rolling back code is another thing I have no tolerance for anymore. The only optio…

Rolling back isn't about avoiding rebuilds, it's about restoring to a known-good state. Making an emergency patch is typically far riskier than going back to last week's build. We always favor rollbacks, unless there were critical fixes in the current release we absolutely cannot afford to lose.

Re: The $440M software error at Knight Capital (2019)

#73

Post by Aperocky »

throwyawayyyy wrote: post #46

Random, but I interviewed at Knight Capital for a software engineering position a few weeks before this all went down. I was in London, so the interview was done over the phone. Picture me in the evening, handwriting C to solve some problem (the fog of time too thick to remember what that problem was), then reading out what I'd written, semicolons and all, to the interviewer. Because of course there was no shared doc…

this is hilarious, apparently someone on their end thought it was an acceptable interview format, thanks for sharing.

Re: The $440M software error at Knight Capital (2019)

#74

Post by civilized »

gzer0 wrote: post #5

The incident happened after a technician forgot to copy the new Retail Liquidity Program (RLP) code to one of the eight SMARS computer servers, which was Knight's automated routing system for equity orders. RLP code repurposed a flag that was formerly used to activate an old function known as 'Power Peg'. Power Peg was designed to move stock prices higher and lower in order to verify the behavior of trading algorithm…

So... you're saying the company activated a tool called "Power Peg"...

and.... they got f***ed?

Well, what did they expect?

Re: The $440M software error at Knight Capital (2019)

#75

Post by aaronharnly »

pclmulqdq wrote: post #65
isogon wrote: post #62

Earlier quoted context omitted.

Repurposing flags not always well-motivated, but one legitimate reason to do this is the memory (and particularly cache) footprint. Often flags are local to a particular object. If there are lots of such objects, you want each to take as little space as possible. You should check out the contortions linux devs go through to make struct page small [0]. This is important, because there is one such struct per page of ph…

This is pretty much why flags get repurposed. It's also important to mention that things like JSON and protobufs are too expensive for HFT, so you are likely going to be sending structs over the wire. Repurposing flags lets you change a wire format with a lot less friction than adding a byte to a struct. Essentially, it lets you change the minor version number on a protocol and only recompile the endpoints without ch…

Thank you both!

Re: The $440M software error at Knight Capital (2019)

#76

Post by analog31 »

throwyawayyyy wrote: post #46

Random, but I interviewed at Knight Capital for a software engineering position a few weeks before this all went down. I was in London, so the interview was done over the phone. Picture me in the evening, handwriting C to solve some problem (the fog of time too thick to remember what that problem was), then reading out what I'd written, semicolons and all, to the interviewer. Because of course there was no shared doc…

I would have started them off with a couple of preprocessor macros.

Re: The $440M software error at Knight Capital (2019)

#77

Post by munificent »

coolhoody wrote: post #64
throwyawayyyy wrote: post #46

Random, but I interviewed at Knight Capital for a software engineering position a few weeks before this all went down. I was in London, so the interview was done over the phone. Picture me in the evening, handwriting C to solve some problem (the fog of time too thick to remember what that problem was), then reading out what I'd written, semicolons and all, to the interviewer. Because of course there was no shared doc…

> handwriting C /…/ reading out what I'd written, semicolons and all, to the interviewer. I had to re-read it to make sure you are not joking. The fact that you were not made me laugh harder. I'm now just saying "retuuurn" in various exaggerated accents.

I'm picturing them arguing vigorously over what character the word "bracket" specifies in American versus British English.

Re: The $440M software error at Knight Capital (2019)

#78

Post by munificent »

kevstev wrote: post #11
bfm wrote: post #2

The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…

I worked in algo trading for years, eventually got out because quite frankly the level of risk I was carrying on my shoulders everyday for what I was being paid were just way out of whack, I at least personally never got the huge pay days that people talked about until after I left finance for more pure tech. Interestingly, I worked at Knight and my team pioneered trying to blow up the firm, but that was in 2004, and…

> I understand why they may have chosen to do so, at the Big Bank I was working at, getting a new fix tag to be passed through all the layers properly would involve at least two other teams and coordinating releases and probably several weeks worth of meetings. If you just reuse an old value you can avoid all that since everything is already set up.

There's a good meta-lesson here which is that smart people will do dumb things if you make the smart thing require too much red tape or process.

Processes can block stupidity, but they can also block intelligence if not well designed.

Re: The $440M software error at Knight Capital (2019)

#79

Post by brigandish »

munificent wrote: post #77
coolhoody wrote: post #64

Earlier quoted context omitted.

> handwriting C /…/ reading out what I'd written, semicolons and all, to the interviewer. I had to re-read it to make sure you are not joking. The fact that you were not made me laugh harder. I'm now just saying "retuuurn" in various exaggerated accents.

I'm picturing them arguing vigorously over what character the word "bracket" specifies in American versus British English.

Are brackets different in American English? I'd say the Queen should order a second burning down of Washington but, the way things are going, the locals will probably get to it soon.

Re: The $440M software error at Knight Capital (2019)

#80

Post by photochemsyn »

Really good write-up. Perhaps there's a dawning realization that the model of 'move fast and break things' is fatally flawed?

> "Knight’s IT project managers and CIO should have pushed back on the hyper-aggressive delivery schedule... Thirty days to implement, test, and deploy major changes to an algorithmic trading system that is used to make markets daily worth billions of dollars is impulsive, naive, and reckless."

The fact that since 2008, the portion of all stock trades in the U.S. taking place away from public markets has risen from 15 percent to more than 40 percent is also kind of astonishing. It's long past time to re-erect the walls between commercial and investment banking.

Post reply on HN