Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The $440M software error at Knight Capital (2019)

henricodolfing.com

Post reply on HN
Print view View original

61–70 of 177 posts

Re: The $440M software error at Knight Capital (2019)

#61

Post by rosege »

kevstev wrote: post #45
fullsend wrote: post #29

Earlier quoted context omitted.

I appreciate your comment about pay. Recruiters will often tell me "it's finance so of course the pay will be substantial." Then when we get to talking numbers they're like "300k a year". Oh, you mean the going rate at a FAANG? And I have to move to New York or Chicago, work more hours, and actively work for people who I know are taking home paychecks with 7+ zeroes on them? Come on. Sometimes it's 400 plus bonus or…

Yeah, pay at the big banks is shit really, especially when you consider the utter lack of work/life balance. I left in 2013 making 150k, which was supposed to be supplemented by a ~40% bonus for the level I was at, but each year was "well its been a tough year..." and after getting a token amount one year, and then zeroes the next 2, after working 50-60 hour weeks, I was like I am not only done with this place, but t…

I spent a few years at an investment bank, not in the US, and the only people on serious money were some of the top managers. But my overall opinion of these people were that they did very little but the lower downs I met were some of the most talented people I ever worked with.

The top ones would spend all their time traveling the world to the offices and meeting with staff in each location and the sending emails to the rest of the department about what the staff in that location were working on. They would harvest ideas from the staff as they went and then present that as their own or approve projects that staff have suggested to them. I really didn't see how they were worth the $5M they were earning since they didn't come up with the ideas for what would be done and didn't do any real work.

Re: The $440M software error at Knight Capital (2019)

#62

Post by isogon »

aaronharnly wrote: post #50
pclmulqdq wrote: post #30

Earlier quoted context omitted.

I used to work in HFT. I have seen highly variable practices in this case, including a "mini-knight" incident in the single-digit millions due to tech debt and poor test coverage. However, the most useful change that has resulted from the KCG debacle was adding several layers of kill switches, a dedicated ops team to watch trading and flip the kill switches, and embracing devops automation. There is a much more serio…

I’m curious about the “repurposed flags” part. I wouldn’t think of flags as expensive / effortful to make more of, but clearly they must be if people are tempted to reuse them. Can you help me understand what is meant by a flag in this context, and why it would be repurposed?

Repurposing flags not always well-motivated, but one legitimate reason to do this is the memory (and particularly cache) footprint.

Often flags are local to a particular object. If there are lots of such objects, you want each to take as little space as possible. You should check out the contortions linux devs go through to make struct page small [0]. This is important, because there is one such struct per page of physical memory. The memory use is a near-constant percentage of your total memory, and you wouldn't want it to be any larger than necessary.

Even when there are not a lot of these objects, in low-latency software it's important to hit the cache. Your program should always just be as compact in memory as possible.

Semantically flags are booleans (is proposition P true of this object). They are stored compactly as bitsets, often implicitly, say:

    #define FLAG_1 0x01
    #define FLAG_2 0x02
    /* ... */
    #define FLAG_8 0x80

   struct order {
       u32 qty;
       u16 id;
       u8  type;
       u8  flags;
   };
This struct will fit into 8 bytes. This is great, as you probably won't waste space to alignment in many cases -- 8 is a good multiple. But if you wanted to add FLAG_9 here, your flags would become a u16, and your struct would, frustratingly, stop fitting into 8 bytes. To avoid this, one might repurpose flags.

Another example of this is intrustive flagging, using, for example, the high or low bits of a pointer aligned to 2^n bytes. If you run out of bits there, not much you can do.

[0] https://github.com/torvalds/linux/blob/master/include/linux/...

Re: The $440M software error at Knight Capital (2019)

#63

Post by snotrockets »

spacemanmatt wrote: post #44
caffeine wrote: post #38

Earlier quoted context omitted.

The attitude that finance pays more is a leftover from a previous era. 10-15 years ago it was true: the profits from HFT were so also way, way bigger and split up amongst a much smaller group of firms. Now those firms are all in a completely competitive industry squeezing each other for basis points. Meanwhile the definition of a FAANG is that it has an effective monopoly, and these companies are taking in way more m…

> but we can’t really drop N from the acronym without a replacement Huh, yeah. That would be quite a GAAF. Gotta come up with something before Netflix is forced out of the FAANG club.

I’ve seen MAAM being used.

Re: The $440M software error at Knight Capital (2019)

#64

Post by coolhoody »

throwyawayyyy wrote: post #46

Random, but I interviewed at Knight Capital for a software engineering position a few weeks before this all went down. I was in London, so the interview was done over the phone. Picture me in the evening, handwriting C to solve some problem (the fog of time too thick to remember what that problem was), then reading out what I'd written, semicolons and all, to the interviewer. Because of course there was no shared doc…

> handwriting C /…/ reading out what I'd written, semicolons and all, to the interviewer.

I had to re-read it to make sure you are not joking. The fact that you were not made me laugh harder.

I'm now just saying "retuuurn" in various exaggerated accents.

Re: The $440M software error at Knight Capital (2019)

#65

Post by pclmulqdq »

isogon wrote: post #62
aaronharnly wrote: post #50

Earlier quoted context omitted.

I’m curious about the “repurposed flags” part. I wouldn’t think of flags as expensive / effortful to make more of, but clearly they must be if people are tempted to reuse them. Can you help me understand what is meant by a flag in this context, and why it would be repurposed?

Repurposing flags not always well-motivated, but one legitimate reason to do this is the memory (and particularly cache) footprint. Often flags are local to a particular object. If there are lots of such objects, you want each to take as little space as possible. You should check out the contortions linux devs go through to make struct page small [0]. This is important, because there is one such struct per page of ph…

This is pretty much why flags get repurposed. It's also important to mention that things like JSON and protobufs are too expensive for HFT, so you are likely going to be sending structs over the wire. Repurposing flags lets you change a wire format with a lot less friction than adding a byte to a struct. Essentially, it lets you change the minor version number on a protocol and only recompile the endpoints without changing the major version number and recompiling everything.

Re: The $440M software error at Knight Capital (2019)

#67

Post by chmod775 »

bfm wrote: post #40
user3939382 wrote: post #18

Here’s a 225 million dollar oopsie from 2005 https://www.foxnews.com/story/typing-error-causes-225m-loss-...

Today there was a 300B oopsie in Europe caused by a Citibank "glitch" bloomberg.com/news/articles/2022-05-02/citi-s-london-trading-desk-behind-rare-european-flash-crash

It only was a sudden drop in share prices, which quickly rebounded. The amount of money that actually changed hands due to that mistake will be tiny in comparison.

Re: The $440M software error at Knight Capital (2019)

#69

Post by Bilal_io »

nogridbag wrote: post #60
throwyawayyyy wrote: post #46

Random, but I interviewed at Knight Capital for a software engineering position a few weeks before this all went down. I was in London, so the interview was done over the phone. Picture me in the evening, handwriting C to solve some problem (the fog of time too thick to remember what that problem was), then reading out what I'd written, semicolons and all, to the interviewer. Because of course there was no shared doc…

Same! Although I interviewed in person in their NYC office. I was very junior at the time and the team I interviewed with was awesome. I (luckily) didn't get the job. I did a few more interviews and accepted an offer from another company where I met my wife!

Do you return the wife when you leave the job?

In all seriousness, you dodged a bullet. Do you remember what problem or set of problems they asked you to solve?

Also, what made the interviewing team awesome in your opinion?

Post reply on HN