Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The $440M software error at Knight Capital (2019)

henricodolfing.com

Post reply on HN
Print view View original

51–60 of 177 posts

Re: The $440M software error at Knight Capital (2019)

#51

Post by 22SAS »

bfm wrote: post #2

The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…

Currently work at an HFT firm. Most of the firms invest well into good DevOps, Trading Systems and SRE teams, to ensure that everything from installing a trading server at the colocation facility, to CI/CD and making changes to the systems configs, is done well. There are also guards in place to ensure that if the system seems to make trades that are way too odd then pull the plug and go down immediately.

Also, any code that does not need to be there, is promptly removed right away.

Where I work at, we have a few people from KCG i.e what was formed after Knight Capital merged with GETCO, after this incident. Sometimes this incident is bought up, although none of them I think ever worked for Knight Capital before this incident.

Re: The $440M software error at Knight Capital (2019)

#52

Post by bob1029 »

bfm wrote: post #2

The OP details how poor software engineering practices brought down a 1.4B market marker with 1400 employees in 2012. Some of the issues mentioned include: - Keeping synthetic test data generation as part of a production build. - Keeping dead code for years. - Re-purposing a feature flag. - Refactoring without regression tests. - Manual deployments without peer reviews. They forgot to update one of their servers with…

Repurposing feature flags is some kind of next dimension horror for me. We've got quite a few of these to deal with, and if someone started changing what they mean we'd be fucked super fast. Simply suggesting that we alter the meaning of an existing FF would result in the resignation of a non-zero number of project managers on my team.

Rolling back code is another thing I have no tolerance for anymore. The only option we entertain these days is a roll-forward. If your software takes so long to iterate/build that you need to go back to and old version in an emergency, you need to review your languages/tools/frameworks/processes. We maintain a contractual obligation to our customers for same-day code updates (in cases of production/regulatory emergencies) because we have enough confidence in our processes.

Re: The $440M software error at Knight Capital (2019)

#53

Post by 22SAS »

caffeine wrote: post #38
fullsend wrote: post #29

Earlier quoted context omitted.

I appreciate your comment about pay. Recruiters will often tell me "it's finance so of course the pay will be substantial." Then when we get to talking numbers they're like "300k a year". Oh, you mean the going rate at a FAANG? And I have to move to New York or Chicago, work more hours, and actively work for people who I know are taking home paychecks with 7+ zeroes on them? Come on. Sometimes it's 400 plus bonus or…

The attitude that finance pays more is a leftover from a previous era. 10-15 years ago it was true: the profits from HFT were so also way, way bigger and split up amongst a much smaller group of firms. Now those firms are all in a completely competitive industry squeezing each other for basis points. Meanwhile the definition of a FAANG is that it has an effective monopoly, and these companies are taking in way more m…

Tbf, most of us don't really prefer to be called as HFT's but as Market Makers. Different name, but we still use the same ultra low latency techniques to get the job done.

Re: The $440M software error at Knight Capital (2019)

#54

Post by isogon »

22SAS wrote: post #49
hatesinterviews wrote: post #47

Earlier quoted context omitted.

At our firm, the numbers are similar: $600k TC for new grads ($200k base, $100k minimum first year bonus, $300k signing bonus)

WTF! I am at an HFT firm in Chicago, this is insane. This seems to be a lot like an offer from Radix, or Headland, or maybe Algo Dev at HRT.

There is certainly much variance between the firms, especially the sign-on IME. People I know have turned down HRT core dev for big tech because their offers were unimpressive.

I think an interesting target for comparison with big tech is Jane Street, since their culture and WLB are good, so the main QoL drawbacks of finance don't apply. A new grad will get ~300k at Jane Street, though probably not with this large a sign-on.

Re: The $440M software error at Knight Capital (2019)

#55

Post by 22SAS »

isogon wrote: post #43
fullsend wrote: post #29

Earlier quoted context omitted.

I appreciate your comment about pay. Recruiters will often tell me "it's finance so of course the pay will be substantial." Then when we get to talking numbers they're like "300k a year". Oh, you mean the going rate at a FAANG? And I have to move to New York or Chicago, work more hours, and actively work for people who I know are taking home paychecks with 7+ zeroes on them? Come on. Sometimes it's 400 plus bonus or…

I cannot confirm this. ~300k is pay (excluding sign-on) fresh out of college at a big HFT -- sufficiently senior devs make 7 figures.

Honestly, that depends on the firm. There are same that do pay very well like this, eg: HRT, Jane Street (they are not an HFT though), Headlands, Radix. Some others like Jump, Optiver the pay varies depending on whether it's front office or back office.

Where I work at, the new grad offers are slightly better than FAANG, but the growth is very good based on performance, we also pay very well to people coming in from a competitor.

Re: The $440M software error at Knight Capital (2019)

#56

Post by 22SAS »

isogon wrote: post #54
22SAS wrote: post #49

Earlier quoted context omitted.

WTF! I am at an HFT firm in Chicago, this is insane. This seems to be a lot like an offer from Radix, or Headland, or maybe Algo Dev at HRT.

There is certainly much variance between the firms, especially the sign-on IME. People I know have turned down HRT core dev for big tech because their offers were unimpressive. I think an interesting target for comparison with big tech is Jane Street, since their culture and WLB are good, so the main QoL drawbacks of finance don't apply. A new grad will get ~300k at Jane Street, though probably not with this large a…

This is interesting, didn't know this about HRT Core Dev where offers were below FAANG. My understanding is that core devs are basically the folks who work on all the low latency stuff, so they'd be pretty well.

Jane Street, from what I recall, is 300K (base + bonus) and 125K sign-on, and also it is non-negotiable. No idea what their numbers are like for experienced hires from competitors.

Re: The $440M software error at Knight Capital (2019)

#57

Post by 22SAS »

kevstev wrote: post #45
fullsend wrote: post #29

Earlier quoted context omitted.

I appreciate your comment about pay. Recruiters will often tell me "it's finance so of course the pay will be substantial." Then when we get to talking numbers they're like "300k a year". Oh, you mean the going rate at a FAANG? And I have to move to New York or Chicago, work more hours, and actively work for people who I know are taking home paychecks with 7+ zeroes on them? Come on. Sometimes it's 400 plus bonus or…

Yeah, pay at the big banks is shit really, especially when you consider the utter lack of work/life balance. I left in 2013 making 150k, which was supposed to be supplemented by a ~40% bonus for the level I was at, but each year was "well its been a tough year..." and after getting a token amount one year, and then zeroes the next 2, after working 50-60 hour weeks, I was like I am not only done with this place, but t…

Most quantitative hedge funds and prop trading firms are now following a very tech like culture since they realize now that technology is just as important as the strategies. To get the best engineers, especially from FAANG, they need to have a similar culture otherwise they'll have a hard time getting new hires.

Re: The $440M software error at Knight Capital (2019)

#58

Post by 22SAS »

asjre34marakf wrote: post #48
fullsend wrote: post #29

Earlier quoted context omitted.

I appreciate your comment about pay. Recruiters will often tell me "it's finance so of course the pay will be substantial." Then when we get to talking numbers they're like "300k a year". Oh, you mean the going rate at a FAANG? And I have to move to New York or Chicago, work more hours, and actively work for people who I know are taking home paychecks with 7+ zeroes on them? Come on. Sometimes it's 400 plus bonus or…

Why pay more than market rate of a replaceable ML person? Is there any realistic path for a demonstrably smart and hardworking person into that 7+zeros club? Evidence suggests no: leetcode grinders and FAANGers are not in that club, and most of them will never even make it into the 6+zeros club. Net wealth -- sure, but not income.

It's all about making $$ for the firm. If the strategies developed are very profitable then 7-figures is definitely reachable for the researchers at a prop trading firm.

Re: The $440M software error at Knight Capital (2019)

#59

Post by ceejayoz »

strgcmc wrote: post #32
ceejayoz wrote: post #27

Earlier quoted context omitted.

Because those transactions cause other transactions, which cause others, and so on and so forth. You'd have to reset the market for the day. Imagine how pissed you'd be if you made money off Knight's mistake and it all just disappeared the next day.

Except, well, cancelling transactions obviously does happen, sometimes: https://www.reuters.com/business/lme-suspends-nickel-trading... Knight was probably too messy to rollback cleanly, but that just means it's a matter of cost/complexity/politics... if you're a big enough player, then the exchange will do you favors, like in the LME case. Free markets, lol

> The LME announced that all trades will be voided from midnight until 8:15 a.m. on Tuesday when trading stopped and added that it was considering a closure of several days.

> "People will be asking if this really a functioning market... This is meant to be a market of last resort and people can't get inventories to deliver against positions," said Colin Hamilton, managing director of commodities research at BMO Capital Markets.

There’s gonna be a pretty high threshold for this sort of thing. Higher than “one company fucked up and wants a do-over”.

Re: The $440M software error at Knight Capital (2019)

#60

Post by nogridbag »

throwyawayyyy wrote: post #46

Random, but I interviewed at Knight Capital for a software engineering position a few weeks before this all went down. I was in London, so the interview was done over the phone. Picture me in the evening, handwriting C to solve some problem (the fog of time too thick to remember what that problem was), then reading out what I'd written, semicolons and all, to the interviewer. Because of course there was no shared doc…

Same! Although I interviewed in person in their NYC office. I was very junior at the time and the team I interviewed with was awesome. I (luckily) didn't get the job. I did a few more interviews and accepted an offer from another company where I met my wife!
Post reply on HN