How to lose $172,222 a second for 45 minutes • Hacker News Board

Re: How to lose $172,222 a second for 45 minutes

#221

Post by sitkack » Tue, Oct 22, 2013, 10:11 PM UTC

frankc wrote: post #219

sitkack wrote: post #207

Dead code takes down another system. A perfect storm of failures that they made themselves. My gut feeling is that most trading firms could suffer a similar loss. Having worked for a 3rd party accounting management firm that kept logs for smaller traders I really realized how borked the whole system is. 60s era pen and paper stuff moving at the speed of light. > Sadly, the primary cause was found to be a piece of sof…

Actually, I would characterize this as the removal of dead code that brought down the system, which is basically the opposite.

Dead code (latent functionality) in both systems was reactivated without proper testing or knowledge and both systems failed as result. We need to be very weary of latent functionality.

Re: How to lose $172,222 a second for 45 minutes

#222

Post by gngeal » Tue, Oct 22, 2013, 10:12 PM UTC

fosk wrote: post #53

jpatokal wrote: post #46

Just another reminder of how systems that you'd think are rock solid often aren't. In my previous life working with telcos, I once tried to teach a particularly huge customer how to use CVS how to manage configurations across a 10+ machine cluster of machines. They didn't see any value in it, so they stuck to their good old process of SSHing into each machine individually, "cp config.xml config.xml.20131022", and the…

Ha, probably no one ever got fired for manually SSHing into each machine. It seems like a joke, but it really isn't.

What about SSHing into each machine not manually, but automatically with OKing all the changes before reloading the configuration?

SSH is great at least in the sense that it's configuration-format-agnostic, mostly-OS-agnostic, client-side-automation-agnostic, and many other things like that.

Re: How to lose $172,222 a second for 45 minutes

#223

Post by minimax » Tue, Oct 22, 2013, 10:34 PM UTC

ryporter wrote: post #101

sirsar wrote: post #4

I'm shocked they didn't have a killswitch or automated stop-loss of some kind. A script that says "We just lost $5M in a few minutes; maybe there's a problem." Or, a guy paid minimum wage to watch the balance, with a button on his desk. $172,222 is a lot of minimum-wage years.

I work for a small automated trading firm (in foreign exchange), and marking positions to market is one of the difficulties in designing an effective kill switch, because these marks can easily make the difference between a large gain and a large loss. In fast-moving markets (which is when a kill switch is most useful), it's very hard to determine the true mid-market rate. Our system of course always has such a notio…

They unintentionally built up a huge position well beyond their internal capital thresholds. You don't have worry about mark to market to detect that. If you think your FX strategy could end up with max notional of let's say £50MM, you can put something in place to stop trading if you exceed that. If you are a smaller shop, your prime broker is probably already doing something like that for you.

Those types of risk controls (i.e. capital thresholds) are bog standard in our industry and it still kind of blows my mind that Knight managed to get around them so easily.

Re: How to lose $172,222 a second for 45 minutes

#224

Post by [deleted] » Tue, Oct 22, 2013, 10:40 PM UTC

chollida1 wrote: post #214

adambratt wrote: post #9

The week after this we had a trader in our office who had a meeting at Knight on the morning it happened. He said he saw the whole dev team just power off and go home at 11am, followed quickly by the rest of the employees. At that point, there was nothing they could do. The craziest thing is that it went on for so long. No one caught it until their own traders so it come across Bloomberg and CNBC. They actually thoug…

I dont' want to call your friend a liar, but this is most likely false. > The week after this we had a trader in our office who had a meeting at Knight on the morning it happened. > He said he saw the whole dev team just power off and go home at 11am, followed quickly by the rest of the employees. 1) Dev and Trading/Sales happen at different physical locations. 2) I actually know someone who spent their day cleaning…

[deleted]

Re: How to lose $172,222 a second for 45 minutes

#225

Post by sergiosgc » Tue, Oct 22, 2013, 11:57 PM UTC

kylebrown wrote: post #204

ansible wrote: post #197

Earlier quoted context omitted.

I really hate to go down this road because it's been rehashed thousands of times on Hacker News... And I've yet to see a convincing argument that HFT is of any benefit to anyone else other than themselves. At the most basic level, HFT firms make money, right? Otherwise they wouldn't be doing this. Where does this money come from? Entities that are holding these same stocks for longer periods of time. It increases the…

HFT profits come from the money that the specialists/dealers used to make. They were the guys standing at the podiums quoting bid/ask prices all day.

If that were true, then removing HFT would bring back floor dealers. Clearly, it is false. Electronic trading, not HFT, deprecated floor tradrs.

Re: How to lose $172,222 a second for 45 minutes

#226

Post by Zigurd » Wed, Oct 23, 2013, 12:29 AM UTC

mathattack wrote: post #169

hrjet wrote: post #111

Earlier quoted context omitted.

I heard similar stories from a friend working for a big telco. The other day he was describing a strange bug which was triggering in the field of a large telco. It was strange because it used to get triggered after exactly 85 days of deployment. It turned out to be a debug script that was pinging a development server and timing out (because the development server was not accessible from the field). The series of retr…

Legacy telcos are the epitome of large big institutions where many of the best talents leave. After just a few years of working with them, I couldn't believe they could get a dial tone. How did they produce so much great R&D?

> How did they produce so much great R&D?

By having negotiated profit margins. Bell Labs was meant to soak up excess money. What we are now nostalgic for would never have happened without an interval during which a telco was considered a "natural monopoly."

Re: How to lose $172,222 a second for 45 minutes

#227

Post by simoncion » Wed, Oct 23, 2013, 1:21 AM UTC

mgav wrote: post #118

Very interesting, though I was happy to see Knight Capital take the huge loss, since they were such complete scumbags who stole hundreds of millions of dollars by backing away from trades* during the dotcom boom and bust. *Backing away is when a market maker makes a firm offer to buy or sell shares, receives an order to execute that transaction (which they are ethically and legally obligated to do) and instead cancel…

Would you kindly provide the un-minified version of your link? This isn't Twitter, and HN has its own method of shortening overlong links while also displaying the original link target.

Re: How to lose $172,222 a second for 45 minutes

#228

Post by kevinpet » Wed, Oct 23, 2013, 1:57 AM UTC

mcphilip wrote: post #47

notaddicted wrote: post #44

Earlier quoted context omitted.

Yes, these criminal masterminds found a way to swindle honest hard-working themselves out of 440 million dollars and "get away with" a cool extra -12 million.

They also amassed over $3 billion net short positions spread across 75 stocks during those 45 minutes causing significant losses to investors with stop loss positions triggered that would not have happened without Knight's erroneous trades. They didn't just harm themselves...

Anyone using stop loss orders harmed themselves. It's like saying that you wouldn't have gotten hurt rolling down the steepest hill in town on a skateboard with no helmet if someone hadn't parked in a red zone.

Re: How to lose $172,222 a second for 45 minutes

#229

Post by Ricapar » Wed, Oct 23, 2013, 2:09 AM UTC

simoncion wrote: post #227

mgav wrote: post #118

Very interesting, though I was happy to see Knight Capital take the huge loss, since they were such complete scumbags who stole hundreds of millions of dollars by backing away from trades* during the dotcom boom and bust. *Backing away is when a market maker makes a firm offer to buy or sell shares, receives an order to execute that transaction (which they are ethically and legally obligated to do) and instead cancel…

Would you kindly provide the un-minified version of your link? This isn't Twitter, and HN has its own method of shortening overlong links while also displaying the original link target.

Un-minified link:

http://books.google.com/books?id=geCHWBx-e9EC&pg=PA126&lpg=P...

Re: How to lose $172,222 a second for 45 minutes

#230

Post by jrochkind1 » Wed, Oct 23, 2013, 3:16 AM UTC

comrade_ogilvy wrote: post #189

jrochkind1 wrote: post #153

Earlier quoted context omitted.

I don't know if it's excusable or not, but it may be incompatible with typical human cognition to expect someone to be able to do that. Maybe you have to figure out a way to test people for unusually high aptitude at looking at mind-numbingly dull repetitive things over and over again, but then still being able to notice the aberrant ones. And then only put people in that job with unusually high aptitude there. Or ha…

Yours is the correct viewpoint: it is incompatible with human cognition. If an alert system is not perceived as highly reliable in directing positive action, then the humans involved will inevitably disable the alert system, either by pulling out a screwdriver or rewriting their mental rubrics to ignore the messages as noise. Knight Capital is just the finance version of Three Mile Island and Deepwater Horizon -- the…

in the ancestral comment here, it wasn't even an alert system! It was just "your job is to sit there all day and watch every single transaction and flag the aberrant one"!!