Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to lose $172,222 a second for 45 minutes

pythonsweetness.tumblr.com

Post reply on HN
Print view View original

101–110 of 243 posts

Re: How to lose $172,222 a second for 45 minutes

#101

Post by ryporter »

sirsar wrote: post #4

I'm shocked they didn't have a killswitch or automated stop-loss of some kind. A script that says "We just lost $5M in a few minutes; maybe there's a problem." Or, a guy paid minimum wage to watch the balance, with a button on his desk. $172,222 is a lot of minimum-wage years.

I work for a small automated trading firm (in foreign exchange), and marking positions to market is one of the difficulties in designing an effective kill switch, because these marks can easily make the difference between a large gain and a large loss. In fast-moving markets (which is when a kill switch is most useful), it's very hard to determine the true mid-market rate. Our system of course always has such a notion, but if we used it to shut down our system every time it looked like we had lost money, then a market data glitch (which is not at all uncommon) would impose a large opportunity cost as a human intervened during an active market.

Instead, we designed our system so that there's a very low threshold for it stop trading if it appears to have lost money, but to only do so temporarily. If our marked-to-market position recovers shortly thereafter while the trading system is idle, then the apparent loss was probably due to a market data glitch. On the other hand, if our position does not recover, then the temporary stoppage becomes permanent, and a human intervenes. (Obviously, there are more details here, but this is the general idea, and it's worked very well for us.)

Re: How to lose $172,222 a second for 45 minutes

#102

Post by michaelt »

babuskov wrote: post #95
paganel wrote: post #86

Earlier quoted context omitted.

> running a shadow copy Honest question, it's that really doable at this scale?

It's doable. The real question is how much it would cost and does it make sense then.

In this case the system was deploying new functionality, so your shadow couldn't just run last week's version or you'd get false alarms all the time. And obviously deploying the same broken version to the primary and the shadow wouldn't detect anything.

So you'd need two codebases and two developer teams, coordinated enough that their code produced exactly the same output yet independent enough that they didn't make the same mistakes. With the challenges of coordination this would more than double your costs.

Of course, with the benefit of hindsight, the costs might have been worthwhile...

Re: How to lose $172,222 a second for 45 minutes

#103

Post by marincounty »

telephonetemp wrote: post #70

I assumed they had redundant servers with consensus algorithms in place in finance but apparently they don't. Would it be impractical?

Ya, I just assumed they would have at least 3 levels of redundancy, or at least a person there who could pull 8 cords?

Re: How to lose $172,222 a second for 45 minutes

#104

Post by chalst »

Houshalter wrote: post #68

They fined them for losing money? What?

Trading companies are supposed to have effective risk policy (i.e., they are supposed to think about what happens if things do wrong and formulate rules to ameliorate risks) and compliance controls (i.e., bureaucratic controls that ensure that policy actually is followed). The SEC charge does not paint a pretty picture on these counts, but the key point seems to be this one:

-- Relied on financial risk controls that were not capable of preventing the entry of orders that exceeded pre-set capital thresholds for the firm in the aggregate.

https://www.sec.gov/News/PressRelease/Detail/PressRelease/13...

(The charge also states that Knight violated rules on covering shorts, but I guess this is not so important).

The SEC is quite right to fine firms that have lost money with poor risk controls: the point is that bad risk management can hurt the whole sector. It is like fining a factory owner who lets their plant break pollution regulations.

Re: How to lose $172,222 a second for 45 minutes

#105

Post by confluence »

vecter wrote: post #92
scrrr wrote: post #87

High Frequency Trading seems so abstract. There's no value created, it seems. It's like something in between imperfect systems, scraping off the margin created by that imperfection. It's fascinating, and interesting from an algorithmic point of view (like a computer game), but at the same time I don't feel sympathy for this company going out of business.

I really hate to go down this road because it's been rehashed thousands of times on Hacker News, but high frequency traders add value to the market by adding liquidity (and therefore reducing spreads --> cost to you for executing) and price discovery.

HFT does not add liquidity, but they do squeeze spreads in liquid contracts. It's some what good for retail, but terrible for institutional investors, which hurt retail on the backend. HFT is just a front running operation in those cases.

Re: How to lose $172,222 a second for 45 minutes

#106

Post by kops »

candydance wrote: post #62
MBCook wrote: post #7

Earlier quoted context omitted.

How many millions in orders do they normally process per minute? Since there were no procedures in place, would you like to be the guy who pulled the plug on the (let's guess) $100 million/minute processing system? Do you think you could get another job after that? What would the costs be for violating contracts? You could single handedly sink the company (which, in the end, this issue basically did). I don't blame t…

Have them in pairs or teams, war games style. Guy1 "Sir, we're losing money much faster than predicted, turn your key" Guy2 "It could just be market variance" Guy1 pulls out gun "Turn your key, sir"

Despite the gun brandishing, that conversation looks too polite to have ever taken place on a trading floor :-)

Re: How to lose $172,222 a second for 45 minutes

#107

Post by gd1 »

nhebb wrote: post #66
tempestn wrote: post #37

Earlier quoted context omitted.

The standard answer to this is that they provide liquidity. Whether that benefit outweighs the drawbacks is a subject of debate.

A non-economist wants to know: if liquidity is beneficial to our economy and liquidity is a function of time, how how does the time-benefit curve look as t approaches 0? I don't know if you can quantify the benefit and map this curve, but if you could I don't imagine it would scale to infinity as time approached zero.

Liquidity isn't a function of time, it is a function of relative time between the predators (arbitrageurs) and the prey (market makers). If the predators are much faster than the prey, liquidity will disappear since the market makers can't survive (their prices are too stale and they are getting taken advantage of). It has always been this way, even since Nathan Rothschild used carrier pigeons to get news of the Battle of Waterloo.

Re: How to lose $172,222 a second for 45 minutes

#108

Post by strongvigilance »

marincounty wrote: post #96
scrrr wrote: post #87

High Frequency Trading seems so abstract. There's no value created, it seems. It's like something in between imperfect systems, scraping off the margin created by that imperfection. It's fascinating, and interesting from an algorithmic point of view (like a computer game), but at the same time I don't feel sympathy for this company going out of business.

I think HIgh Frequency Trading will eventually be neutralized through competition. I noticed the their profits aren't as staggering as they were a few years ago. I would like to see a law that limits how close an company(individual) can set up shop next to an exchange though. I think Insider Trading is more of a problem than the public knows?

Why have a limit on how close people can be to the exchange? For one thing, everyone would just colo at exactly the minimum distance, achieving nothing. For another, ability to colo is not really an issue. If you wanted to create an HFT startup, colo is not going to be a major cost compared to hiring developers.

Re: How to lose $172,222 a second for 45 minutes

#109

Post by SideburnsOfDoom »

malbs wrote: post #11

Just one of the risks of automation, and a good reminder why human monitoring is necessary. Having said that, we deployed a system that was mostly automated, with the human operator to oversee investments and if any out-of-the-ordinary transactions (based on experience) were taking place, to shut it down. She happily sat there approving the recommendations even though the recommendations were absolutely outside of an…

> Just one of the risks of automation

Automated deployments would have helped them. They made lots of mistakes, but IMHO not automating the deploy was likely the #1 mistake here.

Post reply on HN