Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Trading Program Ran Amok, With No ‘Off’ Switch

dealbook.nytimes.com

Post reply on HN
Print view View original

81–90 of 97 posts

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#81

Post by white_devil »

yummyfajitas wrote: post #56
SagelyGuru wrote: post #32

'Prediction is difficult, especially with regard to the future'. It looks like they were using some new algorithm, which should have made them a lot of money, had the market gone up after their massive purchases. In that case, they would have pocketed fat bonuses and would not be on the news. However, it has not happened, so the crying and the search for a scapegoat is on. It sounds like the case of the banking busin…

Huh? Knight Capital lost a bunch of money, and will likely go bankrupt. Essentially, Knight's software bug transferred a bunch of money from Knight to everyone else. This poses minimal systematic risk to anyone else, and they will almost certainly get no bailout. The markets have already recovered. The S&P was down a little bit on thurs and recovered by friday. Knight is down 60%. http://www.google.com/finance?q=INDE…

> This is ultimately a situation of the market being a robust and stable dynamical system.

Not while it's being shaped by algorithms competing against each other, which you're a part of.

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#82

Post by sp332 »

pheon wrote: post #30
fsckin wrote: post #12

I would rarely suggest this, but if something is so incredibly broken that you're loosing money at a rate of 800 million dollars per hour, screw the customers. Turn it off at any cost. If you are forthcoming and transparent, customers will understand.

Point is you dont know what the loss is. 1) They bought too much stock (incorrectly) 2) realized WTF, stopped everything 2a) more likely their clients said WTF is wrong first 3) had to sell the stock for the rest of the day. Its only after they sold everything did the $440MM price tag surface. Hopefully they sold most of their positions to goldman (instead free market) so one of their investors made a boatload of cas…

This is wrong. The algorithm was buying and selling constantly, sometimes losing small amounts of money (usually about $15) each time, sometimes as often as 20-40 times per second for each of about 150 symbols.

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#83

Post by sp332 »

rorrr wrote: post #5
nmcfarl wrote: post #2

They lost $440 million (and amount greater than their market cap), and possibly the company, on what the world knows to be incompetence. At some point if I couldn’t stop it - I’d be tempted to just kill the power to the server rooms, all of them. There just has to be a way to cut your losses.

It happened very quickly, after years of operations. Technically, it can take seconds to lose that much. What they did wrong is do all their trading with the same algorithm. Way to put all eggs in one basket.

I'm pretty sure they don't do all their trading with one algorithm. What makes you think they do?

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#84

Post by sp332 »

Devilboy wrote: post #31
jonah wrote: post #23

Earlier quoted context omitted.

Seems like maybe they couldn't hold on to the stock for long enough to unload the enormous volume they were dealing with. It sounded like at one point they were doing AS MUCH VOLUME AS EVERYONE ELSE on the exchange combined. http://news.ycombinator.com/item?id=4337750

Since there's 2 parties to every trade doesn't that make 50% the limit?

Nope, sometimes they bought stock they sold themselves!

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#85

Post by toomuchtodo »

wglb wrote: post #54
wpietri wrote: post #35

Earlier quoted context omitted.

Having talked with people who write life-critical code, the regulation isn't really what makes it safe. Safety comes from good engineering. The regulation just makes it much harder to bring an unsafe product to market, and makes it clearer who to blame when people die.

But don't you think the existence of regulation influences the culture?

Penalties influence the culture. As we all know, the first lesson in economics is that incentives matter.

Sometimes, the right people aren't being incentivized to do the right thing.

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#86

Post by ewood »

jriley wrote: post #79
ewood wrote: post #40

Earlier quoted context omitted.

I have experience in HFT, there are similarities to market making and I have plenty of colleagues who've worked in market making. Just like any company the culture is largely dependent on those in charge. Founders of these companies fall into three buckets - traders, techies, and mathematicians/physicists - and quality control will generally be a function of the founder mix. Mostly techies: strong software culture, u…

Thanks for link. Given your experience, do you have any thoughts about a small circuit-breaker on every security that trips for, say any 3-standard deviation event? In a quick Google search, it seems circuit breakers do not kick in for first 15 minutes of trading: http://www.nytimes.com/2012/08/02/business/unusual-volume-ro...

In derivative markets such as futures there are predefined price and volume limits (in the jargon 'limit up' and 'limit down' - ie: max up and down movements.) These limits exist primarily to prevent market manipulation or cornering a market (buying the entire supply of a commodity) but have also been triggered by trading around events such as the Japan earthquake. Doing a quick search it seems that there are plans to trial these controls on equity markets (http://blogs.law.harvard.edu/corpgov/2012/06/13/limit-up-lim...)

The problem here though was that while some stocks had dramatic price movements that might have triggered a limit control, more heavily traded stocks were able to absorb the additional volume and the price did not move significantly. Knight were not doing anything outside of normal bands, they were buying normal volumes of stocks close to the current market price and selling close to the current price. What they were doing was illogical in a profit sense because they were buying at a high price and selling at a lower price and thus immediately losing money. I think it would be very difficult for an exchange to trap this kind of problem.

In all the issue of determining 'normal' trading is very difficult as markets tend to be much noisier than you might expect. The majority of trading occurs near the open and closes of major markets (Hong Kong, London, New York) or data releases (eg US Unemployment) so large spikes in volume and price are a regular occurrence. In equities this is even more difficult as smaller stocks will tend to be more volatile and profit reporting season increases this volatility even further. Markets are ruled by fear and greed, and falsely triggering a limit may cause larger issues than it solves.

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#87

Post by teyc »

I read the nanex article. Regardless whether it is true or not, the general trend is towards development of more sophisticated load testing programs.

The most benign ones were developed for use in IT systems. E.g. Apache bench. While these can cause disruptions if aimed at production services, this does not necessarily threaten the health of an entire enterprise.

However, the trend is that all software sectors are starting to adopt this particular technique of testing software with not sufficient regard to what happens if it is released into live systems.

For example, we have chaos monkey, from Netflix, which randomly shutdown services in a cloud based system.

What would happen if software which simulated meltdown at a nuclear facility was accidentally bundled into the build system by a tired operator? Or some one does the same with flight software?

The main software running trading platforms would presumably be supervised by another program to ensure that bad algorithms do not lose e company too much money. However there was no such tool for the component that generated the test data.

To me, it sounds like the supervision should be done at a higher level, e.g. A wrapper around existing APIs. All software running against live systems must call into the wrapper.

Secondly, test software should conduct some kind of verification. E.g. Check for evidence that it is testing against a Test system. This might be the presence of a nonexistent company, et c.

I am more than happy to compile any other ideas you may have so that the IT industry is able to build more fail safes into software.

We are starting to see some of these fail safes in practice. E.g. When you try to send out an email to everyone in the organization, email software may warn you if you are sure you want to do that. The problem is we haven't thought enough about these scenarios that we don't adequately address them.

Incidentally, over in Australia, the Commonwealth Bank suffered a major downtime when it's outsourcer HP accidentally pushed out system wide updates instead of doing this to select machines as originally intended.

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#88

Post by teyc »

ewood wrote: post #86
jriley wrote: post #79

Earlier quoted context omitted.

Thanks for link. Given your experience, do you have any thoughts about a small circuit-breaker on every security that trips for, say any 3-standard deviation event? In a quick Google search, it seems circuit breakers do not kick in for first 15 minutes of trading: http://www.nytimes.com/2012/08/02/business/unusual-volume-ro...

In derivative markets such as futures there are predefined price and volume limits (in the jargon 'limit up' and 'limit down' - ie: max up and down movements.) These limits exist primarily to prevent market manipulation or cornering a market (buying the entire supply of a commodity) but have also been triggered by trading around events such as the Japan earthquake. Doing a quick search it seems that there are plans t…

Perhaps one way to mitigate this is by wrapping the calls to another system such that all trading is supervised.

Another technique is to provide API keys in the wrapper so that test programs will not have the keys to a live system.

The real problem is that the risk of these test systems have not been sufficiently identified or recognized. We are all too busy creating mock systems instead of devoting sufficient oversight to the development of test software.

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#89

Post by jriley »

ewood wrote: post #86
jriley wrote: post #79

Earlier quoted context omitted.

Thanks for link. Given your experience, do you have any thoughts about a small circuit-breaker on every security that trips for, say any 3-standard deviation event? In a quick Google search, it seems circuit breakers do not kick in for first 15 minutes of trading: http://www.nytimes.com/2012/08/02/business/unusual-volume-ro...

In derivative markets such as futures there are predefined price and volume limits (in the jargon 'limit up' and 'limit down' - ie: max up and down movements.) These limits exist primarily to prevent market manipulation or cornering a market (buying the entire supply of a commodity) but have also been triggered by trading around events such as the Japan earthquake. Doing a quick search it seems that there are plans t…

Good insight.

Made me wonder what would happen to volume if markets were open 24/7...but I'll leave that thought for another day.

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#90

Post by davidmr »

SkyMarshal wrote: post #66
wtracy wrote: post #51

Earlier quoted context omitted.

"We think the two periods of time when there was a sudden drop in trading (9:48 and 9:52) are when they restarted the system. Once it came back, the Tester, being part of the package, fired up too and proceeded to continue ..." Ouch!

Worse dumb mistake I've ever made was to accidentally send a test email to several thousand live customers instead of the test accounts. That was a sinking feeling. But creating a bug that loses your company half a billion dollars in thirty minutes and bankrupts them, must be stomach-churning.

Much, much worse than the bug in the market making algorithms was the design failure of not having some out-of-band mechanism to kill order traffic (or of said mechanism's failure to be tested adequately).

This is a risk management failure much more than a programming error.

Post reply on HN