Live data from Hacker News

Trading Program Ran Amok, With No ‘Off’ Switch

dealbook.nytimes.com

11–20 of 97 posts

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#11
post #2

They lost $440 million (and amount greater than their market cap), and possibly the company, on what the world knows to be incompetence. At some point if I couldn’t stop it - I’d be tempted to just kill the power to the server rooms, all of them. There just has to be a way to cut your losses.

The server rooms might not be nearby. They might be located closer to the big exchanges' servers.

Very likely. Speed of light matters in these situations.

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#13
post #2

They lost $440 million (and amount greater than their market cap), and possibly the company, on what the world knows to be incompetence. At some point if I couldn’t stop it - I’d be tempted to just kill the power to the server rooms, all of them. There just has to be a way to cut your losses.

I'd love to know what qualifies you to throw a word like incompetence around here. My best guess is the reason it took 45 minutes to shut it off was due to a judgement call: burn through free cash, or take out all their customers too. Bear in mind some of the largest retail brokerages in the world hang off Knight. Their primary functions are acting as an order destination and a market-maker, for efficiency's sake an…

I'm sure they would have tried pushing code in that time at least a few times.

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#14
post #9

"Knight is also working with Goldman Sachs to help unwind the trades behind its extensive loss, according to people briefed on the matter. "Goldman has agreed to buy, at a discount, the shares that the trading firm had accumulated. Such a move would help Knight by taking the portfolio off its hands and freeing up capital." What does this mean? Why would GS do this? Why would Knight do this? Couldn't they just sell th…

Yes and no. Given the kind of volume Knight purchased during the faulty trades, it could be difficult to offload that many shares on the open market in a timely manner. Maybe those stocks are hot Monday, maybe they're not. Knight needs capital yesterday to keep floating, so they're likely looking to sell everything in one basket.

As for GS's motivation, they're buying at a discount. Due to the time sensitive nature of Knight's predicament, they're probably trading the portfolio to Goldman at a reduced rate. Unlike Knight, Goldman has the cash to sit on it for a while and sell the shares directly out into the open market, even if it takes a few days. Given the discount they bought the shares at, they're likely selling with a decent margin.

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#15
What I wonder, following this story this week, is how the software quality controls at a place like Knight compare with those for life-critical systems like those in, e.g., aviation.

On one hand, you'd think the QA in finance would be pretty solid, considering that the survival of the company could be at stake (witness Knight). On the other hand, I have a feeling that even there, people just don't take it that seriously.

Would love to hear from anyone with more experience writing software for these industries.

Re: Trading Program Ran Amok, With No ‘Off’ Switch

#18
This was apparently an infrastructure problem of some sort: http://www.bloomberg.com/video/tom-joyce-knight-is-open-for-...

Infrastructure changes can be notoriously difficult to back out by simply using an "off" switch, particularly if this was some type of a firmware upgrade that impacted all of their production servers. Backing it out at a minimum would require some type of a reboot, which would cause problems with an active trades. It could very well be that they were running an Active-Active environment, they had to go Active-Passive, back out the changes from the passive environment, reboot, and surgically cut over to the passive environment. This could easily take 30 minutes.

Post reply on HN