Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to lose $172,222 a second for 45 minutes

pythonsweetness.tumblr.com

Post reply on HN
Print view View original

121–130 of 243 posts

Re: How to lose $172,222 a second for 45 minutes

#121

Post by jzwinck »

donpdonp wrote: post #59
ycombobreaker wrote: post #52

Earlier quoted context omitted.

> After trading was halted they set the cap at 20% loss for rolling back trades. FYI, NYSE rolled back transactions based on predetermined price rules. There were no discretionary rollbacks associated with Knight's "big day". > The only people that came out of this ahead were aggressive algos on the other side Don't forget the other market makers; they took the other side of those Knight trades and made the spread ev…

is there more information on what rollbacks did happen and what the price rules are?

Search for "clearly erroneous" and the name of your favorite exchange. The policies are often spelled out fairly clearly, though most (all) will include an exchange discretion clause.

Re: How to lose $172,222 a second for 45 minutes

#122

Post by 6d0debc071 »

lmm wrote: post #114
6d0debc071 wrote: post #85

Earlier quoted context omitted.

This was the bit that most shocked me: "Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called." From thereon they purely and simply deserved everything that happened to them.

They didn't retest some code that hadn't been used since 2003. Why would they? Sure, removing that code would've been a good idea, but that's a forgivable mistake. Reusing a flag that did something different in a currently-deployed version, without having a "transition" version that ignores that flag? Dodgy, but makes sense if you're in a rush. Needing to manually deploy code to 8 different servers? Just stupid.

Because not testing it can have unintended side effects at some low level of probability, and we all know this - mostly we just don't think it's going to happen to us. However, as the significance of the risk associated with that low level of probability goes up the demands for securing against that risk go up.

Computers are very powerful when placed in certain configurations. The more powerful the system you're dealing with the more cautious you should be. If they were dealing with an app then, sure, I'd have a lot more pity for them not taking precautions - such precautions would not be reasonable to expect of them. But if you're not being excessively paranoid about such a powerful system as was deployed here, then you're doing it wrong.

I do feel some pity for them based on the fact that there's not a tradition of caution in programming. And I do agree that there were multiple points of failure in there. But testing all the code that's going to be on a system like this is a base level of caution that should be used - whether or not you intend to use that code. If you think it's too much bother to test, then it shouldn't be there - but if it's gonna be there then for god's sake test.

Re: How to lose $172,222 a second for 45 minutes

#123

Post by nandemo »

confluence wrote: post #105
vecter wrote: post #92

Earlier quoted context omitted.

I really hate to go down this road because it's been rehashed thousands of times on Hacker News, but high frequency traders add value to the market by adding liquidity (and therefore reducing spreads --> cost to you for executing) and price discovery.

HFT does not add liquidity, but they do squeeze spreads in liquid contracts. It's some what good for retail, but terrible for institutional investors, which hurt retail on the backend. HFT is just a front running operation in those cases.

What is that definition of adding liquidity that doesn't include "tightening the spread"?

Re: How to lose $172,222 a second for 45 minutes

#124

Post by SideburnsOfDoom »

moath wrote: post #63
malbs wrote: post #11

Just one of the risks of automation, and a good reminder why human monitoring is necessary. Having said that, we deployed a system that was mostly automated, with the human operator to oversee investments and if any out-of-the-ordinary transactions (based on experience) were taking place, to shut it down. She happily sat there approving the recommendations even though the recommendations were absolutely outside of an…

I'm not seeing anything here that makes me think they had any kind of automation at all! In my experience, if they had automation things would have consistently failed on all servers the deployment was executed on. That said, if you're going to fly the jet liner in full manual mode, you better make sure your co-pilot is watching the instruments.

The parent post may be pointing out that the point of this software is to automate trading on the stock market. It's risky if your software testing, rollout, monitoring and rollback process is not sufficiently automated. This second kind of automation is the kind that you or I are most likely more familiar with. And was lacking.

Re: How to lose $172,222 a second for 45 minutes

#125

Post by SideburnsOfDoom »

vincie wrote: post #71

I would love to hear from an ex-Knight tech. Wouldn't be surprised if they wrote something along the lines of: "Management just wanted this thing in ASAP!", or perhaps "Tests weren't part of the kpi's". I may sound biased against non-techs, but I have seen this time and time again. Testing is a barrier to quick deployment, and "How much money are we losing while doing all that stoopid testing?".

> Testing is a barrier to quick deployment

I really feel bad for people who think like that. A process where tests and deployments are automated and repeatable are vital to quick, robust deployment. Quick deployment without tests just isn't going to work well.

Re: How to lose $172,222 a second for 45 minutes

#126

Post by segmondy »

fiatmoney wrote: post #8

"The best part is the fine: $12m, despite the resulting audit also revealing that the system was systematically sending naked shorts." Cool - all you have to do to get away with financial crimes is create a system with no protections against breaking the law.

Be very careful with what you are suggesting. If we are going to call them criminals because of bug in their code and system. Then don't be upset the day you are called a criminal for a bug in your code, for not having a test unit or deploying the wrong thing and not having the right process in place.

Re: How to lose $172,222 a second for 45 minutes

#127

Post by jzwinck »

beagle3 wrote: post #64
thaumasiotes wrote: post #38

Earlier quoted context omitted.

> How can it be otherwise? You'd need scheduled downtime to do it any other way. Roll out the code in advance, and have the production machines switch to it at a defined, synchronized time? I mean, imagine you only have one production machine. If you're willing to admit that you can have it switch from version X to version Y with no downtime, then synchronization is the only barrier to doing the same on n machines. W…

Synchronization is non-trivial, but the question is mostly how fine you need the synchronization to be. E.g., if you are doing a live upgrade using e.g Erlang or Nginx, you can sort-of decide when new processes will be served by the new server, but existing processes and requests may linger with old code until much later. But there's at least 30 minutes of downtime per week per market (usually per day), and the vast…

You're right, yet nearly everybody operates on a daily restart schedule. I guess it's more intuitive, but it's wrong. Still, nobody ever got fired for bouncing systems once a day!

Re: How to lose $172,222 a second for 45 minutes

#128

Post by dijit »

lhaussknecht wrote: post #73
jpatokal wrote: post #46

Just another reminder of how systems that you'd think are rock solid often aren't. In my previous life working with telcos, I once tried to teach a particularly huge customer how to use CVS how to manage configurations across a 10+ machine cluster of machines. They didn't see any value in it, so they stuck to their good old process of SSHing into each machine individually, "cp config.xml config.xml.20131022", and the…

Reminds me of a colleague who RDPed into each of our 140 subsidiaries to change a config file. He had a list of servers on his desk and ticked off every server. Took him the whole day to apply the changes.

I did this with nagios configs in my first job :(

Re: How to lose $172,222 a second for 45 minutes

#129

Post by fhars »

marijn wrote: post #91
scrrr wrote: post #87

High Frequency Trading seems so abstract. There's no value created, it seems. It's like something in between imperfect systems, scraping off the margin created by that imperfection. It's fascinating, and interesting from an algorithmic point of view (like a computer game), but at the same time I don't feel sympathy for this company going out of business.

Looking at systems by considering whether they 'create value' in some generalized utilitarian sense is unproductive. Such systems survive by being able to extract energy somehow, in this case by exploiting properties of the stock trading system. I guess you could say that they create a lot of value—for the people doing it. Very few modern economic activities make sense in a broader perspective, they exist purely beca…

Looking at systems by considering whether they 'create value' in some generalized utilitarian sense is unproductive. Such systems survive by being able to extract energy somehow

A perfect description of many forms of organized crime.

Re: How to lose $172,222 a second for 45 minutes

#130

Post by Pitarou »

michaelt wrote: post #75
tantalor wrote: post #72

That explains how the deprecated "Power Peg" model was activated, but why was that model so flawed?

You may have heard people with things like backups and emergency generators saying "you have to test this stuff weekly, in case someone has broken it so it'll fail the moment you call on it." Software is the same. Knight had code that hadn't been run in 8 years. Sure, the code worked 8 years ago, but things have changed around it since then. As the problem code never ran, no-one noticed it getting broken, or had any…

If I understand this correctly, this isn't like having untested code around. It's more like leaving highly toxic medicine in the bathroom cabinet when you no longer need it, or leaving an electrically powered band saw plugged in when it's not in use.
Post reply on HN