I would love to hear from an ex-Knight tech. Wouldn't be surprised if they wrote something along the lines of: "Management just wanted this thing in ASAP!", or perhaps "Tests weren't part of the kpi's". I may sound biased against non-techs, but I have seen this time and time again. Testing is a barrier to quick deployment, and "How much money are we losing while doing all that stoopid testing?".
How to lose $172,222 a second for 45 minutes
71–80 of 243 posts
Re: How to lose $172,222 a second for 45 minutes
#72That explains how the deprecated "Power Peg" model was activated, but why was that model so flawed?
Re: How to lose $172,222 a second for 45 minutes
#73Just another reminder of how systems that you'd think are rock solid often aren't. In my previous life working with telcos, I once tried to teach a particularly huge customer how to use CVS how to manage configurations across a 10+ machine cluster of machines. They didn't see any value in it, so they stuck to their good old process of SSHing into each machine individually, "cp config.xml config.xml.20131022", and the…
Reminds me of a colleague who RDPed into each of our 140 subsidiaries to change a config file.
He had a list of servers on his desk and ticked off every server.
Took him the whole day to apply the changes.
Re: How to lose $172,222 a second for 45 minutes
#74They fined them for losing money? What?
They fined them for (accidentally) selling shorts they couldn't cover.
Re: How to lose $172,222 a second for 45 minutes
#75That explains how the deprecated "Power Peg" model was activated, but why was that model so flawed?
You may have heard people with things like backups and emergency generators saying "you have to test this stuff weekly, in case someone has broken it so it'll fail the moment you call on it."
Software is the same.
Knight had code that hadn't been run in 8 years. Sure, the code worked 8 years ago, but things have changed around it since then. As the problem code never ran, no-one noticed it getting broken, or had any reason to fix it if it broke in testing.
Most likely the code worked fine 8 years ago, broke in the intervening 8 years, and hence was broken when activated.
Re: How to lose $172,222 a second for 45 minutes
#76They fined them for losing money? What?
No, the SEC fined them for losing money stupidly. In order to have access to the market like they did, they had to follow certain laws that are enforced by the SEC. When they were losing all that money they weren't following those laws.
It's like if you cause an accident while you're driving by breaking the law; you get a traffic citation (and the accompanying fine), even if your car is totaled as a result of the accident, because you did something illegal.
Re: How to lose $172,222 a second for 45 minutes
#77Just another reminder of how systems that you'd think are rock solid often aren't. In my previous life working with telcos, I once tried to teach a particularly huge customer how to use CVS how to manage configurations across a 10+ machine cluster of machines. They didn't see any value in it, so they stuck to their good old process of SSHing into each machine individually, "cp config.xml config.xml.20131022", and the…
We often get frustrated with strict change management processes, excruciating verification etc .. but there is a reason why the most mature (not always the biggest..) operators have these processes in place.
Re: How to lose $172,222 a second for 45 minutes
#78Earlier quoted context omitted.
Yeah ours was monitoring just rubber stamped it. Afterwards everyone remarked, everyone could tell they were bad just looking at what was in front of them, our theory was she was too busy watching the breaking bad final episode or something.
You know how if the first 100 times a dialog box comes up, the correct response is to click 'ok', then people start just clicking 'ok' on every dialog box, and then the 201st one comes up "Destroy everything? [cancel] [ok], and they click 'ok' too, and don't think anything of it?
Sure, you click reflexively, but you should notice the text was bad either after clicking or within a couple more clicks. Letting the first few errors through is reasonable, but letting a wall of them through without noticing anything wrong, when reading them is your job, is inexcusable.
Re: How to lose $172,222 a second for 45 minutes
#79That explains how the deprecated "Power Peg" model was activated, but why was that model so flawed?
They roughly explained it. The counters were in a different state because of refactoring moving their updates around, so the Power Peg code had an incorrect view of reality.
Re: How to lose $172,222 a second for 45 minutes
#80I'm shocked they didn't have a killswitch or automated stop-loss of some kind. A script that says "We just lost $5M in a few minutes; maybe there's a problem." Or, a guy paid minimum wage to watch the balance, with a button on his desk. $172,222 is a lot of minimum-wage years.
It looks like they saw positions accumulating in one of theirs accounts, but couldn't identify the source. And maybe they saw positions too late, because of lag. Here's a description from that SEC report, onto what went wrong with their monitoring system: "Moreover, because the 33 Account held positions from multiple sources, Knight personnel could not quickly determine the nature or source of the positions accumulat…
I think your formatting above has forced a very wide page width....