How to lose $172,222 a second for 45 minutes
pythonsweetness.tumblr.com
How to lose $172,222 a second for 45 minutes
1–10 of 243 posts
Re: How to lose $172,222 a second for 45 minutes
#2I remember when Knight was in the news regarding this but never the technical details about what took place. It's scary stuff especially given the money on the line, and it makes a good case study for devops. I understand the temptation to re-use a field but normally I'm for using new values in those fields.
Re: How to lose $172,222 a second for 45 minutes
#3Hindsight is 20/20
Re: How to lose $172,222 a second for 45 minutes
#4I'm shocked they didn't have a killswitch or automated stop-loss of some kind. A script that says "We just lost $5M in a few minutes; maybe there's a problem." Or, a guy paid minimum wage to watch the balance, with a button on his desk. $172,222 is a lot of minimum-wage years.
Re: How to lose $172,222 a second for 45 minutes
#5The title reminds me of hosting clients, who would complain about losing thousands of dollars per minute when their $10/month website was experiencing downtime.
Re: How to lose $172,222 a second for 45 minutes
#6Powder Keg is a distinctly un-reassuring name for finance related functionality.
Re: How to lose $172,222 a second for 45 minutes
#7I'm shocked they didn't have a killswitch or automated stop-loss of some kind. A script that says "We just lost $5M in a few minutes; maybe there's a problem." Or, a guy paid minimum wage to watch the balance, with a button on his desk. $172,222 is a lot of minimum-wage years.
How many millions in orders do they normally process per minute?
Since there were no procedures in place, would you like to be the guy who pulled the plug on the (let's guess) $100 million/minute processing system? Do you think you could get another job after that? What would the costs be for violating contracts? You could single handedly sink the company (which, in the end, this issue basically did).
I don't blame the guy for not killing all operations. He never should have been put in that situation. Proper QA, regression testing, monitoring, running a shadow copy and verifying it's output, there are tons of things that could have prevented/mitigated this.
Re: How to lose $172,222 a second for 45 minutes
#8"The best part is the fine: $12m, despite the resulting audit also revealing that the system was systematically sending naked shorts."
Cool - all you have to do to get away with financial crimes is create a system with no protections against breaking the law.
Re: How to lose $172,222 a second for 45 minutes
#9The week after this we had a trader in our office who had a meeting at Knight on the morning it happened.
He said he saw the whole dev team just power off and go home at 11am, followed quickly by the rest of the employees. At that point, there was nothing they could do.
The craziest thing is that it went on for so long. No one caught it until their own traders so it come across Bloomberg and CNBC. They actually thought it was a rival HFT and tried to play against it.
The only people that came out of this ahead were aggressive algos on the other side and a few smart individual traders. A lot of retail guys had stop losses blown through that normally would never have been hit. After trading was halted they set the cap at 20% loss for rolling back trades. So if you lost 19% of your position in that short period of craziness, tough luck.
Re: How to lose $172,222 a second for 45 minutes
#10Hindsight is 20/20
Having code on your production servers that runs billions of dollars of business per day, which you haven't run for 8 years, is obviously bad.
Deploying in such a way that all your servers are not running the same codebase is obviously bad.
Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad.
Not having anyone monitor things closely enough, including the hundreds of warning emails they got before the market opened, is obviously bad.
There is no hindsight necessary here. You could look at what they were doing and predict a catastrophe.