Earlier quoted context omitted.
How is it fake?
As opposed to all the "real" money in the world?
How is it "real"?
How to lose $172,222 a second for 45 minutes
61–70 of 243 posts
Re: How to lose $172,222 a second for 45 minutes
#62I'm shocked they didn't have a killswitch or automated stop-loss of some kind. A script that says "We just lost $5M in a few minutes; maybe there's a problem." Or, a guy paid minimum wage to watch the balance, with a button on his desk. $172,222 is a lot of minimum-wage years.
How many millions in orders do they normally process per minute? Since there were no procedures in place, would you like to be the guy who pulled the plug on the (let's guess) $100 million/minute processing system? Do you think you could get another job after that? What would the costs be for violating contracts? You could single handedly sink the company (which, in the end, this issue basically did). I don't blame t…
Have them in pairs or teams, war games style.
Guy1 "Sir, we're losing money much faster than predicted, turn your key"
Guy2 "It could just be market variance"
Guy1 pulls out gun "Turn your key, sir"
Re: How to lose $172,222 a second for 45 minutes
#63Just one of the risks of automation, and a good reminder why human monitoring is necessary. Having said that, we deployed a system that was mostly automated, with the human operator to oversee investments and if any out-of-the-ordinary transactions (based on experience) were taking place, to shut it down. She happily sat there approving the recommendations even though the recommendations were absolutely outside of an…
I'm not seeing anything here that makes me think they had any kind of automation at all! In my experience, if they had automation things would have consistently failed on all servers the deployment was executed on.
That said, if you're going to fly the jet liner in full manual mode, you better make sure your co-pilot is watching the instruments.
Re: How to lose $172,222 a second for 45 minutes
#64Earlier quoted context omitted.
I'm not sure I agree with "Deploying in such a way that all your servers are not running the same codebase is obviously bad." I have a lot of experience in large scale systems (although this incident with 8 machines does not qualify) and I would say there is _always_ a period of transition where versions X and Y are online in production simultaneously. How can it be otherwise? You'd need scheduled downtime to do it a…
> How can it be otherwise? You'd need scheduled downtime to do it any other way. Roll out the code in advance, and have the production machines switch to it at a defined, synchronized time? I mean, imagine you only have one production machine. If you're willing to admit that you can have it switch from version X to version Y with no downtime, then synchronization is the only barrier to doing the same on n machines. W…
Synchronization is non-trivial, but the question is mostly how fine you need the synchronization to be. E.g., if you are doing a live upgrade using e.g Erlang or Nginx, you can sort-of decide when new processes will be served by the new server, but existing processes and requests may linger with old code until much later.
But there's at least 30 minutes of downtime per week per market (usually per day), and the vast majority of downtimes coincide during the weekend - so this is all moot discussion and needlessly complex solution. If you can afford the downtime, switch midnight GMT between Saturday and Sunday, when all markets are closed.
Re: How to lose $172,222 a second for 45 minutes
#65Don't humans also make similar large scale mistakes? Merill Lynch's infamous London whale comes to mind. Also. I could be wrong but aren't most of derivatives a zero sum game: don't I have to lose money on my puts for you to make money on your calls ? Didn't so many people lose money on securities because they misunderstood their exposure ? The Knight computer error was spectacular and catastrophic but us humans have…
Options are complicated. At their most basic level they are no different than a bet, so yes zero-sum. However when used in a spread or as a hedge or any other way to avoid risk or when sold against stock you own as an income generator, it's tough to call them zero-sum. Puts and Calls are confusing as they are both something you buy. It's not like a sports bet where you're betting on the team to win so the other side…
Yea I always used options as a form of protection, not the rest.
Re: How to lose $172,222 a second for 45 minutes
#66Is there any benefit to the market as a whole to have these high speed transactions trying to game the system? Seems like as a rule, they're likely to cause instability, and I have a hard time seeing any benefits in them.
The standard answer to this is that they provide liquidity. Whether that benefit outweighs the drawbacks is a subject of debate.
A non-economist wants to know: if liquidity is beneficial to our economy and liquidity is a function of time, how how does the time-benefit curve look as t approaches 0? I don't know if you can quantify the benefit and map this curve, but if you could I don't imagine it would scale to infinity as time approached zero.
Re: How to lose $172,222 a second for 45 minutes
#67I'm shocked they didn't have a killswitch or automated stop-loss of some kind. A script that says "We just lost $5M in a few minutes; maybe there's a problem." Or, a guy paid minimum wage to watch the balance, with a button on his desk. $172,222 is a lot of minimum-wage years.
Stop loss orders are not a panacea here. You could lock in a loss for no other reason than the market becoming very volatile due to some news item or some other reason like the flash crash. And you can't even guarantee to limit your losses at the stop loss price. It's just the price that triggers an attempt to unwind the position.
Also, the market may not go against you immediately. What if the glitch in the system means you're opening positions in stocks and you drive up the price by doing so? The losses are not immediately apparent. There's no screen where you could watch your losses run up in real time. The losses only become apparent once you try to unwind those positions and that's the case in many kinds of scenarios.
I believe it took J.P Morgan months to unwind the London Whale positions and really know what losses were incurred.
I think there's a better chance of catching a glitch at the point where the positions are opened.
Re: How to lose $172,222 a second for 45 minutes
#68They fined them for losing money? What?
Re: How to lose $172,222 a second for 45 minutes
#69Is there any benefit to the market as a whole to have these high speed transactions trying to game the system? Seems like as a rule, they're likely to cause instability, and I have a hard time seeing any benefits in them.
I don't know how much of a benefit there is, but I doubt it's a problem. If they make a mistake, worst case is they lose money and everyone else gains from it.
Re: How to lose $172,222 a second for 45 minutes
#70I assumed they had redundant servers with consensus algorithms in place in finance but apparently they don't. Would it be impractical?