Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to lose $172,222 a second for 45 minutes

pythonsweetness.tumblr.com

Post reply on HN
Print view View original

31–40 of 243 posts

Re: How to lose $172,222 a second for 45 minutes

#31

Post by 4hthth4 »

malbs wrote: post #28
fiatmoney wrote: post #17

Earlier quoted context omitted.

You should read the linked PDF - they had systems that were 100% dependent on human monitoring, that no one was checking, or where no one recognized anything unusual. If anything, their failures were due to massive lack of automation in deployment, testing, and monitoring.

Yeah ours was monitoring just rubber stamped it. Afterwards everyone remarked, everyone could tell they were bad just looking at what was in front of them, our theory was she was too busy watching the breaking bad final episode or something.

It makes me think you shouldn't allow only a single person to monitor something like that...

Re: How to lose $172,222 a second for 45 minutes

#32

Post by malbs »

fiatmoney wrote: post #17
malbs wrote: post #11

Just one of the risks of automation, and a good reminder why human monitoring is necessary. Having said that, we deployed a system that was mostly automated, with the human operator to oversee investments and if any out-of-the-ordinary transactions (based on experience) were taking place, to shut it down. She happily sat there approving the recommendations even though the recommendations were absolutely outside of an…

You should read the linked PDF - they had systems that were 100% dependent on human monitoring, that no one was checking, or where no one recognized anything unusual. If anything, their failures were due to massive lack of automation in deployment, testing, and monitoring.

Oh the other amazing thing we used to have - a system that generated so many alerts that everyone just created outlook rules to automatically delete all the alerts. I mean really? I wondered if we should modify the system to only generate messages when there are actual exceptional cases? People looked at me like I was an idiot

Re: How to lose $172,222 a second for 45 minutes

#33

Post by manishsharan »

Don't humans also make similar large scale mistakes? Merill Lynch's infamous London whale comes to mind. Also. I could be wrong but aren't most of derivatives a zero sum game: don't I have to lose money on my puts for you to make money on your calls ? Didn't so many people lose money on securities because they misunderstood their exposure ?

The Knight computer error was spectacular and catastrophic but us humans have a longer track record of making catastrophic financial decisions in the market.

Re: How to lose $172,222 a second for 45 minutes

#34

Post by jrochkind1 »

malbs wrote: post #28
fiatmoney wrote: post #17

Earlier quoted context omitted.

You should read the linked PDF - they had systems that were 100% dependent on human monitoring, that no one was checking, or where no one recognized anything unusual. If anything, their failures were due to massive lack of automation in deployment, testing, and monitoring.

Yeah ours was monitoring just rubber stamped it. Afterwards everyone remarked, everyone could tell they were bad just looking at what was in front of them, our theory was she was too busy watching the breaking bad final episode or something.

You know how if the first 100 times a dialog box comes up, the correct response is to click 'ok', then people start just clicking 'ok' on every dialog box, and then the 201st one comes up "Destroy everything? [cancel] [ok], and they click 'ok' too, and don't think anything of it?

Re: How to lose $172,222 a second for 45 minutes

#36

Post by drill_sarge »

phyalow wrote: post #35
drill_sarge wrote: post #30

I still find just the fact scary that at this moment automated systems are shoving billions of fake money back and forth around the world.

How is it fake?

Or better call it virtual money. (but we all know theres also "fake" money made and destroyed in that system, too)

Re: How to lose $172,222 a second for 45 minutes

#37

Post by tempestn »

dror wrote: post #23

Is there any benefit to the market as a whole to have these high speed transactions trying to game the system? Seems like as a rule, they're likely to cause instability, and I have a hard time seeing any benefits in them.

The standard answer to this is that they provide liquidity. Whether that benefit outweighs the drawbacks is a subject of debate.

Re: How to lose $172,222 a second for 45 minutes

#38

Post by thaumasiotes »

thrownaway2424 wrote: post #15
MBCook wrote: post #10

Earlier quoted context omitted.

Having code on your production servers that runs billions of dollars of business per day, which you haven't run for 8 years , is obviously bad. Deploying in such a way that all your servers are not running the same codebase is obviously bad. Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad. Not having anyone monitor things closely enough, including the hundreds of…

I'm not sure I agree with "Deploying in such a way that all your servers are not running the same codebase is obviously bad." I have a lot of experience in large scale systems (although this incident with 8 machines does not qualify) and I would say there is _always_ a period of transition where versions X and Y are online in production simultaneously. How can it be otherwise? You'd need scheduled downtime to do it a…

> How can it be otherwise? You'd need scheduled downtime to do it any other way.

Roll out the code in advance, and have the production machines switch to it at a defined, synchronized time?

I mean, imagine you only have one production machine. If you're willing to admit that you can have it switch from version X to version Y with no downtime, then synchronization is the only barrier to doing the same on n machines. Why would you need scheduled downtime?

Re: How to lose $172,222 a second for 45 minutes

#39

Post by drill_sarge »

4hthth4 wrote: post #31
malbs wrote: post #28

Earlier quoted context omitted.

Yeah ours was monitoring just rubber stamped it. Afterwards everyone remarked, everyone could tell they were bad just looking at what was in front of them, our theory was she was too busy watching the breaking bad final episode or something.

It makes me think you shouldn't allow only a single person to monitor something like that...

Yes, the more people monitoring the better. But in this situation someone has to be in charge and make a decision. And nobody wants to be that guy.

Re: How to lose $172,222 a second for 45 minutes

#40

Post by adambratt »

manishsharan wrote: post #33

Don't humans also make similar large scale mistakes? Merill Lynch's infamous London whale comes to mind. Also. I could be wrong but aren't most of derivatives a zero sum game: don't I have to lose money on my puts for you to make money on your calls ? Didn't so many people lose money on securities because they misunderstood their exposure ? The Knight computer error was spectacular and catastrophic but us humans have…

Options are complicated. At their most basic level they are no different than a bet, so yes zero-sum. However when used in a spread or as a hedge or any other way to avoid risk or when sold against stock you own as an income generator, it's tough to call them zero-sum.

Puts and Calls are confusing as they are both something you buy. It's not like a sports bet where you're betting on the team to win so the other side will lose if they win. You can buy a put and a call in the same stock and profit on both if your strike prices are aligned properly.

The opposite side of buying a call is selling a call. Just like shorting a stock, when selling an option the most you can make is 100% and your loss potential is infinite. As an option seller, you're hoping the option expires worthless and out of the money so that you can keep the initial outlay. Most brokerages require a high level of clearance to allow you to sell naked puts and calls as it's generally a bad idea if you can't cover the potential upside.

Post reply on HN