Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to lose $172,222 a second for 45 minutes

pythonsweetness.tumblr.com

Post reply on HN
Print view View original

161–170 of 243 posts

Re: How to lose $172,222 a second for 45 minutes

#161

Post by avn2109 »

marincounty wrote: post #96
scrrr wrote: post #87

High Frequency Trading seems so abstract. There's no value created, it seems. It's like something in between imperfect systems, scraping off the margin created by that imperfection. It's fascinating, and interesting from an algorithmic point of view (like a computer game), but at the same time I don't feel sympathy for this company going out of business.

I think HIgh Frequency Trading will eventually be neutralized through competition. I noticed the their profits aren't as staggering as they were a few years ago. I would like to see a law that limits how close an company(individual) can set up shop next to an exchange though. I think Insider Trading is more of a problem than the public knows?

"Only a fool would trade stocks without inside information."

Re: How to lose $172,222 a second for 45 minutes

#162

Post by bradleyjg »

adambratt wrote: post #40
manishsharan wrote: post #33

Don't humans also make similar large scale mistakes? Merill Lynch's infamous London whale comes to mind. Also. I could be wrong but aren't most of derivatives a zero sum game: don't I have to lose money on my puts for you to make money on your calls ? Didn't so many people lose money on securities because they misunderstood their exposure ? The Knight computer error was spectacular and catastrophic but us humans have…

Options are complicated. At their most basic level they are no different than a bet, so yes zero-sum. However when used in a spread or as a hedge or any other way to avoid risk or when sold against stock you own as an income generator, it's tough to call them zero-sum. Puts and Calls are confusing as they are both something you buy. It's not like a sports bet where you're betting on the team to win so the other side…

For most products there is a legal framework that forbids buying an insurance contract unless you have an insurable interest. However, under the Commodity Futures Modernization Act of 2000, designed by Summers, Greenspan, Levitt, and Rainer, state insurance regulators are forbidden from regulating OTC derivatives as insurance products. They were already forbidden from regulating exchange traded derivatives (e.g. options and futures).

Re: How to lose $172,222 a second for 45 minutes

#163

Post by xntrk »

washedup wrote: post #144
levosmetalo wrote: post #136

Earlier quoted context omitted.

I would support idea to just price every transaction with a 0.0001% transaction fee. It should be enough to bring the whole HFT industry out of business while not having any impact on markets themselves.

There already are trading fees. Each exchange charges a small fee for every trade. It's how they make money and is why they have an incentive to increase volume. For example, on the CME futures exchange the fee per energy contract is usually 50 cents. Of course, those are only for trades that are actually filled. For quoting there are no transaction costs, but there are often rules to limit quoting (transactions/fill…

This isn't true. "Each exchange charges a small fee for every trade" Not all exchanges charge fees for posting liquitity. Smaller ECNs will actually pay traders to provide liquitity. If you hit or lift someones bid or offer you will get charged a fee but if you post an order that does not cross against the current market you can get paided for. these rebates are very importatn for HFT desks.

Re: How to lose $172,222 a second for 45 minutes

#164

Post by hderms »

koliber wrote: post #143
levosmetalo wrote: post #136

Earlier quoted context omitted.

I would support idea to just price every transaction with a 0.0001% transaction fee. It should be enough to bring the whole HFT industry out of business while not having any impact on markets themselves.

A lot of the HFT high-frequencyness comes from lots of orders beng sent to market. Only a very small number of these orders result in transactions. A transaction fee would not affect most of the order flow which never gets executed.

there should be a small tax per order put out, even if cancelled and a small percentage-based tax on each trade. If that behavior has been determined to exacerbate price declines that externality should be captured in a tax and de-incentivized properly.

Re: How to lose $172,222 a second for 45 minutes

#165

Post by jbrechtel »

adambratt wrote: post #9

The week after this we had a trader in our office who had a meeting at Knight on the morning it happened. He said he saw the whole dev team just power off and go home at 11am, followed quickly by the rest of the employees. At that point, there was nothing they could do. The craziest thing is that it went on for so long. No one caught it until their own traders so it come across Bloomberg and CNBC. They actually thoug…

They left at 11am because they realized how bad the problem was and just gave up? Or left at 11am as a normal thing (e.g. before they noticed the problem) ?

Re: How to lose $172,222 a second for 45 minutes

#166

Post by MBCook »

thrownaway2424 wrote: post #15
MBCook wrote: post #10

Earlier quoted context omitted.

Having code on your production servers that runs billions of dollars of business per day, which you haven't run for 8 years , is obviously bad. Deploying in such a way that all your servers are not running the same codebase is obviously bad. Deploying to production with no plan for how to roll it back if something goes wrong is obviously bad. Not having anyone monitor things closely enough, including the hundreds of…

I'm not sure I agree with "Deploying in such a way that all your servers are not running the same codebase is obviously bad." I have a lot of experience in large scale systems (although this incident with 8 machines does not qualify) and I would say there is _always_ a period of transition where versions X and Y are online in production simultaneously. How can it be otherwise? You'd need scheduled downtime to do it a…

You're right. I understand a slow rollout, I was referring to the fact that they thought they deployed to all servers but didn't.

If you plan is for the software to be on all servers, it needs to be on all servers.

Re: How to lose $172,222 a second for 45 minutes

#167

Post by jdbernard »

6d0debc071 wrote: post #85
sirsar wrote: post #4

I'm shocked they didn't have a killswitch or automated stop-loss of some kind. A script that says "We just lost $5M in a few minutes; maybe there's a problem." Or, a guy paid minimum wage to watch the balance, with a button on his desk. $172,222 is a lot of minimum-wage years.

This was the bit that most shocked me: "Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called." From thereon they purely and simply deserved everything that happened to them.

If I understand correctly, they were planning to remove the Power Peg code. Do you re-test the left-overs of your refactoring: the parts you delete?

The real problem here is that they were using incremental deployment and did not have a good process for ensuring the same changes were successfully made to all server.

Re: How to lose $172,222 a second for 45 minutes

#168

Post by protomyth »

"During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review."

That is just painful to read. How many times do we hear a company couldn't figure out how to migrate code properly? Do any software engineering programs teach proper code migration?

Next time a manager questions money spent on integration or system testing, hand them a printout of this SEC document and explain how much the problem can cost.

Re: How to lose $172,222 a second for 45 minutes

#169

Post by mathattack »

hrjet wrote: post #111
jpatokal wrote: post #46

Just another reminder of how systems that you'd think are rock solid often aren't. In my previous life working with telcos, I once tried to teach a particularly huge customer how to use CVS how to manage configurations across a 10+ machine cluster of machines. They didn't see any value in it, so they stuck to their good old process of SSHing into each machine individually, "cp config.xml config.xml.20131022", and the…

I heard similar stories from a friend working for a big telco. The other day he was describing a strange bug which was triggering in the field of a large telco. It was strange because it used to get triggered after exactly 85 days of deployment. It turned out to be a debug script that was pinging a development server and timing out (because the development server was not accessible from the field). The series of retr…

Legacy telcos are the epitome of large big institutions where many of the best talents leave. After just a few years of working with them, I couldn't believe they could get a dial tone. How did they produce so much great R&D?

Re: How to lose $172,222 a second for 45 minutes

#170

Post by mathattack »

fosk wrote: post #53
jpatokal wrote: post #46

Just another reminder of how systems that you'd think are rock solid often aren't. In my previous life working with telcos, I once tried to teach a particularly huge customer how to use CVS how to manage configurations across a 10+ machine cluster of machines. They didn't see any value in it, so they stuck to their good old process of SSHing into each machine individually, "cp config.xml config.xml.20131022", and the…

Ha, probably no one ever got fired for manually SSHing into each machine. It seems like a joke, but it really isn't.

Is that job security?
Post reply on HN