How I made $500k with machine learning and high frequency trading • Hacker News Board

Re: How I made $500k with machine learning and high frequency trading

#301

Post by gd1 » Wed, Nov 07, 2012, 11:58 PM UTC

loup-vaillant wrote: post #281

gd1 wrote: post #279

Earlier quoted context omitted.

You're still missing it. Every single additional limit order in the market creates real value. If I want to buy 500 shares of company A, and one market participant is offering 250 at $10. Another is offering 250 at $11. It is going to cost me 250 x 10 + 250 x 11 = $5250. Now, the "short term speculator" you are complaining about shows up and puts his tiny little order in, he is offering 1 share at $10. Now the equati…

(Edit: "limit order" looks like technical jargon. If so, I don't know what it means. Maybe you didn't say what I thought you said) In your example, it looked like you wanted your 500 share for something else than selling them. So you're talking about "real transactions (with non-speculators) at the boundaries". So yes, those transactions are a useful service provided to you by the speculators. I'm not denying that. O…

I think we are getting closer to understanding here. You're kind of quibbling over the definition of value in your other post - someone can provide a "valuable" service even if it isn't currently being used (by your 'real transactors'). I might sleep better at night knowing that if I want to liquefy my investment in Apple tomorrow, I can do so without having to worry that nobody will be there to take the other side of the trade, that I'll have to pay an enormous spread or be left holding a position I want to be out of. An international corporation can be more confident of planning their cashflows knowing that the FX market will always be liquid, and will be quoting at most 2 ticks wide in the majors 24 hours a day every day. This is the oil of an economy, the freedom to allocate capital when you want, where you want and pay the smallest possible friction costs to do so.

Many things in our economy can be described as a zero-sum game. Society only needs so much of any given good, and at the retailer level you aren't involved in increasing demand. Two corner stores might get into a price/advertising war, but let's argue that there is still only a fixed amount of product they can sell to a small community. So advertiser's get fat on fees (in our case, exchanges do). So what is the point you ask? Evolution, competition, it can only be good for the consumer. You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost? It takes a tin foil hat to think that it isn't better and cheaper to invest now.

" they are not selected by the quality of the service, just ability to rip each other off" - people have a bit of a naive view that there are some sort of magical harlem globetrotter moves you can pull on an order book. The mechanics are dead simple I'm afraid. It isn't chess, not even checkers. You can't even see the other participants, everything is anonymous on the exchanges I've dealt with. Spoofing is illegal, and that is about as clever as it gets. Very little you can do to make your algo better will put it at odds with providing a better quality service.

To prove that, remember the two types of algo. If I write a liquidity provider that doesn't offer a tighter spread than the competition, I will miss out on trade flow and make little or nothing. If I write a liquidity taker that has an opinion that is wrong about the correct value of a security, someone else who is right will smash me. So effectively we are selecting for more liquidity, tighter spreads, and more accurate prices reflecting available information.

Re: How I made $500k with machine learning and high frequency trading

#302

Post by kolbe » Thu, Nov 08, 2012, 1:17 AM UTC

kevinrpope wrote: post #299

kolbe wrote: post #285

Earlier quoted context omitted.

How about efficiency? People call the liquidity providing aspects of HFT 'bullshit', but computers have vastly reduced the manpower necessary to manage a market. Each futures pit used to have hundreds of traders, who required several assistants/support and commanded a huge salary. Many firms needed multiple traders in a pit, just to be able to make sure they could provide liquidity to all possible market participants…

An argument can also be made that this is a net negative contribution, as instead of a market employing hundreds of people, it's only employing dozens. Ergo, more unemployed people. While this is good for the market's owners and those currently employed to trade there, it is bad for the economy as a whole.

You're on Hacker News, but you think that destroying jobs with technological innovation is a bad thing?

Re: How I made $500k with machine learning and high frequency trading

#303

Post by loup-vaillant » Thu, Nov 08, 2012, 1:23 AM UTC

gd1 wrote: post #301

loup-vaillant wrote: post #281

Earlier quoted context omitted.

(Edit: "limit order" looks like technical jargon. If so, I don't know what it means. Maybe you didn't say what I thought you said) In your example, it looked like you wanted your 500 share for something else than selling them. So you're talking about "real transactions (with non-speculators) at the boundaries". So yes, those transactions are a useful service provided to you by the speculators. I'm not denying that. O…

I think we are getting closer to understanding here. You're kind of quibbling over the definition of value in your other post - someone can provide a "valuable" service even if it isn't currently being used (by your 'real transactors'). I might sleep better at night knowing that if I want to liquefy my investment in Apple tomorrow, I can do so without having to worry that nobody will be there to take the other side o…

My definition of value is simple: something is valuable when it manages to raise people's utility functions (right now, that's about as rigorous as I can be). A trade that happens to be closed by a plain old investor is valuable. If it's another speculator, however, the value is zero. Only consequences ultimately matter.

But, if I got you right, the anonymity of it all mean we cannot separate the two… hmm… Then we've got to multiply the potential utility of the trade by the (quite low) probability of it being closed by a non-speculator. Still valuable, but much less. And I'm back wondering to what extent this is worth the (collective) effort. You did change my mind a little, though. I'll need to learn more.

> You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost?

The consumer winning does count as creating value. However there are 2 parts in the retail store competition example: the part where they compete on quality, price, diversity… and the part where they pay fat fees to the advertiser. The first part benefits the consumer, the second just add inefficiency in the loop: if both stores could only agree to not use ads, everybody would win.

When Pareto Optimum and Nash Equilibrium are at odds, life sucks.

Re: How I made $500k with machine learning and high frequency trading

#304

Post by genwin » Thu, Nov 08, 2012, 1:43 AM UTC

kevinrpope wrote: post #300

genwin wrote: post #258

Earlier quoted context omitted.

People will tell you that you were just a lucky monkey. But you could have run your algorithm on past data, for hundreds or thousands of fake portfolios, to tell, statistically, what the odds of your algorithm being simply lucky are. In early 2000s I wrote a machine learning algorithm that beat the S&P 100 with over 1 trillion to 1 odds against it being luck. It predicted a full trading day in advance. But that was a…

What did you end up doing with your code? Would you be able to run it today with the low-cost broker APIs?

The code sits in one of my archive folders. I ran it for a few years, perhaps to 2004, and saw the market steadily becoming more efficient, lowering my results (like the OP did). It may well be that it no longer predicts skillfully or profitably. As I recall, to beat the market the costs had to be very low, like pennies per trade, with no bid/ask spread, which I understood to be possible for large trading firms.

Re: How I made $500k with machine learning and high frequency trading

#305

Post by jspaulding » Thu, Nov 08, 2012, 1:55 AM UTC

nashequilibrium wrote: post #239

Pls do not follow the advice of the OP. I started a hedgefund in 2004 doing HF platform arbitrage and ran it for 5yrs and i can honestly tell you that this is just survivorship bias. This is a very complex field and being off slightly, having a slight bias, a fraction of a point off your execution pricing and a slightly flawed money management system is recipe for disaster. The biggest issue is the confusion that you…

There are plenty of shops making tons of money with HFT who do not have deal flow at all - it's got nothing to do with luck.

Survivorship bias would mean I simply got lucky. If you're going to say that you're at least going to need to look at my P&L charts and say how I could possibly achieve that much success with luck alone.

Finally, machine learning has everything to do with my success. There were hundreds of variables in my algorithm that were ALL optimized using ML. If you read the article you would know that I built an accurate model for backtesting that I used to optimize variables as well as confirm that I was going to make money before I even started live trading.

I'm pretty confident that whatever you were doing in 2004 has nothing to do with what I was doing.

Re: How I made $500k with machine learning and high frequency trading

#306

Post by kevinrpope » Thu, Nov 08, 2012, 1:56 AM UTC

kolbe wrote: post #302

kevinrpope wrote: post #299

Earlier quoted context omitted.

An argument can also be made that this is a net negative contribution, as instead of a market employing hundreds of people, it's only employing dozens. Ergo, more unemployed people. While this is good for the market's owners and those currently employed to trade there, it is bad for the economy as a whole.

You're on Hacker News, but you think that destroying jobs with technological innovation is a bad thing?

Not at all, but in re-reading my comment I can see why you'd think that. My intention was to make a devil's advocate comment: 2 sides to every coin, etc.

Re: How I made $500k with machine learning and high frequency trading

#307

Post by kevinrpope » Thu, Nov 08, 2012, 1:58 AM UTC

genwin wrote: post #304

kevinrpope wrote: post #300

Earlier quoted context omitted.

What did you end up doing with your code? Would you be able to run it today with the low-cost broker APIs?

The code sits in one of my archive folders. I ran it for a few years, perhaps to 2004, and saw the market steadily becoming more efficient, lowering my results (like the OP did). It may well be that it no longer predicts skillfully or profitably. As I recall, to beat the market the costs had to be very low, like pennies per trade, with no bid/ask spread, which I understood to be possible for large trading firms.

OK, cool. There are some places that offer equity trading for ~$0.005/share, but that says nothing about overcoming bid/ask. Looks like the OP did that by throwing a bit of market making into the mix.

Re: How I made $500k with machine learning and high frequency trading

#308

Post by lotu » Thu, Nov 08, 2012, 4:44 AM UTC

mikeash wrote: post #57

washedup wrote: post #44

Earlier quoted context omitted.

Well, in the article he said tat he did not care about direction, he would simple buy when his expected price was up, and sell when down. However, there could have easily been a bias in his model that "preferred" and performed better during upward movements. If so, he got lucky.

You don't need a bias to accidentally make money when the market is overall moving up, do you? Picking stocks by throwing darts while blindfolded will, on average, make you money in a market that's moving up.

The problem is he was also selling short. If you pick stocks randomly and randomly pick to buy or sell you shouldn't make money, ever.

Re: How I made $500k with machine learning and high frequency trading

#309

Post by quant123 » Thu, Nov 08, 2012, 6:42 AM UTC

jspaulding wrote: post #305

nashequilibrium wrote: post #239

Pls do not follow the advice of the OP. I started a hedgefund in 2004 doing HF platform arbitrage and ran it for 5yrs and i can honestly tell you that this is just survivorship bias. This is a very complex field and being off slightly, having a slight bias, a fraction of a point off your execution pricing and a slightly flawed money management system is recipe for disaster. The biggest issue is the confusion that you…

There are plenty of shops making tons of money with HFT who do not have deal flow at all - it's got nothing to do with luck. Survivorship bias would mean I simply got lucky. If you're going to say that you're at least going to need to look at my P&L charts and say how I could possibly achieve that much success with luck alone. Finally, machine learning has everything to do with my success. There were hundreds of vari…

agreed. jspaulding got it right although I can see nashequilibrium's skepticism as the US futures market is incredibly crowded (read: competitive, no free cookies).

Re: How I made $500k with machine learning and high frequency trading

#310

Post by dschiptsov » Thu, Nov 08, 2012, 10:32 AM UTC

curationary wrote: post #292

dschiptsov wrote: post #71

Basically this is a story about a guy who was smart enough to script up his trading tool (he discovered that there is an API and wrote some code to use it). He trade other people's money, using other people's (probably employer's) account and resources, I suppose. His employer have paid all the fees, and, took all the risks - if there is profit - it is mine, if there is a lose - it is theirs.) The essence of trading…

Basically this is a guy who does what many other have done, look no further than elitetrader.com, but smart enough to add "machine learning" to his story headline, which is the hottest buzz term on the street this year, everyone and his mom talks about machine learning last I checked.

So, it is just plain SEO for his startup - a primitive on-line course aggregator.