How I made $500k with machine learning and high frequency trading • Hacker News Board

Re: How I made $500k with machine learning and high frequency trading

#171

Post by lusr » Tue, Nov 06, 2012, 7:20 PM UTC

jspaulding wrote: post #163

tarr11 wrote: post #137

Earlier quoted context omitted.

Ie., if you made 500k in one period, but then lost 50k a year for the next 10 years, you've made a net profit of 0. All I know is that you had one good run, similar to how some mutual funds have a good run for a while. How much did you spend before you "tuned" it? How much did you spend afterwards? What were the tax consequences of your trades? Did you make exactly 500k? Have you traded at all since then? You mention…

In my final month I lost $600. I didn't include this on the chart to keep things simpler visually. Since then I have not traded and the reason is that it was abundantly clear that my program was no longer working. That's why I shut it off. With regard to tuning I may have lost $1000 or so but as I wrote in the article I was able to build a backtesting model that accurately simulated live trading. So once I had that I…

If you do release the source, what's the best way to be notified of this? Your Twitter account looks pretty active? I'm particularly interested in your risk management strategies (this is where my previous efforts fell short).

Re: How I made $500k with machine learning and high frequency trading

#172

Post by incision » Tue, Nov 06, 2012, 7:31 PM UTC

jcromartie wrote: post #6

jspaulding wrote: post #2

If anyone has questions for me happy to answer as best I can.

I have one question: Why doesn't every hacker do this to make extra money? Is it within the grasp of anybody who can program to automate trading like this? EDIT: Sounds like it's not really for everybody. You have to own or rent a server with access to direct lines to the exchanges, or else your lag will be such that profiting from HFT is impossible. How much do these cost?

>Why doesn't every hacker do this to make extra money?

I wondered the same thing and tried to answer it for myself not that long ago.

In short, it's hard, time-consuming, stressful and costly.

1) Your code has to work going forward. Coming up with something that works in backtesting is easy, doing it moving forward while staying within your risk envelope and considering all the associated costs is hard.

2) Closely related to #1. Whatever you put together is surely going to need plenty adaptation and oversight. It's a very fluid problem, you're just one player among countless others. How much time do you have in the day? Where is your capital coming from? For most people, the source of capital is a job that they probably can't ignore while fiddling with a trading program that would have to be remarkably successful to replace that income.

3) Everything costs money. You need a certain amount of capital to start with and there are all sorts of running costs. From the basic costs of execution, to market data and eventually co-location if you get that far. The fact that the OP kicked off with an amount that's essentially the barest minimum for any kind active trading is exceptional.

4) Can you maintain discipline and continue executing on all of the above while losing money? How many people struggle to simply get up on time, control their desire to check a particular website or to get over and move on from some frustration? Even if there's automation involved, trading will test your mental fortitude.

Re: How I made $500k with machine learning and high frequency trading

#173

Post by mcarvin » Tue, Nov 06, 2012, 7:32 PM UTC

fr0sty wrote: post #165

mcarvin wrote: post #161

Very important to understand that making $500k speculatively is not evidence of an 'edge', nor is trading frequency evidence of the absence of luck. From March 2009 through much of 2010, the market was strongly bullish - if his algorithm showed a positive market bias then his returns would primarily be a function of timing (read luck: and there are a million variants on the nature of the bias that could be unwittingl…

1. Author graphed his daily returns which should give you a handle on his volatility. He also stated his max drawdown was $2k. 2. Author has stated elsewhere than he began with $10k in seed capital. 3. The "bullish" market you cite increased only about %70 in that timeframe while the author's returns were multiples of that number. 4. Author walked away from a _previously successful_ strategy that no longer produced p…

1. I cannot get even a remote sense for the nature of his risk exposure from looking at his daily returns. 2. ok. 3. The point here is that a systematic bias in his algorithm will expose his trading strategy to the good graces of market fortune (luck) regardless of whether he trades a million, billion or once a day. The source of the bias is irrelevant. 4. did not see where he said that but that very much confirms 'timing' / which in this case I interpret as luck as being at least a contributing factor.

Re: How I made $500k with machine learning and high frequency trading

#174

Post by [deleted] » Tue, Nov 06, 2012, 7:36 PM UTC

tarr11 wrote: post #23

This is just glorified gambling. I am not sure what special insight or advantage he had, other than his own model. Every trader has a model. It could have easily been called "how i lost 500k with machine learning". Like gambling, it's easy to manipulate statistics to show that you did well in some period of time. I worked for a large investment bank about 10 years ago, writing trading programs for quant traders who w…

[deleted]

Re: How I made $500k with machine learning and high frequency trading

#175

Post by jspaulding » Tue, Nov 06, 2012, 7:45 PM UTC

runjake wrote: post #138

Relax with the vitriol. The guy is sharing an interesting personal story, not providing a step-by-step HOWTO or recommending people follow his suit. In fact, the article is really an ad for his startup Courseware.

CourseTalk. CourseTalk! http://coursetalk.org .. You are right kind of :) But I've made a decision to start reaching out generally so I can attract cool people to work with on whatever projects I may be interested in in the future.

Re: How I made $500k with machine learning and high frequency trading

#176

Post by gweinberg » Tue, Nov 06, 2012, 7:48 PM UTC

beagle3 wrote: post #48

tarr11 wrote: post #23

This is just glorified gambling. I am not sure what special insight or advantage he had, other than his own model. Every trader has a model. It could have easily been called "how i lost 500k with machine learning". Like gambling, it's easy to manipulate statistics to show that you did well in some period of time. I worked for a large investment bank about 10 years ago, writing trading programs for quant traders who w…

It is only gambling in the sense that any business is gambling: Your customers might stop coming tomorrow because the fad wore off, or a competitor provides a better/cheaper/hipper alternative. (And indeed, living is gambling. It's all just a matter of the risk/reward portfolie). But jspauld has apparently made $2/trade after fees on 250,000 trading, with a very small standard deviation (I would guess less than $2/tr…

No. Every business has a risk element, but what makes this gambling is that there is no good or service being produced. It's a game of trying to outguess the other players, with one trader's gain being another trader's loss (relative to market returns).

Because there's a commission on trades, and because you pay taxes on net gains but your minimum tax is zero, high frequency trading by its very nature must a loss for most players.

Re: How I made $500k with machine learning and high frequency trading

#177

Post by gabemart » Tue, Nov 06, 2012, 7:49 PM UTC

gabemart wrote: post #116

martinced wrote: post #75

Earlier quoted context omitted.

Exactly. I've played millions (!) of real-money online poker hands and won some money in the process (not anywhere near what most people who've played that many hands did). Most people don't understand that, when you're able to recognize patterns, playing millions of hands while never exposing more than 1% of your bankroll on any deal is not "gambling" but "printing money" (a tiny amount of money in my case compared…

I think the real problem (as with so many problems) is definitional. What does "gambling" actually mean? If I tried to kludge together a definition, I might come up with something like: >Risking the loss of something of value in exchange for the possibility of gaining something of greater value in a situation where the determining factor of losing value or gaining value is random chance The problem with a definition…

For the sake of clarity, I meant:

>If a lottery has a total of ten $1 tickets for sale and each ticket has an equal chance of winning, there is an obvious difference between the prize being $11 and $9, but buying a ticket at either price is just as much "gambling" in the common parlance.

Re: How I made $500k with machine learning and high frequency trading

#178

Post by aggronn » Tue, Nov 06, 2012, 7:51 PM UTC

bloaf wrote: post #118

danso wrote: post #53

I don't have much experience with finance or working experience with machine learning, but I've always wondered how much room there was for a clever amateur to profit in this space, even as it's crowded with much more sophisticated professionals with much more sophisticated algorithms and machines. I'm thinking back to Garry Kasparov's piece in the NY Book Review a couple years back: http://www.nybooks.com/articles/a…

Good point, I also wonder about the potential to exploit the algorithms used by the "professionals." In other words, if you can come up with a reasonable approximation of what the pros will do, can you use that information to beat them?

'Theoretically', no. Its hard to be optimistic about these two ideas because while the chess example is a good story, its not analogous for many reasons, ranging from disparity in available information to players to a difference of several magnitudes in saturation. Not to mention HFT just isn't chess.

HF traders are just as much hackers as anyone on HN (and there are plenty of HF traders on HN). So 'theoretically', they've already done what is being suggested here. If someone comes along and develops a winning strategy, it really shouldn't be considered as having anything to do with 'professional strategy vs novice strategies'. It would just be about one person either getting really lucky or coming up with something that is genius in its own right.

--

If there are 'professionals' and then other 'professionals' whose strategy depends on information about how other 'professionals' trade (and there is), you end up with strategies at all valid points in the sample space of possible strategies and counter strategies. Theoretically, there should be no other possible strategies. Inevitably someone will come up with one though, and the 'sample space' will grow. But its extremely unlikely that additional unique strategies are successful just because they 'counter' the strategies in the sample space. But then again, this is real life and these things aren't impossible.

Re: How I made $500k with machine learning and high frequency trading

#179

Post by jspaulding » Tue, Nov 06, 2012, 7:52 PM UTC

nrmn wrote: post #10

Could you comment on how your "curve fitting" algorithm worked? Did you end up with an equation for each curve? Im working on something that requires curve fitting and any kind of tip would be helpful.

I basically just brute forced it. I came up with a cost function which would measure the difference between a possible curve and each data point. I think you're supposed to do the squared difference but I can't remember if I did that. With a cost function in place it's just a matter of zooming in on variables that minimize the cost function.

Re: How I made $500k with machine learning and high frequency trading

#180

Post by jspaulding » Tue, Nov 06, 2012, 7:54 PM UTC

wtvanhest wrote: post #9

You said "Growth stopped" but your P&L shows negative growth approaching zero. Would it be more fair to say that your profitability turned to zero?

To be honest I'd given up and moved on to other stuff well before I actually shut down the program.