I'll invoke the Black Swan (https://en.wikipedia.org/wiki/The_Black_Swan_%28Taleb_book%2...) since it hasn't been done yet in this thread.
Machine learning for financial prediction
71–80 of 84 posts
Re: Machine learning for financial prediction
#72It's just so ridiculously easy to overfit these models, and so so many ways to shoot yourself in the foot as a result. For example, "I split the data set into 5 random segments and then trained a model on 4 of the 5 segments and then tested it on 5th." Such data is serially correlated (it's not good old iid) so already it looks like you have poisoned the test set with information from the training set. The hard part…
> Such data is serially correlated (it's not good old iid) so already it looks like you have poisoned the test set with information from the training set.
Kind of. If it was that simple making money off of an autoregressive model would be trivial -> everyone would do it -> serial correlation would disappear.
I agree with your observation that figuring out what to feed the beast is one of the bigger challenges though. Case and point: train a mean reversion model on the last seven years of S&P data to buy dips and train a momentum model to buy higher highs. That equity curve would look very encouraging. Do it on a fifteen year basis, and not so much. Now the question becomes: how long of a lookback do you use when training your models? Chopping up data at random will mux out useful correlations. Subsetting into periods leads to poorly generalized models. Not fun.
Re: Machine learning for financial prediction
#73It's just so ridiculously easy to overfit these models, and so so many ways to shoot yourself in the foot as a result. For example, "I split the data set into 5 random segments and then trained a model on 4 of the 5 segments and then tested it on 5th." Such data is serially correlated (it's not good old iid) so already it looks like you have poisoned the test set with information from the training set. The hard part…
It's so easy to do machine learning and think you're a genius when you are in fact overfitting. It's almost like casino gambling. You tweak some hyperparameter, pull the slot machine lever, and wham, your model says you should be rich real soon...
Re: Machine learning for financial prediction
#74Earlier quoted context omitted.
There are trading firms that have made fortunes letting the algorithms do the trading: http://www.fool.com/investing/general/2014/02/09/this-man-ma... https://en.m.wikipedia.org/wiki/Renaissance_Technologies Yes, you have to adapt but these guys play with billions.
Yes, everyone's heard of RenTec, particularly those of us in the industry. Their existence doesn't seem seem germane to my point. You were claiming that a little guy's edge won't be rapidly eroded by the market. This is only true if his trading strategy is uncorrelated to every other little guy's trading strategy. But this is unlikely. The space of unsophisticated strategies is not all that large.
So what are those hundreds of people doing at RenTec if "the space of unsophisticated strategies is not all that large"? Do you think their $100 million algorithmic trades go unnoticed? A $10,000 trade is going to make someone take notice of a little guy?
At any rate, is there some reason that you are driving the discussion to "it's not possible". No one learns anything by saying "it's impossible". Perhaps we should have ignored the post.
Re: Machine learning for financial prediction
#75Earlier quoted context omitted.
Who do I contact? I have a deep learning startup that is trading forex right now, I would like to make some contacts and see if I can integrate
Why FX? More direct access to exchanges?
it's the most liquid market on earth, it's 24 hour for the majors, its depth is enormous, and it is less prone to event risk than individual equities (as long as you stick to unpegged currencies) so you can get tons of leverage on it.
Re: Machine learning for financial prediction
#76Earlier quoted context omitted.
Exactly. Plus let's remember you are adding nothing to life here. All you are doing is collecting 30k off other people with a slightly less optimal "strategy" than you. And no I don't believe these people are "adding liquidity and assisting price discovery". to the reply as I can't post as HN censors detractors of big finance: I don't believe the benefits of liquidity added by HFT are worth the enormous costs firms s…
How about just making a transaction because you feel that you gain something from it? Does it need any other motivation? I mean, I buy food, go to a concert, rent a car, and buy a stock, not for the purpose of gifting something to society, but for my own sake.
This is the opposite - you don't buy and then resell food in a few seconds for gain.
Re: Machine learning for financial prediction
#77Earlier quoted context omitted.
Yes, everyone's heard of RenTec, particularly those of us in the industry. Their existence doesn't seem seem germane to my point. You were claiming that a little guy's edge won't be rapidly eroded by the market. This is only true if his trading strategy is uncorrelated to every other little guy's trading strategy. But this is unlikely. The space of unsophisticated strategies is not all that large.
So what are those hundreds of people doing at RenTec if "the space of unsophisticated strategies is not all that large"? Do you think their $100 million algorithmic trades go unnoticed? A $10,000 trade is going to make someone take notice of a little guy? At any rate, is there some reason that you are driving the discussion to "it's not possible". No one learns anything by saying "it's impossible". Perhaps we should…
They're developing sophisticated strategies. Duh.
Re: Machine learning for financial prediction
#78Anyone know where he got all the raw data to feed his algo? Clearly he used a lot of data and the two main sources of free info i know of are google finance and yahoo finance. At least with google finance i run into issues with their api if you execute too many calls simultaneously, a bunch end up not returning any data
Agreed with Quandl being a good source for financial data. Their APIs are also quite well-maintained. Within Quandl, I've found Zack's to be a good resource. I work with both Quandl and Zack's pretty frequently, let me know if you're interested in buying large amounts of data from Zack's, and I can perhaps get you a discount from the listed prices on the Quandl website.
If I was interested in such a discount, how would I contact you? (Also, how much is a "large amount"?)
Re: Machine learning for financial prediction
#79Earlier quoted context omitted.
Exactly. Plus let's remember you are adding nothing to life here. All you are doing is collecting 30k off other people with a slightly less optimal "strategy" than you. And no I don't believe these people are "adding liquidity and assisting price discovery". to the reply as I can't post as HN censors detractors of big finance: I don't believe the benefits of liquidity added by HFT are worth the enormous costs firms s…
>> And no I don't believe these people are "adding liquidity and assisting price discovery". The nice thing about reality is that it remains even when your belief persists against it. In large-cap stocks during rising markets, high frequency trading does improve liquidity[1]. While the effect may not be as prevalent during a downturn and it may not impact smaller stocks as much, I'd like to see your evidence that it…
Generating a profit does not equate to generating wealth.
Re: Machine learning for financial prediction
#80Earlier quoted context omitted.
So what are those hundreds of people doing at RenTec if "the space of unsophisticated strategies is not all that large"? Do you think their $100 million algorithmic trades go unnoticed? A $10,000 trade is going to make someone take notice of a little guy? At any rate, is there some reason that you are driving the discussion to "it's not possible". No one learns anything by saying "it's impossible". Perhaps we should…
They're developing sophisticated strategies. Duh.
ok, what's the difference between sophisticated and unsophisticated algorithmic trading?