Why on earth there is supposed to be even a correlation between so-called past and future (bullshitting as a business aside)?
It is like predicting Trump by Nickson or, even better - predicting a next Trump.
Using a Keras Long Short-Term Memory Model to Predict Stock Prices
161–170 of 178 posts
Re: Using a Keras Long Short-Term Memory Model to Predict Stock Prices
#162This is pure nonsense. This isn’t even the right way to begin thinking about this as a forecasting task — the target series should be log-normal returns, not raw asset price. The performance of this model is laughably bad, which is probably why he spends zero time evaluating its effectiveness. You could trivially get better forecasts than this by naively repeating the last-observed price . This isn’t ML. It’s cargo-c…
http://scikit-learn.org/stable/modules/generated/sklearn.pre... Should normalize the data and raw prices won’t be used and instead the normalized coefficients are effectively a percentage
Re: Using a Keras Long Short-Term Memory Model to Predict Stock Prices
#163Earlier quoted context omitted.
Would it be the same (valid) with percentage returns?
Less so. Log-normal returns are better because they have the property that a summation of log-normal returns over contiguous intervals is equal to the log-normal returns of the combined interval. In other words: Losing 5% and then gaining 5% doesn’t put you back at exactly 100%, and log-normal fixes that.
Ah, that makes sense. Many thanks.
Re: Using a Keras Long Short-Term Memory Model to Predict Stock Prices
#164I cannot fault your enthusiam but I can fault your method :)
Read Tsay's Financial Time Series Analysis for a better idea how to forecast financial data.
The biggest problem you will face is stationarity: the statistical properties of the data is not constant over time. For example, the mean and std dev is not constant over time. Using returns instead of raw prices helps to make better financial forecasts.
Two methods to explore:
1. You are better off predicting stock prices by predicting future returns and then forecasting is the current price plus predicted future return.
2. You could use your neural model to predict absolute size of returns using realized volatility.
Re: Using a Keras Long Short-Term Memory Model to Predict Stock Prices
#165I'm not a purist believer in the efficient market hypothesis. BUT, I doubt there's much alpha to be gained simply from looking at price data which is widely and publicly available. Also keep in mind that markets are dynamic feedback loops so even if this model had an edge, the act of publishing this article would work against you to neutralize that edge in the future. There's a good reason the most successful 'quanti…
Rentech and the other more secretive top quant firms such as TGS and PDT use nearly exclusively public data. Funds like Two Sigma that haven't had as good (or scalable) returns are actually the ones that focus on novel data sources. In reality, everyone in this industry is drowning in data, and the real edge comes from learning how to more efficiently parse and analyze data rather than acquiring more of it. The top p…
Funny things I've heard about Two Sigma. Everyone on the outside: Two Sigma is the best. People at the top of the financial industry: Two Sigma is cheesy.
Re: Using a Keras Long Short-Term Memory Model to Predict Stock Prices
#166This is pure nonsense. This isn’t even the right way to begin thinking about this as a forecasting task — the target series should be log-normal returns, not raw asset price. The performance of this model is laughably bad, which is probably why he spends zero time evaluating its effectiveness. You could trivially get better forecasts than this by naively repeating the last-observed price . This isn’t ML. It’s cargo-c…
Even more so since test data can't be from the same time range: i.e. for time series you need to split train/test by date, not randomly, otherwise your model just memorizes the series.
It's standard practice to validate forecasts on non-randomized test/validation splits of the same time series, since this simulates the conditions where the model will be deployed in reality: It will know everything there is to know about the past, and it will know nothing about the future.
See Hyndman's fpp2 — https://otexts.org/fpp2/accuracy.html
Also, his description of rolling window validation: https://robjhyndman.com/hyndsight/rolling-forecasts/
Re: Using a Keras Long Short-Term Memory Model to Predict Stock Prices
#167Earlier quoted context omitted.
> that's all ML seems to be. You have to be very selective about what you consider "ML" to come to that conclusion. There has been a constant parade of incredible, mind-blowing results out of ML over the past decade, advancing the state of the art by leaps and bounds both in research and in real applications. Do you not remember how terrible speech recognition and speech synthesis were just a few short years ago? Did…
I concur totally, however think we should add in a caveat that is relevant for this audience: ...only the major consumer tech companies are actually reaping the benefit of applied ML in a way that is profitable. That's because they have the platforms and applications that people are using at scale. So ML is a force multiplier if you already have a consistent and strong user base for a good product. If you're trying t…
These days, startups founded by people who have worked in big tech companies and have witnessed the power of ML/AI will definitely apply it from day-1.
You don't have to invent or implement the algorithm yourself. You can use AI/ML intelligent/cognitive services provided by the big-3 cloud companies to reduce your starting cost significantly.
To have positive ROI, your problem complexity should have crossed the threshold where common-sense traditional solutions don't work any more.
If you do embark on ML research yourself, then be sure to walk the path from simpler models to complex ones while carefully establishing performance metrics.
Re: Using a Keras Long Short-Term Memory Model to Predict Stock Prices
#16811:15, restate my assumptions: 1. Mathematics is the language of nature. 2. Everything around us can be represented and understood through numbers. 3. If you graph these numbers, patterns emerge. Therefore: There are patterns everywhere in nature.
12:50 Press Return (For anyone who didn't get the reference, you have an excellent movie & coding soundtrack to catch up on: https://www.youtube.com/watch?v=ShdmErv5jvs )
I haven't thought about this in years. Love that sound track. Thanks
Re: Using a Keras Long Short-Term Memory Model to Predict Stock Prices
#169Earlier quoted context omitted.
Well, being an employee gives you exclusive access to the best fund in existence by far (35% return every year for decades). Nobody is going to want to kill their golden goose.
Extreme secrecy plus consistent very high returns sounds like the hallmarks of a ponzi scheme
[deleted]
Re: Using a Keras Long Short-Term Memory Model to Predict Stock Prices
#170Earlier quoted context omitted.
As someone outside the tech sphere on either coast, that's all ML seems to be. What I've seen from companies marketing to Higher Education is - we have a lot of data, you set arbitrary flags to the data that you believe indicate 'x' (or even better, they have pre-built data expectations) and you will get 'y' outcome. And none of it is actually based on anything real. It's all anecdotal applied to extreme amounts of a…
The problem isn’t that ML has gotten worse. It’s just as rigorous and far more powerful than it ever was. The problem is ML is hard, it hasn’t gotten orders-of-magnitude easier to understand, and there’s enormous incentive now to pass off amateur understanding as complete. The real ML still happens — it’s just drowned out.
I think people are saying that that's something they're not happy with. The big methods in AI, like backprop,
1) work a LOT better for specific problems than statistics or statistical learning ever has (and at this point, I think we can safely say: ever will)
2) a lot of methods either can't be explained, or outright shouldn't work, according to statistical theory.
The use of statistics in machine learning is limited to evaluating performance and individual element performance (and even that is tenuous at best in many cases). If you ask, say, why would an autoencoder, with an LSTM on it's compressed representation and Q-learning evaluation have somewhat decent performance on half the computer games humans ever designed ? Statistics will not be useful in formulating an answer.
If you ask extremely valid questions, like "why would an LSTM predict anything ?". Statistics draws a blank. There is no good reason to assume an LSTM will ever converge (and on a truly random dataset, it won't, whereas statistical methods will still allow you to say something).
I think there's 2 reasons for this
1) the "upper limit" of complexity a human can understand in a statistical model is lower than the upper limit a neural network can "understand". In statistics the human understanding is critical to getting to a valid model, in machine learning ... it is not. Meaning machine learning can learn relationships a human mind cannot.
2) There must be some fundamental property of the world we live in that matches neural network architecture. In order for backprop to work on real-world problems, it has to be the case that almost all real world phenomena are continuous, both "raw" and in the frequency domain. If this wasn't the case, machine learning would never be able to learn anything.