Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Time Series Prediction Using LSTM Deep Neural Networks

altumintelligence.com

Post reply on HN
Print view View original

61–70 of 72 posts

Re: Time Series Prediction Using LSTM Deep Neural Networks

#61

Post by GChevalier »

I see here that original poster (OP) of the post tried to use many-to-one LSTMs instead of many-to-many LSTMs. I tell that first by looking at the charts. Then I saw the method named "predict_point_by_point" with the comment "Predict each timestep given the last sequence of true data, in effect only predicting 1 step ahead each time" in his code here: https://github.com/jaungiers/LSTM-Neural-Network-for-Time-Se...

I strongly think the system would be better to perform many predictions at once instead, using seq2seq neural networks. The problem is properly explained here at the beginning of this other post: https://github.com/LukeTonin/keras-seq-2-seq-signal-predicti... This other post is, in turn, derived from my original project here doing seq2seq predictions with TensorFlow: https://github.com/guillaume-chevalier/seq2seq-signal-predic...

OP also forgot to cite the image I made: https://en.wikipedia.org/wiki/Long_short-term_memory#/media/...

Well, glad to see that some similar work as mine can get this much traction on HN. I would have loved to get this much traction when I did my post, too. Anyway, I would suggest OP to take a look at seq2seq, as it objectively performs better (and without the "laggy drift" visual effect observed as in OP's figure named "S&P500 multi-sequence prediction").

In other words, using many-to-one neural architectures creates some kind of feedback which doesn't happen with seq2seq which doesn't build on its own accumulated error. It has a decoder with different weights than the encoder, and can be deep (stacked).

Re: Time Series Prediction Using LSTM Deep Neural Networks

#62

Post by RA_Fisher »

mlthoughts2018 wrote: post #56
curiousgal wrote: post #46

Earlier quoted context omitted.

Because saying "relevant username" is frowned upon I'll just point out that R A Fisher is "a genius who almost single-handedly created the foundations for modern statistical science"[0] 0. https://en.m.wikipedia.org/wiki/Ronald_Fisher

It’s funny to me, as a professional statistician, because most methods popularized by Fischer et al in the early 1900s are wildly inappropriate for practical problems, especially policy decision science or causal inference. All the theory behind t-testing, Wald testing, using the detivatives of the log likelihood near to the MLE point estimate in order to also estimate standard errors when no analytical solution exis…

I don't disagree with anything you've written. The only thing I'd take issue with is placing NHST at the feet of statisticians. Scientists deserve a fair share as well. :-p

Re: Time Series Prediction Using LSTM Deep Neural Networks

#63

Post by zwaps »

jkabrg wrote: post #31
zwaps wrote: post #28

I find it interesting that Computer Scientists are basically rediscovering statistics. Now when predicting time series, an issue is that most model (like ARIMA, GARCH etc.) are short-memory processes. When you look at the full-series prediction of LSTMs, you observe the same thing. So in terms of Time Series, Machine Learning is currently in the mid to late 80's compared to Financial Econometrics. So if you are a CS,…

Nassim Taleb had some negative things to say about GARCH. "GARCH does not work out of sample. It is a good story, but I was unable to use it in predicting squared deviations or mean deviations" I haven't found it in Rob J Hyndman's forecasting tutorial either. How does it fare in the Makridakis competitions?

GARCH, like I said, is a short memory process and is inherently inadequate for (longer) out of sample predictions. Doing this is possible, but not really correct. Taleb is basically right, of course what he says is probably inflammatory and half wrong, as usual.

Don't forget that most econometrics models are also concerned with identification and causality, less with prediction.

Re: Time Series Prediction Using LSTM Deep Neural Networks

#64

Post by zwaps »

md2be wrote: post #48
zwaps wrote: post #28

I find it interesting that Computer Scientists are basically rediscovering statistics. Now when predicting time series, an issue is that most model (like ARIMA, GARCH etc.) are short-memory processes. When you look at the full-series prediction of LSTMs, you observe the same thing. So in terms of Time Series, Machine Learning is currently in the mid to late 80's compared to Financial Econometrics. So if you are a CS,…

I agree with your sentiments, but there is a contribution that the CS departments made that the statistics, math, Econ (as in econometrics departments) seemed to have overlooked. I remember going to each of these departments in 2002 and asking them why don’t we split the data sets to train and update the coefficients and automate the process. The answer was always the same “that’s trivial and adds nothing to the fiel…

I think that ML is very useful, but remember that forecasting is really not the main objective of econometric models.

Basically, forecasting implies you have a good handle on all properties of the relevant distributions, which in my opinion is a lost cause in social sciences (think external validity).

Instead, econometrics is nowadays mainly concerned with the identification of causal effect using non-parametric or semi-parametric approaches. Basically, you can believably estimate the directionality of some mechanism, but you probably never have the data or model to make a good out of sample prediction. You can, but it's basically implied that approaches that consistently estimate some marginal of a conditional expectation will NOT be that useful to predict a whole stochastic process.

Also, using training and test sets kind of predicates that your process is very stable. Otherwise the "test" set is not really a good test, is it? Again, in social sciences these things are hard to argue. You usually wanna generalize some mechanism from this industry to that industry, not find a good predictor in the same industry. Test datasets still run on the same data!

ML is successful because in practice we DO care about prediction. This allows us to do all the cool things. Because econometrics/stats is so conservative and comes from a causal standpoint, people are just really shy to develop a model for prediction (not everywhere true, but that's the gist). For ML, the primary question is basically how good the thing predicts. When I first tried scikit learn way back, I was so confused it didn't offer standard errors or some other statistical measure. But then I saw how ingrained the in-sample, out-sample process is and I thought well - that's really useful.

tl;dr: Stats and ML have different objectives, but there is a lot to learn in stats for ML

Re: Time Series Prediction Using LSTM Deep Neural Networks

#65

Post by srean »

salty_biscuits wrote: post #60
srean wrote: post #55

Earlier quoted context omitted.

But what you describe is still curve fitting. I say this in spite of some expertise in ML myself. There are some parts of ML that are not fall in the curve fitting family but they are still a small part, for example Markov logic network, some parts of reinforcement learning. What you are saying is curve fitting with good predictive ability is not trivial, and that is indeed true.

Markov Logic Networks are still about finding coefficients for a probability distribution over some process. My opinion is that there is only curve fitting. There is data and a minimum complexity model that can reproduce the data with minimum error. So do you really believe that there are physical processes where this approach will fail?

There is more to Markov Logic than estimating parameters. That's the "unification" part of the Markov logic, the analogue of https://en.wikipedia.org/wiki/Unification_(computer_science)

Re: Time Series Prediction Using LSTM Deep Neural Networks

#66

Post by [deleted] »

mlthoughts2018 wrote: post #56
curiousgal wrote: post #46

Earlier quoted context omitted.

Because saying "relevant username" is frowned upon I'll just point out that R A Fisher is "a genius who almost single-handedly created the foundations for modern statistical science"[0] 0. https://en.m.wikipedia.org/wiki/Ronald_Fisher

It’s funny to me, as a professional statistician, because most methods popularized by Fischer et al in the early 1900s are wildly inappropriate for practical problems, especially policy decision science or causal inference. All the theory behind t-testing, Wald testing, using the detivatives of the log likelihood near to the MLE point estimate in order to also estimate standard errors when no analytical solution exis…

[deleted]

Re: Time Series Prediction Using LSTM Deep Neural Networks

#67

Post by RA_Fisher »

srean wrote: post #53
curiousgal wrote: post #46

Earlier quoted context omitted.

Because saying "relevant username" is frowned upon I'll just point out that R A Fisher is "a genius who almost single-handedly created the foundations for modern statistical science"[0] 0. https://en.m.wikipedia.org/wiki/Ronald_Fisher

... and someone who would be very difficult to "out asshole" or to out do in male chauvinism. Those are criticism on personality, on the technical side it took the community a long time to undo the damage of promoting non robust parametric statistics. But this much is certain he pulled statistics into the realms of math -- no mean feat.

That's true and it's well documented. I think E.T. Jaynes gives a poigntent reflection in his Theory of Probability.

Re: Time Series Prediction Using LSTM Deep Neural Networks

#68

Post by luanton »

GChevalier wrote: post #61

I see here that original poster (OP) of the post tried to use many-to-one LSTMs instead of many-to-many LSTMs. I tell that first by looking at the charts. Then I saw the method named "predict_point_by_point" with the comment "Predict each timestep given the last sequence of true data, in effect only predicting 1 step ahead each time" in his code here: https://github.com/jaungiers/LSTM-Neural-Network-for-Time-Se... I…

https://news.ycombinator.com/item?id=17902967

The aim of this post is to explain why sequence to sequence models appear to perform better than "many to one" RNNs on signal prediction problems. It also describes an implementation of a sequence 2 sequence model using the Keras API.

Re: Time Series Prediction Using LSTM Deep Neural Networks

#69

Post by sidr »

srean wrote: post #54
laichzeit0 wrote: post #43

Earlier quoted context omitted.

I don't think it's entirely fair to say "Computer Scientists are basically rediscovering statistics". LSTMs are used beyond just time series prediction. It is also quite common in language modelling tasks, which is also a sequence modelling task, and where it works quite well. I'm not familiar at all with using GARCH/ARIMA for something like this. Also, with neural networks it's very easy and natural to build complex…

> What are the statistical equivalents for this? That would be nonparametric statistics.

> That would be nonparametric statistics.

No, it wouldn't. Firstly, nonparametrics in general can be a little misleading. The most common instantiations place function ("process") priors on modeling decisions that are otherwise found through trial and error. Those process priors do have their own parameters though. But more importantly, LSTMs and neural networks are very much parametric - their success come from the advances in computing and optimization that have enabled estimating these parameters in very complicated model structures.

Re: Time Series Prediction Using LSTM Deep Neural Networks

#70

Post by srean »

sidr wrote: post #69
srean wrote: post #54

Earlier quoted context omitted.

> What are the statistical equivalents for this? That would be nonparametric statistics.

> That would be nonparametric statistics. No, it wouldn't. Firstly, nonparametrics in general can be a little misleading. The most common instantiations place function ("process") priors on modeling decisions that are otherwise found through trial and error. Those process priors do have their own parameters though. But more importantly, LSTMs and neural networks are very much parametric - their success come from the…

Why are you talking about priors ? Nonparametric vs parametric is an axis completely orthogonal to Bayesian vs Frequentist.

We weren't talking about the "success" though, I was responding to the question "where in the body of stats literature would a neural net model lie".

I argue that would be non-parametric stats. In parametric stats the limit (#params/#data) goes to 0. For models where this is not the case, statisticians and probabilists call them non-parametric (and in certain cases semi-parametric models). Neural net, especially the deep kind (and certainly not the single layer kind) have the property that #params/#data is finite and large.

Post reply on HN