Statistical vs. Deep Learning forecasting methods
1–10 of 85 posts
Re: Statistical vs. Deep Learning forecasting methods
#2Comparison of several Deep Learning models and ensembles to classical statistical models for the 3,003 series of the M3 competition.
Re: Statistical vs. Deep Learning forecasting methods
#3It is something that bothers me about the ML literature is that they frequently present a large number of evaluation results such as precision and AUC but these are not qualified by error bars. Typically they make a table which has different algorithms on one side and different problems on the other side and the highest score for a given problem gets bolded.
I know if you did the experiment over and over against with different splits you'd get slightly different scores so I'd like to see some guidance as to significance in terms of ① statistical significance, and ② is it significant on a business level. Would customers notice the difference? Would it make better decisions that move the needle for revenue or other business metrics?
This study is an example where a drastically more expensive algorithm seems to produce a practically insignificant improvement.
Re: Statistical vs. Deep Learning forecasting methods
#4What is the point of this kind of comparison? It is completely dependent on the 3000 datasets they chose to use. You're not going to find that one method is better than another in-general or find some type of time series for which you can make a specific methodological recommendation (unless that series is specifically constructed with a mathematical feature, like stationarity).
What matters is "which method is better for MY data?" but that's not something an academic can study. You just have a test a few different things.
Re: Statistical vs. Deep Learning forecasting methods
#5It is something that bothers me about the ML literature is that they frequently present a large number of evaluation results such as precision and AUC but these are not qualified by error bars. Typically they make a table which has different algorithms on one side and different problems on the other side and the highest score for a given problem gets bolded. I know if you did the experiment over and over against with…
This is one of my default suggestions when I act as reviewer: t test with bonferroni correction please. ML, ironically, has absolutely horrible practices in terms of distinguishing signal from noise( which at least is partially offset by the social pressure to share code, but still)
Re: Statistical vs. Deep Learning forecasting methods
#6What is the point of this kind of comparison? It is completely dependent on the 3000 datasets they chose to use. You're not going to find that one method is better than another in-general or find some type of time series for which you can make a specific methodological recommendation (unless that series is specifically constructed with a mathematical feature, like stationarity). What matters is "which method is bette…
My thoughts exactly. Unless the method can be shown to be inferior in certain or all dimensions, it is a meaningless comparison.
Re: Statistical vs. Deep Learning forecasting methods
#7I wish we could start moving to better approaches for evaluating time series forecasts. Ideally, the forecaster reports a probability distribution over time series, then we evaluate the predictive density with regard to an error function that is optimal for the intended application of the forecast at hand.
Re: Statistical vs. Deep Learning forecasting methods
#8It is something that bothers me about the ML literature is that they frequently present a large number of evaluation results such as precision and AUC but these are not qualified by error bars. Typically they make a table which has different algorithms on one side and different problems on the other side and the highest score for a given problem gets bolded. I know if you did the experiment over and over against with…
This is one of my default suggestions when I act as reviewer: t test with bonferroni correction please. ML, ironically, has absolutely horrible practices in terms of distinguishing signal from noise( which at least is partially offset by the social pressure to share code, but still)
Question: why do we care about the Bonferroni correction if the model being reviewed shows high performance on holdout/test samples?
I mean, it's nice to know that the p-values of coefficients on models you are submitting for publication are appropriately reported under the conservative approach Bonferroni applies, but I would think making it a _default_ is an inappropriate forcing function when the performance on holdout is more appropriate. Data leakage would be a much, much larger concern IMHO. Variance of the performance metrics is also important.
What am I missing?
Re: Statistical vs. Deep Learning forecasting methods
#9Timeseries data can sometimes be deceptive, depending on what you are trying to model.
I have been hacking on a peroneal research project to predict hurricane tracks furcating using deep learning. Only given track and intensity data at different points in time (every 6 hours) and some simple feature engineering, you will not get any good results close to the official NHC forecast, no matter what model you use.
In hindsight, this is a little obvious. Hurricane forecasting time series models depend more on other factors than time itself. A sales forecast can depend on seasonal trends and key events in time, but a hurricane forecast is much more dependent on long-range spatial data like the state atmosphere and ocean that are very non-trivial to model simply using just track data.
However, deep leading models and techniques in this scenario are helpful because they can allow you to integrate multiple modalities like images, graphs, and volumetric data into this one model, which may not be possible with statistical models alone.
Re: Statistical vs. Deep Learning forecasting methods
#10It is something that bothers me about the ML literature is that they frequently present a large number of evaluation results such as precision and AUC but these are not qualified by error bars. Typically they make a table which has different algorithms on one side and different problems on the other side and the highest score for a given problem gets bolded. I know if you did the experiment over and over against with…
This is one of my default suggestions when I act as reviewer: t test with bonferroni correction please. ML, ironically, has absolutely horrible practices in terms of distinguishing signal from noise( which at least is partially offset by the social pressure to share code, but still)