why are middle-ground (but SOTA) techniques like guassian processes and GBM regression not in this comparo
Statistical vs. Deep Learning forecasting methods
11–20 of 85 posts
Re: Statistical vs. Deep Learning forecasting methods
#12What is the point of this kind of comparison? It is completely dependent on the 3000 datasets they chose to use. You're not going to find that one method is better than another in-general or find some type of time series for which you can make a specific methodological recommendation (unless that series is specifically constructed with a mathematical feature, like stationarity). What matters is "which method is bette…
so your corollary to the No Free Lunch theorem is "Lunch Is Impossible"?
Re: Statistical vs. Deep Learning forecasting methods
#13It is something that bothers me about the ML literature is that they frequently present a large number of evaluation results such as precision and AUC but these are not qualified by error bars. Typically they make a table which has different algorithms on one side and different problems on the other side and the highest score for a given problem gets bolded. I know if you did the experiment over and over against with…
The tests sets are large enough to render this moot, as the confidence intervals are almost certainly smaller than the precisions typically reported, i. e. 0.1 %.
Re: Statistical vs. Deep Learning forecasting methods
#14It is something that bothers me about the ML literature is that they frequently present a large number of evaluation results such as precision and AUC but these are not qualified by error bars. Typically they make a table which has different algorithms on one side and different problems on the other side and the highest score for a given problem gets bolded. I know if you did the experiment over and over against with…
Thanks for the comment!
In Machine Learning literature, the variance of accuracy measurements originates from different network parameters initialization. Since the deep learning ensembles already use aggregate computation in the hundreds of days, computing the variance would elevate the computational time into thousands of days.
In contrast, statistical methods that we report optimize convex objectives; their optimal parameters are deterministic.
That being said, we like the idea of including cross-validation with different splits for future experiments.
Re: Statistical vs. Deep Learning forecasting methods
#15Earlier quoted context omitted.
This is one of my default suggestions when I act as reviewer: t test with bonferroni correction please. ML, ironically, has absolutely horrible practices in terms of distinguishing signal from noise( which at least is partially offset by the social pressure to share code, but still)
Question: why do we care about the Bonferroni correction if the model being reviewed shows high performance on holdout/test samples? I mean, it's nice to know that the p-values of coefficients on models you are submitting for publication are appropriately reported under the conservative approach Bonferroni applies, but I would think making it a _default_ is an inappropriate forcing function when the performance on ho…
Because the variance can be uniformly high, making it difficult to properly judge the improvement of one method vs the baseline method: did you actually improve, or did you just get a few lucky seeds? It's much harder to get a paper debunking new "SotA" methods so I default to showing a clear improvement over a good baseline. Simply looking at the performance is also not enough because a task can look impressive, but be actually quite simple (and vice versa), so using these statistical measures makes it easy to distinguish good models on hard tasks from bad models on easy tasks.
I should also note 1) this is about testing whether the performance of a model is meaningfully different from another, not the coefficient of the models 2) I don't reject papers just because they lack this, or if they fail to achieve a statistical significance, I just want it in the paper so the reader can use that to judge (and it also helps suss out cherry picked results)
Re: Statistical vs. Deep Learning forecasting methods
#16Earlier quoted context omitted.
This is one of my default suggestions when I act as reviewer: t test with bonferroni correction please. ML, ironically, has absolutely horrible practices in terms of distinguishing signal from noise( which at least is partially offset by the social pressure to share code, but still)
Question: why do we care about the Bonferroni correction if the model being reviewed shows high performance on holdout/test samples? I mean, it's nice to know that the p-values of coefficients on models you are submitting for publication are appropriately reported under the conservative approach Bonferroni applies, but I would think making it a _default_ is an inappropriate forcing function when the performance on ho…
You'd want to do some sort of test because it can help assess whether your method did better than the alternatives by chance. For example can you really say Method A is better than B if A got 88% accuracy on the holdout set and B got 86% accuracy? Would that be true of all possible datasets?
t-test with Bonferroni isn't necessarily the best test for all metrics either.
Re: Statistical vs. Deep Learning forecasting methods
#17It is something that bothers me about the ML literature is that they frequently present a large number of evaluation results such as precision and AUC but these are not qualified by error bars. Typically they make a table which has different algorithms on one side and different problems on the other side and the highest score for a given problem gets bolded. I know if you did the experiment over and over against with…
This is one of my default suggestions when I act as reviewer: t test with bonferroni correction please. ML, ironically, has absolutely horrible practices in terms of distinguishing signal from noise( which at least is partially offset by the social pressure to share code, but still)
Bonferroni's correction on hold-out data is an excellent suggestion. To adapt it into time series forecasting, one could perform temporal cross-validation with rolling windows and follow the performance's variance through time.
Unfortunately, the computational time would explode if the ML method's optimization is performed naively. Precise measurements of the statistical significance would crowd out researchers except for Big Tech.
Re: Statistical vs. Deep Learning forecasting methods
#18It is something that bothers me about the ML literature is that they frequently present a large number of evaluation results such as precision and AUC but these are not qualified by error bars. Typically they make a table which has different algorithms on one side and different problems on the other side and the highest score for a given problem gets bolded. I know if you did the experiment over and over against with…
The tests sets are large enough to render this moot, as the confidence intervals are almost certainly smaller than the precisions typically reported, i. e. 0.1 %.
I've worked on commercial systems where Nhttps://ir-datasets.com/gov2.html
and sometimes it as bad as N=50 queries with judgements. I don't see papers that are part of TREC or based on TREC data dealing with sampling errors in any systematic way.
Re: Statistical vs. Deep Learning forecasting methods
#19why are middle-ground (but SOTA) techniques like guassian processes and GBM regression not in this comparo
A lot of M3 datasets we use are high-frequency, with large seasonal inputs. Considering Gaussian Processes (GP) complexity is O(N^3), a careful study of their performance would be challenging.
Also... I'm not aware of any efficient GP Python implementations.
Re: Statistical vs. Deep Learning forecasting methods
#20why are middle-ground (but SOTA) techniques like guassian processes and GBM regression not in this comparo
A lot of M3 datasets we use are high-frequency, with large seasonal inputs. Considering Gaussian Processes (GP) complexity is O(N^3), a careful study of their performance would be challenging. Also... I'm not aware of any efficient GP Python implementations.
Just write your GP model in Pyro or something like that.