41–50 of 85 posts
I wish we could start moving to better approaches for evaluating time series forecasts. Ideally, the forecaster reports a probability distribution over time series, then we evaluate the predictive density with regard to an error function that is optimal for the intended application of the forecast at hand.
Earlier quoted context omitted.
Every researcher would love to include error bars but it's a matter of limited computing resources at universities. Unless you're training on a tiny dataset like MNIST, these training runs get expensive. Also, unless you parallelize from the start and risk wasting a lot of resources if something goes wrong, it could take longer to get the results.
Using bootstrap and/or repeated runs is a great way to get error bars but there are low cost ways to do it. For instance they estimate error bars on public opinion polls based on simple formulas and not redoing the poll a large number of times.
There’s definitely use for these classical, model-based methods, for sure. But a contrived comparison claiming they’re king is just misinformation.
Eg, here are a number of issues with classical techniques where dl succeeds (‘they’ here refers to classical techniques):
- they often don’t support missing/corrupt data
- they focus on linear relationships and not complex joint distributions
- they focus on fixed temporal dependence that must be diagnosed and specified a priori
- they take as input univariate, not multiple interval, data
- they focus on one-step forecasts, not long time horizons
- they’re highly parameterized and rigid to assumptions
- they fail for cold start problems
A more nuanced comparison would do well to mention these.
Ok, I fully agree with their foundational premise: Start simple.
But: They've overstated their case a bit. Saying that deep learning will cost $11,000 and need 14 days on this data set is not reasonable. I believe you can find some code that will cost that much. The readme suggests that this is typical of deep learning, which is not true. DL models have enormous variety. You can train a useful, high-performance model on a laptop CPU in a seconds-to-minutes timeframe; examples include multilayer perceptrons for simple classification, a smaller-scale CNN, or a collaborative filtering model.
While I don't endorse all details of their argument, I do think the culture of applied ML/data science has shifted too far toward default-DL. The truth is that many problems faced by real companies can be solved with simple techniques or pre-trained models.
Another perspective: A DL model is a spacecraft (expensive, sophisticated, powerful). Simple models like logistic regression are bikes and cars (affordable, efficient, less powerful). Using heuristics is like walking. Often your goal is just a few blocks away, in which case it would be inefficient to use a spacecraft.
Seems like these guys just wasted $11k to erroneously claim, “deep learning bad! Simple is better!” There’s definitely use for these classical, model-based methods, for sure. But a contrived comparison claiming they’re king is just misinformation. Eg, here are a number of issues with classical techniques where dl succeeds (‘they’ here refers to classical techniques): - they often don’t support missing/corrupt data -…
You gotta be kidding right, that's one thing that they do well.
This readme lands to me like this: "People say deep learning killed stats, but that's not true; in fact, DL can be a huge mistake." Ok, I fully agree with their foundational premise: Start simple. But: They've overstated their case a bit. Saying that deep learning will cost $11,000 and need 14 days on this data set is not reasonable. I believe you can find some code that will cost that much. The readme suggests that…
After glancing at the paper they're criticising, I really wonder how they arrived at these insane figures. From what I saw, they were mostly using stuff like MLPs with a handful layers at O(100) neurons at most. Yeah, if you put a hundred million parameter transformer in there you will train forever (and waste tons of compute since that would be complete overkill), but not with simple perceptrons. I don't know the extent of the data, but given these architectures I very much doubt a practical model would take this long to train - even on a CPU - given that you could run a statistical ensemble in 5 minutes.
Earlier quoted context omitted.
The tests sets are large enough to render this moot, as the confidence intervals are almost certainly smaller than the precisions typically reported, i. e. 0.1 %.
I've worked on commercial systems where N https://ir-datasets.com/gov2.html and sometimes it as bad as N=50 queries with judgements. I don't see papers that are part of TREC or based on TREC data dealing with sampling errors in any systematic way.
- systematically addressing sampling error;
- more than 50 queries;
- more/all QRELs;
- full evaluation instead of system pooling;
- study IR not just of the English language (this has been picked up by CLEF and NTCIR in Europe and Japan, respectively)
- to devise metrics that take energy efficiency into account.
- ...
At the same time, we have to be very grateful to NIST/TREC for executing an international (open) benchmark annually, which has moved the field forward a lot in the last 25 years.
Earlier quoted context omitted.
This is one of my default suggestions when I act as reviewer: t test with bonferroni correction please. ML, ironically, has absolutely horrible practices in terms of distinguishing signal from noise( which at least is partially offset by the social pressure to share code, but still)
Bonferroni's correction on hold-out data is an excellent suggestion. To adapt it into time series forecasting, one could perform temporal cross-validation with rolling windows and follow the performance's variance through time. Unfortunately, the computational time would explode if the ML method's optimization is performed naively. Precise measurements of the statistical significance would crowd out researchers excep…
Holm-Šidák would be better--but something like false discovery rate might be easier to interpret.
It is something that bothers me about the ML literature is that they frequently present a large number of evaluation results such as precision and AUC but these are not qualified by error bars. Typically they make a table which has different algorithms on one side and different problems on the other side and the highest score for a given problem gets bolded. I know if you did the experiment over and over against with…
This is one of my default suggestions when I act as reviewer: t test with bonferroni correction please. ML, ironically, has absolutely horrible practices in terms of distinguishing signal from noise( which at least is partially offset by the social pressure to share code, but still)