Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistical vs. Deep Learning forecasting methods

github.com

Post reply on HN
Print view View original

31–40 of 85 posts

Re: Statistical vs. Deep Learning forecasting methods

#31

Post by fedegr »

nerdponx wrote: post #24
kgarten wrote: post #22

Nice article and interesting comparison. Yet, I have a minor issue with the title: Deep Learning are also statistical methods ... "univariate models vs. " would be a better title.

You could argue that deep learning is not a statistical method in the traditional sense, in that a typical neural network model is not a probability model, and some neural networks are well known to produce specifically bad probability models, requiring some amount of post processing in order to produce correctly "calibrated" probability predictions. However I don't like that there is often a strict dichotomy present…

Co-author here: all in due time. Next iteration we will include LigthGBM, XGBoost, and newer DL models like TFT and NHiTS.

Re: Statistical vs. Deep Learning forecasting methods

#32

Post by brrrrrm »

zmachinaz wrote: post #30
brrrrrm wrote: post #29

I'm heavily involved in this area of research (getting deep learning competitive with computationally efficient statistical methods), and I'd like to note a couple things I've found: 1. Deep learning doesn't require thorough understanding of priors or statistical techniques. This opens the door to more programmers in the same way high level languages empower far more people than pure assembly. The tradeoffs are analo…

Regarding 1) I am not sure if you are not trading "high human efficiency" against increased risk of blowing up at some point. Good luck doing forecasting without thorough understanding of priors and statistics in general.

that's a good point. I guess as an addendum it's not just compute efficiency but also "statistical efficiency" (if that has any meaning?)

Re: Statistical vs. Deep Learning forecasting methods

#33

Post by uoaei »

nerdponx wrote: post #24
kgarten wrote: post #22

Nice article and interesting comparison. Yet, I have a minor issue with the title: Deep Learning are also statistical methods ... "univariate models vs. " would be a better title.

You could argue that deep learning is not a statistical method in the traditional sense, in that a typical neural network model is not a probability model, and some neural networks are well known to produce specifically bad probability models, requiring some amount of post processing in order to produce correctly "calibrated" probability predictions. However I don't like that there is often a strict dichotomy present…

Statistical models and probabilistic models are not synonymous.

Vanilla deep learning models are statistical models (a la linear regression) and not probabilistic models (a la Gaussian mixture). It is important to maintain the distinction.

But to your point about the dichotomy between deep learning and more "traditional" statistical methods: this confusion in common parlance clearly has negative effects on model-building among engineers. You are right that when people think "deep learning" they think of very specific architectures with very specific features, and don't seem to conceive of the possibility that automatic differentiation techniques mean you can incorporate all sorts of new model components that blur the line between deep learning and older methods. For instance, you could feed the results of a kernel SVM to an ARIMA model in such a way that the whole thing is end-to-end differentiable. In fact, the great benefit of deep learning long-term is (in my opinion) that the ability to build these compositional models means you can bake in that much more inductive bias into the models you build, meaning they can be smaller and more stable in training.

Re: Statistical vs. Deep Learning forecasting methods

#34

Post by uoaei »

graycat wrote: post #23

I can have some interest in, hope for, etc. machine learning . One reason is, for the curve fitting methods of classic statistics, i.e., versions of regression , the math assumptions that give some hope of some good results are essentially impossible to verify and look like they will hold closely only rarely. So, even when using such statistics , good advice is to have two steps, (1) apply the statistics, i.e., fit,…

A point about gradient-free methods such as simulated annealing and genetic algorithms: the transition (sometimes called "neighbor") function is the most important part by far. The most important insight is the most obvious one in some way: if your task is to search a problem space efficiently for an optimal solution, it pays to know exactly how to move from where you are to where you want to be in that problem space. To that point, (the structure of) transitions between successive state samples should be refined to your specific problem and encoding of the domain in order to be useful in any reasonable amount of time.

Re: Statistical vs. Deep Learning forecasting methods

#35

Post by PaulHoule »

zone411 wrote: post #28
PaulHoule wrote: post #3

It is something that bothers me about the ML literature is that they frequently present a large number of evaluation results such as precision and AUC but these are not qualified by error bars. Typically they make a table which has different algorithms on one side and different problems on the other side and the highest score for a given problem gets bolded. I know if you did the experiment over and over against with…

Every researcher would love to include error bars but it's a matter of limited computing resources at universities. Unless you're training on a tiny dataset like MNIST, these training runs get expensive. Also, unless you parallelize from the start and risk wasting a lot of resources if something goes wrong, it could take longer to get the results.

Using bootstrap and/or repeated runs is a great way to get error bars but there are low cost ways to do it.

For instance they estimate error bars on public opinion polls based on simple formulas and not redoing the poll a large number of times.

Re: Statistical vs. Deep Learning forecasting methods

#36

Post by nequo »

PaulHoule wrote: post #35
zone411 wrote: post #28

Earlier quoted context omitted.

Every researcher would love to include error bars but it's a matter of limited computing resources at universities. Unless you're training on a tiny dataset like MNIST, these training runs get expensive. Also, unless you parallelize from the start and risk wasting a lot of resources if something goes wrong, it could take longer to get the results.

Using bootstrap and/or repeated runs is a great way to get error bars but there are low cost ways to do it. For instance they estimate error bars on public opinion polls based on simple formulas and not redoing the poll a large number of times.

If you don't have an analytical expression for your asymptotic variance, you do have to use bootstrap though.

For public opinion polls, the estimator is simple (i.e., a sample mean), so we have an analytical expression for its asymptotic variance.

Re: Statistical vs. Deep Learning forecasting methods

#37

Post by time_to_smile »

PaulHoule wrote: post #35
zone411 wrote: post #28

Earlier quoted context omitted.

Every researcher would love to include error bars but it's a matter of limited computing resources at universities. Unless you're training on a tiny dataset like MNIST, these training runs get expensive. Also, unless you parallelize from the start and risk wasting a lot of resources if something goes wrong, it could take longer to get the results.

Using bootstrap and/or repeated runs is a great way to get error bars but there are low cost ways to do it. For instance they estimate error bars on public opinion polls based on simple formulas and not redoing the poll a large number of times.

Simple formulas only work because the models themselves for those polls are incredibly simple and adding a bit more complexity requires a lot of tools to compute these uncertainties (this is part of the reason you see probabilistic programming so popular for people doing non-trivial polling work).

There are no simple approximations for a range of even slightly complex models. Even some nice computational tricks like the Laplace approximation don't work on models with high numbers of parameters (since you need to compute the diagonal of the Hessian).

A good overview of the situation is covered in Efron & Hastie's "Computer Age Statistical Inference".

Re: Statistical vs. Deep Learning forecasting methods

#38

Post by [deleted] »

PaulHoule wrote: post #35
zone411 wrote: post #28

Earlier quoted context omitted.

Every researcher would love to include error bars but it's a matter of limited computing resources at universities. Unless you're training on a tiny dataset like MNIST, these training runs get expensive. Also, unless you parallelize from the start and risk wasting a lot of resources if something goes wrong, it could take longer to get the results.

Using bootstrap and/or repeated runs is a great way to get error bars but there are low cost ways to do it. For instance they estimate error bars on public opinion polls based on simple formulas and not redoing the poll a large number of times.

[deleted]

Re: Statistical vs. Deep Learning forecasting methods

#39

Post by singhrac »

brrrrrm wrote: post #32
zmachinaz wrote: post #30

Earlier quoted context omitted.

Regarding 1) I am not sure if you are not trading "high human efficiency" against increased risk of blowing up at some point. Good luck doing forecasting without thorough understanding of priors and statistics in general.

that's a good point. I guess as an addendum it's not just compute efficiency but also "statistical efficiency" (if that has any meaning?)

I think that term already has usage as a proxy for "lowest sampling variance"; for example the Gauss Markov theorem shows that OLS is the most efficient unbiased linear estimator.

I guess this is echoing your point 2, but I would have generally said that "principled" statistical models are less efficient these days than DL (see: HMC being much slower than variational Bayes). Priors are usually overrated but I think the risk is more that basic mistakes are made because people don't understand what assumptions go into "basic" machine learning ideas like train/test splits or model selection. I'm not sure it warrants a lot of panic though.

Re: Statistical vs. Deep Learning forecasting methods

#40

Post by dumb1224 »

uoaei wrote: post #33
nerdponx wrote: post #24

Earlier quoted context omitted.

You could argue that deep learning is not a statistical method in the traditional sense, in that a typical neural network model is not a probability model, and some neural networks are well known to produce specifically bad probability models, requiring some amount of post processing in order to produce correctly "calibrated" probability predictions. However I don't like that there is often a strict dichotomy present…

Statistical models and probabilistic models are not synonymous. Vanilla deep learning models are statistical models (a la linear regression) and not probabilistic models (a la Gaussian mixture). It is important to maintain the distinction. But to your point about the dichotomy between deep learning and more "traditional" statistical methods: this confusion in common parlance clearly has negative effects on model-buil…

I have very limited statistical background but doesn't variational inference applied in the neural networks make them probabilistic models? The modelling definitely seems so because the math in those papers doesn't even specify whether it's a network (it implies that it can be any model).
Post reply on HN