Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Statistical vs. Deep Learning forecasting methods

github.com

Post reply on HN
Print view View original

61–70 of 85 posts

Re: Statistical vs. Deep Learning forecasting methods

#61

Post by jwilber »

srean wrote: post #46
jwilber wrote: post #43

Seems like these guys just wasted $11k to erroneously claim, “deep learning bad! Simple is better!” There’s definitely use for these classical, model-based methods, for sure. But a contrived comparison claiming they’re king is just misinformation. Eg, here are a number of issues with classical techniques where dl succeeds (‘they’ here refers to classical techniques): - they often don’t support missing/corrupt data -…

> they often don’t support missing/corrupt data You gotta be kidding right, that's one thing that they do well.

You can fit a state space model to missing data in many api’s, sure, but that doesn’t mean it’s supported in any meaningful way.

Imagine you have a new product with 6 months of missing data. You can feed that into ets/arima/whatever, but you’re not getting any valuable output for those missing point value estimates.

Re: Statistical vs. Deep Learning forecasting methods

#62

Post by nerdponx »

uoaei wrote: post #54
salty_biscuits wrote: post #53

Earlier quoted context omitted.

"Vanilla deep learning models are statistical models (a la linear regression) and not probabilistic models (a la Gaussian mixture). It is important to maintain the distinction." Isn't this just a matter of interpretation of the models? You can interpret linear regression in a Bayesian way and say that the prediction of the linear model is the MAP of the mean, you can also calculate the variance, the l2 norm objective…

> Isn't this just a matter of interpretation of the models? Not really. This is the classic frequentist vs Bayesian debate. In frequentist-land, you are computing point estimates of the model parameters. In Bayesian-land, you are computing distribution estimates of the model parameters. It is true that there is a difference in interpretation of the generative process but the two choices demand fundamentally different…

Statistics as practiced today (1930s until now?) consists almost entirely of making inferences about unobserved probability distributions. That includes nonparametric statistics, and frequentist versus Bayesian has nothing to do with it.

There are some probability models that are not really statistical models, but there are few or no statistical models that are not also probability models.

Least-squares regression is a probability model. Even if you don't particularly care about the error distribution, you are still estimating a conditional expectation and setting a conditional independence assumption on the residuals. If that's not a probability model, then I don't know what it is!

Re: Statistical vs. Deep Learning forecasting methods

#63

Post by Salgat »

ford wrote: post #58

I've done some work in this area and have indeed found that simpler statistical models often out perform ML/DL. This is true for single time series, where we are predicting P(x_t+1 | x_0..t) DL has advantages when you a) have additional context at each time step, or b) you have multiple related time series. For example, consider Amazon who predicts demands for all of their products. At each time step, they know about…

My understanding is that the biggest advantage between traditional machine learning and neural networks is that neural networks are useful when features either need to be generated or are poorly understood (such as a binary blob of an image or sound sample). So for data that is already neatly organized and labeled in a spreadsheet, DL loses its main advantage.

Re: Statistical vs. Deep Learning forecasting methods

#64

Post by graycat »

uoaei wrote: post #34
graycat wrote: post #23

I can have some interest in, hope for, etc. machine learning . One reason is, for the curve fitting methods of classic statistics, i.e., versions of regression , the math assumptions that give some hope of some good results are essentially impossible to verify and look like they will hold closely only rarely. So, even when using such statistics , good advice is to have two steps, (1) apply the statistics, i.e., fit,…

A point about gradient-free methods such as simulated annealing and genetic algorithms: the transition (sometimes called "neighbor") function is the most important part by far. The most important insight is the most obvious one in some way: if your task is to search a problem space efficiently for an optimal solution, it pays to know exactly how to move from where you are to where you want to be in that problem space…

> the transition (sometimes called "neighbor") function is the most important part by far.

And, indeed, in the 0-1 integer linear programming with Lagrangian relaxation I used there is nothing differentiable so should be counted as "gradient free". And the linear programming part and the Lagrangian part do "move" from where are to closer to "where want to be".

A thing is, the bag of tricks, techniques, that work here is large. So, right, should use knowledge of the real problem to pick what tricks to use.

Re: Statistical vs. Deep Learning forecasting methods

#65

Post by uoaei »

nerdponx wrote: post #62
uoaei wrote: post #54

Earlier quoted context omitted.

> Isn't this just a matter of interpretation of the models? Not really. This is the classic frequentist vs Bayesian debate. In frequentist-land, you are computing point estimates of the model parameters. In Bayesian-land, you are computing distribution estimates of the model parameters. It is true that there is a difference in interpretation of the generative process but the two choices demand fundamentally different…

Statistics as practiced today (1930s until now?) consists almost entirely of making inferences about unobserved probability distributions. That includes nonparametric statistics, and frequentist versus Bayesian has nothing to do with it. There are some probability models that are not really statistical models, but there are few or no statistical models that are not also probability models. Least-squares regression is…

Maybe the more intuitively obvious way to put it is:

probability : distributions :: statistics : expected values

Re: Statistical vs. Deep Learning forecasting methods

#66

Post by flusteredBias »

uoaei wrote: post #65
nerdponx wrote: post #62

Earlier quoted context omitted.

Statistics as practiced today (1930s until now?) consists almost entirely of making inferences about unobserved probability distributions. That includes nonparametric statistics, and frequentist versus Bayesian has nothing to do with it. There are some probability models that are not really statistical models, but there are few or no statistical models that are not also probability models. Least-squares regression is…

Maybe the more intuitively obvious way to put it is: probability : distributions :: statistics : expected values

I don’t think I have read anything more false on the internet. XD

Re: Statistical vs. Deep Learning forecasting methods

#67

Post by flusteredBias »

Xcelerate wrote: post #7

I wish we could start moving to better approaches for evaluating time series forecasts. Ideally, the forecaster reports a probability distribution over time series, then we evaluate the predictive density with regard to an error function that is optimal for the intended application of the forecast at hand.

You mean CRPS?

Re: Statistical vs. Deep Learning forecasting methods

#68

Post by flusteredBias »

flusteredBias wrote: post #67
Xcelerate wrote: post #7

I wish we could start moving to better approaches for evaluating time series forecasts. Ideally, the forecaster reports a probability distribution over time series, then we evaluate the predictive density with regard to an error function that is optimal for the intended application of the forecast at hand.

You mean CRPS?

I use my package https://github.com/alexhallam/tablespoon to generate naive forecasts then evaluate the crps of the naive vs the crps of the alternative method. This “skill score” approach is very good.

Re: Statistical vs. Deep Learning forecasting methods

#69

Post by flusteredBias »

jwilber wrote: post #43

Seems like these guys just wasted $11k to erroneously claim, “deep learning bad! Simple is better!” There’s definitely use for these classical, model-based methods, for sure. But a contrived comparison claiming they’re king is just misinformation. Eg, here are a number of issues with classical techniques where dl succeeds (‘they’ here refers to classical techniques): - they often don’t support missing/corrupt data -…

I am going to defined the readme a little

- they often don’t support missing/corrupt data

Clean data would benefit most models, not just non-deep learning models. Missing data introduces bias even in DL models.

- they focus on linear relationships and not complex joint distributions

1. If seasonality is present. Which is usually the case in practical business problems then you will find that actual ~ lag_actuals explains most of the variance with a linear relationship. non-linearities in time series is not something that I see often. You can usually make a feature that explains those away linearly.

- they focus on fixed temporal dependence that must be diagnosed and specified a priori

Not sure sure you are saying here.

- they take as input univariate, not multiple interval, data

That is not the case. Time series regression can take many lags for inputs.

- they focus on one-step forecasts, not long time horizons

False. Time series regression models are used to forecast revenue many years into the future in the business world.

- they’re highly parameterized and rigid to assumptions

So is F=ma

- they fail for cold start problems

Because a cold start is not a time series data set. Why would time series methods work on non time series data.

Re: Statistical vs. Deep Learning forecasting methods

#70

Post by flusteredBias »

flusteredBias wrote: post #66
uoaei wrote: post #65

Earlier quoted context omitted.

Maybe the more intuitively obvious way to put it is: probability : distributions :: statistics : expected values

I don’t think I have read anything more false on the internet. XD

I apologize. ^This comment was to harsh.

Statistics can be summarizes as one thing n -> N. Does ‘little n’ represent ‘big N’. In other words, does the sample generalize to the population. Statistics means something like “description of the state”. It was born out of census samples where larger population samples had to be estimated. “n” could be a handful of fish in a “N” lake. “n” could also be the parameter estimated in a linear regression with the sample of data collected while “N” is the true parameter of the relationship if we had all the data. Point estimation is about finding the needle in the haystack, but much more often statistics is about finding the haystack given the needle. One tool statistics uses to get to the haystack is probability.

Post reply on HN