Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 91 posts

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#51

Post by hrzn »

I would recommend Darts in Python [1]. It's easy to use (think fit()/predict()) and includes

* Statistical models (ETS, (V)ARIMA(X), etc)

* ML models (sklearn models, LGBM, etc)

* Many recent deep learning models (N-BEATS, TFT, etc)

* Seamlessly works on multi-dimensional series

* Models can be trained on multiple series

* Several models support taking in external data (covariates), known either in the past only, or also in the future

* Many models offer rich support for probabilistic forecasts

* Model evaluation is easy: Darts has many metrics, offers backtest etc

* Deep learning scales to large datasets, using GPUs, TPUs, etc

* You can do reconciliation of forecasts at different hierarchical levels

* There's even now an explainability module for some of the models - showing you what matters for computing the forecasts

* (coming soon): an anomaly detection module :)

* (also, it even include FB Prophet if you really want to use it)

Warning: I'm probably biased because I'm Darts creator.

[1] https://github.com/unit8co/darts

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#52

Post by nerdponx »

Lots of people have already made good library recommendations, so I will make a non-recommendation for all the data science students out there: stop thinking about libraries, and start thinking about models.

"What library do I use?" is the wrong question. "What model do I use?" is the right question. Libraries are just part of the process of answering that question.

That said, high quality implementations of interesting times series models seem hard to come by, so it's still a legitimate question to ask about libraries. but consider the goal of asking about libraries: you want to find high-quality implementations of useful models, not a magic black box that you can crank data through.

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#53

Post by Imanari »

isoprophlex wrote: post #21
Imanari wrote: post #12

For feature engineering check out tsfresh and sktime, especially the minirocket algorithm. https://tsfresh.readthedocs.io/en/latest/ https://www.sktime.org/en/v0.8.2/api_reference/auto_generate...

I've had someone in a team implement feature engineering using tsfresh. It lead to a malignantly under-performing, complicated heap of spaghetti that was a nightmare to get into production. Weird API, slow code, little added value over simple features found in a day of manual exploration. Person doing the implementation wasn't a rock star coder so we couldn't fix the performance and complexity issues in time; it was…

Actually, I had a similar off-putting experience with tsfresh. I just thought it was due to me not understanding how to use it properly. The API is indeed pretty weird.

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#54

Post by dxbydt »

Peter Cotton has atleast a dozen very credible studies/results on prophet vs other timeseries libraries. Before committing to prophet, please check out a few of these (all over linkedin). His tone is acerbic because he believes prophet is suboptimal & makes poor forecasts compared to the other contenders. That said, you can ignore the tone, just download the packages & test out the scenarios for yourself. I personally will not use prophet. Like most stat tools in the python ecosystem, it is super easy to deploy & code up, but often inaccurate if you actually care about the results. ofcourse, if its some sales prediction forecast where everything’s pretty much made up & data is sparse/unverifiable, then prophet ftw.

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#55

Post by nerdponx »

em500 wrote: post #30
isoprophlex wrote: post #19

Can you reframe the problem to suit a more classical approach - regression using xgboost or lgbm? If so, go for that! As an example, imagine you want to calculate only a single sample into the future. Say furthermore that you have six input timeseries sampled hourly, and you don't expect meaningful correlation beyond 48h old samples. You create 6x48 input features, take the single target value that you want to predic…

fbprophet is mostly just regression though, with features for trend, yearly and weekly periodicity (smoothed a bit using trigonometric regressors), and holiday features. The only non-standard linear regression part is that it includes a flexible piecewise linear trend, with regularization to select where the trend is allowed to change. Once the change points are selected, it's literally linear regression, that you co…

I generally think it's better to start with something even simpler than Prophet, like EMA and ETS.

One of the "auto ARIMA" packages in R (Forecast or Fable) could also be a good choice.

However I think Prophet is a good next step from there. It's at least a sensible starting point for your own feature engineering.

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#56

Post by dxbydt »

bayan1234 wrote: post #17

Forecast package in R is quite useful. Even if you don’t use R, this book by Rob Hyndman is very approachable and easy to follow. https://otexts.com/fpp2/

This is a good book and forecast is a very good R package, imo.

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#57

Post by moralestapia »

Fiahil wrote: post #27
chirau wrote: post #25

Earlier quoted context omitted.

What does that make you titlewise? Data Engineer? ML Engineer?

A Software Engineer. I'm just specialised a bit in DevOps, Data Engineering, and (beware buzzword) MLOps

What's MLops? Is it what I imagine?

Maintaining repos of training/data, APIfying the pipeline, deploying an ML processing pipeline with CI/CD, etc?

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#58

Post by nerdponx »

dxbydt wrote: post #54

Peter Cotton has atleast a dozen very credible studies/results on prophet vs other timeseries libraries. Before committing to prophet, please check out a few of these (all over linkedin). His tone is acerbic because he believes prophet is suboptimal & makes poor forecasts compared to the other contenders. That said, you can ignore the tone, just download the packages & test out the scenarios for yourself. I personall…

I think acerbity is warranted to some extent. We are data scientists. We get paid the big bucks because we have big brains and have the skills and training to use those big brains in order to reason about the work we are doing.

Data science has become so easy and accessible nowadays that basically anyone who can write code can fit and use models. That's a great thing in general, but it means that those of us who do this for a living really should hold ourselves to higher standards.

Even if you are thoroughly mediocre at your job (like me) and aren't smart enough to come up with something like Prophet on your own, you absolutely do need the ability to reason about the models that you use, and to evaluate their performance correctly.

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#59

Post by nerdponx »

qsort wrote: post #18
bayan1234 wrote: post #17

Forecast package in R is quite useful. Even if you don’t use R, this book by Rob Hyndman is very approachable and easy to follow. https://otexts.com/fpp2/

+1 for the book, an excellent reference. As someone who's primarily a developer, it's been extremely useful as a study guide. Also, there's a new version; s/2/3 :)

It's useful as a study guide and reference even for someone who ostensibly learned all this stuff in school. It's a tremendously good book, and it's even more impressive that it's free to read online in a high-quality HTML document.

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#60

Post by nerdponx »

isoprophlex wrote: post #19

Can you reframe the problem to suit a more classical approach - regression using xgboost or lgbm? If so, go for that! As an example, imagine you want to calculate only a single sample into the future. Say furthermore that you have six input timeseries sampled hourly, and you don't expect meaningful correlation beyond 48h old samples. You create 6x48 input features, take the single target value that you want to predic…

Your description of this technique as "6x48" reminds me of 1D convolutional neural networks.

I always had a gut feeling that there was some kind of unifying principle between NN and tree ensemble models. I wonder if the latter is kind of like a highly quantized, compressed version of the former, which explains why tree ensembles seem to work better than NNs on "lower resolution" data.

Post reply on HN