Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 91 posts

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#21

Post by isoprophlex »

Imanari wrote: post #12

For feature engineering check out tsfresh and sktime, especially the minirocket algorithm. https://tsfresh.readthedocs.io/en/latest/ https://www.sktime.org/en/v0.8.2/api_reference/auto_generate...

I've had someone in a team implement feature engineering using tsfresh. It lead to a malignantly under-performing, complicated heap of spaghetti that was a nightmare to get into production. Weird API, slow code, little added value over simple features found in a day of manual exploration.

Person doing the implementation wasn't a rock star coder so we couldn't fix the performance and complexity issues in time; it was left it out of the releas. Maybe with more expertise, tsfresh can add value. The experience was pretty off-putting for me, all in all. Maybe others have different experiences?

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#23

Post by isoprophlex »

istinetz wrote: post #10
unlikelymordant wrote: post #6

Earlier quoted context omitted.

Xgboost is a classifier for tabular data, prophet is for time series prediction. They are different use cases, though you can likely massage xgboost to do time series prediction if you really wanted to. So the question of which is better is "it depends"

I'm not even sure how you can get it to work on time series...

Check out my top level comment in this thread for a (hopefully clear) example. Sometimes you can rephrase a time series problem into boring classical regression.

It can make the implementation and maintainability of a codebase better (IMHO), without sacrificing predictive power.

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#25

Post by chirau »

Fiahil wrote: post #24

XGboost, LGBM, pmdarima, stanpy (for bayesian modelling). Plus a few others. Don't ask me what they do with all of these, I'm just the guy who make sure the forecast keeps being reproducible.

What does that make you titlewise? Data Engineer? ML Engineer?

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#26

Post by RamblingCTO »

isoprophlex wrote: post #22
curiousgal wrote: post #14

Time series analysis is where R shines compared to Python.

Can you substantiate your comment? What would you like to see in a python tool that R does uniquely well?

Not parent, but easy: https://cran.r-project.org/web/packages/forecast/index.html

/e: I'm a bit out of this game for 2-3 years now, but Python had nothing comparable. Prophet is suboptimal at best. Some things are implemented here or there, but it's all over the place. So I agree that R really shines here.

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#27

Post by Fiahil »

chirau wrote: post #25
Fiahil wrote: post #24

XGboost, LGBM, pmdarima, stanpy (for bayesian modelling). Plus a few others. Don't ask me what they do with all of these, I'm just the guy who make sure the forecast keeps being reproducible.

What does that make you titlewise? Data Engineer? ML Engineer?

A Software Engineer. I'm just specialised a bit in DevOps, Data Engineering, and (beware buzzword) MLOps

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#29

Post by esquire_900 »

isoprophlex wrote: post #21
Imanari wrote: post #12

For feature engineering check out tsfresh and sktime, especially the minirocket algorithm. https://tsfresh.readthedocs.io/en/latest/ https://www.sktime.org/en/v0.8.2/api_reference/auto_generate...

I've had someone in a team implement feature engineering using tsfresh. It lead to a malignantly under-performing, complicated heap of spaghetti that was a nightmare to get into production. Weird API, slow code, little added value over simple features found in a day of manual exploration. Person doing the implementation wasn't a rock star coder so we couldn't fix the performance and complexity issues in time; it was…

I've tried it a number of times, and had a similar experience. The whole stack is orders of magnitudes slower to compute compared to "simple" features (i.e. rolling averages), without showing real predictive improvements.

Re: Ask HN: Data Scientists, what libraries do you use for timeseries forecasting?

#30

Post by em500 »

isoprophlex wrote: post #19

Can you reframe the problem to suit a more classical approach - regression using xgboost or lgbm? If so, go for that! As an example, imagine you want to calculate only a single sample into the future. Say furthermore that you have six input timeseries sampled hourly, and you don't expect meaningful correlation beyond 48h old samples. You create 6x48 input features, take the single target value that you want to predic…

fbprophet is mostly just regression though, with features for trend, yearly and weekly periodicity (smoothed a bit using trigonometric regressors), and holiday features. The only non-standard linear regression part is that it includes a flexible piecewise linear trend, with regularization to select where the trend is allowed to change. Once the change points are selected, it's literally linear regression, that you could fit with anything that can handle regression (statsmodels, sklearn, xgboost, keras, tf, scipy or even just plain numpy).

So you could roll your own using lots of different libraries and do the feature engineering, but for plain jane business time series, fbprophet usually works pretty decently with minimum effort. Because most of the predictable variation in business time series is are patterns in human behavior, which are mostly regulated by the weekly and yearly cycle, interrupted a few times per year by holidays. (And the 24h cycle if you forecast intra-day).

That said, fbprophet is somewhat tuned for a few years of regular data (ideally not interrupted by pandemics), sampled at the daily frequency. If your data is different (e.g. weekly or intra-day) it can start to break in unexpected ways, and it becomes worthwhile to dive into the model and customize or roll your own.

Post reply on HN