Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model

parseable.com

Post reply on HN
Print view View original

31–36 of 36 posts

Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model

#31

Post by 3abiton »

wenc wrote: post #9

I wonder how this would perform on the M4 Makridakis competitions (time series competitions) https://github.com/Mcompetitions/M4-methods https://en.wikipedia.org/wiki/Makridakis_Competitions Makridakis' conclusion remained true for many years: "statistically sophisticated and complex methods do not necessarily provide more accurate forecasts than simpler ones." Maybe things have changed? (side: Nixtla showed a simple…

When I worked in Demand prediction (multivariate), it was lgbm that was outperformong across the board.

Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model

#32

Post by parmesant »

mvATM99 wrote: post #24
parmesant wrote: post #11

Earlier quoted context omitted.

Author here, we're trying these out for the first time for our use-cases so these are great points for us to improve upon!

Good to see positive reception to feedback! Sorry if my message came out as condescending, was not the intent. I recommend reading this piece on metrics https://openforecast.org/wp-content/uploads/2024/07/Svetunko... . It's easy to grasp, yet it contains great tips.

we're grateful for the honest feedback (and the awesome resource!), makes it easier to identify areas for improvement. Also, your point about using multiple metrics (based on use-cases, audience, etc) makes a lot of sense. Will incorporate this in our next experiment.

Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model

#33

Post by parmesant »

sheepscreek wrote: post #14

> Our dataset consisted of Kubernetes pod metrics collected from a production retail checkout application. That sums it up and it’s no surprise why Datadog’s toto model performed exceptionally well. The results would have been much more useful had they opted for a heterogenous mix of data sets. I am thinking of census data and statistics, or financial forecasting (GDP, interest rates), or clinical trial drop-out rate…

At the moment our focus is on observability, hence the narrow scope of our dataset. A pretty good benchmark for observability seems to be Datadog's BOOM- https://huggingface.co/datasets/Datadog/BOOM

But for general purpose time-series forecasting, benchmarks mentioned in other comments like GIFT or M4 might come in handy. We might include them in the follow-up experiment.

Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model

#34

Post by parmesant »

spmurrayzzz wrote: post #21

I'd be curious what the results would be with the automated Autogluon fit/evals. I suspect given the results here, a weighted average model would likely win out.

We'll definitely include it in our next experiment (shaping up to be quite big!)

Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model

#35

Post by parmesant »

fumeux_fume wrote: post #17

I'm a bit confused by the results table. Were these models tested against the same dataset? Also, a visualization of the test data and forecasts would be helpful as well.

Based on the feedback, we could have done a much better job with these results (lessons for our next experiment). But yes, the models were tested against the same dataset which was aggregated over different granularities (1 minute, 1 hour, 1 day)

Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model

#36

Post by stevenae »

ted_dunning wrote: post #28
mvATM99 wrote: post #23

Earlier quoted context omitted.

Short answer: i use multiple metrics, never rely on just 1 metric. Long answer: Is the metric for people with subject-matter knowledge? Then (Weighted)RMSSE, or the MASE alternative for a median forecast. WRMSSE is is very nice, it can deal with zeroes, is scale-invariant and symmetrical in penalizing under/over-forecasting. The above metrics are completely uninterpretable to people outside of the forecasting sphere…

MAPE can be a problem also if you have a problem where rare excursions are what you want to predict and the cost of missing an event is much higher than predicting a non-event. A model that just predicts no change would have very low MAPE because most of the time nothing happens. When the event happens, however, the error of predicting status quo ante is much worse than small baseline errors.

My reading of this situation is that MAPE would do the opposite. Means are skewed towards outliers.
Post reply on HN