I'd be curious what the results would be with the automated Autogluon fit/evals. I suspect given the results here, a weighted average model would likely win out.
Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model
21–30 of 36 posts
Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model
#22Earlier quoted context omitted.
That's actually one of the use-cases that we set out to explore with these models. We'll release a head-to-head comparison soon!
That's the thing I'm most interested in out of these. Super interested to see what you find out. Did you or do you plan to publish any of your code or data sets from this?
Author here, we’re just getting started with these experiments and plan to apply them to more features on our roadmap. Future posts will be more detailed, based on the feedback we received here. Once we finish implementing these features, we’ll be happy to share the code and dataset.
Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model
#23Look i'm optimistic about time-series foundation models too, but this post is hard to take seriously when the test is so flawed: - Forward filling missing short periods of missing values. Why keep this in when you explictly mention this is not normal? Either remove it all or don't impute anything - Claiming superiority over classic models and then not mentioning any in the results table - Or let's not forget, the car…
To clarify, you'd prefer rmsle?
Short answer: i use multiple metrics, never rely on just 1 metric.
Long answer: Is the metric for people with subject-matter knowledge? Then (Weighted)RMSSE, or the MASE alternative for a median forecast. WRMSSE is is very nice, it can deal with zeroes, is scale-invariant and symmetrical in penalizing under/over-forecasting.
The above metrics are completely uninterpretable to people outside of the forecasting sphere though. For those cases i tend to just stick with raw errors; if a percentage metric is really necessary then a Weighted MAPE/RMSE, the weighing is still graspable for most, and it doesn't explode with zeroes.
I've also been exploring FVA (Forecast Value Added), compared against a second decent forecast. FVA is very intuitive, if your base-measures are reliable at least. Aside from that i always look at forecast plots. It's tedious but they often tell you a lot that gets lost in the numbers.
RMSLE i havent used much. From what i read it looks interesting, though more for very specific scenarios (many outliers, high variance, nonlinear data?)
Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model
#24Look i'm optimistic about time-series foundation models too, but this post is hard to take seriously when the test is so flawed: - Forward filling missing short periods of missing values. Why keep this in when you explictly mention this is not normal? Either remove it all or don't impute anything - Claiming superiority over classic models and then not mentioning any in the results table - Or let's not forget, the car…
Author here, we're trying these out for the first time for our use-cases so these are great points for us to improve upon!
Good to see positive reception to feedback! Sorry if my message came out as condescending, was not the intent. I recommend reading this piece on metrics https://openforecast.org/wp-content/uploads/2024/07/Svetunko.... It's easy to grasp, yet it contains great tips.
Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model
#25Earlier quoted context omitted.
To clarify, you'd prefer rmsle?
Short answer: i use multiple metrics, never rely on just 1 metric. Long answer: Is the metric for people with subject-matter knowledge? Then (Weighted)RMSSE, or the MASE alternative for a median forecast. WRMSSE is is very nice, it can deal with zeroes, is scale-invariant and symmetrical in penalizing under/over-forecasting. The above metrics are completely uninterpretable to people outside of the forecasting sphere…
Thanks for the reply! I am outside the forecasting sphere.
RMSLE gives proportional error (so, scale-invariant) without MAPE's systematic under-prediction bias. It does require all-positive values, for the logarithm step.
Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model
#26I think that the concept of a "foundation model" for time series is actually a bit flawed as presented in this blog post. A foundation model is interesting because it is capable of many tasks _beyond the target tasks_ that it was trained to do, whereas what the author is looking for is a time-series model that can make out-of-distribution predictions without re-training - which is, in my opinion, a problem that is pr…
A lot of people try to hedge this kind of sober insight along with their personal economic goals to say all manner of unfalsifiable statements of adequate application in some context, but it is refreshing to try to deal with the issues separately and I think a lot of people miss the insufficiency compared to traditional methods in all cases that I've heard of so far.
Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model
#27[deleted]
Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model
#28Earlier quoted context omitted.
To clarify, you'd prefer rmsle?
Short answer: i use multiple metrics, never rely on just 1 metric. Long answer: Is the metric for people with subject-matter knowledge? Then (Weighted)RMSSE, or the MASE alternative for a median forecast. WRMSSE is is very nice, it can deal with zeroes, is scale-invariant and symmetrical in penalizing under/over-forecasting. The above metrics are completely uninterpretable to people outside of the forecasting sphere…
MAPE can be a problem also if you have a problem where rare excursions are what you want to predict and the cost of missing an event is much higher than predicting a non-event. A model that just predicts no change would have very low MAPE because most of the time nothing happens. When the event happens, however, the error of predicting status quo ante is much worse than small baseline errors.
Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model
#29I think that the concept of a "foundation model" for time series is actually a bit flawed as presented in this blog post. A foundation model is interesting because it is capable of many tasks _beyond the target tasks_ that it was trained to do, whereas what the author is looking for is a time-series model that can make out-of-distribution predictions without re-training - which is, in my opinion, a problem that is pr…
A lot of people try to hedge this kind of sober insight along with their personal economic goals to say all manner of unfalsifiable statements of adequate application in some context, but it is refreshing to try to deal with the issues separately and I think a lot of people miss the insufficiency compared to traditional methods in all cases that I've heard of so far.
Ai slop
Re: Zero-Shot Forecasting: Our Search for a Time-Series Foundation Model
#30[flagged]
Author here, apologies for not making it clear on the post regarding the definition of Zero-shot time-series forecasting, but it's quite widely used and here's the definition of it "Zero-shot time-series forecasting is a framework for time-series prediction that does not require fine-tuning with specific time-series data to be predicted."
Thanks for the reply!