Is there a github repo for her time series forecasting experiments? I'd like to see her code and be able to run it against my own data sets.
The unreasonable difficulty of time series forecasting
61–69 of 69 posts
Re: The unreasonable difficulty of time series forecasting
#62> I’ve been thinking recently about what makes time series forecasting problems so difficult compared to other sequence learning tasks Whenever I teach people time series forecasting, I always point out that one of the biggest challenges is that you will always have values at prediction time that are out side the range of values observed during training (specifically the value of t ). In plenty of other machine learn…
> Whenever I teach people time series forecasting, I always point out that one of the biggest challenges is that you will always have values at prediction time that are out side the range of values observed during training (specifically the value of t). I don't get this, time is usually not a covariate in ts models, so why is it a challenge?
Well you know you will likely have an outlier. The question is when.
Re: The unreasonable difficulty of time series forecasting
#63> Given the lack of forecasting signal, the obvious next step then is to seek out external (exogenous) features in the real world that can help prediction models. this seems right to me. maybe another interesting approach would be a fusion llm+ts model that does multiple-input-single-output with input metadata and causality narrative. so it "thinks" about what data it has and how predictive it may be of the target va…
There's a couple "AI forecaster" companies now, Preseen and FutureSearch. My understanding is you give them a query and they research every possible relevant factor and give an estimate of the likelihood of your query, and supposedly they're comparable to if you asked a human to do the same thing, but they're new enough that they're not really proven yet.
The best advertisement for these companies would probably be running a public Polymarket account so everyone else can quickly check how accurate they are.
Re: The unreasonable difficulty of time series forecasting
#64This article focuses mostly on point forecasting . There are many other things that are interesting to forecast. Consider for example the use case of forecasting the average speed on a road segment with a maximum speed of 70mph. Forecasting whether that will be 69.8 or 70.3 is not very relevant. What is relevant is forecasting when the speed drops below a traffic jam threshold. But the exact timing of that might be i…
Indeed. For example, volatility is of interest in financial data, because it relates to the pricing of risk. So are the correlations between asset classes.
Interestingly market volatility often goes hand-in-hand with increased correlation between asset prices: https://en.wikipedia.org/wiki/Anna_Karenina_principle#Order_...
This corresponds to a restatement of Murphy's Law, namely "life is a bitch and then you die". When you most need a diversified portfolio, diversification is hardest to achieve.
Re: The unreasonable difficulty of time series forecasting
#65Earlier quoted context omitted.
There's a couple "AI forecaster" companies now, Preseen and FutureSearch. My understanding is you give them a query and they research every possible relevant factor and give an estimate of the likelihood of your query, and supposedly they're comparable to if you asked a human to do the same thing, but they're new enough that they're not really proven yet.
something like that would be interesting. i just get the sense that doing it metadata blind would result in something that learns a somewhat brittle short-time covariance-like structure that doesn't use all available information to achieve best possible generalization and that in the end the benchmark numbers start to reflect some luck with that for predicting non-stationary behavior combined with a lot coming from a…
I did some energy price forecasting and my neural network learned the morning evening patterns just from historical data, it worked great. Then I tried adding more information to this like wind and solar (and many more). None of these external factors made any difference while being very important for energy prices. It seems like the market already incorporated all the factors in the price (as they should).
The takeaway might be that historic data of market data might often be enough to make a reasonable prediction. Only external data that nobody else has (used) can make your prediction better then the market.
Making a prediction with same accuracy as the market: easy Making a prediction with more accuracy then the market: very hard
Re: The unreasonable difficulty of time series forecasting
#66Earlier quoted context omitted.
something like that would be interesting. i just get the sense that doing it metadata blind would result in something that learns a somewhat brittle short-time covariance-like structure that doesn't use all available information to achieve best possible generalization and that in the end the benchmark numbers start to reflect some luck with that for predicting non-stationary behavior combined with a lot coming from a…
I did some energy price forecasting and my neural network learned the morning evening patterns just from historical data, it worked great. Then I tried adding more information to this like wind and solar (and many more). None of these external factors made any difference while being very important for energy prices. It seems like the market already incorporated all the factors in the price (as they should). The takea…
> The takeaway might be that historic data of market data might often be enough to make a reasonable prediction. Only external data that nobody else has (used) can make your prediction better then the market.
that's basically the definition of an efficient market. demand forecasting and insuring against price shifts is the actually useful thing that the commodities futures markets do.
also a fundamental difference from other time series prediction problems. there are all sorts of weird dynamics that go on before one biosignal effects another, or one metric predicts a failure, where a reasonably efficient market price reacts quickly to well known exogenous factors.
Re: The unreasonable difficulty of time series forecasting
#67> I’ve been thinking recently about what makes time series forecasting problems so difficult compared to other sequence learning tasks Whenever I teach people time series forecasting, I always point out that one of the biggest challenges is that you will always have values at prediction time that are out side the range of values observed during training (specifically the value of t ). In plenty of other machine learn…
I don't understand, this is just a sampling problem?
> You can train on every token you'll ever see and every pixel value you'll ever see, you can do regression analysis on every categorical value you include and an least an observation from within a range of every continuous and discrete value you'll observe.
Can you give an example of this? Lets say you are developing DLSS, you don't have the of a game that have not yet been developed.
> But with forecasting you will always have values you predict that are outside the range of anything you trained on.
A time series of my body temperature will only ever range from 20C to 50C. Outside of that range, I have bigger problems than my prediction being wrong.
Re: The unreasonable difficulty of time series forecasting
#68> I’ve been thinking recently about what makes time series forecasting problems so difficult compared to other sequence learning tasks Whenever I teach people time series forecasting, I always point out that one of the biggest challenges is that you will always have values at prediction time that are out side the range of values observed during training (specifically the value of t ). In plenty of other machine learn…
I don't understand, this is just a sampling problem? > You can train on every token you'll ever see and every pixel value you'll ever see, you can do regression analysis on every categorical value you include and an least an observation from within a range of every continuous and discrete value you'll observe. Can you give an example of this? Lets say you are developing DLSS, you don't have the of a game that have no…
It's not a sampling problem, but more a gambling problem, as in the gambler's fallacy?
Re: The unreasonable difficulty of time series forecasting
#69> I’ve been thinking recently about what makes time series forecasting problems so difficult compared to other sequence learning tasks Whenever I teach people time series forecasting, I always point out that one of the biggest challenges is that you will always have values at prediction time that are out side the range of values observed during training (specifically the value of t ). In plenty of other machine learn…
This is incredibly well stated. I might trot out this comment the next time the bosses ask for more accurate next-year financial projections.