41–50 of 69 posts
Well, Jane Street cracked it
A very interesting article, suzyahyah. I especially appreciated your definition of stationarity, a concept with which I struggled in my own time series class. If I understand correctly, it sounds like the basic premise is that a fundamentally statistical methodology (LLMs) can't realistically predict a non-stationary data generation, which makes sense. Separately, I've wondered for some time if there might be some re…
This brings up the philosophical question of whether humans have free will or is everything deterministic following the laws of physics.
Well. Lots of math that boils down to 'predicting the future is hard'. Especially when the future is one of social construction, that's what gets lost a bit here. Predicting the future is easier for planetary motions than for Bitcoin.
However, the aim of math in these situations is often to give explanations for intuitive impressions like "predicting the future is hard". The concept of NP-completeness gives one (very partial) explanation why certain computing problems are "hard", for example. So that theory doesn't "boil down to saying programming is hard". Unfortunately, I don't think the text really gives strong explanation in this case.
Earlier quoted context omitted.
It's true that markets are more _adversarial_. But there's still a lot of trouble with distribution shifts even in server metrics. As an example, our SRE team got paged a few times in the past month for traffic drops due to the World Cup. This stresses the nowcasting alert in several dimensions: - there's no seasonal pattern to the matches, they happen sorta randomly. - they drive increased query traffic in the hour…
The UK power grid operators famously plan for massive demand surges at eg the end of major football matches. I cant imagine what their forecasters thought of the England-Mexico nail biter. Half the nation heading to put the kettle on, half glued to their seats. Would love to see the charts of demand now that the World Cup is done...
But I’m sure the World Cup is still pretty relevant for operations.
I see that a lot of these are markets. Yes, it’s hard to predict markets. Because anybody who can successfully predict markets, does so, makes money, and changes the market so their predictions lose their edge. Time series forecasts are a lot easier if you are forecasting, say, disk use in your servers or whatnot. (By “easy” I mean you can do a simple prediction and get useful insights.)
Whenever I teach people time series forecasting, I always point out that one of the biggest challenges is that you will always have values at prediction time that are out side the range of values observed during training (specifically the value of t).
In plenty of other machine learning and statistical modeling tasks this is not the case. You can train on every token you'll ever see and every pixel value you'll ever see, you can do regression analysis on every categorical value you include and an least an observation from within a range of every continuous and discrete value you'll observe. But with forecasting you will always have values you predict that are outside the range of anything you trained on.
You would run into similar problems if you tried to create a statistical model of the density of water given a temperature but your training data only included values between 0-100 C and you went out and started predicting values covering all the temperatures found on Earth.
For whatever reason, when time is a variable we somehow think it is immune from the obvious limitation of predicting on values outside of the range of values you trained on.
Well. Lots of math that boils down to 'predicting the future is hard'. Especially when the future is one of social construction, that's what gets lost a bit here. Predicting the future is easier for planetary motions than for Bitcoin.
As I mentioned in another comment, this can also be rephrased as "predicting data with values outside the range you trained on typically doesn't go well". If you tried to predict some health metric based on weight and height but you only had people under 4' 10" and less than 120lbs you wouldn't be shocked at all if it worked terribly when applied to American football players.
Time-series forecasting is hard because you are always going to be predicting based on data outside of your observed range ("forecasting" does go much better when you're trying to fill-in-the-blanks of things that happened in the past).