Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google's 200M-parameter time-series foundation model with 16k context

github.com

Post reply on HN
Print view View original

71–80 of 117 posts

Re: Google's 200M-parameter time-series foundation model with 16k context

#71

Post by thesz »

EmilStenstrom wrote: post #4

I somehow find the concept of a general time series model strange. How can the same model predict egg prices in Italy, and global inflation in a reliable way? And how would you even use this model, given that there are no explanations that help you trust where the prediction comes from…

  > How can the same model predict egg prices in Italy, and global inflation in a reliable way?
For one, there's Benford's law: https://en.wikipedia.org/wiki/Benford%27s_law

So, predict sign (branch predictors in modern CPUs also use neural networks of sorts), exponent (most probably it changes slowly) and then predict mantissa using Benford's law.

Re: Google's 200M-parameter time-series foundation model with 16k context

#72

Post by Foobar8568 »

rockwotj wrote: post #11
Foobar8568 wrote: post #2

Somehow I missed that one. Are there any competition on this? I always had difficulties with ML and time series, I'll need to try that out.

https://www.datadoghq.com/blog/datadog-time-series-foundatio... https://moment-timeseries-foundation-model.github.io/ https://arxiv.org/abs/2403.07815 A friend at work used one to predict when our CEO would post in Slack, which is verry entertaining to see if correct.

Many thanks for the links!

Re: Google's 200M-parameter time-series foundation model with 16k context

#73

Post by drzaiusx11 »

jdthedisciple wrote: post #20

Let me be blunt: Shannon would tell us that time forecasting is bullshit: There is infinitely more entropy in the real world out there than any model can even remotely capture. The world is not minecraft.

Time series forecasting has proven useful in a number of different domains from weather to health monitoring. Sure you can easily over fit on the training data, but in general that's a data source/input problem where you need many high quality data sources to find the signal in the noise.

The world is chaotic sure, but there are still truths to be found in noisy time series data; saying that the world is too random to be knowable is a bit dismissive, no?

Re: Google's 200M-parameter time-series foundation model with 16k context

#74

Post by robot-wrangler »

dist-epoch wrote: post #66
lordgrenville wrote: post #48

Earlier quoted context omitted.

That's what traditional time-series modelling does. This is a foundational model, which means it's just a neural network trained on lots of time series. (So maybe OP's question still stands? But it's the same question as "how can LLMs be good at so many different kinds of conversations?")

Because traditional time-series modelling (ARIMA, GARCH, ...) is too "simple" and "strict". Just like "simple" computer vision (OpenCV, edge-detection, ...) was crushed by neural networks when having to deal with real world images.

This seemed like a good answer at first. But on further thought, images on the whole really do seem to have quite a bit more standard structure / "grammar" to exploit compared to arbitrary time-series. Many images are of the world, where there is gravity so you might see preponderance of blobs at the bottom, or the repetitive types like people, animals, faces, eyes. Wildly abstract images still have some continuity, pixels in a neighborhood are likely to be similar.

Time series in general have none of this kind of structure that's strictly necessary. I'm sure that many real-world sensors typically have some gaussian distribution aspects + noise and/or smoothness and locality types of assumptions that are pretty safe, but presumably that simple stuff is exactly what traditional time-series modelling was exploiting.

Maybe the real question is just what kind of time-series are in the training data, and why do we think whatever implicit structure that is there actually generalizes? I mean, you can see how any training that mixes pictures of dogs and cats with picturing of people could maybe improve drawing hair, detecting hair, or let you draw people AND dogs. It's less clear to me how mixing sensor data / financial data / anything else together could be helpful.

Re: Google's 200M-parameter time-series foundation model with 16k context

#76

Post by dist-epoch »

robot-wrangler wrote: post #74
dist-epoch wrote: post #66

Earlier quoted context omitted.

Because traditional time-series modelling (ARIMA, GARCH, ...) is too "simple" and "strict". Just like "simple" computer vision (OpenCV, edge-detection, ...) was crushed by neural networks when having to deal with real world images.

This seemed like a good answer at first. But on further thought, images on the whole really do seem to have quite a bit more standard structure / "grammar" to exploit compared to arbitrary time-series. Many images are of the world, where there is gravity so you might see preponderance of blobs at the bottom, or the repetitive types like people, animals, faces, eyes. Wildly abstract images still have some continuity,…

> It's less clear to me how mixing sensor data / financial data / anything else together could be helpful.

Because many of these have the same underlying causal structures - humans doing things, weather correlations, holidays.

Well studied behavioral stuff like "the stock market takes the stairs up and the elevator down" which is not really captured by "traditional" modelling tools.

I'm sure people will be doing mechanical interpretation on these models to extract what they pattern match for prediction.

Re: Google's 200M-parameter time-series foundation model with 16k context

#77

Post by jofzar »

FartyMcFarter wrote: post #53
jofzar wrote: post #34

Earlier quoted context omitted.

I mean it's super obvious, it's directly tied to scrubs popularity. New season of scrubs = new war in the middle east.

Wow, I didn't know. Thank you! Such a great show.

It's suprisingly good, like it's it's 100% worth watching if you liked scrubs.

Re: Google's 200M-parameter time-series foundation model with 16k context

#78

Post by torginus »

dist-epoch wrote: post #76
robot-wrangler wrote: post #74

Earlier quoted context omitted.

This seemed like a good answer at first. But on further thought, images on the whole really do seem to have quite a bit more standard structure / "grammar" to exploit compared to arbitrary time-series. Many images are of the world, where there is gravity so you might see preponderance of blobs at the bottom, or the repetitive types like people, animals, faces, eyes. Wildly abstract images still have some continuity,…

> It's less clear to me how mixing sensor data / financial data / anything else together could be helpful. Because many of these have the same underlying causal structures - humans doing things, weather correlations, holidays. Well studied behavioral stuff like "the stock market takes the stairs up and the elevator down" which is not really captured by "traditional" modelling tools. I'm sure people will be doing mech…

Personally, coming from an EE background and not finance or statistics, I would go about identifying these patterns with an Signals & Systems toolbox, like systems identification, various matched filters/classifiers.

This might be a totall wrong approach, but I think it might make sense to try to model a matched filter based on previous stock selloff/bullrun trigger events, and then see if the it has any predictive ability, likewise the market reaction seems to be usually some sort of delayed impulse-like activity, with the whales reacting quickly, and then a distribution of less savvy investors following up the signal with various delays.

I'm sure other smarter people have explored this approach much more in depth before me.

Re: Google's 200M-parameter time-series foundation model with 16k context

#79

Post by guntars »

jcelerier wrote: post #63
teruakohatu wrote: post #6

Earlier quoted context omitted.

What is not generally understood is that these models don’t predict egg prices or inflation in Italy. They decompose a time series into trends, seasonality and residuals. That’s what they are actually modelling. They cannot predict wars in the Middle East influencing inflation unless there is a seasonal pattern(s).

> They cannot predict wars in the Middle East influencing inflation unless there is a seasonal pattern(s). well...

The Middle East war season is upon us once again

Re: Google's 200M-parameter time-series foundation model with 16k context

#80

Post by carschno »

amelius wrote: post #57
teruakohatu wrote: post #6

Earlier quoted context omitted.

What is not generally understood is that these models don’t predict egg prices or inflation in Italy. They decompose a time series into trends, seasonality and residuals. That’s what they are actually modelling. They cannot predict wars in the Middle East influencing inflation unless there is a seasonal pattern(s).

What makes these models different from models used for e.g. audio? Or other low-dimensional time domain signals?

You could abstract speech or other audio as a series of sounds, where time is indeed a factor. Speech, however, has patterns that are more similar to written language than to seasonal patterns that are typically assumed in time series. While trained on different data, the architecture of TimesFM is actually similar to LLMs. But not identical, as pointed out at https://research.google/blog/a-decoder-only-foundation-model...:

> Firstly, we need a multilayer perceptron block with residual connections to convert a patch of time-series into a token that can be input to the transformer layers along with positional encodings (PE).

> [...]

> Secondly, at the other end, an output token from the stacked transformer can be used to predict a longer length of subsequent time-points than the input patch length, i.e., the output patch length can be larger than the input patch length.

Post reply on HN