Live data from Hacker News

GraphCast: AI model for weather forecasting

deepmind.google

151–160 of 310 posts

Re: GraphCast: AI model for weather forecasting

#151
post #79

Are there any experts around that can chime in on the possible impacts of this technology if widely adopted?

It will get adopted, eventually we will have more accurate weather forecasts. Thats good for anything that depends on weather - e.g. energy consumption and production, transportation costs...

Re: GraphCast: AI model for weather forecasting

#152

From what I can tell from reading & based off https://colab.research.google.com/github/deepmind/graphcast/... , one needs access to ECMWF Era5 or HRES data-sets or something similar to be able to run and use this model. Unknown what licensing options ECMWF offers for Era5, but to use this model in any live fashion, I think one is probably going to need a small fortune. Maybe some other dataset can be adapted (likely…

You can get some of the historical data also from here: https://cloud.google.com/storage/docs/public-datasets/era5 (if the official API is too slow. )

To use the data in live fashion I think you would need to get license from ECMWF...

Re: GraphCast: AI model for weather forecasting

#153

In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…

There is also https://github.com/google-research/weatherbench2 which has baselines of numerical weather models.

Re: GraphCast: AI model for weather forecasting

#154
When will we have enough data that we will be able to apply this to everything? Imagine a model that can predict all kinds of trends - what new consumer good will be the most likely to succeed, where the next war is most likely to break out, who will win the next election, which stocks are going to break out. One gigantic black box with a massive state, with input from everything - planning approvals, social media posts, solar activity, air travel numbers, seismic readings, TV feeds.

Re: GraphCast: AI model for weather forecasting

#156

In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…

Is there somewhere to see historical forecasts? So not "the weather on 25 December 2022 was such and such" but rather "on 20 December 2022 the forecast for 25 December 2022 was such and such"

I’ve always wanted to see something like that. I always wonder if forecasts are a coin flip beyond a window of a few hours.

Re: GraphCast: AI model for weather forecasting

#157

Earlier quoted context omitted.

Is there somewhere to see historical forecasts? So not "the weather on 25 December 2022 was such and such" but rather "on 20 December 2022 the forecast for 25 December 2022 was such and such"

I’ve always wanted to see something like that. I always wonder if forecasts are a coin flip beyond a window of a few hours.

I know at a minimum that hurricane forecasts have gotten significantly better over time. We can now

https://www.nhc.noaa.gov/verification/verify5.shtml

Our 96 hour projections are as accurate today as the 24 hour projections were in 1990.

Re: GraphCast: AI model for weather forecasting

#158
post #12

This is fascinating: > For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the weather 6 hours in the future. This process can then be rolled forward in 6-hour increments to provide state-of-the-art forecasts up to 10 days in advance.

Weather is markovian

That is not strictly true. The weather at time t0 may affect non-weather phenomena at time t1 (e.g. traffic), which in turn may affect weather at time t2.

Furthermore, a predictive model is not working with a complete picture of the weather, but rather some limited-resolution measurements. So, even ignoring non-weather, there may be local weather phenomena detected at time t0, escaping detection at time t1, but still affecting weather at time t2.

Re: GraphCast: AI model for weather forecasting

#159

In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…

This is great. I am very curious about the architectural decisions you've taken here. Is there a blog post / article about them? 80 yrs of historical data -- are you storing that somewhere in PG and the APIs are just fetching it? If so, what indices have you set up to make APIs fetch faster etc. I just fetched 1960 to 2022 in about 12 secs.
Post reply on HN