Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GraphCast: AI model for weather forecasting

deepmind.google

Post reply on HN
Print view View original

171–180 of 310 posts

Re: GraphCast: AI model for weather forecasting

#171

Post by kubiton »

Guestmodinfo wrote: post #166
meteo-jeff wrote: post #20

In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…

I always suspect that they don't tell me the actual temperature. Maybe I am totally wrong but I suspect. I need to get my own physical thermometer not the digital one in my room and outside my house and have a camera focussed on it. So that later I can speed up the video and see how much the weather varied the previous night.

What? Why?

Re: GraphCast: AI model for weather forecasting

#173

Post by meteo-jeff »

Omnipresent wrote: post #159
meteo-jeff wrote: post #20

In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…

This is great. I am very curious about the architectural decisions you've taken here. Is there a blog post / article about them? 80 yrs of historical data -- are you storing that somewhere in PG and the APIs are just fetching it? If so, what indices have you set up to make APIs fetch faster etc. I just fetched 1960 to 2022 in about 12 secs.

Traditional database systems struggle to handle gridded data efficiently. Using PG with time-based indices is memory and storage extensive. It works well for a limited number of locations, but global weather models at 9-12 km resolution have 4 to 6 million grid-cells.

I am exploiting on the homogeneity of gridded data. In a 2D field, calculating the data position for a graphical coordinate is straightforward. Once you add time as a third dimension, you can pick any timestamp at any point on earth. To optimize read speed, all time steps are stored sequentially on disk in a rotated/transposed OLAP cube.

Although the data now consists of millions of floating-point values without accompanying attributes like timestamps or geographical coordinates, the storage requirements are still high. Open-Meteo chunks data into small portions, each covering 10 locations and 2 weeks of data. Each block is individually compressed using an optimized compression scheme.

While this process isn't groundbreaking and is supported by file systems like NetCDF, Zarr, or HDF5, the challenge lies in efficiently working with multiple weather models and updating data with each new weather model run every few hours.

You can find more information here: https://openmeteo.substack.com/i/64601201/how-data-are-store...

Re: GraphCast: AI model for weather forecasting

#174

Post by 3abiton »

meteo-jeff wrote: post #20

In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…

Are multiple data sources supported?

Re: GraphCast: AI model for weather forecasting

#175

Post by brna »

meteo-jeff wrote: post #20

In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…

Hi Jeff, Great work, Respect!

I just hit the daily limit on the second request at https://climate-api.open-meteo.com/v1/climate

I see the limit for non-commercial use should be "less than 10.000 daily API calls". Technically 2 is less than 10.000, I know, but still I decided to drop you a comment. :)

Re: GraphCast: AI model for weather forecasting

#176

Post by EricLeer »

I am in the power forecasting domain, where weather forecasts are one of the most important inputs. What I find surprising is that with all the papers and publications from google in the past years, there seems to be no way to get access to these forecasts! We've now evaluated numerous of the ai weather forecasting startups that are popping up everywhere and so far for all of them their claims fall flat on their face when you actually start comparing their quality in a production setting next to the HRES model from ECMWF.

Re: GraphCast: AI model for weather forecasting

#177

Post by EricLeer »

RandomWorker wrote: post #131
dil8 wrote: post #103

Earlier quoted context omitted.

Graph neural networks are deep learning models that trained on graph data.

Do you have any resources where I could learn more about these networks?

See for instance the pytorch geometric [1] package, which is the main implementation in pytorch. They also link to some papers there that might explain you more.

[1] https://pytorch-geometric.readthedocs.io/en/latest/

Re: GraphCast: AI model for weather forecasting

#179

Post by jjp »

Fatnino wrote: post #116
meteo-jeff wrote: post #20

In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…

Is there somewhere to see historical forecasts? So not "the weather on 25 December 2022 was such and such" but rather "on 20 December 2022 the forecast for 25 December 2022 was such and such"

Are you thinking something like https://www.forecastadvisor.com/?

Re: GraphCast: AI model for weather forecasting

#180

Post by mjs »

berniedurfee wrote: post #156
Fatnino wrote: post #116

Earlier quoted context omitted.

Is there somewhere to see historical forecasts? So not "the weather on 25 December 2022 was such and such" but rather "on 20 December 2022 the forecast for 25 December 2022 was such and such"

I’ve always wanted to see something like that. I always wonder if forecasts are a coin flip beyond a window of a few hours.

Looks like https://sites.research.google/weatherbench/ attempts to "benchmark" different forecast models/systems.

They're very cautious about naming a "best" model though!

> Weather forecasting is a multi-faceted problem with a variety of use cases. No single metric fits all those use cases. Therefore,it is important to look at a number of different metrics and consider how the forecast will be applied.

Post reply on HN