Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GraphCast: AI model for weather forecasting

deepmind.google

Post reply on HN
Print view View original

111–120 of 310 posts

Re: GraphCast: AI model for weather forecasting

#111

Post by miserableuse »

counters wrote: post #90
miserableuse wrote: post #80

Does anybody know if its possible to initialize the model using GFS initial conditions used for the GFS HRES model? If so, where can I find this file and how can I use it? Any help would be greatly appreciated!

You can try, but other models in this class have struggled when initialized using model states pulled from other analysis systems. ECMWF publishes a tool that can help bootstrap simple inference runs with different AI models [1] (they have plugins for several). You could write a tool that re-maps a GDAS analysis to "look like" ERA-5 or IFS analysis, and then try feeding it into GraphCast. But YMMV if the integration…

Thank you for your response. Are these ML models initialized by gridded initial conditions measurements (such as the GDAS pointed out) or by NWP model forecast results (such as hour-zero forecast from the GFS)? Or are those one and the same?

Re: GraphCast: AI model for weather forecasting

#112

Post by sosodev »

vosper wrote: post #85
pyb wrote: post #83

Curious. How can AI/ML perform on a problem that is, as far as I understand, inherently chaotic / unpredictable ? It sounds like a fundamental contradiction to me.

Weather isn’t fundamentally unpredictable. We predict weather with a fairly high degree of accuracy (for most practical uses), and the accuracy getting better all the time. https://scijinks.gov/forecast-reliability

I'm kinda surprised that this government science website doesn't seem to link sources. I'd like to read the research to understand how they're measuring the accuracy.

Re: GraphCast: AI model for weather forecasting

#113

Post by counters »

miserableuse wrote: post #111
counters wrote: post #90

Earlier quoted context omitted.

You can try, but other models in this class have struggled when initialized using model states pulled from other analysis systems. ECMWF publishes a tool that can help bootstrap simple inference runs with different AI models [1] (they have plugins for several). You could write a tool that re-maps a GDAS analysis to "look like" ERA-5 or IFS analysis, and then try feeding it into GraphCast. But YMMV if the integration…

Thank you for your response. Are these ML models initialized by gridded initial conditions measurements (such as the GDAS pointed out) or by NWP model forecast results (such as hour-zero forecast from the GFS)? Or are those one and the same?

They're more-or-less the same thing.

Re: GraphCast: AI model for weather forecasting

#114

Post by comment_ran »

meteo-jeff wrote: post #20

In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…

How about https://pirateweather.net/en/latest/ ?

Does anyone have a compare this API with the latest API we have here?

Re: GraphCast: AI model for weather forecasting

#115

Post by bberenberg »

stabbles wrote: post #102

If you live in a country where local, short-term rain / shower forecast is essential (like [1] [2]), it's funny to see how incredibly bad radar forecast is. There are really convenient apps that show an animated map with radar data of rain, historical data + prediction (typically). The prediction is always completely bonkers. You can eyeball it better. No wonder "AI" can improve that. Even linear extrapolation is bet…

Interesting that you say this. I spent in month in AMS 7-8 years ago and buienradar was accurate down to the minute when I used it. Has something changed?

Re: GraphCast: AI model for weather forecasting

#116

Post by Fatnino »

meteo-jeff wrote: post #20

In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…

Is there somewhere to see historical forecasts?

So not "the weather on 25 December 2022 was such and such" but rather "on 20 December 2022 the forecast for 25 December 2022 was such and such"

Re: GraphCast: AI model for weather forecasting

#117

Post by supdudesupdude »

stabbles wrote: post #102

If you live in a country where local, short-term rain / shower forecast is essential (like [1] [2]), it's funny to see how incredibly bad radar forecast is. There are really convenient apps that show an animated map with radar data of rain, historical data + prediction (typically). The prediction is always completely bonkers. You can eyeball it better. No wonder "AI" can improve that. Even linear extrapolation is bet…

Funny to mention. None of the AI forecasts can actually predict precip. None of them mention this and i assume everyone thinks this means the rain forecasts are better. Nope just temperature and humidity and wind. Important but come on, it's a bunch of shite

Re: GraphCast: AI model for weather forecasting

#118

Post by meteo-jeff »

comment_ran wrote: post #114
meteo-jeff wrote: post #20

In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…

How about https://pirateweather.net/en/latest/ ? Does anyone have a compare this API with the latest API we have here?

Both APIs use weather models from NOAA GFS and HRRR, providing accurate forecasts in North America. HRRR updates every hour, capturing recent showers and storms in the upcoming hours. PirateWeather gained popularity last year as a replacement for the Dark Sky API when Dark Sky servers were shut down.

With Open-Meteo, I'm working to integrate more weather models, offering access not only to current forecasts but also past data. For Europe and South-East Asia, high-resolution models from 7 different weather services improve forecast accuracy compared to global models. The data covers not only common weather variables like temperature, wind, and precipitation but also includes information on wind at higher altitudes, solar radiation forecasts, and soil properties.

Using custom compression methods, large historical weather datasets like ERA5 are compressed from 20 TB to 4 TB, making them accessible through a time-series API. All data is stored in local files; no database set-up required. If you're interested in creating your own weather API, Docker images are provided, and you can download open data from NOAA GFS or other weather models.

Re: GraphCast: AI model for weather forecasting

#119

Post by supdudesupdude »

Gys wrote: post #101
dist-epoch wrote: post #72

Earlier quoted context omitted.

There are models which take as input both global forecasts and local ones, and which then can transpose a global forecast into a local one. National weather institutions sometimes do this, since they don't have the resources to run a massive supercomputer model.

Interesting. So what I am looking for is probably an even more scaled down version? Or something that runs in the cloud with an api to upload my local measurements.

Hate to break it but one weather station wont improve a forecast? What are they supposed to do? Ignore the output of our state of the art forecast models and add an if statement for your specific weather station??

Re: GraphCast: AI model for weather forecasting

#120

Post by swells34 »

mdbmdb wrote: post #55
meteo-jeff wrote: post #42

Earlier quoted context omitted.

Extreme weather is predicted by numerical weather models. Correctly representing hurricanes has driven development on the NOAA GFS model for centuries. Open-Meteo focuses on providing access to weather data for single locations or small areas. If you look at data for coastal areas, forecast and past weather data will show severe winds. Storm tracks or maps are not available, but might be implemented in the future.

Appreciate the response. Do you know of any services that provide what I described in the previous comments? I'm specifically interested in extreme weather conditions and their visual representation (hurricanes, tornados, hails etc.) with API capabilities

Go to: nhc.noaa.gov/gis There's a list of data and products with kmls and kmzs and geojsons and all sorts of stuff. I haven't actually used the API for retrieving these, but NOAA has a pretty solid track record with data dissemination.
Post reply on HN