Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GraphCast: AI model for weather forecasting

deepmind.google

Post reply on HN
Print view View original

41–50 of 310 posts

Re: GraphCast: AI model for weather forecasting

#41

Post by serjester »

silveraxe93 wrote: post #38
wenc wrote: post #25

Earlier quoted context omitted.

It builds on top of supercomputer model output and does better at the specific task of medium term forecasts. It is a kind of iterative refinement on the data that supercomputers produce — it doesn’t supplant supercomputers. In fact the paper calls out that it has a hard dependency on the output produced by supercomputers.

Could you point me to the part where it says it depends on supercomputer output? I didn't read the paper but the linked post seems to say otherwise? It mentions it used the supercomputer output to impute data during training. But for prediction it just needs: > For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the…

You can read about it more in their paper. Specifically page 36. Their dataset, ERA5, is created using a process called reanalysis. It combines historical weather observations with modern weather models to create a consistent record of past weather conditions.

https://storage.googleapis.com/deepmind-media/DeepMind.com/B...

Re: GraphCast: AI model for weather forecasting

#42

Post by meteo-jeff »

mdbmdb wrote: post #34
meteo-jeff wrote: post #20

In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…

Is it able to provide data on extreme events. Say, the current and potential path of a hurricane? similar to .kml that NOAA provides

Extreme weather is predicted by numerical weather models. Correctly representing hurricanes has driven development on the NOAA GFS model for centuries.

Open-Meteo focuses on providing access to weather data for single locations or small areas. If you look at data for coastal areas, forecast and past weather data will show severe winds. Storm tracks or maps are not available, but might be implemented in the future.

Re: GraphCast: AI model for weather forecasting

#43

Post by silveraxe93 »

serjester wrote: post #41
silveraxe93 wrote: post #38

Earlier quoted context omitted.

Could you point me to the part where it says it depends on supercomputer output? I didn't read the paper but the linked post seems to say otherwise? It mentions it used the supercomputer output to impute data during training. But for prediction it just needs: > For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the…

You can read about it more in their paper. Specifically page 36. Their dataset, ERA5, is created using a process called reanalysis. It combines historical weather observations with modern weather models to create a consistent record of past weather conditions. https://storage.googleapis.com/deepmind-media/DeepMind.com/B...

Ah nice. Thanks!

Re: GraphCast: AI model for weather forecasting

#44

Post by tash9 »

user_7832 wrote: post #31

(If someone with knowledge or experience can chime in, please feel free.) To the best of my knowledge, poor weather (especially wind shear/microbursts) are one of the most dangerous things possible in aviation. Is there any chance, or plans, to implement this in the current weather radars in planes?

If you're talking about small scale phenomena (less than 1km), then this wouldn't help other than to be able to signal when the conditions are such that these phenomena are more likely to happen.

Re: GraphCast: AI model for weather forecasting

#45

Post by whatever1 »

wenc wrote: post #25
serjester wrote: post #16

To call this impressive is an understatement. Using a single GPU, outperforms models that run on the world's largest super computers. Completely open sourced - not just model weights. And fairly simple training / input data. > ... with the current version being the largest we can practically fit under current engineering constraints, but which have potential to scale much further in the future with greater compute re…

It builds on top of supercomputer model output and does better at the specific task of medium term forecasts. It is a kind of iterative refinement on the data that supercomputers produce — it doesn’t supplant supercomputers. In fact the paper calls out that it has a hard dependency on the output produced by supercomputers.

So best case scenario we can avoid some computation for inference, assuming that historical system dynamics are still valid. This model needs to be constantly monitored by full scale simulations and rectified over time.

Re: GraphCast: AI model for weather forecasting

#46

Post by goosinmouse »

counters wrote: post #32
boxed wrote: post #29

Earlier quoted context omitted.

I don't get this. Surely past and real weather should be the input training data, not the output of numerical modeling?

Well, what is "real weather data?" We have dozens of complementary and contradictory sources of weather information. Different types of satellites measuring EM radiation in different bands, weather stations, terrestrial weather radars, buoys, weather balloons... it's a massive hodge-podge of different systems measuring different things in an uncoordinated fashion. Today, it's not really practical to assemble that dat…

Great comment, thank you for sharing your insights. I don't think many people truly understand just how massive these weather models are and the sheer volume of data assimilation work that's been done for decades to get us to this point today.

I always have a lot of ideas about using AI to solve very small scale weather forecasting issues, but there's just so much to it. It's always a learning experience for sure.

Re: GraphCast: AI model for weather forecasting

#49

Post by WhitneyLand »

amluto wrote: post #27

I've never studied weather forecasting, but I can't say I'm surprised. All of these models, AFAICT, are based on the "state" of the weather, but "state" deserves massive scare quotes: it's a bunch of 2D fields (wind speed, pressure, etc) -- note the 2D . Actual weather dynamics happen in three dimensions, and three dimensional land features, buildings, etc as well as gnarly 2D surface phenomena (ocean surface tempera…

How does it make sense to say this is something you’ve “never studied”, followed by how they “ought to be” doing it better?

It also seems like some of your facts differ from theirs, may I ask how far you read into the paper?

Re: GraphCast: AI model for weather forecasting

#50

Post by danielmarkbruce »

carbocation wrote: post #37
wenc wrote: post #25

Earlier quoted context omitted.

It builds on top of supercomputer model output and does better at the specific task of medium term forecasts. It is a kind of iterative refinement on the data that supercomputers produce — it doesn’t supplant supercomputers. In fact the paper calls out that it has a hard dependency on the output produced by supercomputers.

I don't understand why this is downvoted. This is a classic thing to do with deep learning: take something that has a solution that is expensive to compute, and then train a deep learning model from that. And along the way, your model might yield improvements, too, and you can layer in additional features, interpolate at finer-grained resolution, etc. If nothing else, the forward pass in a deep learning model is almo…

Because "iterative refinement" is sort of wrong. It's not a refinement and it's not iterative. It's an entirely different model to physical simulation which works entirely differently and the speed up is order of magnitude.

Building a statistical model to approximate a physical process isn't a new idea for sure.. there are literally dozens of them for weather.. the idea itself isn't really even iterative, it's the same idea... but it's all in the execution. If you built a model to predict stock prices tomorrow and it generated 1000% pa, it wouldn't be reasonable for me to call it iterative.

Post reply on HN