31–40 of 310 posts
To the best of my knowledge, poor weather (especially wind shear/microbursts) are one of the most dangerous things possible in aviation. Is there any chance, or plans, to implement this in the current weather radars in planes?
Earlier quoted context omitted.
Not to take away from the excitement but ML weather prediction builds upon the years of data produced by numerical models on supercomputers. It cannot do anything without that computation and its forecasts are dependent on the quality of that computation. Ensemble models are already used to quantify uncertainty (it’s referenced in their paper). But it is exciting that they are able to recognize patterns in multi year…
I don't get this. Surely past and real weather should be the input training data, not the output of numerical modeling?
We have dozens of complementary and contradictory sources of weather information. Different types of satellites measuring EM radiation in different bands, weather stations, terrestrial weather radars, buoys, weather balloons... it's a massive hodge-podge of different systems measuring different things in an uncoordinated fashion.
Today, it's not really practical to assemble that data and directly feed it into an AI system. So the state-of-the-art in AI weather forecasting involves using an intermediate representation - "reanalysis" datasets which apply a sophisticated physics based weather model to assimilate all of these data sets into a single, self-consistent 3D and time-varying record of the state of the atmosphere. This data is the unsung hero of the weather revolution - just as the WMO's coordinated synoptic time observations for weather balloons catalyzed effective early numerical weather prediction in the 50's and 60's, accessible re-analysis data - and the computational tools and platforms to actually work with these peta-scale datasets - has catalyzed the advent of "pure AI" weather forecasting systems.
Unknown what licensing options ECMWF offers for Era5, but to use this model in any live fashion, I think one is probably going to need a small fortune. Maybe some other dataset can be adapted (likely at great pain)...
In case someone is looking for historical weather data for ML training and prediction, I created an open-source weather API which continuously archives weather data. Using past and forecast data from multiple numerical weather models can be combined using ML to achieve better forecast skill than any individual model. Because each model is physically bound, the resulting ML model should be stable. See: https://open-me…
I've never studied weather forecasting, but I can't say I'm surprised. All of these models, AFAICT, are based on the "state" of the weather, but "state" deserves massive scare quotes: it's a bunch of 2D fields (wind speed, pressure, etc) -- note the 2D . Actual weather dynamics happen in three dimensions, and three dimensional land features, buildings, etc as well as gnarly 2D surface phenomena (ocean surface tempera…
To call this impressive is an understatement. Using a single GPU, outperforms models that run on the world's largest super computers. Completely open sourced - not just model weights. And fairly simple training / input data. > ... with the current version being the largest we can practically fit under current engineering constraints, but which have potential to scale much further in the future with greater compute re…
It builds on top of supercomputer model output and does better at the specific task of medium term forecasts. It is a kind of iterative refinement on the data that supercomputers produce — it doesn’t supplant supercomputers. In fact the paper calls out that it has a hard dependency on the output produced by supercomputers.
To call this impressive is an understatement. Using a single GPU, outperforms models that run on the world's largest super computers. Completely open sourced - not just model weights. And fairly simple training / input data. > ... with the current version being the largest we can practically fit under current engineering constraints, but which have potential to scale much further in the future with greater compute re…
It builds on top of supercomputer model output and does better at the specific task of medium term forecasts. It is a kind of iterative refinement on the data that supercomputers produce — it doesn’t supplant supercomputers. In fact the paper calls out that it has a hard dependency on the output produced by supercomputers.
I didn't read the paper but the linked post seems to say otherwise? It mentions it used the supercomputer output to impute data during training. But for prediction it just needs:
> For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the weather 6 hours in the future. This process can then be rolled forward in 6-hour increments to provide state-of-the-art forecasts up to 10 days in advance.
This is fascinating: > For inputs, GraphCast requires just two sets of data: the state of the weather 6 hours ago, and the current state of the weather. The model then predicts the weather 6 hours in the future. This process can then be rolled forward in 6-hour increments to provide state-of-the-art forecasts up to 10 days in advance.
I don't know much about weather prediction, but if a model can improve the state of the art only with that data as input, my conclusion is that previous models were crap... or am I missing something?