Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google says AI weather model masters 15-day forecast

phys.org

Post reply on HN
Print view View original

61–70 of 216 posts

Re: Google says AI weather model masters 15-day forecast

#61

Post by staunton »

mjburgess wrote: post #6
bad_haircut72 wrote: post #5

Obviously this team knows way more about this donain than me but I have to ask, wouldnt this only be able to predict weather which is in line with past weather patterns/indicators? I can imagine a weather analyst might be able to see "between the data" and recognise when some anomaly might be brewing, but an AI model would not

Yup. There's fundamentally no way to automatically verify predictive accuracy, the assumptions in ML papers about such accuracy is (the almost always false presumption) that there are no distributional shifts in the data generating process. Here, since weather certainly changes its fundamental patterns over time, there is no way of reliably predicting out-sample performance. It's highly likely that in 1-2 years time…

Ultimately, a hybrid approach might win out in the end.

Use equations for what we know, use machine learning to fit parameters in the equations, as well as terms we don't know.

Prople nowadays can solve differential equations where some terms are "neural networks" and train those networks on data while numerically solving the equation. (some people call it "neural differential equations", if you want a search query to start)

Re: Google says AI weather model masters 15-day forecast

#62

Post by kolinko »

atonse wrote: post #7

This is great from a practical standpoint (being able to predict weather), but does it actually improve our understanding of the weather, or WHY those predictions are better? That is my issue with some of these AI advances. With these, we won't have actually gotten better at understanding the weather patterns, since it's all just a bunch of weights which nobody really understands.

Define "our understanding". With complex / chaotic systems there sometimes are no higher level laws that govern them - all we have is just modeling and prediction.

Re: Google says AI weather model masters 15-day forecast

#63

Post by dgrin91 »

n4r9 wrote: post #57
BadHumans wrote: post #56

I'm friends with a meteorologist and the 15+ day forecast is the bane of their existence because you can't accurately forecast beyond a week so I would love to know how they are measuring accuracy. The article doesn't say and I know the paper is going to go over my head.

I would guess that everyday they're comparing the current weather against the forecast from 15 days ago. Not a lot of data points to be sure, but perhaps enough to have confidence of very high accuracy.

Alternatively the can do back testing - using historical data they feed a subset into the predictor, then compare it's predictions to actual history

Re: Google says AI weather model masters 15-day forecast

#64

Post by [deleted] »

dylan604 wrote: post #12
xnx wrote: post #3

Earlier quoted context omitted.

New model. Much better than the last one.

It's so new that it has no long term track record. I want to see how its record looks after say 1/3/5 years. Any 15 day forecast is just too far into the future for me to take seriously. Even the 3 day forecast is loose at best. By the time it reaches 7-10 days, some of the forecast is completely off when it reaches the 3 day window. "But DeepMind said GenCast surpassed the precision of the center's forecasts in more…

[deleted]

Re: Google says AI weather model masters 15-day forecast

#65

Post by mjburgess »

wcoenen wrote: post #38
mjburgess wrote: post #6

Earlier quoted context omitted.

Yup. There's fundamentally no way to automatically verify predictive accuracy, the assumptions in ML papers about such accuracy is (the almost always false presumption) that there are no distributional shifts in the data generating process. Here, since weather certainly changes its fundamental patterns over time, there is no way of reliably predicting out-sample performance. It's highly likely that in 1-2 years time…

Weather models can't compute some of the important physics that happens at sub-grid scale, and use various tricks to deal with that. How sure are you that these traditional models aren't also heavily tuned on past data to maximize performance? Perhaps they will also perform badly "out of distribution".

Causal physical models have no distributional requirements, so they are not sensitive to distribution shifts. ie., A causal model accounts for all possible distributions.

The kind of risk with a causal model is that the model itself, of reality, is incorrect. This is a different kind of risk than there being 'no model at all' as in the case of curve-fitting over historical data, which is radically more fragile to expected shifts.

In general, we're quite good at knowing the limitations of causal models, ie., specifying model risk here is much easier. You even, exactly, point out a known problem in the modelling which is impossible to state for an AI model.

Since the AI model is just a weak induction, there are no terms/structures/etc. within that model which can be analysed to understand what parts of it are sensitive to what aspects of the system from which the data was taken.

All we can say is, we know that in general, train/test distributions have to be "nearly exactly the same" for any of these methods to show anything like cross-val levels of accuracy. So we can very confidently predict when we know train/test wont be the same, that this is a mumbojumbo metric.

Indeed, in the vast majority of common ML examples you can, right now, just go and look at real out-sample data collected later than the "common dataset" and you'll find the val accuracy is random or worse-than-random despite arbitarily high cross-val scores.

The dataset which drives me most up-the-wall on this is house price datasets, or pricing datasets in general. Prices, generally, follow geometric brownian motion and nearly all ML models are extremelhy bad at modelling prices. So it's basically pseudoscience whenver anyone uses these datasets to demonstrate anything, esp. predictive accuracy.

Re: Google says AI weather model masters 15-day forecast

#66

Post by raincole »

atonse wrote: post #7

This is great from a practical standpoint (being able to predict weather), but does it actually improve our understanding of the weather, or WHY those predictions are better? That is my issue with some of these AI advances. With these, we won't have actually gotten better at understanding the weather patterns, since it's all just a bunch of weights which nobody really understands.

A fisherman was relaxing on a sunny beach, enjoying the day with his fishing line in the water. A businessman, stressed from work, walked by and criticized him for not working harder.

"If you worked more, you'd catch more fish," the businessman said.

"And what would my reward be?" asked the fisherman with a smile.

"You could earn money, buy bigger nets, and catch even more fish!" the businessman replied.

"And then what?" the fisherman asked again.

"Then, you could buy a boat and catch even larger hauls!" said the businessman.

"And after that?"

"You could buy more boats, hire a crew, and eventually own a fleet, freeing you to relax forever!"

The fisherman, still smiling, replied, "But isn't that what I'm already doing?"

Re: Google says AI weather model masters 15-day forecast

#67

Post by fancyfredbot »

It takes 8 minutes to produce a 15 day forecast. That's actually quite a long time for an AI model. I should probably read the paper to find out why but does anyone know? Is the model predicting the weather in 10 minutes time and just run iteratively 2000 times for a 14 day forecast?

Re: Google says AI weather model masters 15-day forecast

#68

Post by SpaceManNabs »

jacobgkau wrote: post #23
SpaceManNabs wrote: post #9

Earlier quoted context omitted.

do you want a causal weather model or one that can predict the weather well?

When climate change occurs (which it is), we're going to want a causal one so we can actually make a forecast instead of predicting only based on past (unchanged) data.

ahhhh i was hoping someone wouldn't mention this. i wanted to add a caveat but it made my comment look ugly lol.

but like someone else says weather and climate models forecast on different scales and for different purposes usually.

Re: Google says AI weather model masters 15-day forecast

#69

Post by SpaceManNabs »

hotstickyballs wrote: post #26
jacobgkau wrote: post #23

Earlier quoted context omitted.

When climate change occurs (which it is), we're going to want a causal one so we can actually make a forecast instead of predicting only based on past (unchanged) data.

What does a causal weather model even mean?

Doesn't exactly answer your question, but maybe this link will give enough intuition so that i can be lazy and hand-wave away an answer

https://en.wikipedia.org/wiki/The_Book_of_Why#Chapter_1:_The...

Re: Google says AI weather model masters 15-day forecast

#70

Post by joe_the_user »

raincole wrote: post #66
atonse wrote: post #7

This is great from a practical standpoint (being able to predict weather), but does it actually improve our understanding of the weather, or WHY those predictions are better? That is my issue with some of these AI advances. With these, we won't have actually gotten better at understanding the weather patterns, since it's all just a bunch of weights which nobody really understands.

A fisherman was relaxing on a sunny beach, enjoying the day with his fishing line in the water. A businessman, stressed from work, walked by and criticized him for not working harder. "If you worked more, you'd catch more fish," the businessman said. "And what would my reward be?" asked the fisherman with a smile. "You could earn money, buy bigger nets, and catch even more fish!" the businessman replied. "And then wh…

But AI models in no way free you from work and they are hardly relaxing.

If anything, they "free" people from understanding but that's an activity that many if not most people value highly.

Post reply on HN