Independently I was just wondering tonight why weather forecasting does not use some Machine Learning techniques as there is an abundance of training data. Wouldn't any number of regression ML algorithms find a more accurate prediction model? If Google did weather....
Global Forecast System source code
21–29 of 29 posts
Re: Global Forecast System source code
#22Independently I was just wondering tonight why weather forecasting does not use some Machine Learning techniques as there is an abundance of training data. Wouldn't any number of regression ML algorithms find a more accurate prediction model? If Google did weather....
Sometimes Machine Learning is the wrong approach. Direct mathematical modeling, if the underlying causal phenomenon are well known, bypasses the whole learning aspect of an explanatory system. Machine learning might help make sense of the residual noise that can't be accounted for with direct mathematical modeling. But at the level of sophistication that I've seen (predicting wind patterns using Finite Element Analysis methods mapped to topographical patterns fine-grained enough to model small buildings), I can't imagine any machine learning methods that could make any sense of the residual noise.
Re: Global Forecast System source code
#23Independently I was just wondering tonight why weather forecasting does not use some Machine Learning techniques as there is an abundance of training data. Wouldn't any number of regression ML algorithms find a more accurate prediction model? If Google did weather....
Sometimes Machine Learning is the wrong approach. Direct mathematical modeling, if the underlying causal phenomenon are well known, bypasses the whole learning aspect of an explanatory system. Machine learning might help make sense of the residual noise that can't be accounted for with direct mathematical modeling. But at the level of sophistication that I've seen (predicting wind patterns using Finite Element Analys…
I agree that Machine Learning is not always the right approach but wouldn't it be a relatively inexpensive experiment to see if it could be used for more accurate predictions? I'm surprised that topographical patterns are mapped that precisely when ASOS and AWOs stations are so far apart. Also the weather data from commercial airplanes is so far above the surface that modeling small buildings would be meaningless.
I heard a talk from ex-Googler climate.com (now Monsanto) guys. Their forecasts were accurate enough to build a very profitable insurance business. I bet they used Bayesian rather than FEM techniques.
Re: Global Forecast System source code
#24Independently I was just wondering tonight why weather forecasting does not use some Machine Learning techniques as there is an abundance of training data. Wouldn't any number of regression ML algorithms find a more accurate prediction model? If Google did weather....
ML techniques certainly are being used to predict the weather, maybe just not at NOAA or the Weather Channel. The weather derivative industry[0] is an $8 billion dollar industry, making it larger than the entire rocket launch industry (public and private combined). Just as there are Elon Musks in the rocket launch industry, there are maverick hedge funds out there making a killing by predicting the weather using the latest Big Data tools.
Re: Global Forecast System source code
#25This is really cool. If I knew the slightest bit about Fortran, I'd be distracted for weeks.
imo, fortran90 is about the easiest language to follow - if language is all that is stopping you (and you know some other language already), then you should just pile straight in.
From looking at some of the code, what is hard for me to follow is the 'overall view'. The documentation (of the model) looks quite nicely done, but there must be a huge amount of fluid dynamics etc etc knowledge, that would take me years to learn.
Re: Global Forecast System source code
#26Earlier quoted context omitted.
Sometimes Machine Learning is the wrong approach. Direct mathematical modeling, if the underlying causal phenomenon are well known, bypasses the whole learning aspect of an explanatory system. Machine learning might help make sense of the residual noise that can't be accounted for with direct mathematical modeling. But at the level of sophistication that I've seen (predicting wind patterns using Finite Element Analys…
I agree that Machine Learning is not always the right approach but wouldn't it be a relatively inexpensive experiment to see if it could be used for more accurate predictions? I'm surprised that topographical patterns are mapped that precisely when ASOS and AWOs stations are so far apart. Also the weather data from commercial airplanes is so far above the surface that modeling small buildings would be meaningless. I…
Is it that 'physics simulation' forecasting is best for short timescales (e.g. Presumably any short-term ML forecasting would use the physics simulation as it's main input, and then try to improve slightly on it (e.g. maybe you observe that in a particular small area, the rainfall is on average 10% more than the physics simulation, so your ML method would add 10%, giving you a slight forecast improvement)?
Re: Global Forecast System source code
#27Earlier quoted context omitted.
WCOSS (Tide & Gyre): IBM iDataPlex/Intel Sandy Bridge/Linux 208 trillion calculations/sec; 10,048 processing cores; 2,590 trillion bytes of storage [xref: http://www.emc.ncep.noaa.gov/GFS/doc.php#comopesys ]
Here's a really interesting read about the trials of upgrading to WCOSS: http://www.washingtonpost.com/blogs/capital-weather-gang/wp/...
I'd assumed that 'trials' meant 'snafu in a big software project'.
But actually it's about the butterfly effect making it impossible to get exact agreement between the old and new systems for forecasts longer than about 5 days.
Re: Global Forecast System source code
#28Fortran 90 is still the language computational stuff is made in today. I'm not convinced, however, that that choice is really for performance reasons over cultural reasons. Still, Fortran 90 is hell of a lot better than Fortran 77, which unfortunately is the language of choice for some of the people I'm being asked to work with.
If you look at Julia[0] they have quite a lot of performance benchmarks and you will see that Fortran is always at or near the top. What is really important, from a numerical code point of view, is that with Fortran it is easy to be at the top by operating on matrices and vectors etc. You do not have to torture your mind and think too much to optimize your code, "remove the IF in the loops" is often just what you need.
I am writing code in Fortran 77/90 every day, I really really enjoy it. Calling your Fortran code from Python is also so easy that you can really have clear cut between data management in Python and computational work in Fortran.
Re: Global Forecast System source code
#29Independently I was just wondering tonight why weather forecasting does not use some Machine Learning techniques as there is an abundance of training data. Wouldn't any number of regression ML algorithms find a more accurate prediction model? If Google did weather....
Probably because if it worked they'd already be doing it.