Live data from Hacker News

Technical milestone reached: global earth system simulations with 1.2 km resoln

mpimet.mpg.de

81–90 of 105 posts

Re: Technical milestone reached: global earth system simulations with 1.2 km resoln

#81
post #66

Earlier quoted context omitted.

Most of what they do in the Climate simulation arena uses fortran, this case not being an exception, reason being other language would be less efficient --- and thus more climate endangering emissions would be produced. Plotting and data wrangling is another story.

They don't use FORTRAN because of efficiency or emissions reductions lol, they use FORTRAN because they always did and the sort of scientists who do modelling rarely see any reason to upgrade their skills or tools. Programming is something scientists tend to study only as far as needed to get results that look right. This is how the most influential COVID model ended up being a 15,000 line student-quality C program w…

They definitely stick to it, and it might be that they build convenient narratives about it. Mind you that the energy use of climate simulations is not trivial, or do you think such machine can be fed with a typical electric installation from a regular office?

Re: Technical milestone reached: global earth system simulations with 1.2 km resoln

#82
post #71

Earlier quoted context omitted.

They're complete different classes of models. The modeling system in the linked article is a high-fidelity numerical simulation of the coupled Earth system. It's a giant PDE solver for Navier-Stokes applied to the Earth's atmospheres and oceans, coupled together with a great deal of additional physics simulation. The intent is to reproduce, in simulation, the Earth's atmospheric and oceans with the highest fidelity.…

You realize "facsimile" means "copy", right? You didn't explain how they are different.

It’s an example of a surrogate model. It’s an ML model trained on the output of large numerical simulations like the OP, rather doing the simulation itself.

Surrogate models are nice because they can emulate the output of the full fidelity calculation in a fraction of the runtime, but they typically are trained within a range of validity outside of which they cannot reliably extrapolate.

Re: Technical milestone reached: global earth system simulations with 1.2 km resoln

#83
post #65

Being curious about the implementation language. Viewing the code is not easy. If you guess Fortran, you might be right: (different ICON Project) "The infrastructure, ICON-Land, for this ICON-A land component has been newly designed in a Fortran2008 object-oriented, modular, and flexible way." https://mpimet.mpg.de/fileadmin/publikationen/Reports/WEB_Bz... Fortran alive and kicking: https://developer.nvidia.com/cuda-…

I had hoped for Julia.

For reference: https://clima.caltech.edu/

Re: Technical milestone reached: global earth system simulations with 1.2 km resoln

#84

The article talks about 1.2km horizontal resolution. Is it a 3d grid? If so what is the vertical resolution? Or is the vertical dimension integrated within 2d boxes?

Yes, these are fully 3D simulations. The vertical resolution is slightly harder to describe because models like these typically use more complicated vertical coordinates than just "meters above ground". E.g. a technical article on the ICON-A formulation [1] says that it uses a "hybrid sigma" coordinate - so something that follows a combination of pressure and terrain. For a very high-resolution simulation that resolves clouds, you'd need to crank this up to O(100 m) in the bottom of the model, but it can space out as you get higher in the atmosphere.

[1]: https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2017MS00...

Re: Technical milestone reached: global earth system simulations with 1.2 km resoln

#85
post #9

This is fantastic - would be really exciting to see this level of resolution making its way to global operational weather forecasts (currently at ~10km).

It's far too expensive with dubious impacts on forecast quality. Adaptive mesh approaches are far more suitable for high res global weather modeling... Why simulate the area under boring, dynamically unimportant areas?

At least three industries will be grateful for high-res wind predictions: aerospace, maritime and wind-generation.

Re: Technical milestone reached: global earth system simulations with 1.2 km resoln

#86
post #77
post #70

Earlier quoted context omitted.

How do you know they would lose a significant margin with Julia?

There were talks about it in the Climate modelling arena, and around a 10% slower was mentioned in some discussions.

Well, if that's not hard data!

Re: Technical milestone reached: global earth system simulations with 1.2 km resoln

#87
post #78

Earlier quoted context omitted.

Are numbers not objects? Is scientific notation not a way of expressing numbers that are too large or too small to be conceived or expressed in decimal form? Does a large database keep every object and every bit of data stored in it in RAM? How is it possible a fractal can be rendered in parts of the whole? Even older computers ordinarily can operate on 2^140 objects. user@decadeoldcomputer:~$ echo 2^140 |bc 13937965…

No computer can operate on 2**140 objects in any meaningful way, because no computer can even remember whether it's already done with one of the objects or not. Your example operates on a single object, a number.

Your response is a no true scotsman fallacy while also moving the goalposts, though what you've described is either a single-user single-task operating system, or you are conflating the limitations of current conventional single computer technology, which has already been solved with conventional clusters as well as quantum computing, and probably also with modern GPUs. Parallel computing has existed at least since the mid-1960s.

Re: Technical milestone reached: global earth system simulations with 1.2 km resoln

#88
post #86
post #77

Earlier quoted context omitted.

There were talks about it in the Climate modelling arena, and around a 10% slower was mentioned in some discussions.

Well, if that's not hard data!

Please can you specify what do you mean?

Re: Technical milestone reached: global earth system simulations with 1.2 km resoln

#89
post #71

Earlier quoted context omitted.

They're complete different classes of models. The modeling system in the linked article is a high-fidelity numerical simulation of the coupled Earth system. It's a giant PDE solver for Navier-Stokes applied to the Earth's atmospheres and oceans, coupled together with a great deal of additional physics simulation. The intent is to reproduce, in simulation, the Earth's atmospheric and oceans with the highest fidelity.…

You realize "facsimile" means "copy", right? You didn't explain how they are different.

It's a deep learning based emulator of a full-complexity NWP system, but it is far from a production-ready or operational technique.

Re: Technical milestone reached: global earth system simulations with 1.2 km resoln

#90

Earlier quoted context omitted.

Why do you think that we need "femtometer resolution" for "very precise" 100 day weather forecasting? What even is "very precise" 100 day weather forecasting? I think it's very amusing to do the math on how much memory would be required to run a crude primitive equation dycore over even the tiniest of domains at femtometer resolution :) > It's not as though this is part of a growing trend to abandon conventional weat…

> Why do you think that we need "femtometer resolution" for "very precise" 100 day weather forecasting? Due to sensitive dependence on initial conditons. Even using measurements at meter resolution will cause the accuracy of a forecast to begin to break down after only a few days. > What even is "very precise" 100 day weather forecasting? Anywhere from accurate to exact. > I think it's very amusing to do the math on…

> Due to sensitive dependence on initial conditons. Even using measurements at meter resolution will cause the accuracy of a forecast to begin to break down after only a few days.

That's an extremely simplistic take on things. In reality, one of the largest issues with high-resolution weather forecasts (1-3 km scale, convection-permitting simulations) is the fact that you small errors in the initialization or model dynamics lead to changes in small-scale storm structure that feedback onto larger scales of motions, disrupting the forecast. Ultra-fine measurements and simulation resolutions only exacerbate this tendency.

> Anywhere from accurate to exact.

You didn't answer the question. Are you trying to predict convective initiation at 100 days lead time? Are you trying to predict a particular synoptic system? Are you trying to predict whether or not it will be warmer than average or not? These are vastly different weather prediction problems which require different approaches.

> And Bill Gates thought 64K should be enough for anybody. Do you really think computers will only have a few GB of memory 50 years from now?

Modern weather and climate modeling is already a tera- or peta-scale endeavor, depending on exactly what one is trying to do. The sorts of simulations alluded to in the OP push into the exascale.

As other commenters have noted, your odd choice of femotometer (10^-15 meters) would lead to memory requirements larger than the number of atoms in the real atmosphere.

> This straw man does not exactly demonstrate that conventional weather and climate modeling is being abandoned anytime soon. If the unconventional private investments aren't profitable, the market will deal with them.

Of course it does. The age of heterogeneous compute for weather/climate models is just beginning, yet you do not see NVIDIA optimizing NWP systems to run on GPUs or Google porting them to run on TPUs, do you? Instead, you see these organizations pursuing AI/DL, while core NWP development is limited to federal research labs and agencies, but they are increasingly struggling to attract developer and research scientist talent to pursue these activities.

This is a very real challenge that is frequently talked about within the weather community in the United States. I'd hazard the guess that you are not a member of this community?

> much like the local weather, impossible to predict with any accuracy years into the future, and yet the tools used to measure it are consistently getting more accurate, cheaper and smaller. Maybe like bottle-openers, weather sensors may superfluously start appear on everything. The more widespread the measurements, the more data descibing initial conditions, the better the forecast will be at any interval.

There is virtually no data assimilation technology to support the ingestion of the vast majority these data, and we do not even run weather models with suitable configurations to take advantage of them if we had the DA support in the first place. And, as I've mentioned repeatedly, not every measurement leads to an improvement in forecast quality. This is simply _not_ the low- or even high-hanging fruit regarding improvements to weather forecast quality and impact.

I've worked in this exact domain of developing novel weather sensing and observation systems and leveraging them to try to improve forecast quality - across federally-funded research and more than one private company over the past ten years - and it's mostly a fools errand. If one wants to develop improved, impactful, useful weather forecasts, this is not the path to pursue.

Post reply on HN