Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Supercomputer quietly puts U.S. weather resources back on top

usatoday.com

Post reply on HN
Print view View original

51–60 of 79 posts

Re: Supercomputer quietly puts U.S. weather resources back on top

#51

Post by colechristensen »

kuya wrote: post #17
cryptoz wrote: post #3

It's a very exciting time in weather forecasting. We're getting all kinds of new observations available, from cheap LEO satellites to smartphone sensors to smart umbrellas and everything else. And faster computers! And more parallel computers! Think about every single iPhone and every single Galaxy S in the world as a useful sensor array _and_ a parallel processor... Weather forecasting accuracy is still constrained…

just curious, do you have references to smartphone sensors being utilized in numerical weather prediction? i'd been browsing through some WMO reports and other associated NWP lit recently and i'd seen stuff on GPS-RO, but nothing on anyone assimilating "smart" devices.

I don't know much about it, but I know some people doing a startup all about these sorts of things

https://senseai.io

Re: Supercomputer quietly puts U.S. weather resources back on top

#52

Post by al_biglan »

matt2000 wrote: post #2

Is there anyone here who works on weather models who would be able to explain why custom supercomputers are still needed vs. large commodity clusters? Thanks!

I work for the company that provided the storage for this. Our product is not a collection of commodity components but instead something very much more like an "appliance" that provides the performance at scale that NOAA needs. In general, one of the trends we see/fight is the "Hey, we can get components ourselves and use Lustre to build what we need" And.... they are spot on right. What you need to consider doing this is the amount of maintaining and administering that comes with this solution. The trade-off we see is that some groups, "Academia" for example, will prefer the cheaper solution because they have Grad Students, Post Docs and Undergrad research folks to help make the people cost free. Others, who just want to buy something that works in their infrastructure without adding the people costs would prefer to buy something ready made that "just works"

Bottom line: If you are using your system to get work done, the trend is to spend the money for reliability and availability. If you want to save money, it comes at a price (but is certainly do-able)

Related: Look at Apple, Google, Facebook, Amazon, Microsoft... they are large enough to build their custom systems themselves. Not everyone operates at this scale. Individuals can buy a cheap PC and build a NAS... or they can buy one from Netgear, Buffalo, QNAP that "jsut works" at a price premium.

Re: Supercomputer quietly puts U.S. weather resources back on top

#53

Post by avar »

nradov wrote: post #25
cryptoz wrote: post #16

Earlier quoted context omitted.

Sure, that's true - but they can just as happily run their GPUs if they are plugged in and charging without any battery issues. Although for the moment you are right, they aren't that useful as a powerful distributed supercomputer. They ARE, however, really really good at collecting live weather data and sending that to the supercomputer to process. One step at a time.

Smart phones are good at collecting live barometric pressure data but not much else. Temperature sensors aren't very useful because they are poorly calibrated, you can't reliably tell whether someone is outdoors, and the reading can be thrown off by the heat generated by the device or the person holding it. UV sensors are seldom pointed in the right direction. And I haven't seen a phone with a wind speed or precipita…

I genuinely don't mean to sound passive aggressive when I ask, do you know anything about this subject or are you just guessing?

I don't know anything about it, but I can think of numerous ways to mitigate the issues you mention. E.g.:

* Does it actually matter that the absolute value of temperature sensors is inaccurate, or is it enough that they have good relative accuracy (do they?)

* Do you really need to tell if someone is indoors or outdoors given a large enough sample of users in a given area? That also seems like something you could detect with heuristics (work hours, jumps up/down in temperature).

There's a lot of ways to tease useful signals from noisy data. Is it really the case that smart phone temperature sensors in the aggregate are useless as an input into weather forecasting models?

Re: Supercomputer quietly puts U.S. weather resources back on top

#54

Post by cos2pi »

mediocrejoker wrote: post #46
kuya wrote: post #33

for anyone interested, this paper has a nice background on numerical weather prediction http://www.elsevierscitech.com/emails/physics/climate/the_or...

I got about 3 pages into this and felt like I was following the discussion until the discussion of an incorrect prediction result that: > In fact, the spurious tendencies are due to an imbalance between the pressure and wind fields resulting in large amplitude high frequency gravity wave oscillations. Since this paper was published within the last 20 years, I can't imagine what they were referring to by 'gravity wave…

Gravity waves are waves in a fluid that obtain their restoring force from their buoyancy relative to the surrounding fluid. See [0] for the gory details.

Some examples of gravity waves in the atmosphere: [1] [2]

[0] http://glossary.ametsoc.org/wiki/Gravity_wave [1] http://cimss.ssec.wisc.edu/goes/blog/archives/2051 [2] https://www.youtube.com/watch?v=yXnkzeCU3bE

Re: Supercomputer quietly puts U.S. weather resources back on top

#55

Post by mturmon »

thrownaway2424 wrote: post #35
paulmd wrote: post #7

Earlier quoted context omitted.

Modern supercomputers essentially are large commodity clusters. Nobody builds a single fast machine ala Cray-1 anymore. It's just not scaleable past a certain point and we're long since past that. However, they still like to give them fancy names as marketing, because they cost tens of millions of dollars. The main differentiation vs a standard datacenter is low-latency high-throughput interconnects, often with a spe…

What kind of latency do they expect from these things? Less than 50 microseconds for a small RPC or small RDMA access? I'm wondering how far these things really are from commercial cloud clusters.

For more on Cray's Aries interconnect used on the Cray XC-30 referred to in the OP, see this paper from NERSC, which has one: https://www.nersc.gov/assets/pubs_presos/NUG2014Aries.pdf

Depending on topology, you can get ~1 µs latency with Aries. It's based on PCIe-3, so it's different from InfiniBand.

Re: Supercomputer quietly puts U.S. weather resources back on top

#56

Post by semi-extrinsic »

superuser2 wrote: post #50
semi-extrinsic wrote: post #37

Earlier quoted context omitted.

They probably also have most of the bookshelves on rails? The local uni library keeps their old books on a system like this, it's really quite cool. The room can fit, say, 54 bookshelves tight-packed with no room to enter anywhere. So they put 52 shelves in on rails, and between each pair there is a wire going across. Disconnect a wire between two shelves, and all the shelves slide apart so you can enter where you op…

I'd estimate most university libraries have a system like this, though the "disconnect a wire" part is strange. Usually they're just a button on the shelf.

I think the wire is a safety feature to avoid accidentally crushing people. If you try to disconnect a second wire, nothing happens, and you have to go find the first disconnected wire and reconnect it. Then you will see if there is still someone in there. (When you're finished in a row, you're supposed to reconnect behind you when you leave.)

Re: Supercomputer quietly puts U.S. weather resources back on top

#57

Post by Faint »

mediocrejoker wrote: post #46
kuya wrote: post #33

for anyone interested, this paper has a nice background on numerical weather prediction http://www.elsevierscitech.com/emails/physics/climate/the_or...

I got about 3 pages into this and felt like I was following the discussion until the discussion of an incorrect prediction result that: > In fact, the spurious tendencies are due to an imbalance between the pressure and wind fields resulting in large amplitude high frequency gravity wave oscillations. Since this paper was published within the last 20 years, I can't imagine what they were referring to by 'gravity wave…

Are you mixing gravity waves and gravitational waves? Gravity waves are any waves in media in which gravity acts as restoring force, like waves in ocean, or waves between layers with different densities in atmosphere, or, AFAIU, whole atmosphere (think of atmosphere as a big shallow pool, with air sloshing around in the pool instead of some liquid).

https://en.wikipedia.org/wiki/Gravity_wave

Re: Supercomputer quietly puts U.S. weather resources back on top

#58

Post by goodbyegti »

dhsjhdjs wrote: post #9

For a newbie in this domain, how do we start doing our own toy forecasts (lets say for Bay area)?

A really nice intro is this classic paper on numerical integration of the equation for barotropic vorticity:

http://mathsci.ucd.ie/~plynch/eniac/CFvN-1950.pdf

It's something you can solve in a few lines of python and run with minimal compute resources. A nice thing to try is to feed in some data from somewhere like below and watch what happens as you integrate it forwards in time.

http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalys...

Re: Supercomputer quietly puts U.S. weather resources back on top

#59

Post by divreg »

mediocrejoker wrote: post #46
kuya wrote: post #33

for anyone interested, this paper has a nice background on numerical weather prediction http://www.elsevierscitech.com/emails/physics/climate/the_or...

I got about 3 pages into this and felt like I was following the discussion until the discussion of an incorrect prediction result that: > In fact, the spurious tendencies are due to an imbalance between the pressure and wind fields resulting in large amplitude high frequency gravity wave oscillations. Since this paper was published within the last 20 years, I can't imagine what they were referring to by 'gravity wave…

It's funny that you point this out, since some people in the field of astrophysics are quick to correct others when "gravity waves" are used when they really mean "gravitational waves", and sometimes vice versa.

Basically, gravity waves (or g-waves) are a type of perturbation in stratified media where the restoring force on sound waves is buoyancy. The other scenario is where the restoring force is pressure, which are pressure waves (or p-waves).

Gravity waves are important in not just planetary atmospheres but stellar media as well, particularly in the outer region of stars. A popular candidate for /gravitational/ wave sources is binary neutron star systems, so the two words aren't interchangeable there since they refer to two very different phenomena!

Re: Supercomputer quietly puts U.S. weather resources back on top

#60

Post by bannus »

knappe wrote: post #36
munificent wrote: post #23

Earlier quoted context omitted.

"Underground" isn't very feasible in Florida. Most of the state is effectively a sandbar with a very high water table. Underground construction is uncommon.

I was in the south Everglades last week. We went over a pass that was _1_ ft tall. Coming from Colorado, the sign advertising the "pass" was just bizarre.

For the similarly curious -- I assume this is referring to Rock Reef Pass:

http://www.jessstryker.com/national-parks/everglades/rock-re...

Post reply on HN