Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Supercomputer quietly puts U.S. weather resources back on top

usatoday.com

Post reply on HN
Print view View original

31–40 of 79 posts

Re: Supercomputer quietly puts U.S. weather resources back on top

#31

Post by cryptoz »

kuya wrote: post #30
cryptoz wrote: post #19

Earlier quoted context omitted.

Sure. I've been working on this problem for about 5 years now, starting with the collection of barometric pressure from Android devices. I'm currently working on this with iPhones at Sunshine [1], where we collect pressure data (along with other metrics), but pressure is the most valuable. There are researchers who use this data - we have collected about 4 billion atmospheric pressure measurements that we have distri…

ah, thanks, that's fascinating. "It also learns every time you actively report to the community on sky conditions and hazards, translating this information into weather predictions." does sunshine actively run a numerical weather prediction model like WRF? it'd be great to assimilate all these extra sensors into the NWP centre's models, but i imagine things like cal/val and WMO agreements to share data might make thi…

Yes, Sunshine is running WRF!

And yes it would be great to have all these sensor readings available to NOAA, Environment Canada, ECMWF, and everywhere. I have made lots of progress in getting them to talk about it, but it's a long road before any government starts using this data in its own NWP models.

Re: Supercomputer quietly puts U.S. weather resources back on top

#34

Post by kuya »

cryptoz wrote: post #31
kuya wrote: post #30

Earlier quoted context omitted.

ah, thanks, that's fascinating. "It also learns every time you actively report to the community on sky conditions and hazards, translating this information into weather predictions." does sunshine actively run a numerical weather prediction model like WRF? it'd be great to assimilate all these extra sensors into the NWP centre's models, but i imagine things like cal/val and WMO agreements to share data might make thi…

Yes, Sunshine is running WRF! And yes it would be great to have all these sensor readings available to NOAA, Environment Canada, ECMWF, and everywhere. I have made lots of progress in getting them to talk about it, but it's a long road before any government starts using this data in its own NWP models.

you're probably already aware of this, but there is some US legislation trying to open up commercial data to gov't entities:

https://www.congress.gov/bill/114th-congress/house-bill/1561...

there's some stuff being written this year as well for the side that i work (space). with all the upcoming sensor gaps, they're looking at alternative ways to cover.

Re: Supercomputer quietly puts U.S. weather resources back on top

#35

Post by thrownaway2424 »

paulmd wrote: post #7
matt2000 wrote: post #2

Is there anyone here who works on weather models who would be able to explain why custom supercomputers are still needed vs. large commodity clusters? Thanks!

Modern supercomputers essentially are large commodity clusters. Nobody builds a single fast machine ala Cray-1 anymore. It's just not scaleable past a certain point and we're long since past that. However, they still like to give them fancy names as marketing, because they cost tens of millions of dollars. The main differentiation vs a standard datacenter is low-latency high-throughput interconnects, often with a spe…

What kind of latency do they expect from these things? Less than 50 microseconds for a small RPC or small RDMA access? I'm wondering how far these things really are from commercial cloud clusters.

Re: Supercomputer quietly puts U.S. weather resources back on top

#36

Post by knappe »

munificent wrote: post #23
chm wrote: post #20

Earlier quoted context omitted.

Could they have built it underground?

"Underground" isn't very feasible in Florida. Most of the state is effectively a sandbar with a very high water table. Underground construction is uncommon.

I was in the south Everglades last week. We went over a pass that was _1_ ft tall. Coming from Colorado, the sign advertising the "pass" was just bizarre.

Re: Supercomputer quietly puts U.S. weather resources back on top

#37

Post by semi-extrinsic »

superuser2 wrote: post #15
jordache wrote: post #14

Wish they would stop using the library of congress' collection as a reference... I know it's big.. no idea how big.. it's not tangible to me.. never visited.. I'm sure most americans have not visited.

You do not get an idea of how big it is by visiting; the stacks are off limits and much of the material must be ordered in from suburban warehouses.

They probably also have most of the bookshelves on rails? The local uni library keeps their old books on a system like this, it's really quite cool. The room can fit, say, 54 bookshelves tight-packed with no room to enter anywhere. So they put 52 shelves in on rails, and between each pair there is a wire going across. Disconnect a wire between two shelves, and all the shelves slide apart so you can enter where you opened the wire.

Re: Supercomputer quietly puts U.S. weather resources back on top

#38

Post by semi-extrinsic »

thrownaway2424 wrote: post #35
paulmd wrote: post #7

Earlier quoted context omitted.

Modern supercomputers essentially are large commodity clusters. Nobody builds a single fast machine ala Cray-1 anymore. It's just not scaleable past a certain point and we're long since past that. However, they still like to give them fancy names as marketing, because they cost tens of millions of dollars. The main differentiation vs a standard datacenter is low-latency high-throughput interconnects, often with a spe…

What kind of latency do they expect from these things? Less than 50 microseconds for a small RPC or small RDMA access? I'm wondering how far these things really are from commercial cloud clusters.

For a relatively modern cluster the standard is FDR Infiniband, which gives you 56 Gbit/s throughput. The latency for a small (Glenn Lockwood has a nice blogpost with much more detail: http://glennklockwood.blogspot.com/2013/05/fdr-infiniband-vs...

Re: Supercomputer quietly puts U.S. weather resources back on top

#39

Post by oh_sigh »

cryptoz wrote: post #3

It's a very exciting time in weather forecasting. We're getting all kinds of new observations available, from cheap LEO satellites to smartphone sensors to smart umbrellas and everything else. And faster computers! And more parallel computers! Think about every single iPhone and every single Galaxy S in the world as a useful sensor array _and_ a parallel processor... Weather forecasting accuracy is still constrained…

No one in the foreseeable future is going to use cellphones as parallel processors. First of all, the communication latencies are way too large to coordinate anything usefully. Second of all, users are already upset about the lack of battery life available for their phones.

Re: Supercomputer quietly puts U.S. weather resources back on top

#40

Post by CWuestefeld »

Sniffnoy wrote: post #32

I'm bothered by how the article says "three years ago European models delivered a blow to the U.S. weather apparatus". Better models by Europeans do not make the models by Americans any worse. Improved models help everyone.

That was the first thing I noticed, too. The fact that the Euro computers got Hurricane Sandy right was NOT a bad thing - it helped us.

There is no reason to look at this in any way adversarially. I guess it touches a nerve, because of the similar silly comments elsewhere, worrying about how the USA is "losing" if, say, the Chinese economy's size exceeds the size of our own. Other folks doing well doesn't make us worse. When it happens we should be happy: other people are doing well, and we've got an opportunity to learn and improve too!

Post reply on HN