Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

top500.org

Post reply on HN
Print view View original

21–30 of 72 posts

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#21

Post by joe_the_user »

zackmorris wrote: post #3

This might be a good time to ask: my main reservation about TensorFlow is that it's a subset of general purpose computing, so will always be limited to niches like AI or physics simulations or protein folding. If we look at something like MATLAB (or GNU Octave) as general-purpose vector computing, then we need some kind of bridge between the two worlds. I couldn't find much other than this: https://www.quora.com/How-…

I don't know about tensor flow in particular but are little-known methods of running "general purpose" parallel programs on GPUs. Specifically, H. Dietz' MOG, "Mimd on GPU". It's a shame the project hasn't gotten more attention imo.

http://aggregate.org/MOG/

See: https://en.wikipedia.org/wiki/Flynn%27s_taxonomy for explanations of terms.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#22

Post by bmer »

zackmorris wrote: post #3

This might be a good time to ask: my main reservation about TensorFlow is that it's a subset of general purpose computing, so will always be limited to niches like AI or physics simulations or protein folding. If we look at something like MATLAB (or GNU Octave) as general-purpose vector computing, then we need some kind of bridge between the two worlds. I couldn't find much other than this: https://www.quora.com/How-…

What do you mean by "TensorFlow is a subset of general purpose computing, and thus will always be limited to niches"? It's not clear to me at all what one could mean by this. Doesn't TensorFlow have to use matrix math deep down (just like any other digital computing system)?

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#23

Post by bbatha »

improbable22 wrote: post #14
bbatha wrote: post #13

Earlier quoted context omitted.

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like mpi and offer “3d” networking that will take into account physical distance between nodes and can do single rack mesh networking to avoid the overhead of switching. Despite google having lots of compute power they probably can’t leverage it in the way that th…

Thanks, this is interesting. It would be somehow satisfying if their LINPACK benchmarks would actually not be beaten by Google et al. (And their real workloads too.) But how tightly can you really connect 27000 GPUs? Would be curious if anyone has a more technical article handy about what's different.

> But how tightly can you really connect 27000 GPUs?

Not all that well currently, NVidia and others are working on GPU specific interconnects[0] but they don't have anywhere near the scale of traditional interconnects which have supported hundreds of thousands of nodes by the late 90s. On of the big challenges in modern super computer programming is in fact keeping the GPUs hot, which can often mean offloading work that needs high memory usage to CPUs.

Unfortunately my knowledge here is a little dated, I interned at Los Alamos National Lab from 2008 - 2012 when they were doing a lot of rearchitecting of old codes for RoadRunner, the first peta-scale computer. It used Cell chips in accelerator cards and predicated a lot of the challenges in GPU programming, but did not fully elucidate them. For instance we didn't have CUDA!

If I had to take my guess the first exa-scale computer is going to be the one that solves the GPU interconnect problem at scale.

0: https://www.nvidia.com/en-us/data-center/nvlink/

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#24

Post by bwanab »

davrosthedalek wrote: post #2

Since TFA talks about deep learning so much, I wonder how many of the applications run on these machines actually are deep learning, or can make use of the tensor cores in some other way.

Given that the top one is at Oak Ridge National Lab, my guess would be that they're not exploring deep learning. They've got other applications in mind.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#25

Post by dekhn »

improbable22 wrote: post #5

What's the definition of "one supercomputer" for the purposes of TOP500? For example, why doesn't one of Google's warehouses qualify? Or the whole of Google, for that matter. A bit of googling didn't find my anything very satisfactory.

Definition of supercomputer varies, but TOP500 is based on LINPACK. I am not aware of Google running any LINPACK benchmarks on their hardware (except maybe in Cloud VMs)?

As others will say, the classic Google warehouses weren't really supercomputers, but more like massive clusters with a high cross-sectional bandwidth, but with very high latency, and they didn't run an MPI stack.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#26

Post by godelski »

davrosthedalek wrote: post #2

Since TFA talks about deep learning so much, I wonder how many of the applications run on these machines actually are deep learning, or can make use of the tensor cores in some other way.

A lot of people are using GPUs for many other things than ML. The big advantage is the number of cores, and people that run on super computers write algorithms that are highly parallelized (otherwise what's the point). GPUs are getting fast enough that the number of cores they share is gaining an edge. Also the memory on them is MUCH faster than that on a CPU, but the cost is that you have less (20Gb compared to 256Gb).

As far as the TPUs, one big advantage for ML is that they are float16/float32 (normal being f32/f64)(in ML you care very little about precision) and are optimized for tensor calculations. For anything that you don't need that resolution and are doing tensor stuff (lots of math/physics does tensor stuff), then these will give you an advantage. (I'm not aware of anyone using these for things other than ML, but I wouldn't be surprised if people did use them) But other things you need more precision and those won't use the TPUs (AFAIK).

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#27

Post by godelski »

bwanab wrote: post #24
davrosthedalek wrote: post #2

Since TFA talks about deep learning so much, I wonder how many of the applications run on these machines actually are deep learning, or can make use of the tensor cores in some other way.

Given that the top one is at Oak Ridge National Lab, my guess would be that they're not exploring deep learning. They've got other applications in mind.

ORNL, like everyone else, is studying ML. They are a research lab. But there are a lot of other applications that they are interested in. These GPUs do help with the traditional research that they perform.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#28

Post by dwyerm »

jcranmer wrote: post #9
improbable22 wrote: post #5

What's the definition of "one supercomputer" for the purposes of TOP500? For example, why doesn't one of Google's warehouses qualify? Or the whole of Google, for that matter. A bit of googling didn't find my anything very satisfactory.

TOP500 doesn't include distributed systems. Essentially, every computer on TOP500 is a single computer than you can log onto. By contrast, Google's data warehouse would qualify as a large cluster of individual systems. Note that not all supercomputers are on TOP500. Blue Waters is perhaps the most notable one to not bother reporting its performance (it would probably have been #1 had it done so when it came out, and…

I'm not sure that's true. At the very least, EC2 made a showing with C3 instances that made it to #64 in 2013.

https://www.top500.org/system/178321

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#29

Post by jedbrown »

bbatha wrote: post #13
improbable22 wrote: post #5

What's the definition of "one supercomputer" for the purposes of TOP500? For example, why doesn't one of Google's warehouses qualify? Or the whole of Google, for that matter. A bit of googling didn't find my anything very satisfactory.

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like mpi and offer “3d” networking that will take into account physical distance between nodes and can do single rack mesh networking to avoid the overhead of switching. Despite google having lots of compute power they probably can’t leverage it in the way that th…

HPL is not especially sensitive to network latency. There are many data centers that could run HPL and get on the list, but don't care to pull that (relatively expensive) stunt. Among scientific applications, a significant fraction really depend on the high-end networks while others would be fine without.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#30

Post by bbgm »

jedbrown wrote: post #29
bbatha wrote: post #13

Earlier quoted context omitted.

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like mpi and offer “3d” networking that will take into account physical distance between nodes and can do single rack mesh networking to avoid the overhead of switching. Despite google having lots of compute power they probably can’t leverage it in the way that th…

HPL is not especially sensitive to network latency. There are many data centers that could run HPL and get on the list, but don't care to pull that (relatively expensive) stunt. Among scientific applications, a significant fraction really depend on the high-end networks while others would be fine without.

In the past EC2-based clusters have made the Top500. Somewhat unique then because they were the only virtualized systems, but those were just 10 gig ethernet and HPL runs really well on those. In all cases we did it on relatively small number of machines before they had been publicly launched (essentially we used HPL as a stress test).

(Work at AWS)

Post reply on HN