Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

top500.org

Post reply on HN
Print view View original

11–20 of 72 posts

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#11

Post by fnbr »

zackmorris wrote: post #3

This might be a good time to ask: my main reservation about TensorFlow is that it's a subset of general purpose computing, so will always be limited to niches like AI or physics simulations or protein folding. If we look at something like MATLAB (or GNU Octave) as general-purpose vector computing, then we need some kind of bridge between the two worlds. I couldn't find much other than this: https://www.quora.com/How-…

At work, we have a bunch of vectorized computations that we run in Tensorflow, as it's a convenient way to get GPU-optimized code, so that is still an option (albeit an awkward one).

You could also use something like CUDA, or OpenGL to do this; there are some Python libraries to do basic numerical work, such as PyCUDA or gnumpy.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#12

Post by geoalchimista »

zackmorris wrote: post #3

This might be a good time to ask: my main reservation about TensorFlow is that it's a subset of general purpose computing, so will always be limited to niches like AI or physics simulations or protein folding. If we look at something like MATLAB (or GNU Octave) as general-purpose vector computing, then we need some kind of bridge between the two worlds. I couldn't find much other than this: https://www.quora.com/How-…

If you are not confined by the MATLAB platform, then there are a few options. For example, Julia [1] is a general-purpose numerical computing language that has more or less similar syntax with MATLAB for vector/matrix computation, and there is a Julia package TensorFlow.jl [2] that allows you to call TensorFlow in Julia. There are also quite a few packages in development to adapt Julia to GPU-based computation.

And, to be fair, the NumPy/SciPy stack of Python can also be seen as a general-purpose vector computing platform. My feeling is that MATLAB had its days in the 90s. It's just that the most cutting-edge technologies do not seem to be developed in MATLAB any more.

[1] https://github.com/JuliaLang/julia

[2] https://github.com/malmaud/TensorFlow.jl

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#13

Post by bbatha »

improbable22 wrote: post #5

What's the definition of "one supercomputer" for the purposes of TOP500? For example, why doesn't one of Google's warehouses qualify? Or the whole of Google, for that matter. A bit of googling didn't find my anything very satisfactory.

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like mpi and offer “3d” networking that will take into account physical distance between nodes and can do single rack mesh networking to avoid the overhead of switching. Despite google having lots of compute power they probably can’t leverage it in the way that the LINPACK benchmarks need.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#14

Post by improbable22 »

bbatha wrote: post #13
improbable22 wrote: post #5

What's the definition of "one supercomputer" for the purposes of TOP500? For example, why doesn't one of Google's warehouses qualify? Or the whole of Google, for that matter. A bit of googling didn't find my anything very satisfactory.

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like mpi and offer “3d” networking that will take into account physical distance between nodes and can do single rack mesh networking to avoid the overhead of switching. Despite google having lots of compute power they probably can’t leverage it in the way that th…

Thanks, this is interesting. It would be somehow satisfying if their LINPACK benchmarks would actually not be beaten by Google et al. (And their real workloads too.)

But how tightly can you really connect 27000 GPUs? Would be curious if anyone has a more technical article handy about what's different.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#15

Post by ychen306 »

improbable22 wrote: post #5

What's the definition of "one supercomputer" for the purposes of TOP500? For example, why doesn't one of Google's warehouses qualify? Or the whole of Google, for that matter. A bit of googling didn't find my anything very satisfactory.

The difference between a supercomputer and a data center is how "connected" the computations are; supercomputer optimizes the communication between nodes. To put it another way, a data center does a lot of work but, most of the time, for different applications (services) whose dependencies are "sparse".

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#16

Post by deepnotderp »

improbable22 wrote: post #5

What's the definition of "one supercomputer" for the purposes of TOP500? For example, why doesn't one of Google's warehouses qualify? Or the whole of Google, for that matter. A bit of googling didn't find my anything very satisfactory.

It's all in the Interconnects. The hard part of supercomputing is moving data, not computing.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#17

Post by jcranmer »

improbable22 wrote: post #14
bbatha wrote: post #13

Earlier quoted context omitted.

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like mpi and offer “3d” networking that will take into account physical distance between nodes and can do single rack mesh networking to avoid the overhead of switching. Despite google having lots of compute power they probably can’t leverage it in the way that th…

Thanks, this is interesting. It would be somehow satisfying if their LINPACK benchmarks would actually not be beaten by Google et al. (And their real workloads too.) But how tightly can you really connect 27000 GPUs? Would be curious if anyone has a more technical article handy about what's different.

The list of top supercomputers isn't a list of which systems have the most ALUs that you can shove floats through (though that is definitely a strong correlate). The difficult part in HPC is actually being able to keep those ALUs fed with floats. In large HPC applications, the communication is the principle bottleneck in being able to scale up [1]. Communication patterns for HPC application also tend to very much have a bursty everybody-is-sending-at-the-same-time pattern, which makes it very easy to saturate a typical star-like Ethernet network configuration (supercomputers typically use a torus or mesh-style interconnect).

For GPUs, one trick you can do is to do GPU-to-GPU communication that bypasses the CPU. I don't believe the hardware that extends this to do CPU-less transfer systems across different nodes is common on non-HPC systems.

[1] One of the main criticisms of LINPACK as a benchmark is that it is a low-communication benchmark. Essentially, you're doing O(n^3) computation on O(n^2) communication. In many benchmarks, such as grid simulation, the ratio of computation is communication is constant with respect to size.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#18

Post by bmer »

At least for Intel processors, one can run the LINPACK benchmark using pre-built binaries provided by Intel: https://software.intel.com/en-us/articles/intel-linpack-benc...

There must be something similar for AMD processors too, but I can't find it with some quick duckduckgo. Perhaps someone else can link it?

Just a silly thing to compare your PC with the big dogs.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#19

Post by albertzeyer »

zackmorris wrote: post #3

This might be a good time to ask: my main reservation about TensorFlow is that it's a subset of general purpose computing, so will always be limited to niches like AI or physics simulations or protein folding. If we look at something like MATLAB (or GNU Octave) as general-purpose vector computing, then we need some kind of bridge between the two worlds. I couldn't find much other than this: https://www.quora.com/How-…

Why do you think TensorFlow is a subset of general purpose computing? What do you think what is missing? I think nothing is really missing, only that it's maybe more difficult to perform certain kind of tasks. But compared to Matlab/Octave, I don't really see much lacking (in the platform). I would even say the opposite, that the Matlab/Octave platform seems to me like a subset of what TensorFlow offers.
Post reply on HN