Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

top500.org

Post reply on HN
Print view View original

31–40 of 72 posts

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#31

Post by bbgm »

dwyerm wrote: post #28
jcranmer wrote: post #9

Earlier quoted context omitted.

TOP500 doesn't include distributed systems. Essentially, every computer on TOP500 is a single computer than you can log onto. By contrast, Google's data warehouse would qualify as a large cluster of individual systems. Note that not all supercomputers are on TOP500. Blue Waters is perhaps the most notable one to not bother reporting its performance (it would probably have been #1 had it done so when it came out, and…

I'm not sure that's true. At the very least, EC2 made a showing with C3 instances that made it to #64 in 2013. https://www.top500.org/system/178321

CC2 was #42, which was cool just because of the number.

https://aws.amazon.com/blogs/aws/next-generation-cluster-com...

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#32

Post by monocasa »

bbatha wrote: post #13
improbable22 wrote: post #5

What's the definition of "one supercomputer" for the purposes of TOP500? For example, why doesn't one of Google's warehouses qualify? Or the whole of Google, for that matter. A bit of googling didn't find my anything very satisfactory.

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like mpi and offer “3d” networking that will take into account physical distance between nodes and can do single rack mesh networking to avoid the overhead of switching. Despite google having lots of compute power they probably can’t leverage it in the way that th…

The interconnects have gotten waaaayyy better in datacenters over the past five years or so when compared to Infiniband. Stuff like FPGAs doing data plane routing, and all of the "converged Ethernet" standards like RoCE have really narrowed the gap between Ethernet and Infiniband.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#33

Post by dahart »

zackmorris wrote: post #3

This might be a good time to ask: my main reservation about TensorFlow is that it's a subset of general purpose computing, so will always be limited to niches like AI or physics simulations or protein folding. If we look at something like MATLAB (or GNU Octave) as general-purpose vector computing, then we need some kind of bridge between the two worlds. I couldn't find much other than this: https://www.quora.com/How-…

Maybe something like Jupyter or plain Python with GPU enabled numpy is what you’re looking for?

Tensorflow is a library and not a language, it’s meant to be used from a host language that is Turing complete. It’s goal is to make construction of graphs of vector evaluations easier and more performant, but not really to provide a general purpose computation environment, it’s assumed you’re calling it from a general purpose computation environment.

So, if you’re using Tensorflow, you normally have general purpose computing available to you, with the option to bake your vector tasks into graphs easily and/or speed them up.

Jupyter is becoming a decent alternative to Matlab, and you have many options for running vector computations from python, with or without a GPU.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#34

Post by garmaine »

godelski wrote: post #26
davrosthedalek wrote: post #2

Since TFA talks about deep learning so much, I wonder how many of the applications run on these machines actually are deep learning, or can make use of the tensor cores in some other way.

A lot of people are using GPUs for many other things than ML. The big advantage is the number of cores, and people that run on super computers write algorithms that are highly parallelized (otherwise what's the point). GPUs are getting fast enough that the number of cores they share is gaining an edge. Also the memory on them is MUCH faster than that on a CPU, but the cost is that you have less (20Gb compared to 256G…

All modern gpus support f16.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#35

Post by zackmorris »

bmer wrote: post #22
zackmorris wrote: post #3

This might be a good time to ask: my main reservation about TensorFlow is that it's a subset of general purpose computing, so will always be limited to niches like AI or physics simulations or protein folding. If we look at something like MATLAB (or GNU Octave) as general-purpose vector computing, then we need some kind of bridge between the two worlds. I couldn't find much other than this: https://www.quora.com/How-…

What do you mean by "TensorFlow is a subset of general purpose computing, and thus will always be limited to niches"? It's not clear to me at all what one could mean by this. Doesn't TensorFlow have to use matrix math deep down (just like any other digital computing system)?

I'm not sure if you or albertzeyer asked first, but what I meant by that is that MATLAB is similar to any other C-like language, except uses the vector as its primitive instead of something like integer or float. That's really all there is to it. Other than a few details about notation, every major concept of MATLAB stems from that and is easily understood and predictable. MATLAB (or non-proprietary analogs like GNU Octave) let you write C-like code and then its runtime deals with architecture and optimization details internally (so there are no limitations on vector size or number of samplers or anything like that).

Whereas things like TensorFlow, CUDA, OpenCL, OpenGL etc seem to deal more with DSP processing of buffer(s). They all have their own abstractions and lingo which work extremely well for certain use cases, kind of like domain-specific languages (DSLs).

The end result is that it's trivial (at least in theory) to go from a TensorFlow implementation to a MATLAB implementation. But it's very difficult to go the other direction. Another way to think of this is that any solution written in TensorFlow can be run by MATLAB, but the reverse is not necessarily true. Trying to run MATLAB code within TensorFlow might encounter hardware limitations or other restrictions that makes the code run thousands of times slower.

Now I could be wrong about this - maybe they truly are equivalent. But until I'm able to transpile MATLAB code directly to TensorFlow or OpenCL or whatever and have it be performant, I'm going to continue working under this assumption.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#36

Post by zackmorris »

albertzeyer wrote: post #19
zackmorris wrote: post #3

This might be a good time to ask: my main reservation about TensorFlow is that it's a subset of general purpose computing, so will always be limited to niches like AI or physics simulations or protein folding. If we look at something like MATLAB (or GNU Octave) as general-purpose vector computing, then we need some kind of bridge between the two worlds. I couldn't find much other than this: https://www.quora.com/How-…

Why do you think TensorFlow is a subset of general purpose computing? What do you think what is missing? I think nothing is really missing, only that it's maybe more difficult to perform certain kind of tasks. But compared to Matlab/Octave, I don't really see much lacking (in the platform). I would even say the opposite, that the Matlab/Octave platform seems to me like a subset of what TensorFlow offers.

I wasn't sure who was first, so see my reply to bmer here: https://news.ycombinator.com/item?id=17420821

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#37

Post by stephencanon »

jcranmer wrote: post #17
improbable22 wrote: post #14

Earlier quoted context omitted.

Thanks, this is interesting. It would be somehow satisfying if their LINPACK benchmarks would actually not be beaten by Google et al. (And their real workloads too.) But how tightly can you really connect 27000 GPUs? Would be curious if anyone has a more technical article handy about what's different.

The list of top supercomputers isn't a list of which systems have the most ALUs that you can shove floats through (though that is definitely a strong correlate). The difficult part in HPC is actually being able to keep those ALUs fed with floats. In large HPC applications, the communication is the principle bottleneck in being able to scale up [1]. Communication patterns for HPC application also tend to very much hav…

> One of the main criticisms of LINPACK as a benchmark is that it is a low-communication benchmark. Essentially, you're doing O(n^3) computation on O(n^2) communication. In many benchmarks, such as grid simulation, the ratio of computation is communication is constant with respect to size.

This is critical and poorly communicated to most people outside the HPC world.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#38

Post by ianbertolacci »

zackmorris wrote: post #3

This might be a good time to ask: my main reservation about TensorFlow is that it's a subset of general purpose computing, so will always be limited to niches like AI or physics simulations or protein folding. If we look at something like MATLAB (or GNU Octave) as general-purpose vector computing, then we need some kind of bridge between the two worlds. I couldn't find much other than this: https://www.quora.com/How-…

> niches like AI or physics simulations or protein folding These are 99% of HPC workloads. Nothing `niche` about them.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#39

Post by 67_45 »

I wish that you could buy a simple computer where all processing is integrated. The cores form a pyramid with a few really fast ones on top and tons and tons of slow ones below. All is exposed, with no speculation, in a very low level raw API. All abstractions like speculation etc are layers on top ala vulkan

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#40

Post by InclinedPlane »

bbatha wrote: post #13
improbable22 wrote: post #5

What's the definition of "one supercomputer" for the purposes of TOP500? For example, why doesn't one of Google's warehouses qualify? Or the whole of Google, for that matter. A bit of googling didn't find my anything very satisfactory.

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like mpi and offer “3d” networking that will take into account physical distance between nodes and can do single rack mesh networking to avoid the overhead of switching. Despite google having lots of compute power they probably can’t leverage it in the way that th…

Infiniband used to be great but in an era of 40 or 100 gig ethernet it's not particularly special anymore.
Post reply on HN