Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

top500.org

Post reply on HN
Print view View original

51–60 of 72 posts

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#51

Post by bmer »

Koshkin wrote: post #46
bmer wrote: post #45

Earlier quoted context omitted.

Yeah, I am pretty certain you're not right in this understanding. I am not the most knowledgeable, but basically all I know is scientific computing, so here are my two cents: Any C-like language does not use the "vector" as its "primitive" (not clear what you mean by primitive, and I am instead interpreting it as "machine type"), since integers, floats, characters, are the "primitives" (machine types), which are the…

> does not use the "vector" as its "primitive" Well, doesn't the SIMD architecture make vectors "primitives", even if in some limited sense?

As I said:

-------------------------------------

Some things I have not mentioned:

* vectorization: basically, how to do vector operations "smarter" on non-parallelized hardware[SIMD?], something that many languages (e.g. MATLAB, NumPy) and hardware now support

-------------------------------------

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#52

Post by shaklee3 »

shaklee3 wrote: post #42
bbatha wrote: post #23

Earlier quoted context omitted.

> But how tightly can you really connect 27000 GPUs? Not all that well currently, NVidia and others are working on GPU specific interconnects[0] but they don't have anywhere near the scale of traditional interconnects which have supported hundreds of thousands of nodes by the late 90s. On of the big challenges in modern super computer programming is in fact keeping the GPUs hot, which can often mean offloading work t…

Exactly. First there was nvswitch, which dramatically increased the bandwidth over pcie. But that didn't scale to a large number of GPUs. Then there was nvswitch, which solved the scaling problem inside a node. I wouldn't be surprised if the next leap is something like nvlink cables between nodes that don't need traditional routing capabilities.

First there was nvlink, rather.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#53

Post by throwaway2048 »

jcranmer wrote: post #9
improbable22 wrote: post #5

What's the definition of "one supercomputer" for the purposes of TOP500? For example, why doesn't one of Google's warehouses qualify? Or the whole of Google, for that matter. A bit of googling didn't find my anything very satisfactory.

TOP500 doesn't include distributed systems. Essentially, every computer on TOP500 is a single computer than you can log onto. By contrast, Google's data warehouse would qualify as a large cluster of individual systems. Note that not all supercomputers are on TOP500. Blue Waters is perhaps the most notable one to not bother reporting its performance (it would probably have been #1 had it done so when it came out, and…

What you are talking about is called "single system image", that is a single address space, storage is globaly visible, etc.

This concept is pretty much completely dead in the supercomputing world, and has been for decades.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#54

Post by fulafel »

mrb wrote: post #47

The number one supercomputer on the TOP500 list, Summit, is able to majority-attack about 95% of cryptocurrencies that are GPU-mined: https://twitter.com/zorinaq/status/1007005472505978880 That's one advantage that ASIC-mined currencies have over them. Specialized chips raise the security bar so high that the pre-existing installed base of GPUs cannot attack them.

Supercomputers are expensive because of the investment in interconnect & i/o bandwidth & latency.

So using one for a trivially parallelizable, low-communication task like crypto mining would be wasteful - very low bang for the buck.

(There are other cryptanalysis workloads that benefit though, eg parallel number field sieve).

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#55

Post by albertzeyer »

zackmorris wrote: post #35
bmer wrote: post #22

Earlier quoted context omitted.

What do you mean by "TensorFlow is a subset of general purpose computing, and thus will always be limited to niches"? It's not clear to me at all what one could mean by this. Doesn't TensorFlow have to use matrix math deep down (just like any other digital computing system)?

I'm not sure if you or albertzeyer asked first, but what I meant by that is that MATLAB is similar to any other C-like language, except uses the vector as its primitive instead of something like integer or float. That's really all there is to it. Other than a few details about notation, every major concept of MATLAB stems from that and is easily understood and predictable. MATLAB (or non-proprietary analogs like GNU…

In TensorFlow, a vector (or a tensor) is also one of the fundamental datatypes.

> The end result is that it's trivial (at least in theory) to go from a TensorFlow implementation to a MATLAB implementation. But it's very difficult to go the other direction.

I would say just the opposite. From Matlab to TF should be trivial, but the other way not so much. Or can you give me an example in Matlab which would be hard to translate to TF?

I can give you one in TF which would be very hard in Matlab: E.g. how to implement async SGD, with a parameter server and all that logic?

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#56

Post by DannyBee »

deepnotderp wrote: post #16
improbable22 wrote: post #5

What's the definition of "one supercomputer" for the purposes of TOP500? For example, why doesn't one of Google's warehouses qualify? Or the whole of Google, for that matter. A bit of googling didn't find my anything very satisfactory.

It's all in the Interconnects. The hard part of supercomputing is moving data, not computing.

What makes you think google doesn't have good enough interconnects in their data centers? infiniband is not that impressive anymore

"2009: of the top 500 supercomputers in the world, Gigabit Ethernet is the internal interconnect technology in 259 installations, compared with 181 using InfiniBand.[20] "

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#57

Post by DannyBee »

InclinedPlane wrote: post #40
bbatha wrote: post #13

Earlier quoted context omitted.

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like mpi and offer “3d” networking that will take into account physical distance between nodes and can do single rack mesh networking to avoid the overhead of switching. Despite google having lots of compute power they probably can’t leverage it in the way that th…

Infiniband used to be great but in an era of 40 or 100 gig ethernet it's not particularly special anymore.

Right.

As https://en.wikipedia.org/wiki/InfiniBand#cite_note-20 says

"2009: of the top 500 supercomputers in the world, Gigabit Ethernet is the internal interconnect technology in 259 installations, compared with 181 using InfiniBand.[20]"

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#58

Post by DannyBee »

bbatha wrote: post #13
improbable22 wrote: post #5

What's the definition of "one supercomputer" for the purposes of TOP500? For example, why doesn't one of Google's warehouses qualify? Or the whole of Google, for that matter. A bit of googling didn't find my anything very satisfactory.

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like mpi and offer “3d” networking that will take into account physical distance between nodes and can do single rack mesh networking to avoid the overhead of switching. Despite google having lots of compute power they probably can’t leverage it in the way that th…

This is just wrong. 100 gigabit ethernet switches are commonplace in the datacenters of companies like google, facebook, etc.

Facebook even open sourced their second generation 100 gigabit switch design years ago.

Combine this with the fact that LINPACK is not a benchmark that involves a lot of communication in the relative scheme of things. If any of them big players cared, they could probably destroy the top LINPACK numbers instantly.

There are HPC benchmarks that they would have a harder time with, just not LINPACK.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#59

Post by tntn »

davrosthedalek wrote: post #2

Since TFA talks about deep learning so much, I wonder how many of the applications run on these machines actually are deep learning, or can make use of the tensor cores in some other way.

One other point not mentioned in other comments: some work was presented at GTC regarding using tensor cores for a low precision solution followed by iterative refinement to a fp64-equivalent solution. IIRC, 2-4x speed up for fp64 dense system solvers.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#60

Post by mrb »

fulafel wrote: post #54
mrb wrote: post #47

The number one supercomputer on the TOP500 list, Summit, is able to majority-attack about 95% of cryptocurrencies that are GPU-mined: https://twitter.com/zorinaq/status/1007005472505978880 That's one advantage that ASIC-mined currencies have over them. Specialized chips raise the security bar so high that the pre-existing installed base of GPUs cannot attack them.

Supercomputers are expensive because of the investment in interconnect & i/o bandwidth & latency. So using one for a trivially parallelizable, low-communication task like crypto mining would be wasteful - very low bang for the buck. (There are other cryptanalysis workloads that benefit though, eg parallel number field sieve).

You are right that Summit is overbuilt if it was going to be used just for raw crypto mining hashpower. However, even so, it would be quite profitable... The last double spend attack on BTG stole ~$20M. There is no reason to think it couldn't steal $50M or so. Repeat the attack on a handful of other cryptocurrencies and you would quickly recoup the cost of Summit ($200M)...
Post reply on HN