Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

top500.org

Post reply on HN
Print view View original

61–70 of 72 posts

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#61

Post by gnufx »

monocasa wrote: post #32
bbatha wrote: post #13

Earlier quoted context omitted.

One thing google et all are missing from a typical super computer is infiniband style interconnects. They provide integrations with parallel data libraries like mpi and offer “3d” networking that will take into account physical distance between nodes and can do single rack mesh networking to avoid the overhead of switching. Despite google having lots of compute power they probably can’t leverage it in the way that th…

The interconnects have gotten waaaayyy better in datacenters over the past five years or so when compared to Infiniband. Stuff like FPGAs doing data plane routing, and all of the "converged Ethernet" standards like RoCE have really narrowed the gap between Ethernet and Infiniband.

However much Ethernet has changed, it's not appropriate for typical HPC use (c.f. IB, OPA, Aries, ...). I investigated Cisco's UCS at some length, for instance, which they had persuaded an important person to buy, and that's at least partly the baby of an MPI person. (I recall Infiniband was designed as a "datacentre" thing.)

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#62

Post by gnufx »

DannyBee wrote: post #57
InclinedPlane wrote: post #40

Earlier quoted context omitted.

Infiniband used to be great but in an era of 40 or 100 gig ethernet it's not particularly special anymore.

Right. As https://en.wikipedia.org/wiki/InfiniBand#cite_note-20 says "2009: of the top 500 supercomputers in the world, Gigabit Ethernet is the internal interconnect technology in 259 installations, compared with 181 using InfiniBand.[20]"

There are (at least) two parameters that are relevant for the fabric, and latency is the critical one for HPC. Have you seen a materials science computation, of the sort that typically takes much of the time on HPC systems, running on 1GbE and SDR IB, even on three nodes? It sometimes happened on our system by mistake in the 2009 era.

HPL is simply not a good measure of the utility of a system for HPC, which generally accepted amongst HPC people, who deserve to be listened to. There's a good reason you don't see petascale DFT calculations on one of these "cloud provider"s which comprise so much of the top 500. (That might work on Azure, or something else with an Infiniband-ish fabric, but I've still not seen it.)

You can, and to some extent should, model network effects with systems like Simgrid or Dimemas.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#63

Post by ChrisRackauckas »

bmer wrote: post #45
zackmorris wrote: post #35

Earlier quoted context omitted.

I'm not sure if you or albertzeyer asked first, but what I meant by that is that MATLAB is similar to any other C-like language, except uses the vector as its primitive instead of something like integer or float. That's really all there is to it. Other than a few details about notation, every major concept of MATLAB stems from that and is easily understood and predictable. MATLAB (or non-proprietary analogs like GNU…

Yeah, I am pretty certain you're not right in this understanding. I am not the most knowledgeable, but basically all I know is scientific computing, so here are my two cents: Any C-like language does not use the "vector" as its "primitive" (not clear what you mean by primitive, and I am instead interpreting it as "machine type"), since integers, floats, characters, are the "primitives" (machine types), which are the…

>say, Python+NumPy, because then you can use everything the Python ecosystem has to offer, and aren't limited to the MATLAB ecosystem

That has an implicit assumption that Python's ecosystem is better for this kind of work than MATLAB which just isn't true in many areas of scientific computing. Take differential equations for example. MATLAB has matcont which is a great bifurcation analysis library which is unrivaled by Python (PyDSTool is a toy in comparison). Python doesn't really have usable DDE solvers. Python's ODE (SciPy, Sundials wrappers, etc) solvers are much less developed than MATLAB's, doing okay since it's wrapping some standard software but with a lot less flexibility than MATLAB. And you can keep going. And to top it off, compilation with Numba or Cython doesn't work well in the case of differential equations since there are a lot of small function calls. So it's not so clear that Python is good at scientific computing at all, and in fact in some regions like differential equations it's quite a step downwards. Generally, the ecosystem in this area is well developed in Julia + the commercial offerings (MATLAB, Maple, Mathematica).

That said, Python's ecosystem outside of scientific computing is so much better than MATLAB's. However, at that point I would think Julia is a good choice.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#64

Post by tux1968 »

mrb wrote: post #60
fulafel wrote: post #54

Earlier quoted context omitted.

Supercomputers are expensive because of the investment in interconnect & i/o bandwidth & latency. So using one for a trivially parallelizable, low-communication task like crypto mining would be wasteful - very low bang for the buck. (There are other cryptanalysis workloads that benefit though, eg parallel number field sieve).

You are right that Summit is overbuilt if it was going to be used just for raw crypto mining hashpower. However, even so, it would be quite profitable... The last double spend attack on BTG stole ~$20M. There is no reason to think it couldn't steal $50M or so. Repeat the attack on a handful of other cryptocurrencies and you would quickly recoup the cost of Summit ($200M)...

Genuinely curious about what would make this stealing? Isn't it just playing the crypto game better than the competition? What makes it criminal?

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#65

Post by bmer »

ChrisRackauckas wrote: post #63
bmer wrote: post #45

Earlier quoted context omitted.

Yeah, I am pretty certain you're not right in this understanding. I am not the most knowledgeable, but basically all I know is scientific computing, so here are my two cents: Any C-like language does not use the "vector" as its "primitive" (not clear what you mean by primitive, and I am instead interpreting it as "machine type"), since integers, floats, characters, are the "primitives" (machine types), which are the…

>say, Python+NumPy, because then you can use everything the Python ecosystem has to offer, and aren't limited to the MATLAB ecosystem That has an implicit assumption that Python's ecosystem is better for this kind of work than MATLAB which just isn't true in many areas of scientific computing. Take differential equations for example. MATLAB has matcont which is a great bifurcation analysis library which is unrivaled…

> MATLAB has matcont which is a great bifurcation analysis library which is unrivaled by Python (PyDSTool is a toy in comparison).

I know lots of people who work with dynamical systems (math bio), and most people don't use MATLAB's matcont, as it is a toy compared to XPP/AUT. People tolerate XPP/AUT's archaic userface for its power, and can also export data easily to Python for grahping.

Also, the DDE thing is no longer true: https://aip.scitation.org/doi/10.1063/1.5019320

> Python's ODE (SciPy, Sundials wrappers, etc) solvers are much less developed than MATLAB's

What do you mean by "much less developed"? ODE solving is an "old problem" in the sense that it has been optimally addressed in C/Fortran/C++ and its just better to make wrappers around that existing code. ODEPACK/SUNDIALS/PETSc are great examples of a cutting edge standard ODE/PDE solvers which have Python wrappers, and if you've got wrappers around them, you're going to be hard pressed to find anything better.

Then there are people who are developing new ways of integrating ODEs numerically, and for them, MATLAB is only good as a first pass/prototype thing. A professor I know working on something like this doesn't bother using MATLAB for it (C++/Python).

> compilation with Numba or Cython doesn't work well in the case of differential equations since there are a lot of small function calls

Again, I have no clue what you mean by this. I just had a paper published which modelled a system involving hundreds of ODEs, with every involved function in the dynamics being compiled using numba's "nopython" mode. The stuff is blazing fast, only about 10 times slower than a C implementation. The structure of the program looked like this:

scipy.integrate.odeint -> f -> sub-function ... -> sub-function ...

`f` (the function to compute derivatives at each time point) and `sub-function`(s) (which are, I guess, the small function calls you were referring to?) were compiled using numba's nopython mode. It was super easy. You can have a look at the relevant code here: https://github.com/bzm3r/numba-ncc/blob/master/core/dynamics...

So I don't get what you mean by "Numba/Cython doesn't work well in the case of DEs since there are a lot of small function calls".

Note that MAPLE/Mathematica are not in the same camp as MATLAB, since they are primarily symbolic mathematics engines. Both are fantastic though, I agree.

No complaints about Julia. Awesome stuff. Just like Rust. I am not a huge Python fan, in the sense that I am actively moving away from it to "better tools", but I still think Python mostly beats MATLAB.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#66

Post by ChrisRackauckas »

bmer wrote: post #65
ChrisRackauckas wrote: post #63

Earlier quoted context omitted.

>say, Python+NumPy, because then you can use everything the Python ecosystem has to offer, and aren't limited to the MATLAB ecosystem That has an implicit assumption that Python's ecosystem is better for this kind of work than MATLAB which just isn't true in many areas of scientific computing. Take differential equations for example. MATLAB has matcont which is a great bifurcation analysis library which is unrivaled…

> MATLAB has matcont which is a great bifurcation analysis library which is unrivaled by Python (PyDSTool is a toy in comparison). I know lots of people who work with dynamical systems (math bio), and most people don't use MATLAB's matcont, as it is a toy compared to XPP/AUT. People tolerate XPP/AUT's archaic userface for its power, and can also export data easily to Python for grahping. Also, the DDE thing is no lon…

>I know lots of people who work with dynamical systems (math bio), and most people don't use MATLAB's matcont, as it is a toy compared to XPP/AUT. People tolerate XPP/AUT's archaic userface for its power, and can also export data easily to Python for grahping.

No, XPP/AUT is almost strictly less powerful than matcont. It can recognize and handle a much smaller set of bifurcations. Even PyCont (part of PyDSTool) can do some things XPP/AUT can't (though there it's much more of a tradeoff). XPP/AUT is good enough for most math bio though since these higher order bifurcations are much more rare to actually find in models.

>Also, the DDE thing is no longer true: https://aip.scitation.org/doi/10.1063/1.5019320

That's matching dde23 which is only non-stiff with constant lags. Still very very simple and cannot handle a lot of DDEs. Mathematica and MATLAB handles state-dependent DDEs. Maple, Julia, and Fortran via Harier's RADAR5 handle stiff state-dependent DDEs.

>The stuff is blazing fast, only about 10 times slower than a C implementation.

This is the overhead I was mentioning. I say it's slow since it's 10x slower than the C implementation. If that's fast enough for you, that's fine, but there's still a lot to be gained there.

>Note that MAPLE/Mathematica are not in the same camp as MATLAB, since they are primarily symbolic mathematics engines. Both are fantastic though, I agree.

Look at their differential equation solver merits in full detail and you'll see that there's a ton of things these cover that Python libraries don't. I was surprised at first two, but they aren't just symbolic engines. Maple has some of the best stuff for stiff DDEs for example, and Mathematica's Verner + interpolation setup is very modern and matches the Julia stuff while MATLAB/SciPy etc. is still using Dormand-Prince (dopri5, ode45).

I am not saying Python's libraries aren't fine. They definitely are fine if you don't need every little detail and if you don't need every lick of speed. But as you said, it's leaving behind 10x on the table. Also, a lot of its integrators don't allow complex numbers. Also it doesn't have access to much IMEX and exponential integrator stuff. So the Python libraries are fine, but the are far from

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#67

Post by monocasa »

gnufx wrote: post #61
monocasa wrote: post #32

Earlier quoted context omitted.

The interconnects have gotten waaaayyy better in datacenters over the past five years or so when compared to Infiniband. Stuff like FPGAs doing data plane routing, and all of the "converged Ethernet" standards like RoCE have really narrowed the gap between Ethernet and Infiniband.

However much Ethernet has changed, it's not appropriate for typical HPC use (c.f. IB, OPA, Aries, ...). I investigated Cisco's UCS at some length, for instance, which they had persuaded an important person to buy, and that's at least partly the baby of an MPI person. (I recall Infiniband was designed as a "datacentre" thing.)

Can you expand on why?

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#68

Post by bmer »

ChrisRackauckas wrote: post #66
bmer wrote: post #65

Earlier quoted context omitted.

> MATLAB has matcont which is a great bifurcation analysis library which is unrivaled by Python (PyDSTool is a toy in comparison). I know lots of people who work with dynamical systems (math bio), and most people don't use MATLAB's matcont, as it is a toy compared to XPP/AUT. People tolerate XPP/AUT's archaic userface for its power, and can also export data easily to Python for grahping. Also, the DDE thing is no lon…

>I know lots of people who work with dynamical systems (math bio), and most people don't use MATLAB's matcont, as it is a toy compared to XPP/AUT. People tolerate XPP/AUT's archaic userface for its power, and can also export data easily to Python for grahping. No, XPP/AUT is almost strictly less powerful than matcont. It can recognize and handle a much smaller set of bifurcations. Even PyCont (part of PyDSTool) can d…

Hmm. Interesting stuff. I am surprised that Julia has come along so far. Where can I read more about Julia's integrators?

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#69

Post by zackmorris »

albertzeyer wrote: post #55
zackmorris wrote: post #35

Earlier quoted context omitted.

I'm not sure if you or albertzeyer asked first, but what I meant by that is that MATLAB is similar to any other C-like language, except uses the vector as its primitive instead of something like integer or float. That's really all there is to it. Other than a few details about notation, every major concept of MATLAB stems from that and is easily understood and predictable. MATLAB (or non-proprietary analogs like GNU…

In TensorFlow, a vector (or a tensor) is also one of the fundamental datatypes. > The end result is that it's trivial (at least in theory) to go from a TensorFlow implementation to a MATLAB implementation. But it's very difficult to go the other direction. I would say just the opposite. From Matlab to TF should be trivial, but the other way not so much. Or can you give me an example in Matlab which would be hard to t…

I'm much more familiar with MATLAB than I am with TensorFlow so I might be wrong about the lowest-level computation stuff. Part of it is an architecture issue. I can imagine writing a transpiler from MATLAB that automatically dices up the matrix calls to work over a distributed system, maybe with something like Go or Elixir running under the hood and sending the jobs in batches to other processors, then joining all of those results, which abstracts away most concurrency issues.

I find it more difficult to think about TensorFlow this way, because to me it seems more similar to OpenCL or OpenGL, where you are dealing with multiple cores reading from a single memory space and transpiling the TensorFlow code to something like a shader. That type of code works more like SIMD or VLIW and has trouble with things like branching or dynamic codepaths or even reading/writing from/to a computer's main memory.

So what I'm looking for is a way to write at the high-level abstraction of MATLAB syntax and then have the runtime dice that up internally as TensorFlow or Elixir or whatever (that part doesn't matter to me as much). Maybe Julia can do that but I haven't learned it yet. But I want to stay away from the boilerplate - things like binding buffers or carrying the mental burden of having separate vector and system memories.

Re: New GPU-Accelerated Supercomputers Change the Balance of Power on the TOP500

#70

Post by ChrisRackauckas »

bmer wrote: post #68
ChrisRackauckas wrote: post #66

Earlier quoted context omitted.

>I know lots of people who work with dynamical systems (math bio), and most people don't use MATLAB's matcont, as it is a toy compared to XPP/AUT. People tolerate XPP/AUT's archaic userface for its power, and can also export data easily to Python for grahping. No, XPP/AUT is almost strictly less powerful than matcont. It can recognize and handle a much smaller set of bifurcations. Even PyCont (part of PyDSTool) can d…

Hmm. Interesting stuff. I am surprised that Julia has come along so far. Where can I read more about Julia's integrators?

It's all in the docs for DifferentialEquations.jl. For example, here's the page for the first order ODE solvers: http://docs.juliadiffeq.org/latest/solvers/ode_solve.html .
Post reply on HN