Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HPC is dying, and MPI is killing it

dursi.ca

Post reply on HN
Print view View original

111–120 of 125 posts

Re: HPC is dying, and MPI is killing it

#111

Post by jabl »

sgt101 wrote: post #105
markhahn wrote: post #55

Earlier quoted context omitted.

that's silly: HPC has been pinching pennies before big data was a thing. and the computer industry is biz: you get what you pay for. if you can live with Gb performance, you can drop around $2k (IB card, cables, switches) off your price. But it's not as if the hardware is any different, faster or more accessible.

I think it's economics, GPU's are sold by the million, super computers interconnects are sold by the thousands. Commodity kit is mass produced spreading design, vvt and manufacturing tooling costs. The hardware is different in terms of the layout. Aggregations of small cores on boards (gpus) vs. very high speed large cores with lots of local memory. Highly localised connections vs. an interconnect fabric. And it is m…

Huh, where did GPU's enter the discussion?

Anyway, for a typical HPC cluster, it's bog standard x86 hardware, the only remotely exotic thing is the Infiniband network. Common wisdom says that since Infiniband is a niche technology, it's hugely expensive, but strangely(?) it seems to have (MUCH!) better bang per buck than ethernet. A 36-port FDR IB (56 Gb/s) switch has a list price of around $10k, whereas a quick search seems to suggest a 48-port 10GbE switch has a list price of around $15k. So the per-port price is roughly in the same ballpark, but IB gives you >5 times better bandwidth and 2 orders of magnitude lower MPI latency. Another advantage is that IB supports multipathing, so you can build high bisection bandwidth networks (all the way to fully non-blocking) without needing $$$ uber-switches on the spine.

Re: HPC is dying, and MPI is killing it

#112

Post by shepardrtc »

rz2k wrote: post #100
shepardrtc wrote: post #87

Earlier quoted context omitted.

You certainly can do that. In fact, I believe I saw an example of that in a tutorial somewhere talking about what Erlang/Elixir would be good for (I include Elixir because its the same VM controlling processes underneath, but a "nicer" syntax on top). Now, would it be the best language for that? Depends. If you're looking just for speed and massive computations, then no. In the end, C++ pretty much rules everything i…

I actually started going through Programming Elixir this past week because I was thinking it would eventually be a fun way to explore models through ad hoc substitution of rules and exogenous conditions. There are a lot of models available in NetLogo, but it doesn't look like it would be very powerful in terms of running larger scale simulations, and the scope of the language's usefulness is pretty limited compared t…

As an example, here's the game of life written in Elixir:

https://github.com/spikegrobstein/elixir-conway

How much better that is than a C++ version, I don't know. But if you want to get better at C++ and give it a try, forget what you know and read this book: http://www.amazon.com/Programming-Principles-Practice-Using-...

Re: HPC is dying, and MPI is killing it

#113

Post by DannoHung »

dalke wrote: post #109
DannoHung wrote: post #104

Earlier quoted context omitted.

Why does the system go down if a node goes down? Shouldn't the system keep chugging along at a slightly reduced capacity until the node comes back online? Sorry, I'm not involved in HPC at all. I know a little bit about Hadoop. I'm mostly interested in building online message processing and blended real-time/historical analytics. Our problem domain wouldn't want to lose all capacity if part of the system became unava…

There are several different aspects which make recovery hard. HPC tries to push the edge of what's possible with hardware. It does this by throwing redundancy out the window. First, the simulation can be set up to match the hardware. One simulation program I used expected that the nodes would be set up in a ring, so that messages between i and (i+1)%N were cheap. It ran on hardware with two network ports, one forward…

Ah, I see. I didn't realize so much of the problem domain was at the network layer.

For our domain, we'll gladly accept the increased network cost and node redundancy for durability because most of work ends up being not involved with other nodes (most of our computations can occur wherever the data is stored and mutations, aside from append, are infrequent).

Thank you for giving me some context.

Re: HPC is dying, and MPI is killing it

#114

Post by lvkale »

ljdursi wrote: post #19
dalke wrote: post #2

I came into the essay with suspicion. A map-reduce system like Hadoop isn't a good fit for HPC problems, and I thought it would argue that MPI is old => it's stuck in the past. Instead, and to my joy, it was a well-reasoned essay with good, solid points. My only quibble is that Charm++ is not "a framework for particle simulation methods". While the molecular dynamics program NAMD has been using it for 20 years, which…

Thanks! I'm probably mischaracterizing Charm++ a bit, because I'm most familiar with it in particle context (OpenAtom, ChaNGa, NAMD). I guess it's probably particularly used in that context just because it's so good with very fine-grained distributions of work units. I'll edit that line in the article.

(Thanks Andrew Dalke. You remember it after all these years!) Jonathan, actually, even in that set, OpenAtom is not a particle code. Its a quantum chemistry code where each electronic state is represented often by a large 3D array spread over processors.. For more recent examples of representative miniApps, see http://charmplusplus.org/benchmarks/ or our upcoming workshop (sorry for the plug: http://charm.cs.illinois.edu/charmWorkshop

The broader article deserves further though, and I hope to find time to respond. But it is clear that raising the level of abstraction beyond MPI is necessary.

Re: HPC is dying, and MPI is killing it

#115

Post by sgt101 »

jabl wrote: post #111
sgt101 wrote: post #105

Earlier quoted context omitted.

I think it's economics, GPU's are sold by the million, super computers interconnects are sold by the thousands. Commodity kit is mass produced spreading design, vvt and manufacturing tooling costs. The hardware is different in terms of the layout. Aggregations of small cores on boards (gpus) vs. very high speed large cores with lots of local memory. Highly localised connections vs. an interconnect fabric. And it is m…

Huh, where did GPU's enter the discussion? Anyway, for a typical HPC cluster, it's bog standard x86 hardware, the only remotely exotic thing is the Infiniband network. Common wisdom says that since Infiniband is a niche technology, it's hugely expensive, but strangely(?) it seems to have (MUCH!) better bang per buck than ethernet. A 36-port FDR IB (56 Gb/s) switch has a list price of around $10k, whereas a quick sear…

That's interesting, things may have changes with IB since I last looked.

The GPU thing seems to have fallen out of my original comment, I meant to write "I can build you a 100,000 core system for $300k" but some how the decimal point jumped left three times! To do that I would definitely have to use GPU's...

I am seriously lusting after such a device, I feel that there is much to be done.

Re: HPC is dying, and MPI is killing it

#116

Post by dorfsmay »

dalke wrote: post #84
dorfsmay wrote: post #46

Earlier quoted context omitted.

> I thought MPI was mainly geared towards communication-heavy tasks where the underlying network is specialized The beauty of mpi is: * its definition is completely open * it segregate the high level message passing interface from the low level stuff This means that code that was written on cheap old commodity network gear over tcp/ip will work on brand new specialised hardware using their own protocol. Because it's…

I think you've omitted something. PVM had that same beauty, but MPI won over PVM. Do you have any idea why?

I'm not sure... My only exposure to HPC was the install of a cluster with MPI over Infiniband. At the time I looked into MPI a bit and played with it at home (over wifi and ethernet).

Re: HPC is dying, and MPI is killing it

#117

Post by MichaelCrawford »

twic wrote: post #66
MichaelCrawford wrote: post #28

This happened with Smalltalk. I was a Smalltalk coder. I thought it was the best thing since sliced bread. It has always been clear to me that Smalltalk is far superior to Java. I left the company after a little while, to do C++ graphics. I later heard that my former employer rewrote their Smalltalk application in Java. Now no one uses Smalltalk anymore. While Objective-C is based on Smalltalk, Smalltalk was far easi…

What "new methods" are you thinking of? What let Java get ahead of Smalltalk for me personally, as someone getting into programming in 1996, was that i could write it in the text editor i already had, compile it with a compiler i could get for free, and then post the source code on Geocities (actually, Xoom - remember that?) to share with others. Whereas when i tried to get into Smalltalk, the first thing i had to do…

I'd say that your ability to write Java source in any text editor at all was, all by itself, a new method of doing Object-Oriented Programming.

We used Visual Smalltalk Enterprise. It had the cool feature that, at the end of the workday, I could make what amounted to a core dump, then the following morning I would load my core dump into a running program, and there would be all my open windows with the cursors in the right places in the source documents and so on.

That was quite cool I really enjoyed it, however that environment was profoundly non-portable. I expect that much of the success of Java as opposed to Smalltalk was the simple ability one had to post a tarball full of source code on one's FTP site.

Re: HPC is dying, and MPI is killing it

#118

Post by twic »

MichaelCrawford wrote: post #117
twic wrote: post #66

Earlier quoted context omitted.

What "new methods" are you thinking of? What let Java get ahead of Smalltalk for me personally, as someone getting into programming in 1996, was that i could write it in the text editor i already had, compile it with a compiler i could get for free, and then post the source code on Geocities (actually, Xoom - remember that?) to share with others. Whereas when i tried to get into Smalltalk, the first thing i had to do…

I'd say that your ability to write Java source in any text editor at all was, all by itself, a new method of doing Object-Oriented Programming. We used Visual Smalltalk Enterprise. It had the cool feature that, at the end of the workday, I could make what amounted to a core dump, then the following morning I would load my core dump into a running program, and there would be all my open windows with the cursors in the…

That wasn't new, though. C++ had that. Eiffel had that. Every programming language that isn't Smalltalk has that.

Re: HPC is dying, and MPI is killing it

#119

Post by MichaelCrawford »

twic wrote: post #118
MichaelCrawford wrote: post #117

Earlier quoted context omitted.

I'd say that your ability to write Java source in any text editor at all was, all by itself, a new method of doing Object-Oriented Programming. We used Visual Smalltalk Enterprise. It had the cool feature that, at the end of the workday, I could make what amounted to a core dump, then the following morning I would load my core dump into a running program, and there would be all my open windows with the cursors in the…

That wasn't new, though. C++ had that. Eiffel had that. Every programming language that isn't Smalltalk has that.

I'm not dead certain but I think smalltalk may have come before C++. For sure it was in widespread use before C++ was in widespread use.

Re: HPC is dying, and MPI is killing it

#120

Post by dalke »

lvkale wrote: post #114
ljdursi wrote: post #19

Earlier quoted context omitted.

Thanks! I'm probably mischaracterizing Charm++ a bit, because I'm most familiar with it in particle context (OpenAtom, ChaNGa, NAMD). I guess it's probably particularly used in that context just because it's so good with very fine-grained distributions of work units. I'll edit that line in the article.

(Thanks Andrew Dalke. You remember it after all these years!) Jonathan, actually, even in that set, OpenAtom is not a particle code. Its a quantum chemistry code where each electronic state is represented often by a large 3D array spread over processors.. For more recent examples of representative miniApps, see http://charmplusplus.org/benchmarks/ or our upcoming workshop (sorry for the plug: http://charm.cs.illinois…

(Indeed I do!)
Post reply on HN