Live data from Hacker News

HPC is dying, and MPI is killing it

dursi.ca

71–80 of 125 posts

Re: HPC is dying, and MPI is killing it

#71
post #66

This happened with Smalltalk. I was a Smalltalk coder. I thought it was the best thing since sliced bread. It has always been clear to me that Smalltalk is far superior to Java. I left the company after a little while, to do C++ graphics. I later heard that my former employer rewrote their Smalltalk application in Java. Now no one uses Smalltalk anymore. While Objective-C is based on Smalltalk, Smalltalk was far easi…

What "new methods" are you thinking of? What let Java get ahead of Smalltalk for me personally, as someone getting into programming in 1996, was that i could write it in the text editor i already had, compile it with a compiler i could get for free, and then post the source code on Geocities (actually, Xoom - remember that?) to share with others. Whereas when i tried to get into Smalltalk, the first thing i had to do…

Hmm, I don't know. I'm pretty damn productive just programming in notepad(equivalent).

Re: HPC is dying, and MPI is killing it

#73
post #4

Yes, "high performance computing" is dying. There's no commercial market for it. Check the list of the top 500 supercomputers in the world.[1] The top 10 are all Government operations. In the top 25, there are a few oil companies, mostly running big arrays of Intel Xeons. CPU clock speeds maxed out between 3-4GHz a decade ago. Nobody develops special supercomputing CPUs any more. The market is tiny. Old supercomputer…

These days Crays run intel Xeons.

Re: HPC is dying, and MPI is killing it

#74
post #59
post #20

Earlier quoted context omitted.

Oh, sure. I don't think anyone should start rewriting old codes; but as new projects start, I think we have a lot more options out there than we did 10 years ago, and it's worth looking closely at them before starting, rather than defaulting to something. Especially since, once you start, you're probably pretty much locked into whatever you chose for a decade or so.

So, say you wanted to write a weather model, or engineering fluid mechanics model. Which options (besides MPI) you would look at?

Chapel has been used for incompressible moving-grid fluid dynamics, so it's certainly feasible. For that problem the result was ~33% the lines of code of the MPI version. There is a performance hit, but the issues are largely understood; if (say) a meteorological centre were to put its weight behind it, a lot of things could get done.

It's also pretty easy to see how UPC or co-array fortran (which is part of the standard now, so isn't going anywhere any time soon) would work. They'd fall closer to MPI in complexity and performance.

You couldn't plausibly do big 3d simulations in Spark today; that's way outside of what it was designed for. Now analysing the results, esp of a suite of runs, that might be interesting.

Re: HPC is dying, and MPI is killing it

#75
post #2

I came into the essay with suspicion. A map-reduce system like Hadoop isn't a good fit for HPC problems, and I thought it would argue that MPI is old => it's stuck in the past. Instead, and to my joy, it was a well-reasoned essay with good, solid points. My only quibble is that Charm++ is not "a framework for particle simulation methods". While the molecular dynamics program NAMD has been using it for 20 years, which…

People need to be careful about talking about Hadoop as just a map-reduce system.

It's YARN container system is flexible enough to run any JVM application. For example we use it to run an autoscaling ElasticSearch cluster alongside our Hadoop workloads. And we are actively investigating using it to run our Scala microservices.

Re: HPC is dying, and MPI is killing it

#76
post #13
post #6

I've heard there is a new effort being led by Torsten Hoefler to modernize MPI and address a number of the issues mentioned in this article. http://htor.inf.ethz.ch/ I was at a talk of his last year and there are a number of fault tolerant MP algorithms being drawn in. MPI hasn't been updated in ages, I don't think that necessarily means we need to ditch it, it just means the standard needs to be modernized. I don't…

Certainly Prof Hoefler has done a lot of work driving the design of updated remote-memory access for of MPI-3, and any further progress would be welcomed; but I don't think any modernizing of MPI can fix the basic problem. At the end of the day, it's just too low level for application developers, while being too high-level for tool developers. There are parts of MPI which don't share this problem so much - the collec…

How you guys think of Chapel? It compiles so slow.

Re: HPC is dying, and MPI is killing it

#77
post #4

Yes, "high performance computing" is dying. There's no commercial market for it. Check the list of the top 500 supercomputers in the world.[1] The top 10 are all Government operations. In the top 25, there are a few oil companies, mostly running big arrays of Intel Xeons. CPU clock speeds maxed out between 3-4GHz a decade ago. Nobody develops special supercomputing CPUs any more. The market is tiny. Old supercomputer…

> CPU clock speeds maxed out between 3-4GHz a decade ago.

Even for x86 this isn't true (4+GHz is at least possible), let alone platforms like POWER which have already pushed beyond 5GHz. Fancier things like vacuum-channel transistors, graphene transistors, etc. could push that even further once they break into commercial viability.

Not that clock speed alone really matters all that much compared to the other performance benefits of high-performance RISC architectures like POWER and SPARC...

> Nobody develops special supercomputing CPUs any more.

Today I learned that Blue Gene was a figment of my imagination :)

Special supercomputing CPUs are still being developed. The reason why they seem insignificant is because their market size has remained relatively constant, while the markets for general-purpose, non-supercomputing-specific platforms have grown much more rapidly. This doesn't mean supercomputing is dead necessarily, just like how the invention of the microwave oven doesn't mean that ordinary ovens are suddenly dead. Rather, it's just an indicator of different use cases, and the different markets thereof.

> The top 10 are all Government operations.

It's a bit misleading (though I suppose technically accurate) to list academic institutions (like the University of Texas, which holds the #7 spot) as "Government operations"; they're government-funded, yes, but there's a big difference between that and, say, an actual government agency directly managing such an installation. I also fail to see how even a majority of those being government installations has anything to do with anything; governments typically have much greater capital to spend on such things - and greater need for such things - than all but the most massive commercial entities.

HPC was never really the purview of commercial enterprises anyway (unless they had extreme computational requirements). The uptick in the use of COTS products for high-performance computing among enterprises (particularly big Internet-reliant ones like Google) wasn't really at the expense of the HPC crowd losing potential users; it's rather just something that formed very recently alongside HPC already being a niche topic.

Basically, by your arguments, "high-performance computing" has been dying for basically as long as it's existed.

> Grosch's Law [2] stopped working a long time ago.

Only because the world switched to clustering, where Grosch's Law doesn't quite apply, and hasn't addressed the limitations of current transistor technology (like the above-mentioned vacuum-channel and graphene transistor technologies, among many others).

> Maximum price/performance today is achieved with racks of midrange CPUs, which is why that's what every commercial data center has.

That's what "every commercial data center has" (this isn't exactly true, but we'll go with it for now) more because of price alone than because of an actually-calculated price/performance ratio. Businesses tend to think in terms of short-term investments much easier than they tend to think in terms of long-term investments (in contrast with academic and often government institutions, which tend to think in the opposite direction, and therefore have entirely different sets of problems in many cases).

Meanwhile, the big businesses that really do actively calculate an optimal price/performance ratio (like Google) aren't the ones using COTS solutions; they usually have the financial capability to invest in homegrown solutions and cut out any unnecessary expense, and are certainly not just buying a bunch of prebuilt servers from Dell. Google in particular has started to invest heavily in IBM's Open POWER initiative, probably due to a perception that POWER will offer a better price/performance ratio than x86 in their already-very-customized hardware stack.

Re: HPC is dying, and MPI is killing it

#78
post #63
post #12

Earlier quoted context omitted.

Yeah good point. I just felt it might be a misleading of the author to suggest Chapel as an alternative when you cannot possibly write a useful program with it.

There are numerous benchmarks implemented in Chapel, some of which are competitive with other implementations (see paper reference in article). There is a growing standard library and literally thousands of test codes that represent a broad set of functionality. That said, Chapel is not yet a product grade language, nor is it promoted as such. Chapel may not be an appropriate replacement for all MPI programs, but it…

I agree with this, but it also sort of risks being a self-fulfilling prophesy; everyone uses MPI because everyone uses MPI, and no one uses Chapel yet because no one uses Chapel yet. At some point, we who are willing to be early adopters need to just start.

Re: HPC is dying, and MPI is killing it

#79
post #19

Earlier quoted context omitted.

Thanks! I'm probably mischaracterizing Charm++ a bit, because I'm most familiar with it in particle context (OpenAtom, ChaNGa, NAMD). I guess it's probably particularly used in that context just because it's so good with very fine-grained distributions of work units. I'll edit that line in the article.

Another nitpick: 128 cores equals ~ 4 nodes only if your problem is not bound by memory bandwidth. If it is, 128 cores equals ~ 16 cores, and then the interconnect matters a lot. Great writeup though. I do think we need to get more people into the mindset that MPI won't be the standard in 10 years, otherwise it will still be the standard in 10 years.

[deleted]

Re: HPC is dying, and MPI is killing it

#80
post #40

Too bad he didn't talk about GPGPU killing MPI too or not. I don't know enough to say. I'm not familiar with the HPC space but I thought a lot of new work, at least in machine learning, was migrating to GPGPU instead of traditional CPUs. The compute per $ or per watt payoff is too large to ignore.

I wouldn't say GPUs are killing off MPI. You still need some way to pass data between nodes/GPUs (most of these datasets can't fit within a single GPU). What you are seeing though, is less and less use of the CPU. If code developers decide to use GPUs, they try and move their data onto the GPU and keep it there for as long as possible (data movement across PCIe is a killer for performance). ORNL's new machine Sumiit [1] will have 1/4 the nodes of their current machine, Titan, but multiple GPUs per node. Thus emphasizing the move away from CPUs and onto GPUs. Again though, there are still 3K nodes and you need some efficient way to pass data between those nodes.

[1] https://www.olcf.ornl.gov/summit/

Post reply on HN