Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HPC is dying, and MPI is killing it

dursi.ca

Post reply on HN
Print view View original

41–50 of 125 posts

Re: HPC is dying, and MPI is killing it

#41

Post by dhaivatpandya »

spott wrote: post #23
wffurr wrote: post #7

Earlier quoted context omitted.

Did you RTFA? It's about MPI: "MPI is a language-independent communications protocol used to program parallel computers." Runs fine on commodity clusters.

>Runs fine on commodity clusters. Kind of.... For simple, low communication jobs this is true. But when you start trying to find the eigenvectors of a large sparse matrix, communication becomes your bottleneck, at which point MPI on commodity clusters (those without a really fancy interconnect) "works", but not fast enough to be useful.

Communication would become a bottleneck regardless of whether you're using MPI or something else. The problem you're talking about has nothing to do with MPI; it is intrinsic to distributed computing itself.

Re: HPC is dying, and MPI is killing it

#43

Post by ljdursi »

semi-extrinsic wrote: post #35
ljdursi wrote: post #19

Earlier quoted context omitted.

Thanks! I'm probably mischaracterizing Charm++ a bit, because I'm most familiar with it in particle context (OpenAtom, ChaNGa, NAMD). I guess it's probably particularly used in that context just because it's so good with very fine-grained distributions of work units. I'll edit that line in the article.

Another nitpick: 128 cores equals ~ 4 nodes only if your problem is not bound by memory bandwidth. If it is, 128 cores equals ~ 16 cores, and then the interconnect matters a lot. Great writeup though. I do think we need to get more people into the mindset that MPI won't be the standard in 10 years, otherwise it will still be the standard in 10 years.

"won't be the standard in 10 years, otherwise it will still be the standard in 10 years." -- nicely said.

Re: HPC is dying, and MPI is killing it

#44

Post by ljdursi »

bjwbell wrote: post #40

Too bad he didn't talk about GPGPU killing MPI too or not. I don't know enough to say. I'm not familiar with the HPC space but I thought a lot of new work, at least in machine learning, was migrating to GPGPU instead of traditional CPUs. The compute per $ or per watt payoff is too large to ignore.

I think with GPGPU, the issue is a little different; it's local computation, so a bit orthogonal to distributed-memory coordination. But it is interesting to see how many higher-level libraries and other tools (like OpenMP4/OpenACC) are springing up around GPU usage. It's hard not to be a bit jealous...

Re: HPC is dying, and MPI is killing it

#45

Post by ljdursi »

east2west wrote: post #39

I am trying to get into distributed computing so this article is particularly interesting to me. I may be mistaken so please excuse my naivety if my points are off marks. I thought MPI was mainly geared towards communication-heavy tasks where the underlying network is specialized, for example infiniband or bus between CPUs. One use of MPI is to manage distributed memory tasks between different physical CPUs while thr…

I agree that languages that rely on tracing GC seem like they're fundamentally at a disadvantage when it comes to pushing the envelope of single-node performance; the best article I've read arguing this was actually in the context of mobile games, rather than HPC, but I can't for the life of me find the article now.

I don't know if Spark itself is the right way forward; but it's an example of a very productive high-level language for certain forms of distributed memory computing. And some of these issues - like the JVM - aren't fundamental to Spark's approach; there's no inherent reason why something similar couldn't be built based on C++ or the like.

Re: HPC is dying, and MPI is killing it

#46

Post by dorfsmay »

east2west wrote: post #39

I am trying to get into distributed computing so this article is particularly interesting to me. I may be mistaken so please excuse my naivety if my points are off marks. I thought MPI was mainly geared towards communication-heavy tasks where the underlying network is specialized, for example infiniband or bus between CPUs. One use of MPI is to manage distributed memory tasks between different physical CPUs while thr…

> I thought MPI was mainly geared towards communication-heavy tasks where the underlying network is specialized

The beauty of mpi is:

  * its definition is completely open

  * it segregate the high level message passing interface from the low level stuff
This means that code that was written on cheap old commodity network gear over tcp/ip will work on brand new specialised hardware using their own protocol. Because it's fully open, any hardware vendor can provide MPI driver for their hardware at virtually no cost.

Re: HPC is dying, and MPI is killing it

#47

Post by sgt101 »

sampo wrote: post #38

in High Performance Computing, there is (1) 3-dimensional simulations (weather, fluid dynamics, structural mechanics, all kinds of physics simulations, like magnetic storms in space or nuclear reactors etc.) and then there is (2) everything else, like data mining, machine learning, genomics etc. Some of the sparse matrix computations in structural mechanics and in some machine learning algorithms have some overlap. B…

They are doing it (interesting things) for lower capex and lower development costs. On opex, good for operations, bad for power consumption (relatively).

In terms of absolute performance HPC is absolutely faster. In terms of bangs for bucks, Big Data is hands down faster. Also in terms of accessibility Big Data is hugely easier - I can build you a 100 core big data system for $300k

Re: HPC is dying, and MPI is killing it

#48

Post by VLM »

shepardrtc wrote: post #5

The Fortran Standards Committee is attempting to make HPC easier through the use of coarrays, which are essentially massive abstractions over MPI. I really wish people would give Fortran a second chance. It has come a long way from the ancient, all-caps days.

Maybe after that, the cool kids would "permit" us to try Perl again, if they declare it cool again.

Modern Perl is pretty nice. 90s Perl still not so good.

Re: HPC is dying, and MPI is killing it

#49

Post by peterwwillis »

wffurr wrote: post #7
Animats wrote: post #4

Yes, "high performance computing" is dying. There's no commercial market for it. Check the list of the top 500 supercomputers in the world.[1] The top 10 are all Government operations. In the top 25, there are a few oil companies, mostly running big arrays of Intel Xeons. CPU clock speeds maxed out between 3-4GHz a decade ago. Nobody develops special supercomputing CPUs any more. The market is tiny. Old supercomputer…

Did you RTFA? It's about MPI: "MPI is a language-independent communications protocol used to program parallel computers." Runs fine on commodity clusters.

What would you define as a "commodity cluster"? To me it's a 512-core vendor-specific blade server with special interfaces to get more bandwith at lower latency across longer links. But maybe i'm just an old fogey.

Re: HPC is dying, and MPI is killing it

#50

Post by sgt101 »

acadien wrote: post #16
ljdursi wrote: post #13

Earlier quoted context omitted.

Certainly Prof Hoefler has done a lot of work driving the design of updated remote-memory access for of MPI-3, and any further progress would be welcomed; but I don't think any modernizing of MPI can fix the basic problem. At the end of the day, it's just too low level for application developers, while being too high-level for tool developers. There are parts of MPI which don't share this problem so much - the collec…

Again I'm not sure if I agree or disagree with this. My hatred of MPI is only outweighed by the fact that I can use it... and my code works. I think a large part of the inertia behind MPI is legacy code. Often the most complex part of HPC scientific codes is the parallel portion and the abstractions required to perform them (halo decomposition etc). I can't imagine there are too many grad students out there who are e…

I work in a medium sized telco, we have had Spark for prototypes for over a year and have now got a roadmap to put it in production.

I think Spark will totally displace map-reduce in the next 12 months (because it's got map reduce in it, but in memory).

Post reply on HN