Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HPC is dying, and MPI is killing it

dursi.ca

Post reply on HN
Print view View original

31–40 of 125 posts

Re: HPC is dying, and MPI is killing it

#31

Post by rz2k »

shepardrtc wrote: post #24
rz2k wrote: post #21

The article briefly mentions Erlang with its focus on message passing, and the focus on being fault tolerant should be another benefit. I could find a few mentions of Erlang used for simulations, such as here[1], but I am curious whether there is much actual usage in scientific computing, or whether there are some problems in practice. [1] https://books.google.com/books?id=p0h9vAb1m7IC&pg=PA365#v=on...

Erlang itself is not good for the type of numerical computation typical done in HPC. It is amazing as a backend, and there are several ways it can call code written in other languages through the use of ports or NIFS, but if you try to do massive number-crunching using its own libraries then you're going to be unhappy with the results.

How about in simulations where a large number of cellular automata are interacting with each other, but individually only carrying out simple computations?

Re: HPC is dying, and MPI is killing it

#32

Post by tormeh »

FraaJad wrote: post #29
tormeh wrote: post #14

Earlier quoted context omitted.

I LIKE PROGRAMMING IN ALL-CAPS. IT MAKES EVERYTHING I WRITE LOOK OFFICIAL, NO-NONSENSE AND GENERALLY HARD-CORE. INSTEAD OF HAVING A CONVERSATION WITH THE COMPILER I INSTEAD SHOUT!!! COMMANDS AT IT.

Oh so clever to make fun of FORTRAN. This is how the modern FORTRAN Hello world looks like... program hello print *, "Hello World!" end program hello

I was actually partly serious. I voluntarily program in all-caps fortran because I think it makes my code look old/funny/the things I said.

Re: HPC is dying, and MPI is killing it

#33

Post by tormeh »

felixgallo wrote: post #25
rz2k wrote: post #21

The article briefly mentions Erlang with its focus on message passing, and the focus on being fault tolerant should be another benefit. I could find a few mentions of Erlang used for simulations, such as here[1], but I am curious whether there is much actual usage in scientific computing, or whether there are some problems in practice. [1] https://books.google.com/books?id=p0h9vAb1m7IC&pg=PA365#v=on...

erlang is not competitive in math calculation performance, and fault tolerance is not as important when your program has a relatively short, defined lifetime and well-understood inputs and outputs. As a message router and underlying infrastructure for a cluster it'd probably work pretty well, but then you'd have an impedance mismatch and operational concerns between the routing infrastructure and the calculation infr…

Erlang is like Python in that you normally use C for the single-thread numerical stuff.

Re: HPC is dying, and MPI is killing it

#34

Post by sampo »

tormeh wrote: post #14
shepardrtc wrote: post #5

The Fortran Standards Committee is attempting to make HPC easier through the use of coarrays, which are essentially massive abstractions over MPI. I really wish people would give Fortran a second chance. It has come a long way from the ancient, all-caps days.

I LIKE PROGRAMMING IN ALL-CAPS. IT MAKES EVERYTHING I WRITE LOOK OFFICIAL, NO-NONSENSE AND GENERALLY HARD-CORE. INSTEAD OF HAVING A CONVERSATION WITH THE COMPILER I INSTEAD SHOUT!!! COMMANDS AT IT.

Fortran has not been all caps after 1991, when the Fortran 90 standard came out. You knowledge is 24 years old.

Re: HPC is dying, and MPI is killing it

#35

Post by semi-extrinsic »

ljdursi wrote: post #19
dalke wrote: post #2

I came into the essay with suspicion. A map-reduce system like Hadoop isn't a good fit for HPC problems, and I thought it would argue that MPI is old => it's stuck in the past. Instead, and to my joy, it was a well-reasoned essay with good, solid points. My only quibble is that Charm++ is not "a framework for particle simulation methods". While the molecular dynamics program NAMD has been using it for 20 years, which…

Thanks! I'm probably mischaracterizing Charm++ a bit, because I'm most familiar with it in particle context (OpenAtom, ChaNGa, NAMD). I guess it's probably particularly used in that context just because it's so good with very fine-grained distributions of work units. I'll edit that line in the article.

Another nitpick: 128 cores equals ~ 4 nodes only if your problem is not bound by memory bandwidth. If it is, 128 cores equals ~ 16 cores, and then the interconnect matters a lot.

Great writeup though. I do think we need to get more people into the mindset that MPI won't be the standard in 10 years, otherwise it will still be the standard in 10 years.

Re: HPC is dying, and MPI is killing it

#36

Post by tormeh »

acadien wrote: post #16
ljdursi wrote: post #13

Earlier quoted context omitted.

Certainly Prof Hoefler has done a lot of work driving the design of updated remote-memory access for of MPI-3, and any further progress would be welcomed; but I don't think any modernizing of MPI can fix the basic problem. At the end of the day, it's just too low level for application developers, while being too high-level for tool developers. There are parts of MPI which don't share this problem so much - the collec…

Again I'm not sure if I agree or disagree with this. My hatred of MPI is only outweighed by the fact that I can use it... and my code works. I think a large part of the inertia behind MPI is legacy code. Often the most complex part of HPC scientific codes is the parallel portion and the abstractions required to perform them (halo decomposition etc). I can't imagine there are too many grad students out there who are e…

>who in industry has ever heard of Chapel

Chapel's made by Cray. If what you're saying is true then Cray's not done a very good job of advertising Chapel. God knows they have the capability to advertise properly.

Re: HPC is dying, and MPI is killing it

#37

Post by tormeh »

sampo wrote: post #34
tormeh wrote: post #14

Earlier quoted context omitted.

I LIKE PROGRAMMING IN ALL-CAPS. IT MAKES EVERYTHING I WRITE LOOK OFFICIAL, NO-NONSENSE AND GENERALLY HARD-CORE. INSTEAD OF HAVING A CONVERSATION WITH THE COMPILER I INSTEAD SHOUT!!! COMMANDS AT IT.

Fortran has not been all caps after 1991, when the Fortran 90 standard came out. You knowledge is 24 years old.

Yeah, I know. It's just that I like programming in an old-school way for fun. I briefly considered not using structured programming at all, but that's kinda too much.

Re: HPC is dying, and MPI is killing it

#38

Post by sampo »

in High Performance Computing, there is (1) 3-dimensional simulations (weather, fluid dynamics, structural mechanics, all kinds of physics simulations, like magnetic storms in space or nuclear reactors etc.) and then there is (2) everything else, like data mining, machine learning, genomics etc.

Some of the sparse matrix computations in structural mechanics and in some machine learning algorithms have some overlap. But mostly, group 2 has little reason to be interested in what group 1 is doing.

Now, group 2 obviously has more modern tools than the 3d-simulation community, because machine learning came to common use much later that numerical fluid mechanics.

But do 3d-simulation people also have much reason to be interested in what the machine learning people are doing?

The "machine learning / big data" people are probably not doing anything that makes a weather prediction model to run faster? Or are they?

Re: HPC is dying, and MPI is killing it

#39

Post by east2west »

I am trying to get into distributed computing so this article is particularly interesting to me. I may be mistaken so please excuse my naivety if my points are off marks. I thought MPI was mainly geared towards communication-heavy tasks where the underlying network is specialized, for example infiniband or bus between CPUs. One use of MPI is to manage distributed memory tasks between different physical CPUs while threads run on multiple cores of same CPU. Spark, I believe, doesn't handle cases like this well because JVM hides low level details. I have read papers that propose to layer MPI over RDMA rather than expose a flat memory model, which came as a surprise to me but it shows the flexibility of MPI. One thing unclear to me what performance we can expect from MPI when we use commodity network gears, and how it compares to Spark. The article is absolutely correct MPI leaves robustness to user and that is today an oversight.

Modern Hadoop ecosystem is designed for different workload from MPI's. It emphasizes co-localizing date and computation, seamless robustness,and trades off raw power for simple programmingmodels. MapReduce turns out too simple, so Spark implements graph execution, which is nothing new to HPC. As far I know Spark's authors don't believe it is ready for distributed numerical linear algebra yet. But a counterpoint is that I am seeing machine learning libraries using Spark, so perhaps things are improving.

One thing I have learnt today is that MPI isn't gaining popularity. I just have a hard time picturing a JVM language in overall control in HPC where precise control of memory is paramount to performance.

Re: HPC is dying, and MPI is killing it

#40

Post by bjwbell »

Too bad he didn't talk about GPGPU killing MPI too or not. I don't know enough to say.

I'm not familiar with the HPC space but I thought a lot of new work, at least in machine learning, was migrating to GPGPU instead of traditional CPUs. The compute per $ or per watt payoff is too large to ignore.

Post reply on HN