Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HPC is dying, and MPI is killing it

dursi.ca

Post reply on HN
Print view View original

91–100 of 125 posts

Re: HPC is dying, and MPI is killing it

#91

Post by threeseed »

dalke wrote: post #82
threeseed wrote: post #75

Earlier quoted context omitted.

People need to be careful about talking about Hadoop as just a map-reduce system. It's YARN container system is flexible enough to run any JVM application. For example we use it to run an autoscaling ElasticSearch cluster alongside our Hadoop workloads. And we are actively investigating using it to run our Scala microservices.

Even years after Amazon started selling a lot more than just books, if people were asked "what does Amazon sell?", the answer was often "books." I looked at YARN now. I've not heard of it before. It doesn't look like it has anything to do with the topic at hand. How would one build an explicit solver for a 1D diffusion equation, corresponding to the examples given in the "HPC is dying, ..." article, using YARN? How d…

YARN is just a resource manager on top of which Hadoop jobs are run e.g. Hive, Pig.

It is analogous to a set of Docker containers distributed across nodes. The same methods you would use synchronize state in that situation you could use with YARN. For example using a persistent distributed system e.g. Hazelcast to handle system failures and checkpointing.

I am not saying this is some amazing solution to every HPC problem only that Hadoop is far, far more flexible than many people give it credit for.

Re: HPC is dying, and MPI is killing it

#92

Post by cgtyoder »

kenko wrote: post #69
ToastyMallows wrote: post #3

MPI is Message Passing Interface, for those who don't know. Author never explicitly says what MPI stands for. https://en.wikipedia.org/wiki/Message_Passing_Interface

"MPI, the Message Passing Interface, began as a needed standardization above a dizzying array of..." The capitalization and apposition makes it pretty explicit.

Way too far into the article for it to be useful.

Re: HPC is dying, and MPI is killing it

#93

Post by _delirium »

anigbrowl wrote: post #58
ToastyMallows wrote: post #3

MPI is Message Passing Interface, for those who don't know. Author never explicitly says what MPI stands for. https://en.wikipedia.org/wiki/Message_Passing_Interface

Yeah, I had to go back to google after failing to find it defined anywhere on the page. If you don't use something regularly yourself, you may remember the concept but forget the acronym. For want of a couple of sentences of context at the outset, the rest of the article was quite inaccessible. Maybe there's a lesson for the author here.

It wouldn't be hard to define up front, but in context I don't think assuming that the audience knows what MPI is poses much of a problem for the goals of the piece. This isn't a general-interest piece about HPC, but specifically an advocacy piece attempting to convince members of the HPC community that their strong attachment to MPI is detrimental to the field, and that they should refocus their efforts elsewhere. If someone doesn't know what MPI is, they are probably not strongly attached to it (and probably not in HPC), so aren't the people the author is trying to convince.

Re: HPC is dying, and MPI is killing it

#94

Post by dalke »

threeseed wrote: post #91
dalke wrote: post #82

Earlier quoted context omitted.

Even years after Amazon started selling a lot more than just books, if people were asked "what does Amazon sell?", the answer was often "books." I looked at YARN now. I've not heard of it before. It doesn't look like it has anything to do with the topic at hand. How would one build an explicit solver for a 1D diffusion equation, corresponding to the examples given in the "HPC is dying, ..." article, using YARN? How d…

YARN is just a resource manager on top of which Hadoop jobs are run e.g. Hive, Pig. It is analogous to a set of Docker containers distributed across nodes. The same methods you would use synchronize state in that situation you could use with YARN. For example using a persistent distributed system e.g. Hazelcast to handle system failures and checkpointing. I am not saying this is some amazing solution to every HPC pro…

I understand your last paragraph. Looking this time at Hazelcast, what I see is layers of code to understand before being able to do something simple. It really does look like all of the technology you are pointing to is solving a different problem. It's not related to any of the HPC needs I've heard of.

Parts of my simulation are out of phase. I need some gather step to collect the data from individual nodes, when a given timestep is reached, and save the state. A simple solution is to do a barrier every ~30 minutes, send to the master node, and have it save the data.

When I look at Hazelcast I see what looks to be a different sort of clustering - using clusters for redundancy, and not for CPU power. Eg, I see "Hazelcast keeps the backup of each data entry on multiple nodes", and I think "I don't care." If a node goes down, the system goes down, and I restart from a checkpoint. It's much more likely that one of the 512 compute nodes will go down than some database node.

I'll withdraw my original statement that "A map-reduce system like Hadoop" and say simply "a system like Hadoop isn't a good fit for HPC problems".

Here's a lovely essay which agrees with me ;) http://glennklockwood.blogspot.com.au/2014/05/hadoops-uncomf... . It considers the questions:

> Why does Hadoop remain at the fringe of high-performance computing, and what will it take for it to be a serious solution in HPC?

Re: HPC is dying, and MPI is killing it

#95

Post by markhahn »

anigbrowl wrote: post #58
ToastyMallows wrote: post #3

MPI is Message Passing Interface, for those who don't know. Author never explicitly says what MPI stands for. https://en.wikipedia.org/wiki/Message_Passing_Interface

Yeah, I had to go back to google after failing to find it defined anywhere on the page. If you don't use something regularly yourself, you may remember the concept but forget the acronym. For want of a couple of sentences of context at the outset, the rest of the article was quite inaccessible. Maybe there's a lesson for the author here.

a lesson for the author xor the reader.

Re: HPC is dying, and MPI is killing it

#96

Post by noobermin »

sampo wrote: post #54
rz2k wrote: post #31

Earlier quoted context omitted.

How about in simulations where a large number of cellular automata are interacting with each other, but individually only carrying out simple computations?

You cannot make an efficient fluid mechanics simulation on a 4000x4000x4000 grid if you set up a separate process for each individual gridcell. More efficient to just store your numbers in 3d arrays.

why not?

Re: HPC is dying, and MPI is killing it

#97

Post by RhysU »

markhahn wrote: post #65
na85 wrote: post #8

Earlier quoted context omitted.

I spent a few months learning (modern) Fortran a year or two ago. My chief obstacle was the difficulty involved in finding modern tutorials. I don't want to have to read tutorials written in 1994 whose focus is getting people used to F77 up to speed. I've yet to find a tutorial that approached teaching F08 as if it was a new language, which is what I feel is needed. Even in F08 there's a lot of backwards-compatibilit…

"horrid for interactive" is ironic, since interactive and HPC are pretty much disjoint. (well, viz...) From an HPC perspective, Fortran IO should be performed by HDF5...

+1

Re: HPC is dying, and MPI is killing it

#98

Post by ToastyMallows »

kenko wrote: post #69
ToastyMallows wrote: post #3

MPI is Message Passing Interface, for those who don't know. Author never explicitly says what MPI stands for. https://en.wikipedia.org/wiki/Message_Passing_Interface

"MPI, the Message Passing Interface, began as a needed standardization above a dizzying array of..." The capitalization and apposition makes it pretty explicit.

I triple checked before posting this, looks like someone edited the blog ;)

Re: HPC is dying, and MPI is killing it

#99

Post by shepardrtc »

noobermin wrote: post #96
sampo wrote: post #54

Earlier quoted context omitted.

You cannot make an efficient fluid mechanics simulation on a 4000x4000x4000 grid if you set up a separate process for each individual gridcell. More efficient to just store your numbers in 3d arrays.

why not?

If you store the data in arrays, you can use matrix multiplication libraries such as Intel's MKL or OpenBLAS, which are written to be exceptionally optimized for use on multiple cores. I cannot emphasize enough how much time and effort has been put into these libraries to multiply matrices as fast as can possibly be done.

If you use processes such as in the Erlang VM, they're doing calculations, sure, but they're also sending messages back and forth, and they're acting as supervisors, and they're being shuffled around by the VM. There's a lot going on. And that extra stuff that's going on takes away from the time you could be multiplying stuff. And even then, there's been no optimization done for this sort of calculation. There are a lot of tricks you can do. Heck, the better matrix multiplication libraries have individual optimizations for CPUs.

Re: HPC is dying, and MPI is killing it

#100

Post by rz2k »

shepardrtc wrote: post #87
rz2k wrote: post #31

Earlier quoted context omitted.

How about in simulations where a large number of cellular automata are interacting with each other, but individually only carrying out simple computations?

You certainly can do that. In fact, I believe I saw an example of that in a tutorial somewhere talking about what Erlang/Elixir would be good for (I include Elixir because its the same VM controlling processes underneath, but a "nicer" syntax on top). Now, would it be the best language for that? Depends. If you're looking just for speed and massive computations, then no. In the end, C++ pretty much rules everything i…

I actually started going through Programming Elixir this past week because I was thinking it would eventually be a fun way to explore models through ad hoc substitution of rules and exogenous conditions. There are a lot of models available in NetLogo, but it doesn't look like it would be very powerful in terms of running larger scale simulations, and the scope of the language's usefulness is pretty limited compared to that of a general purpose language.

While I have been using C++/Rcpp to extend R as an occasional time saver (for analysis rather than simulations), it's only been little snippets written badly.

Anyway, since its so easy to come up with a too long list of technologies to learn, then only scratch the surface, hearing that Erlang/Elixir isn't the completely wrong tool is helpful.

Post reply on HN