Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CharmPy – A high-level parallel and distributed programming framework

charmpy.readthedocs.io

Post reply on HN
Print view View original

21–27 of 27 posts

Re: CharmPy – A high-level parallel and distributed programming framework

#21

Post by targafarian »

juanjgalvez wrote: post #17
targafarian wrote: post #14

from charmpy import * Please, for the love of God, import names explicitly or use e.g. `import charmpy as cp` and subsequently `cp.foo` so that reading example code we get a better sense of your API without having to guess which names were possibly overwritten.

You must be referring to the example in the README. That is the only example in the source code or docs that uses `import *` as far as I'm aware. But yeah, I agree. It's fixed now.

Much credit to you for changing that in the example. Will give it a try to see what the advantages/disadvantages are next time I need do do more complex tasks in parallel (I generally like Dask, but I would certainly need to try this out to know where each is better or worse--or if it's just a matter of taste).

Re: CharmPy – A high-level parallel and distributed programming framework

#22

Post by juanjgalvez »

p1esk wrote: post #20
juanjgalvez wrote: post #18

Earlier quoted context omitted.

For a single workstation and the task you describe, the pool.map() functionality of the multiprocessing module should be perfectly adequate. Not sure how scheduling overhead would compare between charmpy and multiprocessing, but for this task it shouldn't matter (I assume you need at least a second to convert one file, and even if the conversion is faster, you can chunk the tasks anyway to mask overhead). I would say…

Oh, good point about batching - my files were really small (audio samples for speech recognition), so a conversion of a single file took a lot less than a second. I looked at the par-map.py example, however I can't quite understand where do I enter a server IP or something like that. The whole process is fuzzy to be honest. What do I need to do if I want to run my conversion task on two local workstations? E.g. I ins…

You don't actually have to specify hosts or addresses in your application code. When the application starts, the runtime will know how many processes there are and on which hosts. The key is to use a job launcher.

For the par-map.py example, suppose you want to run it on 4 hosts and 8 processes per host. One way to do this is by launching the application with "charmrun". First, install charmpy on all hosts like you said. Then you would create a nodelist file with the names or addresses of the 4 hosts. Finally, launch like this: `$ charmrun +p32 par-map.py ++nodelist mynodelist.txt`

I have updated the "Running" section of the docs to try to explain this better, also pointing to the charmrun manual. Hopefully things are clearer now.

Re: CharmPy – A high-level parallel and distributed programming framework

#23

Post by p1esk »

juanjgalvez wrote: post #22
p1esk wrote: post #20

Earlier quoted context omitted.

Oh, good point about batching - my files were really small (audio samples for speech recognition), so a conversion of a single file took a lot less than a second. I looked at the par-map.py example, however I can't quite understand where do I enter a server IP or something like that. The whole process is fuzzy to be honest. What do I need to do if I want to run my conversion task on two local workstations? E.g. I ins…

You don't actually have to specify hosts or addresses in your application code. When the application starts, the runtime will know how many processes there are and on which hosts. The key is to use a job launcher. For the par-map.py example, suppose you want to run it on 4 hosts and 8 processes per host. One way to do this is by launching the application with "charmrun". First, install charmpy on all hosts like you s…

Thank you, now it's a lot clearer. I will try it on multiple workstations next time I need to run a large job.

Re: CharmPy – A high-level parallel and distributed programming framework

#24

Post by samtwhite »

juanjgalvez wrote: post #22
p1esk wrote: post #20

Earlier quoted context omitted.

Oh, good point about batching - my files were really small (audio samples for speech recognition), so a conversion of a single file took a lot less than a second. I looked at the par-map.py example, however I can't quite understand where do I enter a server IP or something like that. The whole process is fuzzy to be honest. What do I need to do if I want to run my conversion task on two local workstations? E.g. I ins…

You don't actually have to specify hosts or addresses in your application code. When the application starts, the runtime will know how many processes there are and on which hosts. The key is to use a job launcher. For the par-map.py example, suppose you want to run it on 4 hosts and 8 processes per host. One way to do this is by launching the application with "charmrun". First, install charmpy on all hosts like you s…

And on most clusters and supercomputers you don't need to manually create the nodelist at all, charmrun can do that automatically for you by parsing the batch scheduler's list of allocated hosts.

Re: CharmPy – A high-level parallel and distributed programming framework

#26

Post by juanjgalvez »

Caminha wrote: post #25

Is there a way to interop that with MPI based tools (such as Trilinos or PyTrilinos)? I would really love to be able to write some of my unstructured mesh based simulations on something like that.

For CharmPy this is not currently supported, but Charm++ can interoperate with MPI libraries, so with luck it wouldn't require much effort to get it working. I will open an issue on github to track this task.

Re: CharmPy – A high-level parallel and distributed programming framework

#27

Post by juanjgalvez »

ianbertolacci wrote: post #3

Very cool. However, this documentation is incomplete. It needs the full Python API listing.

We'll update the documentation with the API in the next couple of days. Eventually there will also be a more comprehensive manual explaining every feature.
Post reply on HN