This seems pretty cool, but I'm left with one question. In the example from the repo it states, "The following computes Pi in parallel, using any number of machines and processors." However, after reading through all the docs, I see no reference whatsoever to any multi-machine support, only multi-processing on a single machine. Can this span over a cluster? Although they make the claim that it will "scale to hundreds of thousands of cores" (which I want to believe, I would love to use this) without the API documentation to show me how to do this, their fancy library doesnt do much good.
CharmPy – A high-level parallel and distributed programming framework
11–20 of 27 posts
Re: CharmPy – A high-level parallel and distributed programming framework
#12This seems pretty cool, but I'm left with one question. In the example from the repo it states, "The following computes Pi in parallel, using any number of machines and processors." However, after reading through all the docs, I see no reference whatsoever to any multi-machine support, only multi-processing on a single machine. Can this span over a cluster? Although they make the claim that it will "scale to hundreds…
Hi. Yes, applications can span multiple nodes (e.g. in clusters and supercomputers), and is one of the main use cases of charm++/charmpy. The fact that you don't see anything in the API or examples is that application code is basically transparent to the amount of processes that are launched.
What determines the number of processes used is the launcher (e.g. charmrun, or something like aprun or ibrun on other systems). During initialization, the charmpy runtime will figure out internally how many charmpy processes are active in the job.
With charmrun, you can launch multiple processes in one host, but also across multiple hosts (by ssh'ing into each one and spawning the processes). This is done automatically by charmrun assuming you specify a list of hosts (called nodelist, see http://charm.cs.illinois.edu/manuals/html/charm++/C.html). Again, the application code is not affected by this.
Similarly, on other systems you can launch charmpy applications with the system job launcher (e.g. aprun, sbatch, ibrun…). We have done so for example on Cray supercomputers. It is simple enough but we have to update the documentation to at least show an example of this.
Re: CharmPy – A high-level parallel and distributed programming framework
#13This seems pretty cool, but I'm left with one question. In the example from the repo it states, "The following computes Pi in parallel, using any number of machines and processors." However, after reading through all the docs, I see no reference whatsoever to any multi-machine support, only multi-processing on a single machine. Can this span over a cluster? Although they make the claim that it will "scale to hundreds…
Hi. Yes, applications can span multiple nodes (e.g. in clusters and supercomputers), and is one of the main use cases of charm++/charmpy. The fact that you don't see anything in the API or examples is that application code is basically transparent to the amount of processes that are launched. What determines the number of processes used is the launcher (e.g. charmrun, or something like aprun or ibrun on other systems…
Thanks for the response! This is great, makes it pretty much a better version of mpi. Could you add this to the documentation, or if its already there maybe a link on the main doc page about running on a cluster?
Re: CharmPy – A high-level parallel and distributed programming framework
#14from charmpy import *
Please, for the love of God, import names explicitly or use e.g. `import charmpy as cp` and subsequently `cp.foo` so that reading example code we get a better sense of your API without having to guess which names were possibly overwritten.
Re: CharmPy – A high-level parallel and distributed programming framework
#15Can you please lay out the differences between this and Dask? https://dask.pydata.org/en/latest/ I work on a parallel programming framework for python myself. Not geared towards performance, but the ease of use. http://zproc.readthedocs.io/en/latest/
There are quite a few differences between them. Disclaimer: I work on CharmPy, and I'm not an expert on Dask, so my comments might be biased and not entirely accurate with respect to Dask. Obvious difference between the two is programming style. CharmPy (its current core API) is based on asynchronous method execution between distributed objects. Being objects they can have state and data which allows for a lot of fle…
I had a task recently where I needed to convert several million audio files from one format to another, and I did it with python's multiprocessing module (similar to this: https://stackoverflow.com/questions/50662610/using-multiproc... )
Just like the poster of that question on SO, I'm wondering if that's the best way (in terms of speed or ease of use). Do any of the third party libraries (like yours) offer any advantages for this use case? To clarify, I'm only talking about doing work on a single workstation.
Re: CharmPy – A high-level parallel and distributed programming framework
#16That seems like a very poor choice in name given the existence of PyCharm...
I'm not sure what better to call it. Charm++ (its core) was around ruining the lives of chemistry grad students and their sysadmins long, long before pycharm was a twinkle in its creators' eyes. I'm sure jetbrains won't be coming for it any time soon.
Charmer? CharmIt? Charming? CharmHPD? Charmplus? Superintendentchalmers? The problem here isn't that PyCharm came second or that Jetbrains will have an issue. The problem here is that it's so many conversations will go
> Have you used CharmPy?
> You mean PyCharm?
> No, CharmPy!
> Are we talking about the same thing?
> No
> Oh, that's just confusing, then.
Re: CharmPy – A high-level parallel and distributed programming framework
#17from charmpy import * Please, for the love of God, import names explicitly or use e.g. `import charmpy as cp` and subsequently `cp.foo` so that reading example code we get a better sense of your API without having to guess which names were possibly overwritten.
You must be referring to the example in the README. That is the only example in the source code or docs that uses `import *` as far as I'm aware. But yeah, I agree. It's fixed now.
Re: CharmPy – A high-level parallel and distributed programming framework
#18Earlier quoted context omitted.
There are quite a few differences between them. Disclaimer: I work on CharmPy, and I'm not an expert on Dask, so my comments might be biased and not entirely accurate with respect to Dask. Obvious difference between the two is programming style. CharmPy (its current core API) is based on asynchronous method execution between distributed objects. Being objects they can have state and data which allows for a lot of fle…
I had a task recently where I needed to convert several million audio files from one format to another, and I did it with python's multiprocessing module (similar to this: https://stackoverflow.com/questions/50662610/using-multiproc... ) Just like the poster of that question on SO, I'm wondering if that's the best way (in terms of speed or ease of use). Do any of the third party libraries (like yours) offer any advan…
For a single workstation and the task you describe, the pool.map() functionality of the multiprocessing module should be perfectly adequate. Not sure how scheduling overhead would compare between charmpy and multiprocessing, but for this task it shouldn't matter (I assume you need at least a second to convert one file, and even if the conversion is faster, you can chunk the tasks anyway to mask overhead). I would say the big difference for this task is if you want to run it in parallel on multiple hosts, which pool.map can't do. With charmpy we can provide a distributed parallel map offering the same or similar API as pool.map. There is a simple example in 'examples/parallel-map/par-map.py', but we are working on offering a library on top of charmpy with more features and a solid API.
Re: CharmPy – A high-level parallel and distributed programming framework
#19Earlier quoted context omitted.
I'm not sure what better to call it. Charm++ (its core) was around ruining the lives of chemistry grad students and their sysadmins long, long before pycharm was a twinkle in its creators' eyes. I'm sure jetbrains won't be coming for it any time soon.
Charmer? CharmIt? Charming? CharmHPD? Charmplus? Superintendentchalmers? The problem here isn't that PyCharm came second or that Jetbrains will have an issue. The problem here is that it's so many conversations will go > Have you used CharmPy? > You mean PyCharm? > No, CharmPy! > Are we talking about the same thing? > No > Oh, that's just confusing, then.
I dunno, maybe. They're so completely different that it seems extremely unlikely to me that anyone will confuse the two in a conversation with any context whatsoever. The conversation you made up would almost certainly start with a discussion about scientific parallel programming. Any discussion I've ever had about charm++ (too many, I'm afraid) wouldn't have been confused for a discussion about an IDE.
Either way, it's done.
Re: CharmPy – A high-level parallel and distributed programming framework
#20Earlier quoted context omitted.
I had a task recently where I needed to convert several million audio files from one format to another, and I did it with python's multiprocessing module (similar to this: https://stackoverflow.com/questions/50662610/using-multiproc... ) Just like the poster of that question on SO, I'm wondering if that's the best way (in terms of speed or ease of use). Do any of the third party libraries (like yours) offer any advan…
For a single workstation and the task you describe, the pool.map() functionality of the multiprocessing module should be perfectly adequate. Not sure how scheduling overhead would compare between charmpy and multiprocessing, but for this task it shouldn't matter (I assume you need at least a second to convert one file, and even if the conversion is faster, you can chunk the tasks anyway to mask overhead). I would say…
Oh, good point about batching - my files were really small (audio samples for speech recognition), so a conversion of a single file took a lot less than a second.
I looked at the par-map.py example, however I can't quite understand where do I enter a server IP or something like that. The whole process is fuzzy to be honest. What do I need to do if I want to run my conversion task on two local workstations? E.g. I install CharmPy on both, then what?