Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CharmPy – A high-level parallel and distributed programming framework

charmpy.readthedocs.io

Post reply on HN
Print view View original

1–10 of 27 posts

Re: CharmPy – A high-level parallel and distributed programming framework

#5

Post by davidmr »

leetrout wrote: post #2

That seems like a very poor choice in name given the existence of PyCharm...

I'm not sure what better to call it. Charm++ (its core) was around ruining the lives of chemistry grad students and their sysadmins long, long before pycharm was a twinkle in its creators' eyes. I'm sure jetbrains won't be coming for it any time soon.

Re: CharmPy – A high-level parallel and distributed programming framework

#7

Post by woodson »

Related but somewhat off-topic question: Is there an easy to use parallel processing framework for python that also works well on Windows (anaconda)? I keep having issues with lost processes and other more or less random crashes, no matter whether I use joblib or Dask, etc. All I really need is a parallel for or apply.

Re: CharmPy – A high-level parallel and distributed programming framework

#8

Post by juanjgalvez »

woodson wrote: post #7

Related but somewhat off-topic question: Is there an easy to use parallel processing framework for python that also works well on Windows (anaconda)? I keep having issues with lost processes and other more or less random crashes, no matter whether I use joblib or Dask, etc. All I really need is a parallel for or apply.

You can install charmpy on Windows with pip if that works for you. Launching multiple processes is straightforward (you can check the documentation at charmpy.readthedocs.io).

We don't offer an API yet in charmpy to explicitly do things like parallel apply (but will soon). You can however look at `examples/parallel-map/par-map.py` in the source code which shows a simple example of how to do it with the current API and might be what you are looking for.

Re: CharmPy – A high-level parallel and distributed programming framework

#9

Post by eslaught »

devxpy wrote: post #6

Can you please lay out the differences between this and Dask? https://dask.pydata.org/en/latest/ I work on a parallel programming framework for python myself. Not geared towards performance, but the ease of use. http://zproc.readthedocs.io/en/latest/

As far as I know, Dask is at its core a tasking model (i.e. tasks have input and outputs, and run automatically when inputs are ready in a dataflow-like model).

Charm++ (on which CharmPy is based) is an actor model. Think Erlang for HPC. You've got a set of objects that are all nominally running concurrently, and objects can send messages to one another.

Personally, I prefer the task-based model (but of course I'm biased since I work on one myself). In a proper task-based model, you can't have races or deadlocks, everything looks to a first approximation to be sequential. In actor models there's pretty much no way to hide the conncurrency, and all the traditional pitfalls of parallel programming are exposed to the user.

Re: CharmPy – A high-level parallel and distributed programming framework

#10

Post by juanjgalvez »

devxpy wrote: post #6

Can you please lay out the differences between this and Dask? https://dask.pydata.org/en/latest/ I work on a parallel programming framework for python myself. Not geared towards performance, but the ease of use. http://zproc.readthedocs.io/en/latest/

There are quite a few differences between them. Disclaimer: I work on CharmPy, and I'm not an expert on Dask, so my comments might be biased and not entirely accurate with respect to Dask.

Obvious difference between the two is programming style. CharmPy (its current core API) is based on asynchronous method execution between distributed objects. Being objects they can have state and data which allows for a lot of flexibility. In Dask, you express a workflow as a series of dependent tasks (which as far as I know are stateless so it's more like functional programming) and dask schedules it for you. The scheduling is centralized (done in only one place, so it's like a master-worker pattern) even if you use the "distributed" scheduler (which is needed for multi-node runs). With CharmPy you can have truly distributed applications.

Another thing I observed with the dask model is that, since everything needs to be translated into a task graph before execution, there seems to be poor support for mutable distributed numpy arrays. A mutation operation like modifying a single element of a distributed array is not allowed as far as I know (I have tried), and other mutation operations that are supported actually generate a completely different task graph as a result, with the overhead this entails. In charmpy, this restriction does not exist since you can just invoke a method on the object that holds the data that you want to modify, and do it in-place.

In terms of performance, our initial tests have shown huge performance difference, with CharmPy being up to 200x faster (this is comparing with dask distributed scheduler for a very simple BSP-style program). Of course, difference will vary by workload, but one thing to note is that Dask is pure-python, while CharmPy's core runs in C/C++. The current level of task granularity that we can comfortably support is a few hundred microseconds, and we expect to improve it further. In contrast, the Dask documentation for the distributed scheduler explicitly warns against using small task granularity, recommending tasks larger than 100 ms duration. And something like Jug recommends tasks longer than 20 seconds.

We are planning on adding other APIs on top of the core charmpy API, to accommodate other programming styles. For example, offer better support for the functional parallel programming style (there is a small example of parallel map in the codebase using charmpy), or task scheduling.

Post reply on HN