Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel Programming with Python

chryswoods.com

Post reply on HN
Print view View original

131–140 of 147 posts

Re: Parallel Programming with Python

#131

Post by devxpy »

TickleSteve wrote: post #130
devxpy wrote: post #129

Earlier quoted context omitted.

the esp8266 boards are like pretty cheap. Won't argue about pyboard, because that's more of a gimmick. (Way too expensive for what it does) I think the cost and time of developing on micropython vs a lower level language like C, would superseded the cost associated with wasted cpu cycles. However, I agree with the fact that no real product would use mpy right now in production because of the infancy of the project. I…

No, uPython IS a toy. In embedded s/w, the cost of the h/w outweighs everything else, ok so your 'easily developed' Python app will cost (say) 10K less to develop... but the resource requirements mean you need to go up $0.5 on the processor.... Oops, on your 500'000 devices you've suddenly wasted $250,000. All because you couldn't be bothered to save a few KB of RAM. Python is a toy for embedded s/w.... dont get me g…

Wait what?

How is catching bugs at runtime and dynamically typed language related?

In fact I've had (a lot) better experience debugging mpy apps than Arduino ones (that's the only low level embedded experience I have)

I mostly just let stuff fail, and have them restart gracefully (like the erlang guys)

Re: Parallel Programming with Python

#132

Post by devxpy »

TickleSteve wrote: post #121
devxpy wrote: post #108

Earlier quoted context omitted.

> That doesn't mean to say its "perfect" or "solves" multithreading, just that its easy to write and understand Try saying that out loud?

yes. That makes perfect sense... easy to write and understand is something completely different to correctness, robustness, scalability, etc. All those must be considered if you think you have 'solved' parallelism, but they are orthogonal to 'easy to understand'.

I don't think he meant it like that.

You could easily interpret that as -

Perfect _implies_ that it's easy to write and understand, but it's not the whole picture. It's just a feature that _he_ thinks is _crucial_ to it being perfect.

You get my point right?

Like sure, you could implemented a _perfect_, I don't know like gnome desktop in assembly language, but it wouldn't be easy to write and understand.

He thinks it's essential that it should be easy to read and write for it to be perfect.

Unfortunately, He's not with us now so can't even confirm :(

Re: Parallel Programming with Python

#133

Post by TickleSteve »

devxpy wrote: post #132
TickleSteve wrote: post #121

Earlier quoted context omitted.

yes. That makes perfect sense... easy to write and understand is something completely different to correctness, robustness, scalability, etc. All those must be considered if you think you have 'solved' parallelism, but they are orthogonal to 'easy to understand'.

I don't think he meant it like that. You could easily interpret that as - Perfect _implies_ that it's easy to write and understand, but it's not the whole picture. It's just a feature that _he_ thinks is _crucial_ to it being perfect. You get my point right? Like sure, you could implemented a _perfect_, I don't know like gnome desktop in assembly language, but it wouldn't be easy to write and understand. He thinks it…

Trust me, he's an intelligent guy, he did not mean "perfect" as you're implying it.

He is fully aware that he has not solved parallelism.

Re: Parallel Programming with Python

#134

Post by zbentley »

srean wrote: post #103
zbentley wrote: post #40

Earlier quoted context omitted.

> What you've created here is pretty much what multiprocessing gives you already in a more performant solution (i.e. no zeromq involved) Minor point of pedantry which I'll state because it's an often-overlooked timesaver for folks developing on multiprocessing: not only is MP potentially faster for transferring data between processes compared to this solution, but it can also be way, way faster in situations where yo…

> Because of copy-on-write fork magic, many multiprocessing configurations (including the default) can "send" that data to child processes in constant* time, if the data's already present in e.g. a global when children are created. Have you tried this or got it working ? The fly in the ointment is the reference count . Add a reference and BOOM you suddenly have a huge copy. It can be made to work efficiently in certa…

In practice, I find reference-count related issues with this pattern to be minor.

Most of the situations where I care enough about memory and/or pickling overhead fall into the "take a giant block of binary/string data and process ranges of it in parallel" family, in which case there aren't too many references until the subprocesses get to work. If I had more complex structures of data I'd probably get a little less performance bang for my buck, but even then I suspect it would be much faster than multiprocessing's strategy: pickling and sending data between processes via pipes is many times slower than moving the equivalent amount of data by dirty-writing pages into a forked child.

That's not meant to discount anything y'all are saying, though: refcounts are definitely a very important thing to be mindful of in this situation. A child comment suggests gc.freeze, which can help, but can't entirely save you from thinking about this stuff.

It's also very important to be mindful of what happens with your program at shutdown: if you have a big set of references shared via fork(), and all your children shut down around the same time, your memory usage can shoot up as each child tries to de-refcount all objects in scope. This applies even if each child was only operating on a subset of the references shared to it. If you're processing, say, 1GB of data from the parent in 8 children on a 4 core system (doing M>N(cpu) because e.g. children spend some time writing results out to the FS/network), a near-simultaneous shutdown could allocate 9GB of memory in the very worst case, which can cause OOM or unexpected swapping behavior. Throttled shutdowns using a semaphore or equivalent are the way to go in that case.

Re: Parallel Programming with Python

#135

Post by TickleSteve »

devxpy wrote: post #131
TickleSteve wrote: post #130

Earlier quoted context omitted.

No, uPython IS a toy. In embedded s/w, the cost of the h/w outweighs everything else, ok so your 'easily developed' Python app will cost (say) 10K less to develop... but the resource requirements mean you need to go up $0.5 on the processor.... Oops, on your 500'000 devices you've suddenly wasted $250,000. All because you couldn't be bothered to save a few KB of RAM. Python is a toy for embedded s/w.... dont get me g…

Wait what? How is catching bugs at runtime and dynamically typed language related? In fact I've had (a lot) better experience debugging mpy apps than Arduino ones (that's the only low level embedded experience I have) I mostly just let stuff fail, and have them restart gracefully (like the erlang guys)

Now you're just trolling.

Do yourself a favour and learn your craft instead of trotting out obviously wrong statements to people who know better. Thats not a way to make a good impression.

Oh yes, regarding Erlang, trust me, I'm aware of 'let-it-fail' and have implemented it production on large distributed systems and trust me... that does not justify writing embedded s/w in a dynamically-typed language.

Re: Parallel Programming with Python

#136

Post by devxpy »

TickleSteve wrote: post #135
devxpy wrote: post #131

Earlier quoted context omitted.

Wait what? How is catching bugs at runtime and dynamically typed language related? In fact I've had (a lot) better experience debugging mpy apps than Arduino ones (that's the only low level embedded experience I have) I mostly just let stuff fail, and have them restart gracefully (like the erlang guys)

Now you're just trolling. Do yourself a favour and learn your craft instead of trotting out obviously wrong statements to people who know better. Thats not a way to make a good impression. Oh yes, regarding Erlang, trust me, I'm aware of 'let-it-fail' and have implemented it production on large distributed systems and trust me... that does not justify writing embedded s/w in a dynamically-typed language.

I'm serious.

The "let it fail" strategy is quite useful

Re: Parallel Programming with Python

#137

Post by zbentley »

blattimwind wrote: post #49
zbentley wrote: post #40

Earlier quoted context omitted.

> What you've created here is pretty much what multiprocessing gives you already in a more performant solution (i.e. no zeromq involved) Minor point of pedantry which I'll state because it's an often-overlooked timesaver for folks developing on multiprocessing: not only is MP potentially faster for transferring data between processes compared to this solution, but it can also be way, way faster in situations where yo…

> Because of copy-on-write fork magic Note that this works for big objects, but not for small objects. E.g. if you fork-share a large list of integers or dicts or something like that, then you don't get any memory usage benefits, because every access will cause a refcount-write and that will copy the whole page containing the object. > * Technically it's O(N) for the size of data you have in memory at process pool st…

> this works for big objects, but not for small objects

Very true; I went into some more detail about my typical use case above. Using MP for lots of small objects that you've already extracted from raw data/IO/whatever is a game of diminishing returns. It's in situations like that where traditional shared-memory starts looking more and more attractive. When I get to that point, while multiprocessing and some other packages provide a few nice abstractions over shmem, I start looking for other platforms than Python.

> It's not quite that simple; sharing n pages can take very little time or a bit more time

Definitely; I was simplifying in order to compare the overhead of fork with the overhead of pickling/shipping/unpickling data. Sharing large pieces of data with even very slow fork()ing is, in my experience, so much faster than the [de]serialize approach that it is effectively constant in comparison, but I didn't mean to discount the complexities of what make certain forking situations faster/slower than others.

Re: Parallel Programming with Python

#138

Post by zbentley »

devxpy wrote: post #74
zbentley wrote: post #40

Earlier quoted context omitted.

> What you've created here is pretty much what multiprocessing gives you already in a more performant solution (i.e. no zeromq involved) Minor point of pedantry which I'll state because it's an often-overlooked timesaver for folks developing on multiprocessing: not only is MP potentially faster for transferring data between processes compared to this solution, but it can also be way, way faster in situations where yo…

Performance doesn't equal Better software. In fact, I think Performance centric development is a lesser known evil. > have all your data before creating your processes/pool Zproc exposes the required API for this (Nothing new, just the python API) :) https://zproc.readthedocs.io/en/latest/api.html#zproc.Proces... ( args and kwargs) > a massive dataset Wouldn't you be better off using a Database for that kind of work?…

> Any resources on how to implement that?

  big_data = read_huge_binary_or_string()

  def process_range(rng):
    start, end = rng
    do_something(big_data[start:end])

  pool = multiprocesing.Pool(2)
  pool.map(process_range, [
    (0, 10000),
    (10001, len(big_data),
  ])

Re: Parallel Programming with Python

#139

Post by devxpy »

TickleSteve wrote: post #133
devxpy wrote: post #132

Earlier quoted context omitted.

I don't think he meant it like that. You could easily interpret that as - Perfect _implies_ that it's easy to write and understand, but it's not the whole picture. It's just a feature that _he_ thinks is _crucial_ to it being perfect. You get my point right? Like sure, you could implemented a _perfect_, I don't know like gnome desktop in assembly language, but it wouldn't be easy to write and understand. He thinks it…

Trust me, he's an intelligent guy, he did not mean "perfect" as you're implying it. He is fully aware that he has not solved parallelism.

Does "we have the answer" equal "solved"?

https://www.youtube.com/watch?v=yhGXJ9Jt3-A

Re: Parallel Programming with Python

#140

Post by devxpy »

zbentley wrote: post #138
devxpy wrote: post #74

Earlier quoted context omitted.

Performance doesn't equal Better software. In fact, I think Performance centric development is a lesser known evil. > have all your data before creating your processes/pool Zproc exposes the required API for this (Nothing new, just the python API) :) https://zproc.readthedocs.io/en/latest/api.html#zproc.Proces... ( args and kwargs) > a massive dataset Wouldn't you be better off using a Database for that kind of work?…

> Any resources on how to implement that? big_data = read_huge_binary_or_string() def process_range(rng): start, end = rng do_something(big_data[start:end]) pool = multiprocesing.Pool(2) pool.map(process_range, [ (0, 10000), (10001, len(big_data), ])

Right, that only concerns sending data at startup, which both Python (and zproc) already do.

I thought you were talking about sending data to child processes in constant* time, while it was running.

Post reply on HN