Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster Parallel Python Without Python Multiprocessing

towardsdatascience.com

Post reply on HN
Print view View original

11–20 of 30 posts

Re: Faster Parallel Python Without Python Multiprocessing

#11

Post by mike_mg »

While I appreciate the efforts of authors and believe in long term mission, they seem to not mention anywhere some key shortcomings of Ray, while marketing it pretty hard (eg see the paper).

I have used ray (a year ago) in one of the advertised basic applications: parallelising the environments for RL. It was unusable back then, as it was clogging up the memory.

The plasma store which is backend for arrow was never cleaned which made the computation stop after 3 hours

Here’s the issue:

https://github.com/ray-project/ray/issues/2128

Or perhaps this has been fixed already?

Re: Faster Parallel Python Without Python Multiprocessing

#12

Post by glandium »

zaphirplane wrote: post #9
glandium wrote: post #7

Earlier quoted context omitted.

One problem is that because of the GIL, many python libraries don't use locking on their data, etc., because they don't need to. That makes them thread-unsafe without the GIL.

You mean libraries with c extensions

No, even libraries without. Picking something random in the python 3.7 standard library: collections.OrderedDict.__setitem__ doesn't look thread-safe when it updates its linked list. EDIT: well, in fact, the data race in that one is there whether there is a GIL or not...

Re: Faster Parallel Python Without Python Multiprocessing

#13

Post by gameswithgo »

protomyth wrote: post #6
ATsch wrote: post #5

Earlier quoted context omitted.

The thing is that there's really little benefit from this. Anything performance critical will be written as a native extension anyway, even if Python speeds up by 10x. If you're doing that, you can lift the GIL anyway and run as much in parallel as you want.

Isn’t that a case of the tail wagging the dog? If Python code were 10x faster, the need to write performance sensitive code in a native extension would be a lot lower. There is a cognitive cost in having to switch between languages and knowing when to switch.

its unfortunate that dynamic interpreted languages with easy uptake for small projects get so popular, and then big, serious things have to eat the performance penalty for years and years. The world could use a language that is 'easy' like python/ruby but fast.

Something like a simplified nim/f# with type inference. So it feels dynamic as you code but isn't really.

Or yes if the normal way to run python was with a top notch jit that would be a big plus.

Re: Faster Parallel Python Without Python Multiprocessing

#14

Post by sametmax »

riskneutral wrote: post #4
htfy96 wrote: post #2

Every time I see a performance issue/solution of Python, I was wondering why there is no company maintaining a distribution with a high-performance Python JIT compiler with patched, GIL-free packages. Given the prevalence of Python and what have succeeded in JVM, it seems like a fruitful business.

Because Python is not typed?

Python is strongly typed, by dynamically typed.

Re: Faster Parallel Python Without Python Multiprocessing

#16

Post by grumpyprole »

sametmax wrote: post #14
riskneutral wrote: post #4

Earlier quoted context omitted.

Because Python is not typed?

Python is strongly typed, by dynamically typed.

From a type-theoretic point of view, Python is unityped: https://existentialtype.wordpress.com/2011/03/19/dynamic-lan...

Python has a lot of merit as an easy to use language, a modern BASIC, but "strongly typed" is marketing that confuses rather than enlightens (to quote Prof. Harper). The term "strongly-tagged" is perhaps more apt, but sadly that ship has sailed.

Re: Faster Parallel Python Without Python Multiprocessing

#17

Post by srean »

e12e wrote: post #8
htfy96 wrote: post #2

Every time I see a performance issue/solution of Python, I was wondering why there is no company maintaining a distribution with a high-performance Python JIT compiler with patched, GIL-free packages. Given the prevalence of Python and what have succeeded in JVM, it seems like a fruitful business.

There was/is stackeless with a(n initial?) focus on game engines iirc: https://github.com/stackless-dev/stackless/wiki It got less attention than pypy, but might be more pragmatic wrt parallel perf. There's also some work in pypy to remove the GIL, but not sure if there's been any news on that lately.

Stackless is wonderful, but its for concurrency not parallelism. Long time ago when I checked it out it was faster than go in concurrency but not so in parallelism.

Re: Faster Parallel Python Without Python Multiprocessing

#18

Post by 0_gravitas »

gameswithgo wrote: post #13
protomyth wrote: post #6

Earlier quoted context omitted.

Isn’t that a case of the tail wagging the dog? If Python code were 10x faster, the need to write performance sensitive code in a native extension would be a lot lower. There is a cognitive cost in having to switch between languages and knowing when to switch.

its unfortunate that dynamic interpreted languages with easy uptake for small projects get so popular, and then big, serious things have to eat the performance penalty for years and years. The world could use a language that is 'easy' like python/ruby but fast. Something like a simplified nim/f# with type inference. So it feels dynamic as you code but isn't really. Or yes if the normal way to run python was with a to…

It's not necessarily C-fast and it is dynamically typed but in the ratio of easiness-to-performance elixir seems to be pretty affable, especially for concurrency stuff. It does tend to balk at some heavier stuff like number-crunching, and that's when I see people start to use Rust NIFs.

Re: Faster Parallel Python Without Python Multiprocessing

#19

Post by carlmr »

0_gravitas wrote: post #18
gameswithgo wrote: post #13

Earlier quoted context omitted.

its unfortunate that dynamic interpreted languages with easy uptake for small projects get so popular, and then big, serious things have to eat the performance penalty for years and years. The world could use a language that is 'easy' like python/ruby but fast. Something like a simplified nim/f# with type inference. So it feels dynamic as you code but isn't really. Or yes if the normal way to run python was with a to…

It's not necessarily C-fast and it is dynamically typed but in the ratio of easiness-to-performance elixir seems to be pretty affable, especially for concurrency stuff. It does tend to balk at some heavier stuff like number-crunching, and that's when I see people start to use Rust NIFs.

Elixir also has easy to use environment with mix.

Re: Faster Parallel Python Without Python Multiprocessing

#20

Post by codr7 »

grumpyprole wrote: post #16
sametmax wrote: post #14

Earlier quoted context omitted.

Python is strongly typed, by dynamically typed.

From a type-theoretic point of view, Python is unityped: https://existentialtype.wordpress.com/2011/03/19/dynamic-lan... Python has a lot of merit as an easy to use language, a modern BASIC, but "strongly typed" is marketing that confuses rather than enlightens (to quote Prof. Harper). The term "strongly-tagged" is perhaps more apt, but sadly that ship has sailed.

So what you and Harper are saying is that Python only has one static type? While that is sort of true, I can't see how its leading anywhere. Many static languages have no dynamic types, I think that's worse.

Strongly typed used to mean safe, as in you can't add strings to ints or reinterpret values. As opposed to C and others where anything goes. I think that's a useful distinction.

Imagine all that lovely energy poured into understanding the compromises involved in designing realistic type systems. But its messy, and you don't become famous from dealing with messy problems.

Post reply on HN