Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster Parallel Python Without Python Multiprocessing

towardsdatascience.com

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Faster Parallel Python Without Python Multiprocessing

#21

Post by ris »

protomyth wrote: post #6
ATsch wrote: post #5

Earlier quoted context omitted.

The thing is that there's really little benefit from this. Anything performance critical will be written as a native extension anyway, even if Python speeds up by 10x. If you're doing that, you can lift the GIL anyway and run as much in parallel as you want.

Isn’t that a case of the tail wagging the dog? If Python code were 10x faster, the need to write performance sensitive code in a native extension would be a lot lower. There is a cognitive cost in having to switch between languages and knowing when to switch.

> If Python code were 10x faster

That's a pretty big "if". Unfortunately making python a lot faster is difficult because python is such a dynamic language. The pypy folks have done their best, with a respectable amount of success.

Re: Faster Parallel Python Without Python Multiprocessing

#22

Post by ris »

glandium wrote: post #7
htfy96 wrote: post #2

Every time I see a performance issue/solution of Python, I was wondering why there is no company maintaining a distribution with a high-performance Python JIT compiler with patched, GIL-free packages. Given the prevalence of Python and what have succeeded in JVM, it seems like a fruitful business.

One problem is that because of the GIL, many python libraries don't use locking on their data, etc., because they don't need to. That makes them thread-unsafe without the GIL.

If you look more closely, it often makes them thread-unsafe even with the GIL. Take a look some time at what operations the GIL can pre-empt a python thread in the middle of.

Re: Faster Parallel Python Without Python Multiprocessing

#23

Post by heyflyguy »

I am a terrible python scriptwriter. Still, I use my own scripts that I have written through trial and error and reading lots of stack overflow. I process about 500 images per day and each one takes about 30 seconds each. Adrian Rosenbrock has been a real lifesaver. I have a machine dedicated to this. I tried using multiprocessing once and could not get it to work. Being able to process in a parallel fashion would be a gamechanger for me.

The beauty of python for someone like me is that I can get my job done without actually having to do it. I free up my own time to leverage more of my creativity and have a multiplicative effect on my productivity.

The reason I bring all of this up is that so many of the examples for advanced libraries I see are geared towards seasoned software engineers. The examples include false arrays of data so that you can "just see" how to use it. I don't think anyone realizes how confusing this is to the guy who is a restaurant manager, or the gal who is a researcher that just needs to know how to make this work for them.

If it's an image, how about putting image = cv2.imread("C:\image.jpg") or whatever?

Anyway, a bit of a rant but there are people who are very thankful that smart people in this world like yourself write libraries we can use to make our daily lives better. Including example code that is stupid simple would make me so much happier.

Re: Faster Parallel Python Without Python Multiprocessing

#24

Post by willtim »

codr7 wrote: post #20
grumpyprole wrote: post #16

Earlier quoted context omitted.

From a type-theoretic point of view, Python is unityped: https://existentialtype.wordpress.com/2011/03/19/dynamic-lan... Python has a lot of merit as an easy to use language, a modern BASIC, but "strongly typed" is marketing that confuses rather than enlightens (to quote Prof. Harper). The term "strongly-tagged" is perhaps more apt, but sadly that ship has sailed.

So what you and Harper are saying is that Python only has one static type? While that is sort of true, I can't see how its leading anywhere. Many static languages have no dynamic types, I think that's worse. Strongly typed used to mean safe, as in you can't add strings to ints or reinterpret values. As opposed to C and others where anything goes. I think that's a useful distinction. Imagine all that lovely energy pou…

> Many static languages have no dynamic types, I think that's worse.

I can't think of a static language that doesn't support a dynamic type, it's just that most require explicit casts or unpacking to use with existing typed interfaces, e.g. "Object" in Java.

> Strongly typed used to mean safe

It still does, it's just that type theory doesn't distinguish between a runtime failure with a nice error message versus a segfault, both are programs that "go wrong".

I would be in favour of "strong dynamic types" instead of "strongly typed".

Re: Faster Parallel Python Without Python Multiprocessing

#25

Post by ForHackernews »

gameswithgo wrote: post #13
protomyth wrote: post #6

Earlier quoted context omitted.

Isn’t that a case of the tail wagging the dog? If Python code were 10x faster, the need to write performance sensitive code in a native extension would be a lot lower. There is a cognitive cost in having to switch between languages and knowing when to switch.

its unfortunate that dynamic interpreted languages with easy uptake for small projects get so popular, and then big, serious things have to eat the performance penalty for years and years. The world could use a language that is 'easy' like python/ruby but fast. Something like a simplified nim/f# with type inference. So it feels dynamic as you code but isn't really. Or yes if the normal way to run python was with a to…

> The world could use a language that is 'easy' like python/ruby but fast.

IMHO, Julia is that language: https://julialang.org/

Re: Faster Parallel Python Without Python Multiprocessing

#26

Post by xapata »

glandium wrote: post #12
zaphirplane wrote: post #9

Earlier quoted context omitted.

You mean libraries with c extensions

No, even libraries without. Picking something random in the python 3.7 standard library: collections.OrderedDict.__setitem__ doesn't look thread-safe when it updates its linked list. EDIT: well, in fact, the data race in that one is there whether there is a GIL or not...

Just FYI, there are few remaining use cases for OrderedDict, now that built-in dict keeps insertion order.

Re: Faster Parallel Python Without Python Multiprocessing

#27

Post by protomyth »

ris wrote: post #21
protomyth wrote: post #6

Earlier quoted context omitted.

Isn’t that a case of the tail wagging the dog? If Python code were 10x faster, the need to write performance sensitive code in a native extension would be a lot lower. There is a cognitive cost in having to switch between languages and knowing when to switch.

> If Python code were 10x faster That's a pretty big "if". Unfortunately making python a lot faster is difficult because python is such a dynamic language. The pypy folks have done their best, with a respectable amount of success.

That's a pretty big "if".

The 10x figure I was quoting was from the comment I was responding to. I do think people need to look at what was accomplished by the Strongtalk team and various other projects dealing with dynamic languages including some Lisp implementations. I'm pretty sure Python can be a lot faster, but the legacy code is going to be a big, big problem.

Re: Faster Parallel Python Without Python Multiprocessing

#28

Post by ris »

protomyth wrote: post #27
ris wrote: post #21

Earlier quoted context omitted.

> If Python code were 10x faster That's a pretty big "if". Unfortunately making python a lot faster is difficult because python is such a dynamic language. The pypy folks have done their best, with a respectable amount of success.

That's a pretty big "if". The 10x figure I was quoting was from the comment I was responding to. I do think people need to look at what was accomplished by the Strongtalk team and various other projects dealing with dynamic languages including some Lisp implementations. I'm pretty sure Python can be a lot faster, but the legacy code is going to be a big, big problem.

> various other projects dealing with dynamic languages

Not all dynamic languages are born the same.

> I'm pretty sure Python can be a lot faster

You might like you join the PyPy team then and find out.

Re: Faster Parallel Python Without Python Multiprocessing

#29

Post by codr7 »

willtim wrote: post #24
codr7 wrote: post #20

Earlier quoted context omitted.

So what you and Harper are saying is that Python only has one static type? While that is sort of true, I can't see how its leading anywhere. Many static languages have no dynamic types, I think that's worse. Strongly typed used to mean safe, as in you can't add strings to ints or reinterpret values. As opposed to C and others where anything goes. I think that's a useful distinction. Imagine all that lovely energy pou…

> Many static languages have no dynamic types, I think that's worse. I can't think of a static language that doesn't support a dynamic type, it's just that most require explicit casts or unpacking to use with existing typed interfaces, e.g. "Object" in Java. > Strongly typed used to mean safe It still does, it's just that type theory doesn't distinguish between a runtime failure with a nice error message versus a seg…

I don't know enough Haskell or Rust to claim its impossible (outside of unsafe of course), but it clearly goes against the design of the language and is meant as fallback more than option.

Yes, you can write Java using nothing but Object. And no, the experience won't be comparable to using a dynamically typed language.

Map is not terrain, if theory doesn't match reality then changing the former is the only thing that makes sense.

Categories are generalizations, making them more specific weakens them. A RISC approach works better from my experience, where categories are kept trivial and members belong to multiple categories.

Re: Faster Parallel Python Without Python Multiprocessing

#30

Post by protomyth »

ris wrote: post #28
protomyth wrote: post #27

Earlier quoted context omitted.

That's a pretty big "if". The 10x figure I was quoting was from the comment I was responding to. I do think people need to look at what was accomplished by the Strongtalk team and various other projects dealing with dynamic languages including some Lisp implementations. I'm pretty sure Python can be a lot faster, but the legacy code is going to be a big, big problem.

> various other projects dealing with dynamic languages Not all dynamic languages are born the same. > I'm pretty sure Python can be a lot faster You might like you join the PyPy team then and find out.

You missed the but the legacy code is going to be a big, big problem. part. If, as other posters point out, those C extensions are not designed to work without the the GIL, then its going to be a 2->3 situation.

Frankly, I honor the Python folks and all their achievements, but I really dislike the language. I mostly am researching another paradigm.

Post reply on HN