Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Threading in Python

nryoung.org

Post reply on HN
Print view View original

21–30 of 30 posts

Re: Threading in Python

#21

Post by obviouslygreen »

lifeisstillgood wrote: post #20
exDM69 wrote: post #13

> Generally, you should only use threads if the following is true: - Sharing memory between threads is not an issue. Here's the problem. Threads are really useful only if you can share memory between threads. If you can't share memory, you're usually better off using many processes. Threads in Python (ie. CPython) can still be useful for I/O multiplexing or executing native code in background worker threads via FFI a…

Thank you. I would even go so far as to say that except in simple cases (downloading 10,000 images goes much faster with 100 worker threads than serially - which is I think the origin of "dont share memory) I would say do not use Python - or any other similar language. Got parallel needs at your core? Look at Erlang or Haskell. If parallel or distributed work is mission critical, go with a language that has such thin…

You are technically correct. However, there's at least one invalid assumption at the core of this, which is that people who need to do things that fall into the there's-a-better-language-for-this category always have the opportunity to learn and implement a more appropriate tool.

This is almost always the case on commercial projects. Extremely few companies and clients will be perfectly fine with "yes, I'm a Python expert, but this would be best done in Erlang; I will need an extra week to research, learn, and implement this on top of the month the project would otherwise take." In most situations you either do it the way you know how to do it, eat the extra time (not practical in most cases), or you lose the contract/job.

Of course this is specific to client work, but I think most of us are likely doing that or something similarly limiting for at least half our waking hours, making it fairly relevant when considering ideas like "using tool X for job A is not a good idea when tool Y exists." It's correct but ignores too many practical situations to be very useful advice.

Re: Threading in Python

#23

Post by scott_s »

asynchrony wrote: post #15
ColinWright wrote: post #12

Earlier quoted context omitted.

This turns out not to be the case - some problems and calculations are most naturally expressed with concurrency.

Could you provide an example of a calculation that is more naturally expressed with concurrency?

I wouldn't say calculations, as that implies, to me, a mathematical kernel. I can't think of any low-level calculations that are easier with concurrency. But some problems are most naturally solved with concurrency - that is, it's easier to architect the software with threads. Applications with GUIs are sometimes like this: it may be easier to have a main application thread that does the real computations, and a separate thread that handles all GUI work.

The pattern is that instead of one monolithic process that must know about and do everything, sometimes it's easier to think about several independent processes that only know about one domain, and make requests or give answers to other processes that know about other domains.

Re: Threading in Python

#24

Post by scott_s »

You are not looking for the best optimized performance since threads share memory within a process.

That is a non-sequitur to me. The first half I'm on board with: generally, you use threads to improve performance, but because of the GIL in Python, you may not get the parallelism you want. If you're calling into libraries that don't hold the GIL, then great, but that means you have to be very aware of what's going on below you.

The second half does not follow, though. Typically, that threads share the same address space is the entire reason we use threads over processes. And the reason comes from improved performance: if the thread share an address space, you don't need to copy the data. Copying data is expensive. (It also means you're susceptible to a whole host of synchronization bugs.)

Re: Threading in Python

#25

Post by rozap »

seanp2k2 wrote: post #10

Some developers, when confronted with a problem, think "I know, I'll use threads" have two Now problems they. http://regex.info/blog/2006-09-15/247

Clever. But just like with regexps, the issue arises when a programmer applies them to every situation he/she encounters. Obviously they need not be avoided like the plague, but rather used when the situation calls for them.

Re: Threading in Python

#26

Post by pekk »

scott_s wrote: post #24

You are not looking for the best optimized performance since threads share memory within a process. That is a non-sequitur to me. The first half I'm on board with: generally, you use threads to improve performance, but because of the GIL in Python, you may not get the parallelism you want. If you're calling into libraries that don't hold the GIL, then great, but that means you have to be very aware of what's going on…

Sharing all data between threads means you're susceptible to a whole host of synchronization bugs (in the sense of thread synchronization, not data synchronization). Unless you use synchronization primitives like locks to protect the shared data, which can also easily kill concurrency. It's a trade-off.

If avoiding copying is not a top problem, then you may be wasting your time; there's nothing wrong with using abstractions more appropriate to your environment.

If the program scales out, it should be less important to micro-optimize inside each process because it's so much cheaper just to use another core or another node.

It's getting boring to hear all discussions of concurrency reduced to threads, and threads reduced to the GIL in CPython. It's really not that simple.

Re: Threading in Python

#27

Post by pekk »

stefantalpalaru wrote: post #19

This is Python so threads don't run concurrently because of the GIL (no, using C/C++ code where you can release it is not in the scope of this article). Save yourself the trouble and use multiprocessing.

It's oversimplifying to say "This is Python so threads don't run concurrently because of the GIL".

This is not a Python-language issue, it is an implementation-specific issue. Not all implementations of Python have the GIL.

In reality, threads do run concurrently. Because in the CPython (itself written in C) with the famous GIL, it is normal and realistic to do I/O and heavy computation in C code that releases the GIL, enabling threads to work concurrently. There's no reason this information shouldn't be part of discussions on threads in Python.

That doesn't mean threads are great for everything, but the severity of the case is easily and frequently overstated.

Re: Threading in Python

#28

Post by scott_s »

pekk wrote: post #26
scott_s wrote: post #24

You are not looking for the best optimized performance since threads share memory within a process. That is a non-sequitur to me. The first half I'm on board with: generally, you use threads to improve performance, but because of the GIL in Python, you may not get the parallelism you want. If you're calling into libraries that don't hold the GIL, then great, but that means you have to be very aware of what's going on…

Sharing all data between threads means you're susceptible to a whole host of synchronization bugs (in the sense of thread synchronization, not data synchronization). Unless you use synchronization primitives like locks to protect the shared data, which can also easily kill concurrency. It's a trade-off. If avoiding copying is not a top problem, then you may be wasting your time; there's nothing wrong with using abstr…

Yes, it's a trade-off, which is why I brought it up.

But my point here is that the statement the author made, as far as I'm able to understand it, makes no sense. That is, I think he tried to discuss these issues, but I don't think he understands them well enough to do so. I think you and I are in agreement, unless you are saying that what the author stated does make sense.

Re: Threading in Python

#29

Post by lifeisstillgood »

obviouslygreen wrote: post #21
lifeisstillgood wrote: post #20

Earlier quoted context omitted.

Thank you. I would even go so far as to say that except in simple cases (downloading 10,000 images goes much faster with 100 worker threads than serially - which is I think the origin of "dont share memory) I would say do not use Python - or any other similar language. Got parallel needs at your core? Look at Erlang or Haskell. If parallel or distributed work is mission critical, go with a language that has such thin…

You are technically correct. However, there's at least one invalid assumption at the core of this, which is that people who need to do things that fall into the there's-a-better-language-for-this category always have the opportunity to learn and implement a more appropriate tool. This is almost always the case on commercial projects. Extremely few companies and clients will be perfectly fine with "yes, I'm a Python e…

I used to think that - bit I now believe we can find the clients who want it done right more than right now

To be fair the best way of judging this is the reverse penalty clause - so this job must be done by June 1. Ok and if it is three weeks late as we use erlang? A penalty of 1000 dollars a day? Wow - ok so if I am a month early you can pay a bonus of 20,000 ? No - so perhaps we are not as time critical as we feared ? Would you rather save 20,000 in ongoing maintence costs and general uncertainty over how good the solution is for three weeks delay that would likely creep in anyway?

Have I told you erlang has an uptime of 99.999 % proven over twenty years?

Re: Threading in Python

#30

Post by ColinWright »

asynchrony wrote: post #15
ColinWright wrote: post #12

Earlier quoted context omitted.

This turns out not to be the case - some problems and calculations are most naturally expressed with concurrency.

Could you provide an example of a calculation that is more naturally expressed with concurrency?

Some of the calculations I did for my PhD were inherently parallel - some branches could be cut as others found certain features in the structures. There was no obvious way to order things, it was natural to just set them all off at once and let things trim and prune concurrently.

Various image processing algorithms I use also work concurrently over the image, agents working in different places, and then deciding who is doing best and letting each other know so resources can be concentrated and refocused.

Unless you've worked for a long time in an inherently parallel environment it's hard to see things as anything other than serial processes. For me it can sometimes be hard to see how to serialize things, as many things are naturally parallel. I've worked for nearly 20 years on systems with at least 100 processors and limited communications, so parallel algorithms tend to be what I see first.

Sorting, for example. Merge sort is inherently a parallel algorithm. Why sort that part, then this part? Why not do them both at once, and then start merging while the remainder of the sort is still going on? Or quick sort. Once you have divided your vector into two pieces, why sort them serially?

Sieve of Eratosthenes can be thought of as inherently parallel, as can many factoring algorithms. Why factor this small number, then that one, then that one, and so on, when you can factor them all together?

These are all things that are naturally expressed as concurrent algorithms, processes, or calculations.

Post reply on HN