Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

301–310 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#301

Post by sirfz »

detroitcoder wrote: post #210
eslaught wrote: post #161

The comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively. Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes. I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage sha…

^This. It is a very common usecase for applications I work with to create a very large in memory read-only pd dataframe and then put a flask interface to operations on that dataframe using gunicorn and expose as an API. If I use async workers, the dataframe operations are bound by GIL restraints. If I use sync workers, each process needs a copy of the pd dataframe which the server cannot handle (I have never seen pre…

pd.HDFStore is a good option for storing large DataFrames and it has some power querying capabilities.

Re: Let's Remove the Global Interpreter Lock

#303

Post by CogitoCogito »

throwaway613834 wrote: post #255
peterkelly wrote: post #183

Earlier quoted context omitted.

If CPU load is an issue, why would you be using an interpreter in the first place?

> If CPU load is an issue, why would you be using an interpreter in the first place? You're basically asking why NumPy, SciPy, Numba, etc. even exist. They exist because Python is ridiculously fast to develop in compared to, say, C++.

By using numpy, etc. you're basically _not_ using the interpreter because you're using C/C++/fortran code that's been compiled with python bindings.

To combine both your points, the best approach (if you like python) is to stick with python due to is ease of development and use libraries such as numpy as far as possible. However, if your use case is CPU bound but not served by those libraries, then you'll either need to develop your own extensions or throw away the interpreter altogether (and go with a different language).

Re: Let's Remove the Global Interpreter Lock

#304

Post by gnaritas »

swift wrote: post #274
gnaritas wrote: post #250

Earlier quoted context omitted.

> In languages which don't have a GIL, threads are almost as capable as processes, but lighter weight. But also plagued with shared state concurrency bugs, something multi-processing completely avoids so... > Threads are almost always preferable to processes in most languages. No, they aren't. It's too easy to write buggy code with threads, it's a flawed model. Now it's certainly true that more people choose threads…

The question of shared state vs. message passing is orthogonal to processes vs. threads. Both techniques can be and are commonly used in both situations.

No it isn't. Clearly both techniques "can" be used, but that one "allows" shared state trivially and one doesn't matters greatly; it is not orthogonal, you just don't grasp the point being made about the nature of the choice of abstractions and the problems that come with them.

Re: Let's Remove the Global Interpreter Lock

#305

Post by kllrnohj »

breatheoften wrote: post #263
kllrnohj wrote: post #219

Earlier quoted context omitted.

You seem to be conflating moving with sharing. Moving between threads is always free[1] regardless of if it's mutable or immutable, and there's no concurrency issues at all since it's a move. Move means the sender no longer has a reference. As in, std::move, rust's ownership transfer, webworker's transferables, etc... 1: Yes there's a single synchronize point where the handoff happens, but this is part of sending a m…

Ah I see -- Can you actually describe ownership in python sufficiently well to be able to describe this move operation for any useful python data structures?

Generically ownership is simply who has a reference to the object.

So if a.foo = b, then a 'owns' b. A 'move' is simply handing a different object the reference, then dropping your own reference. For example:

  a.foo = b // a 'owns' b
  c.foo = a.foo // a & c share b, however if the next line is:
  a.foo = None // a has 'moved' b to c, since c now has the only reference to b.
Some languages have codified this to make the contract part of the language, but it doesn't need any first-class language support. It's just a pattern at the end of the day.

Re: Let's Remove the Global Interpreter Lock

#306

Post by goerz »

throwaway613834 wrote: post #254
jzwinck wrote: post #253

Earlier quoted context omitted.

While a lot of NumPy is C and Fortran, Pandas is mostly pure Python and some Cython. And mostly it does not release the GIL. You often end up having to implement your own C extensions or use Numba for the core of your processing. Even with BLAS enabled, NumPy has almost zero intrinsic parallelism, np.dot() being the notable exception which releases the GIL and uses multicore by itself.

> Even with BLAS enabled, NumPy has almost zero intrinsic parallelism, np.dot() being the notable exception which releases the GIL and uses multicore by itself. Is there any sort of list (comprehensive or otherwise) that denotes which NumPy functions are parallelism-friendly? I mean this whether it's in terms of releasing the GIL, in terms of SIMD support, or in terms of being multi-core.

Is there a way to disable this? In an HPC environment, I don't want routines going multi-core without my explicit permission, under any circumstances. I will already have manually set up the parallelization to be at the highest logical level. If using Python, that usually means I have planned out the number of processes to be equal to the number of cores. If each process then starts doing its own multicore calculation (badly load-balanced!) it overtaxes the node and slows everything down.

I really wish numpy/pandas/scipy wouldn't do this kind of uncontrollable parallelization.

Re: Let's Remove the Global Interpreter Lock

#307

Post by sitkack »

eslaught wrote: post #161

The comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively. Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes. I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage sha…

The other glaring issue with CPython is all the globals, meaning there can be only ONE Python VM running in a process space.

The closest work has been done on PyParallel https://news.ycombinator.com/item?id=7861942 but afaik it is only for windows.

Re: Let's Remove the Global Interpreter Lock

#308

Post by tinix »

optimusclimb wrote: post #192
omarforgotpwd wrote: post #141

Earlier quoted context omitted.

Redis could help. Obviously not perfect for every use case but covers many of them.

It doesn't if you need to manage atomic data across the processes, as there's no way to lock and block the other cache consumers (think the data you need to handle cache evictions, etc.) Also, you're describing multiple python processes + an extra server (redis) process - as a "simpler" solution for the limitation that Python doesn't do multi-threads well. Of course there are a ton of use cases out there where you ca…

Blocking workers in a Redis queue is not hard... You can simply put them all on a pubsub control channel and then orchestrate them that way you need to do shit. Or literally just take down the processes, or the network, so they disconnect and stop BLPOPing the queue.

Cache evictions can be handled by Redis natively with TTL.

For retries and failure mitigation, you can still lean on Redis via BRPOPLPUSH/RPOPLPUSH.

If you want to scale beyond one machine, you can't rely on threading to help you. So why not just do it right to begin with, and use a parallel worker queue?

It's not a matter of the GIL being a limitation, a single machine is a limitation too. Don't blame your tools because you're misusing them.

As for threading in Python... on a single machine, for one reason or another... I would still rather use multiple processes, or at the very least, would just simply use eventlet and greenthreads.

Not saying it covers all use cases, it's not a silver bullet, and it doesn't replace threading natively, but damnit, it scales better, and it's the right way to do the task at hand.

Re: Let's Remove the Global Interpreter Lock

#309

Post by sitkack »

sametmax wrote: post #207
eslaught wrote: post #161

The comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively. Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes. I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage sha…

But then multi-interpreters would allow that, and the article discard it as a valid solution. I find it harsh. It seems much easier to implement, doesn't have the same serialization problem than multiprocessing has and allow to utilize all the CPUs. Yes it's not as good proper threads because you do have more overhead, but it's an order of magnitude better than what we currently have, while being way easier to do tha…

The points against sub-interpreters or multi-interpreters is a false dichotomy. There are plenty of scenarios where having multiple interpreters in the same address space would be valuable. Queues could be effectively to communicate between interpreters, no sharing needed. Where sharing is required, those structures can be on their own non-VM heap with the required locks.

Re: Let's Remove the Global Interpreter Lock

#310

Post by optimusclimb »

tinix wrote: post #308
optimusclimb wrote: post #192

Earlier quoted context omitted.

It doesn't if you need to manage atomic data across the processes, as there's no way to lock and block the other cache consumers (think the data you need to handle cache evictions, etc.) Also, you're describing multiple python processes + an extra server (redis) process - as a "simpler" solution for the limitation that Python doesn't do multi-threads well. Of course there are a ton of use cases out there where you ca…

Blocking workers in a Redis queue is not hard... You can simply put them all on a pubsub control channel and then orchestrate them that way you need to do shit. Or literally just take down the processes, or the network, so they disconnect and stop BLPOPing the queue. Cache evictions can be handled by Redis natively with TTL. For retries and failure mitigation, you can still lean on Redis via BRPOPLPUSH/RPOPLPUSH. If…

Wasn't reading data from or using a redis queue, this wasn't a blocking queue issue.

Second, my use case wasn't a simple cache, I was omitting details. So redis having a TTL eviction policy for the values it stores is a moot point. The resources I was dealing with ranged from around 0.5GB to several gigabytes. That was the important working data - but whether or not these objects were available was what had to be coordinated (as well as some other bookkeeping data.)

Also, in this case - of course scaling beyond one machine was important. We were. The issue is that for each machine you allocate, you want to maximize usage of its resources. So each machine gets its own data cache, but nonetheless, we still wanted to max CPU usage per machine. So again, it's multi-process, vs. multi-thread, and in this case - multi-threaded with shared memory was a much easier paradigm than handling co-ordination among separate python processes.

I was just giving an example of reasons one would want true multi-threading in python. I wasn't trying to go into explicit details of an entire project. Please consider this when you reply to people and tell them they're "misusing their tools."

Good day stranger.

Post reply on HN