Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

41–50 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#41

Post by devwastaken »

chocolatebunny wrote: post #28
devwastaken wrote: post #13

This would be great if it means we can run the C portions of Python in threads without performance hits. I recently started a little project that is a cross-platform GUI for batch bzip2 compression, and Python did it quite well with its built-in bzip2 module. But, once I tried to do it parallel, the performance impacts of GIL were obvious. Yes, you can work around that with multi-process, but I'd rather not be spammi…

Were you running on Windows or Linux? It's my understanding that multiple processes doesn't have a big performance penalty on Linux compared to multiple threads.

I was running under Windows, but the application is meant for Windows linux and mac.

Re: Let's Remove the Global Interpreter Lock

#42

Post by dkersten »

devwastaken wrote: post #13

This would be great if it means we can run the C portions of Python in threads without performance hits. I recently started a little project that is a cross-platform GUI for batch bzip2 compression, and Python did it quite well with its built-in bzip2 module. But, once I tried to do it parallel, the performance impacts of GIL were obvious. Yes, you can work around that with multi-process, but I'd rather not be spammi…

If you're calling into C, you can disable the GIL from C for the duration of the call. You have to re-enable it again before the call returns and you have to be careful not to call any of the Python C API calls that rely on the GIL (reference counting and such for sure). Of course, if you want to do this, you can't simply call into a random C library directly but have to write a C stub.

Re: Let's Remove the Global Interpreter Lock

#43

Post by makmanalp »

JulianWasTaken wrote: post #31
macrael wrote: post #11

Do people here use pypy in production? What are the benefits?

We've been running a very large production PyPy deployment across pretty much all our Python apps for about... 4 years now. Saves us a ton of money for essentially no real downside.

Just out of curiousity, would you be willing to answer a few more questions? What has the memory tradeoff been like? What is the workload you're using it for?

Re: Let's Remove the Global Interpreter Lock

#44

Post by dec0dedab0de »

cyphar wrote: post #24
dec0dedab0de wrote: post #19

Earlier quoted context omitted.

in what was is it a pain that threading is not?

You can't just use functions defined in your tool, you need to create a faux-cli interface in order to run each parallel worker. Also, copying large datasets between processes is not efficient. And also, there are cases where the fan-out approach is not the best way of parallelizing a task, and passing information back up to a parent task is more complicated than necessary.

"You can't just use functions defined in your tool, you need to create a faux-cli interface in order to run each parallel worker."

the multiprocessing library allows you to launch multiple processes using your function definitions. It's almost the same as the multithreading library but does not share data.

It seems the real problem, as you pointed out, is the additional memory. I didn't consider situations where each process would need an identical large data set, instead of just a small chunk to work on.

Re: Let's Remove the Global Interpreter Lock

#45

Post by rspeer »

leereeves wrote: post #23
make3 wrote: post #17

Earlier quoted context omitted.

There are many cases where the objects are too big to be passed around. Python is used a huge amount in Machine learning and datascience, where being able to do parallel work on stuff already in memory would be great.

Are those applications often bottlenecked by the CPU, as opposed to GPU or data transfer?

The world of algorithms that run well on a CPU is still much, much bigger than the world of algorithms that run well on a GPU, even in machine learning.

And even if you're fortunate enough that Nvidia designs their GPUs to solve your problem, why should the CPU cores sit idle?

Re: Let's Remove the Global Interpreter Lock

#46

Post by chrisseaton »

dec0dedab0de wrote: post #25
make3 wrote: post #17

Earlier quoted context omitted.

There are many cases where the objects are too big to be passed around. Python is used a huge amount in Machine learning and datascience, where being able to do parallel work on stuff already in memory would be great.

So work on data that can not be broken down into smaller chunks? That makes sense, and is something I never come across.

I'm sure they can be broken down into smaller chunks, but is it more efficient if they aren't broken down and instead shared memory is used? If you want parallelism you're obviously already worried about performance.

Re: Let's Remove the Global Interpreter Lock

#47

Post by dkersten »

dec0dedab0de wrote: post #12

Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…

To add to what everyone else said, if you need transactional semantics, its much simpler in multiple threads. With multiple processes (local or remote), you can't simply share an atomic data structure or a lock, you have to use a distributed lock or consensus algorithm, which are more complex and usually quite "chatty". If memory or network bandwidth are constrained, it may be especially desirable to eliminate this, but even if not, fast locking/transactions may be desirable regardless.

If you're using multiple processes for CPU-bound performance, why not squeeze as much as you can out of each CPU?

Re: Let's Remove the Global Interpreter Lock

#48

Post by chrisseaton »

Beltiras wrote: post #21

Think about a recursive function whose implementation is changed while it is running. The replacement might have an entirely different algorithm. Which version finishes the stack call?

The version that was originally activated. I think that's the case in every single parallel implementation of a programming language ever. I can't imagine it working any other way.

When you redefine a method in any language I'm aware of you just change which method the name points to. You don't modify the original method.

Re: Let's Remove the Global Interpreter Lock

#50

Post by monkmartinez »

devwastaken wrote: post #13

This would be great if it means we can run the C portions of Python in threads without performance hits. I recently started a little project that is a cross-platform GUI for batch bzip2 compression, and Python did it quite well with its built-in bzip2 module. But, once I tried to do it parallel, the performance impacts of GIL were obvious. Yes, you can work around that with multi-process, but I'd rather not be spammi…

> I'd rather not be spamming the running processes list and have to actually handle seperate processes that should be threads.

I may be a bit naive asking this... but why would you care that much?

Looking at activity monitor on my Mac, I count 14 Google Chrome Helper Process instances each spawning upwards of 13 threads. Adobe does something similar, as do several other programs/applications on my machine. Yet, my machine is mostly idle.

I can only speak for myself here. If I want something done on my computer... I don't care if it spams my process list if that is what it takes to complete the task. Don't crash my machine, but do what you have to do to get it done quickly.

Post reply on HN