Let's Remove the Global Interpreter Lock • Hacker News Board

Re: Let's Remove the Global Interpreter Lock

#221

Post by hermitdev » Mon, Aug 14, 2017, 8:04 PM UTC

dec0dedab0de wrote: post #19

ant6n wrote: post #18

Earlier quoted context omitted.

Using multiprocessing is a pain to use, and it's slow.

in what was is it a pain that threading is not?

It's been a while, and my memory is fuzzy, but I recall either pyodbc or pysybase reacting very poorly with the multiprocessing module. With multiprocessing, Python would segfault after fork. With threading, it would "work" albeit slowly. Also, IIRC, it did not matter if the module was imported before or after the fork, still segfaulted. I never had the time to try and track down the issue that was causing it, though, deadlines and all that.

Re: Let's Remove the Global Interpreter Lock

#222

Post by vgy7ujm » Mon, Aug 14, 2017, 8:07 PM UTC

DougN7 wrote: post #71

jrs95 wrote: post #59

Earlier quoted context omitted.

Perl 6 doesn't have a GIL, and already has a sane concurrency model, but the lack of libraries and community interest seems to make that pretty much a non-starter.

Well, that and Perl has to be one of the most unreadable languages out there ;)

Only if you don't know Perl... To me Python is more unreadable ;)

Re: Let's Remove the Global Interpreter Lock

#223

Post by sambe » Mon, Aug 14, 2017, 8:10 PM UTC

detroitcoder wrote: post #210

eslaught wrote: post #161

The comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively. Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes. I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage sha…

^This. It is a very common usecase for applications I work with to create a very large in memory read-only pd dataframe and then put a flask interface to operations on that dataframe using gunicorn and expose as an API. If I use async workers, the dataframe operations are bound by GIL restraints. If I use sync workers, each process needs a copy of the pd dataframe which the server cannot handle (I have never seen pre…

I rarely hear people complain about genuine use-cases but this would seem to be one. However, aren't most/all of the dataframe operations done in C extensions in these cases?

Re: Let's Remove the Global Interpreter Lock

#224

Post by dkersten » Mon, Aug 14, 2017, 8:10 PM UTC

std_throwaway wrote: post #212

dom0 wrote: post #92

Earlier quoted context omitted.

Removing the GIL (in a non-braindead way) likely entails breaking all existing code using the C API. PyPy could do so without breaking cpyext, by maintaining the illusion of a GIL whenever control passes to cpyext.

Does it lock the GIL so numpy can release it again immediately afterwards?

Perhaps it makes the unlock call a no-op before numpy tries to unlock it.

Re: Let's Remove the Global Interpreter Lock

#225

Post by DonbunEf7 » Mon, Aug 14, 2017, 8:15 PM UTC

macrael wrote: post #11

Do people here use pypy in production? What are the benefits?

Switched from CPython+Numpy to PyPy years and years ago, got a 60x speedup on a core numerical kernel and 20x speedup on real-world benchmarks. The codebase was a multiplayer game server. Less memory usage overall, leading to a big improvement in the number of players that could be connected.

You have to not have problematic libraries in your system, but honestly they're all either shitty on CPython too (literally every GUI toolkit that is not Tkinter!) or they're stuff like lxml, where the author/maintainer just has an anti-PyPy bias that they won't drop.

Re: Let's Remove the Global Interpreter Lock

#226

Post by slantedview » Mon, Aug 14, 2017, 8:19 PM UTC

metalliqaz wrote: post #112

crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

> multiprocessing instead of multithreading

There's a reason threads exist.

Re: Let's Remove the Global Interpreter Lock

#227

Post by falcolas » Mon, Aug 14, 2017, 8:30 PM UTC

khedoros1 wrote: post #194

peterkelly wrote: post #183

Earlier quoted context omitted.

If CPU load is an issue, why would you be using an interpreter in the first place?

A lot of the heavy lifting is done through calls to C libraries anyhow, with Python just being a convenient way to pass the data around.

Indeed, and in that case the GIL is effectively a non-issue (there's no requirement for the GIL to be held by non-python code).

Re: Let's Remove the Global Interpreter Lock

#228

Post by tonybaker » Mon, Aug 14, 2017, 8:38 PM UTC

GET RESULTS IN ONE HOUR TIME FOR ALL KIND OF EMAIL HACK.Change School Grades? Hack Banks? Erase Criminal Records? Hack Websites? Hack Database? Hack Drivers license? Hack Call Log? Hack Visichat and Flashchat Rooms? Hack FTP User and Pass? Hack Facebook,Whatssap,Twitter,Instagram,Webcam etc.Hack VB Forum? Hack Wordpress Blog? Hack CC any Country? Hack Money Booker Account? Hack Liberty Reverse Account? Hack Paypal Account? Root Server? By Pass Google Phone Verification? Install Red on Linux Server? Hash Crack? DDOS Server? Retrieval of Lost Files and Documents CONTACT: empiricalhackers4187@gmail.com

Re: Let's Remove the Global Interpreter Lock

#229

Post by falcolas » Mon, Aug 14, 2017, 8:39 PM UTC

vladf wrote: post #143

There seem to be a lot of naysayers in the comments about removing the GIL. Multiprocess parallelism isn't always appropriate, so I find this to be a very promising change that will definitely make me want to switch to PyPy. Here are the use cases I've found multiprocessing to be inappropriate: * High-contention parallel operations. Doing synchronization through a Manager (a separate IPC-based synchronizing broker pr…

> There seem to be a lot of naysayers in the comments about removing the GIL.

That's because it's been attempted over and over and over again. And each time it ends up failing due to the decrease in single-threaded performance (the bevy of necessary memory mutexes aren't free)), and the extensive amount of work required to make all of the standard libraries threadsafe.

I don't buy the $50,000 cost for a second. Sure, you might be able to safely change the interpreter for that little money, but you couldn't fix up performance and the standard library for that.

Re: Let's Remove the Global Interpreter Lock

#230

Post by ericfrederich » Mon, Aug 14, 2017, 8:42 PM UTC

metalliqaz wrote: post #112

crb002 wrote: post #97

Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

The GIL is a legit pain when dealing with GUIs.

When you're jumping between C/C++ code and Python code you don't care much about the GIL... until you have a GUI which needs to be kept responsive and needs the GIL to do so.