Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

281–290 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#281

Post by tinix »

detroitcoder wrote: post #210
eslaught wrote: post #161

The comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively. Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes. I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage sha…

^This. It is a very common usecase for applications I work with to create a very large in memory read-only pd dataframe and then put a flask interface to operations on that dataframe using gunicorn and expose as an API. If I use async workers, the dataframe operations are bound by GIL restraints. If I use sync workers, each process needs a copy of the pd dataframe which the server cannot handle (I have never seen pre…

FWIW, I routinely throw many GBs of pickled dataframes into Redis all the time, and then cluster the workload between multiple processes that are coordinated as a sort of namespaced job queue, all via Redis pubsub, blpop, l/rpush, and set/get. There are much faster and more efficient serialization formats like msgpack or protocol buffers however, compared to pickle, if you really need to squeeze out performance. You just have to chunk your bulk out into pieces and spread the bulk across multiple workers. You have an orchestrator class that puts things onto the queues, pulls things off, loads any modules you need, handles exceptions, etc...

Then you can namespace your queues (and workers), and have separate queues for results handling to push data to the next stage of the pipeline, etc... With stacks of workers, configured as needed. It's all pretty high level from there. GIL has no effect here, and as a side-effect, now you can utilize a massive number of parallel processes for heavy lifting and crunching, even on different machines over the network, where-as that wouldn't be possible with a traditional threaded architecture.

Not saying this necessarily covers your use-case, but it seems strange to use dataframes as a sort of in-memory database, vs using dataframes as the framing to do the munging and heavy lifting. What are you wanting to put multiple cursors on it or something? You could do this with greenlets, for what it's worth... But as someone who has gone down that route (multiple greenlets working over shared stack) I promise doing it with multiple processes and a queue is better, and ultimately way more flexible. Especially if you use something like msgpack or protocol buffers... Then you can have any workers from multiple programming languages and development paradigms doing different work at different stages, all orchestrated and working together via Redis.

Re: Let's Remove the Global Interpreter Lock

#282

Post by PrimHelios »

dom0 wrote: post #92
PrimHelios wrote: post #64

Earlier quoted context omitted.

CPython doesn't have any reservations about breaking the Python API between minor versions, so why care about the C API? I get where you're coming from, but they've already shown they don't care much for compatibility, so I don't see why that's a big obstacle.

Removing the GIL (in a non-braindead way) likely entails breaking all existing code using the C API. PyPy could do so without breaking cpyext, by maintaining the illusion of a GIL whenever control passes to cpyext.

That makes sense, I hadn't thought about the extent of the breakage.

Re: Let's Remove the Global Interpreter Lock

#283

Post by ash_gti »

ash_gti wrote: post #265
smitherfield wrote: post #85

Earlier quoted context omitted.

It’s also still dog-slow for the (Perl 5 / scripting-language) common case, which makes whatever theoretical performance improvements to its semantics a bit academic at this point: https://news.ycombinator.com/item?id=15004977

If you use master instead of the current release that example goes from about two mins to just under one minute for my machine. Also, if you use .subst(‘y’, ’n’) instead of a regex it runs in under 9 seconds locally. Thats still much slower than perl 5 (which locally takes less than half a second) but they’re making great strides at improving performance.

So, doing another simple change to my code brought the runtime down to a few ms. Doing:

  time yes | head -n1000000 | perl6 --profile -e 'for $*IN.readchars { .subst("y", "n").print }’ > /dev/null
says it took 86 ms. Which is pretty decent I’d say.

Re: Let's Remove the Global Interpreter Lock

#284

Post by memracom »

Personally, I don't think the GIL matters. First of all most of us run apps on Linux which has reduced the overhead of processes so much that threads have lost much of their advantage. Secondly, people understand that locks are generally a bad thing to use unless you really are a threading/locking rocket scientist. Most mere mortal developers are better to use message queues. Even the Java world has mostly given up locks in favor of java.util.concurrent which was implemented by serious experts to handle all of the corner cases that you would not think of. Third, using an external Message Queuing system like RabbitMQ gives you other benefits. And fourth, writing distributed apps glued together by message queues helps you avoid the dreaded Big Ball of Mud.

At this stage in Python's evolution, I view the GIL removal as a computer science project that some people will implement again, and again, just to learn or to exercise their chops. Great idea! Just don't demand that the entire community of Python developers goes down your road.

If CPython never gets rid of the GIL that suits me just fine. GIL free programming can be done on other implementations of Python like Jython and IronPython. As far as PyPy is concerned, as long as it does not disrupt the use of PyPy as a means of speeding up a CPython app from time to time, then have fun.

Re: Let's Remove the Global Interpreter Lock

#285

Post by est »

chubot wrote: post #147
wiremine wrote: post #58

> We estimate a total cost of $50k... Just looking at it from a financial perspective, having a great Python interpreter that doesn't have a GIL seems like a no brainer for $50,000, and it creates another reason why people should take a look at PyPy. Side note: if you haven't looked at PyPy, check it out, along with RPython https://rpython.readthedocs.io/en/latest/

Who uses PyPy? I have been hearing about it for so long now, maybe 10 years. And I have been programming in Python almost full-time for 14 years. But still I don't know anybody who uses it? It seems like the C extension API is still an issue, or am I mistaken?

we run a large scale sockjs cluster.

Switched from CPython to PyPy, instant 3x performance boost.

Re: Let's Remove the Global Interpreter Lock

#286

Post by nurettin »

memracom wrote: post #284

Personally, I don't think the GIL matters. First of all most of us run apps on Linux which has reduced the overhead of processes so much that threads have lost much of their advantage. Secondly, people understand that locks are generally a bad thing to use unless you really are a threading/locking rocket scientist. Most mere mortal developers are better to use message queues. Even the Java world has mostly given up l…

Coming from mobile and desktop programming, most use cases I've seen for threads revolve around doing something in the background to keep user interfaces responsive. That use case already has a threadsafe queue. The UI queue.

When your thread finishes or is ready to signal progress, you queue the event to the UI thread and forget about it.

Now I've been following this pattern for a long time and have no a experience dealing with GIL How this removal of GIL going to effect this use case if at all?

Re: Let's Remove the Global Interpreter Lock

#287

Post by freyir »

seunosewa wrote: post #53

The ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.

We could call it Python 3.

Re: Let's Remove the Global Interpreter Lock

#288

Post by vgy7ujm »

bsder wrote: post #247
vgy7ujm wrote: post #222

Earlier quoted context omitted.

Only if you don't know Perl... To me Python is more unreadable ;)

Only if you know ALL of Perl. I used to carry around a Perl program of my own on a printout to take to VLSI interviews. That way when I got the "Do you know Perl?" question I could bring it out and force the interviewer into MY stupid subset of Perl rather than being stuck in his stupid subset of Perl. That's not a compliment to the language.

Is Python really any different or is it just wishful thinking?

Why do I see Python programs that look like a weird mix of Lisp and Java? Surely it is because even with what Python enforce, there are many many ways to produce unclear code that really don't even has to do with the language used.

And why did my employer see the need for the comprehensive Python style guidelines manual... I guess Python bit just as hard as Perl.

Also now that Python is used more by newbies and non-programmers that is where more of the bad code end up (aka Perl late 90s, still pollutes the internet). The quality of Perl frameworks, libs, example code etc is actually increasing and getting easier to find.

Re: Let's Remove the Global Interpreter Lock

#289

Post by astrodust »

neolefty wrote: post #176
omarforgotpwd wrote: post #142

Earlier quoted context omitted.

Motivation for removing the GIL is basically that when people hear about it they go "hmmm that doesn't sound good". Obviously many applications have been written in GIL languages and there aren't really many practical problems that can't be overcome easily.

I think it may be some Stockholm Syndrome -- people have worked very hard to get around the GIL, and they've come to expect its limitations and respect those solutions. But I've never heard of someone asking for a GIL to be added to the JVM.

Exactly! If a feature is good it's worth adding, and the GIL in any language is not good.

People are just as quick to bemoan a language for not having something (generics, templates, pre-processors) because they see some perceived need, but a GIL is never one of those things.

Re: Let's Remove the Global Interpreter Lock

#290

Post by fijal »

twoodfin wrote: post #98

"It mostly works for simple programs, but probably segfaults on anything complicated" is not a promising beginning. Starting with race condition chaos and trying to patch your way out of it with "strategic" locking a) Inspires much less confidence than starting with a known-correct locking model (the degenerate case being a GIL) and preserving it while improving available concurrency. and b) Seems at least 50/50 to e…

The concurrent garbage collector has been already written. It probably has bugs, but the basis has been done for the STM effort and redone now. The mess of race conditions is surely not a great place to start, which is why it would take a man year to finish :-) I don't see a much better starting point tbh, but to look at all the mutable data structures in python (of which there are too many) and try hard
Post reply on HN