Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Let's Remove the Global Interpreter Lock

morepypy.blogspot.com

Post reply on HN
Print view View original

81–90 of 326 posts

Re: Let's Remove the Global Interpreter Lock

#81

Post by sandGorgon »

JulianWasTaken wrote: post #69
makmanalp wrote: post #43

Earlier quoted context omitted.

Just out of curiousity, would you be willing to answer a few more questions? What has the memory tradeoff been like? What is the workload you're using it for?

Certainly! It's a bit hard to answer some of those questions because it's been so long since we've run CPython, and also because we've now got ~10 apps or so that run on PyPy. Initially memory tradeoff was definitely significant, somewhere around 40% or so -- it's going to vary across applications though certainly, and in a lot of cases I'm a bit happy our memory usage went up because it forces us more towards "nicer…

are you using any math packages like Numpy/Pandas or Opencv ?

Re: Let's Remove the Global Interpreter Lock

#82

Post by fijal »

sandGorgon wrote: post #70

This is a PERFECT usecase for Kickstarter. It makes me sad that this is a blog post that made it number 1 on HN with vast readership with open pursestrings.. yet there is not a campaign fundraising link. Use Kickstarter or Plasso to sell a pypy pro license - its so much easier for companies to pay invoices than to donate. If nothing else, I would pay for an official conda pypy package which works seamlessly with pand…

Maybe selling packaging is a good idea.... that said, kickstarter does not work in any jurisdiction we can use.

Re: Let's Remove the Global Interpreter Lock

#83

Post by fijal »

ryanjl wrote: post #80
seunosewa wrote: post #53

The ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.

Jython and Iron Python lack the GIL. It's just an implementation detail of the underlying VM. There's nothing in the language itself which requires a GIL.

C API... You can argue it's not a part of the language, but PyPy was forced to support it at the end

Re: Let's Remove the Global Interpreter Lock

#84

Post by chrisseaton »

Beltiras wrote: post #65
chrisseaton wrote: post #48

Earlier quoted context omitted.

The version that was originally activated. I think that's the case in every single parallel implementation of a programming language ever. I can't imagine it working any other way. When you redefine a method in any language I'm aware of you just change which method the name points to. You don't modify the original method.

So a function: def fun(*args): if not args: return 0 return fun(*(args[1:])) would be call-by-address after the first invocation? It could be lookup-by-name by way of code.

The naive implementation, and the semantic model, is always lookup-by-name on every invocation.

In practice we apply speculative optimisations including inline caching and guard removal with remote dynamic deoptimisation via safe points to make it a direct call instead.

Re: Let's Remove the Global Interpreter Lock

#85

Post by smitherfield »

jrs95 wrote: post #59
seunosewa wrote: post #53

The ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.

Perl 6 doesn't have a GIL, and already has a sane concurrency model, but the lack of libraries and community interest seems to make that pretty much a non-starter.

It’s also still dog-slow for the (Perl 5 / scripting-language) common case, which makes whatever theoretical performance improvements to its semantics a bit academic at this point: https://news.ycombinator.com/item?id=15004977

Re: Let's Remove the Global Interpreter Lock

#86

Post by dom0 »

ryanjl wrote: post #80
seunosewa wrote: post #53

The ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.

Jython and Iron Python lack the GIL. It's just an implementation detail of the underlying VM. There's nothing in the language itself which requires a GIL.

> It's just an implementation detail of the underlying VM.

Python itself is just an implementation detail of the underlying VM.

Re: Let's Remove the Global Interpreter Lock

#87

Post by dTal »

rburhum wrote: post #68
seunosewa wrote: post #53

The ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.

So basically recreate an entire language and library ecosystem because there is one feature that is less than ideal? I hope you realize why a better approach may be to reengineer that one component...

Python has many less-than-ideal features. Do you think we finally got it right, that we will use Python forever, and that the library work of the past decade or so is irreplaceable?

"Is it possible that software is not like anything else, that it is meant to be discarded: that the whole point is to see it as a soap bubble?" -- Alan Perlis

Re: Let's Remove the Global Interpreter Lock

#88

Post by robotresearcher »

monkmartinez wrote: post #50
devwastaken wrote: post #13

This would be great if it means we can run the C portions of Python in threads without performance hits. I recently started a little project that is a cross-platform GUI for batch bzip2 compression, and Python did it quite well with its built-in bzip2 module. But, once I tried to do it parallel, the performance impacts of GIL were obvious. Yes, you can work around that with multi-process, but I'd rather not be spammi…

> I'd rather not be spamming the running processes list and have to actually handle seperate processes that should be threads. I may be a bit naive asking this... but why would you care that much? Looking at activity monitor on my Mac, I count 14 Google Chrome Helper Process instances each spawning upwards of 13 threads. Adobe does something similar, as do several other programs/applications on my machine. Yet, my ma…

A process swap completely wipes the cache. Once swapped in, your process is not up to top speed for a while, until the working set has been copied into cache. You'd like to keep it that way for as long as possible. Best case scenario: one process per core.

Re: Let's Remove the Global Interpreter Lock

#89

Post by dmix »

sandGorgon wrote: post #70

This is a PERFECT usecase for Kickstarter. It makes me sad that this is a blog post that made it number 1 on HN with vast readership with open pursestrings.. yet there is not a campaign fundraising link. Use Kickstarter or Plasso to sell a pypy pro license - its so much easier for companies to pay invoices than to donate. If nothing else, I would pay for an official conda pypy package which works seamlessly with pand…

> yet there is not a campaign fundraising link.

Did you read the article? They said in the article they aren't asking for individual donations at the moment:

>> we would like to judge the interest of the community and the commercial partners to make it happen (we are not looking for individual donations at this point)

Plus I'm sure they will consider using Kickstarter when the time comes.

Re: Let's Remove the Global Interpreter Lock

#90

Post by taeric »

rguillebert wrote: post #79
andruby wrote: post #62

Earlier quoted context omitted.

How can they estimate this? What about all the libraries that might not be compatible with the solution PyPy comes up with? This feels like a number that might in the end blow up to 10x the original estimate.

It's not the PyPy developers' job to make every Python library threadsafe, people writing libraries will have to make their code threadsafe, like in every other language.

There is a clear difference here, though. Making a change that could lead to poorly written libraries now being broken is clearly the fault of the change. Userspace for these libraries is defined by how it is, not how it was intended.

(And really, was it intended to be dangerous in this way?)

Post reply on HN