101–110 of 326 posts
The ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.
But give me some business reasons as to why removing the GIL is critical. Will is save me a ton of money? Will my stack magically just run faster?
I wonder if Google has already done so since they would benefit quite a bit from a GIL-less python.
Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…
To add to what everyone else said, if you need transactional semantics, its much simpler in multiple threads. With multiple processes (local or remote), you can't simply share an atomic data structure or a lock, you have to use a distributed lock or consensus algorithm, which are more complex and usually quite "chatty". If memory or network bandwidth are constrained, it may be especially desirable to eliminate this,…
Earlier quoted context omitted.
It's not the PyPy developers' job to make every Python library threadsafe, people writing libraries will have to make their code threadsafe, like in every other language.
There is a clear difference here, though. Making a change that could lead to poorly written libraries now being broken is clearly the fault of the change. Userspace for these libraries is defined by how it is, not how it was intended. (And really, was it intended to be dangerous in this way?)
Expecting bad code to magically work forever is unrealistic and hinders progress.
Earlier quoted context omitted.
"You can't just use functions defined in your tool, you need to create a faux-cli interface in order to run each parallel worker." the multiprocessing library allows you to launch multiple processes using your function definitions. It's almost the same as the multithreading library but does not share data. It seems the real problem, as you pointed out, is the additional memory. I didn't consider situations where each…
It gets more interesting when you have a large data set that's required for the computation, but as you compute, you may discover partial solutions that can be cached and used by other workers. So not only a large read-only data set, but also a read-write cache used by all workers. This sort of thing is relatively easy with threads, but basically impossible with multiprocessing.
Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is 1. Big enough to need concurrency 2. Not big enough to require multiple boxes. 3. Running in a situation that can not spare the resources for multiprocessing. 4. You want to share memory instead of designing your workflow to handle messages or workin…
There are many cases where the objects are too big to be passed around. Python is used a huge amount in Machine learning and datascience, where being able to do parallel work on stuff already in memory would be great.
Earlier quoted context omitted.
It's the same issue with Python. AFAIK there are a number of Python libraries that are not thread-safe, and the GIL prevents them from being an issue.
I thought the GIL was not held during execution of foreign code in python (at least that was one point given for why the GIL wasn't a big deal in practice).
This is a PERFECT usecase for Kickstarter. It makes me sad that this is a blog post that made it number 1 on HN with vast readership with open pursestrings.. yet there is not a campaign fundraising link. Use Kickstarter or Plasso to sell a pypy pro license - its so much easier for companies to pay invoices than to donate. If nothing else, I would pay for an official conda pypy package which works seamlessly with pand…
So I assume they're not doing a kickstarter to prevent the following from happening:
1. The internet at large will assume they're going to get a GILless PyPy that can actually run their code.
2. A separate PyPy is released that doesn't run their code.
3. People are angry that they didn't get what the thought they were gonna get, like what often happens with kickstarter backed projects.
4. With no coporate support and waning public interest due to the uselessness of a GILless PyPy, the separately released project becomes unmaintained.
Earlier quoted context omitted.
There are many cases where the objects are too big to be passed around. Python is used a huge amount in Machine learning and datascience, where being able to do parallel work on stuff already in memory would be great.
Can't this already be handled by calling out to a C/C++ or FORTRAN procedure that processes the data in multiple threads? For number crunching, Python is almost exclusively used as glue.
Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.