241–250 of 326 posts
Just curious: if they solve it in Python, would it be possible to solve it in Ruby too ?
Ruby does not have that much dynamism compated to "everything is dict of pointers to dicts" Python.
Earlier quoted context omitted.
A lot of the heavy lifting is done through calls to C libraries anyhow, with Python just being a convenient way to pass the data around.
Indeed, and in that case the GIL is effectively a non-issue (there's no requirement for the GIL to be held by non-python code).
Earlier quoted context omitted.
Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload. "Multiprocessing" is…
>Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive Statically and dynamically loaded binaries are resident in the kernel's page cache. Which while each process will have different locations within its process address space for each process (b/c ALSR), they _should_ be de-duplicated in RAM, ultimately all these seperate in process images will be pointing at the same physical…
Earlier quoted context omitted.
I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.
The GIL is a legit pain when dealing with GUIs. When you're jumping between C/C++ code and Python code you don't care much about the GIL... until you have a GUI which needs to be kept responsive and needs the GIL to do so.
There seem to be a lot of naysayers in the comments about removing the GIL. Multiprocess parallelism isn't always appropriate, so I find this to be a very promising change that will definitely make me want to switch to PyPy. Here are the use cases I've found multiprocessing to be inappropriate: * High-contention parallel operations. Doing synchronization through a Manager (a separate IPC-based synchronizing broker pr…
> There seem to be a lot of naysayers in the comments about removing the GIL. That's because it's been attempted over and over and over again. And each time it ends up failing due to the decrease in single-threaded performance (the bevy of necessary memory mutexes aren't free)), and the extensive amount of work required to make all of the standard libraries threadsafe. I don't buy the $50,000 cost for a second. Sure,…
I feel like the GIL is, at this point, Python's most infamous attribute. For a long time I thought it was also the biggest flaw with Python...but over time I care less and less about it. I think the first thing to realize is that single-threaded performance is often significantly better with the GIL than without it. I think Larry Hasting's first Gilectomy talk was extremely insightful (about the GIL in general and ab…
The GIL has been a much bigger problem for perception than it ever has been for performance. Python has lost more mindshare over it than anything else. The few machine cycles that were ever saved by moving away from it were far outweighed by the waste of human cycles.
If Python would simply suck it up and eat the 20% performance hit, we could stop talking about the GIL and start optimizing code to get the 20% back.
Earlier quoted context omitted.
Well, that and Perl has to be one of the most unreadable languages out there ;)
Only if you don't know Perl... To me Python is more unreadable ;)
I used to carry around a Perl program of my own on a printout to take to VLSI interviews. That way when I got the "Do you know Perl?" question I could bring it out and force the interviewer into MY stupid subset of Perl rather than being stuck in his stupid subset of Perl.
That's not a compliment to the language.
Earlier quoted context omitted.
And yet, if you could have what you want, would it actually be faster? The costs of synchronizing mutable data between cores is surprisingly high. Any time your CPU thinks that the data that it has in its cache might not be what some other CPU has in its cache, the two have to coordinate what they are doing. And thanks to the fact that Python uses reference counting, data is constantly being changed even though you d…
Right, but like I said, I'd be fine with a read-only shared data structure. I have a problem that has a hefty data model. The problem can be decomposed and attacked in parallel, but the decomposition doesn't cut across the data. Right now I run n instances on n cores, but that means making n copies of a large data structure. This requires a lot of system memory, ruins any chance I have of not wrecking the cache (not…
If you need to share a large readonly structure, the best way IMO is that approach. Implement the structure in a low-level language that supports mmap (be very sure to make the whole structure be in the mmap'd block - it is easy to wind up with pointers to random other memory and you don't want that!) and have high performance accessors to use in your code.
Earlier quoted context omitted.
I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading. With 80% of the developers out there, they are basically assured of producing better, more stable code this way.
Your parent comment gives good advice, because the GIL is probably here to stay and so there's no use complaining about it. But the idea that multiprocessing gives better results than multithreading is ridiculous. In languages which don't have a GIL, threads are almost as capable as processes, but lighter weight. Threads are almost always preferable to processes in most languages. I understand why the GIL is still ar…
But also plagued with shared state concurrency bugs, something multi-processing completely avoids so...
> Threads are almost always preferable to processes in most languages.
No, they aren't. It's too easy to write buggy code with threads, it's a flawed model. Now it's certainly true that more people choose threads than processes but that's because they vastly overestimate their ability to write bug free lock based code. Processes are better.