Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's up, Python? The GIL removed, a new compiler, optparse deprecated

bitecode.dev

Post reply on HN
Print view View original

81–90 of 302 posts

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#81

Post by jjoonathan »

RayVR wrote: post #71
nwallin wrote: post #53

Earlier quoted context omitted.

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

Multiprocessing is not a real solution, it’s a break-glass procedure when you just need to throw some cores at something without any hope for reliability. Unless something has changed since I used python, it is essentially a wrapper on Fork. This means you need to deal with stuck/dead processes. I’ve used multiprocessing extensively and once you hit a certain amount of usage, even in a pool, you just get hangs and un…

Yep, multiprocessing is a cope.

If processes were a universal substitute for threads we wouldn't have threads. That reasoning only gets stronger when you apply python's heavy limitations, but it gets the most strength when you experience the awkwardness of multiprocessing firsthand.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#82

Post by ActorNightly »

devwastaken wrote: post #13

This is exactly what I have been looking forward to. Allow me to do no-gil, let me the developer make that choice. There are issues with that certainly, but I am conscious of this fact and given an analysis of no-gil benefits it is significantly more beneficial to have no-gil for certain use cases. One of the most significant of these cases to me is threading outside an Operating system context. What if I want to use…

> let me the developer make that choice.

The final push towards making no-GIL as the only option is the big issue here. An optional no-GIL is ok (although a waste of time), making it default is bad.

>What if I want to use both of the cores to a Cortex M0

The solution to anything performant in Python is writing the C extension, just like Numpy did. Python isn't meant to be a performant language. The GIL allows you to write code without thinking about complexities of parallelism.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#83

Post by dpedu »

oivey wrote: post #45
fbdab103 wrote: post #38

Earlier quoted context omitted.

It was correctly written to the invariant promises of the platform at the time. If Python is altering the deal, that does not suddenly make the library writer at fault.

There seems to be some confusion here. The GIL is not an invariant of Python that makes your code thread safe. Python is not altering the deal. You can still use threads to write concurrent code in Python today, and you'll still run into all of the classic concurrency related bugs. People just mostly don't bother writing threaded code in Python today because it provides no performance benefit. That may change, and it…

You're right, the GIL certainly doesn't automatically make all code thread-safe. However, it does make some operations that would normally be problematic in a multithreaded context thread-safe. Such as: appending to a list, updating a dict, modifying object attributes, and others.

https://docs.python.org/3/faq/library.html#what-kinds-of-glo...

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#84

Post by slt2021 »

RayVR wrote: post #71
nwallin wrote: post #53

Earlier quoted context omitted.

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

Multiprocessing is not a real solution, it’s a break-glass procedure when you just need to throw some cores at something without any hope for reliability. Unless something has changed since I used python, it is essentially a wrapper on Fork. This means you need to deal with stuck/dead processes. I’ve used multiprocessing extensively and once you hit a certain amount of usage, even in a pool, you just get hangs and un…

multiprocessing is very good solution for scatter-and-gather (or map/reduce) type workloads: for example ssh to 1000 machines, run some commands, grab output, analyze output, done some action based on output, etc

if you are managing a fleet of machines and have some tasks to do on each machine, then multiprocessing is the life saver.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#85

Post by whywhywhydude »

nwallin wrote: post #53
wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

If you have a non trivial application, multiprocessing just takes a lot of memory. Every child process that you create duplicates the parent memory. There are some interesting hacks like gc.freeze that exploits the copy on write feature of forks to reduce memory, but ultimately you can just create a few hundred of processes compared to thousands of threads because of memory consumption.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#86

Post by coldtea »

nwallin wrote: post #53
wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

>Use the python multiprocessing module. If you've already written it with the multithreading module, it is a drop in replacement. Your data structure will live in shared memory

Only if it can be immutable. So it can't be shared and changed by multiple processes as needed (with synchronization).

And even if you can have it mostly immutable, if you need to refresh it (e.g. after some time read a newer large file from disk to load into your data structure), you can't without restarting the whole server and processes.

So, it could work for this case, but it's hardly a general solution for the problem.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#87

Post by coldtea »

whywhywhydude wrote: post #85
nwallin wrote: post #53

Earlier quoted context omitted.

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

If you have a non trivial application, multiprocessing just takes a lot of memory. Every child process that you create duplicates the parent memory. There are some interesting hacks like gc.freeze that exploits the copy on write feature of forks to reduce memory, but ultimately you can just create a few hundred of processes compared to thousands of threads because of memory consumption.

>If you have a non trivial application, multiprocessing just takes a lot of memory. Every child process that you create duplicates the parent memory.

Not really, unless you want to alter it. The OS uses copy on write behind the scenes for forked processes, so will use the same memory locations already loaded until/if you modify that. So parent memory isn't really duplicated.

As for any new memory allocated by each child process, that's its own.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#88

Post by sgeisenh »

coldtea wrote: post #87
whywhywhydude wrote: post #85

Earlier quoted context omitted.

If you have a non trivial application, multiprocessing just takes a lot of memory. Every child process that you create duplicates the parent memory. There are some interesting hacks like gc.freeze that exploits the copy on write feature of forks to reduce memory, but ultimately you can just create a few hundred of processes compared to thousands of threads because of memory consumption.

> If you have a non trivial application, multiprocessing just takes a lot of memory. Every child process that you create duplicates the parent memory. Not really, unless you want to alter it. The OS uses copy on write behind the scenes for forked processes, so will use the same memory locations already loaded until/if you modify that. So parent memory isn't really duplicated. As for any new memory allocated by each c…

Unfortunately the generational GC modifies bits all over the heap, so you have to use some tricks to really leverage copy on write (as the commenter alludes to).

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#89

Post by codedokode »

> tools like pip-run already support running a script for which you have the deps described with such comments

> Packages are installed in a temporary virtual env and deleted after the run, like npx used to do for the JS world.

Is it efficient? Download packages, install them only to delete several seconds later. Wastes precious SSD cells.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#90

Post by dathinab »

ggm wrote: post #58
dathinab wrote: post #37

Earlier quoted context omitted.

> You have a single big data structure that can't be shared easily between multiple processes. Can't you use multiprocessing with that? Maybe mapping the data structure to a file and mmapping that in multiple processes? Maybe wrapping the whole thing in database instead of just using one huge nested dictionary? ton of additional complexity, not worth it for many use-cases and anything on the line of "using multiple p…

> Basically the moment "use mmap" or "use multi-processing" is a reasonable recommendation for something ad-hocish there is something rally wrong with the tools you use IMHO. Hmm. So you're saying only languages which bury lock and mutex over shared data are appropriate to use for async parallelism over shared data? Because calling explicit lock() and releae() isn't that hard. However it does incur a function call ov…

no I never said that
Post reply on HN