[deleted]
What's up, Python? The GIL removed, a new compiler, optparse deprecated
251–260 of 302 posts
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#252Earlier quoted context omitted.
> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…
Multiprocessing is not a real solution, it’s a break-glass procedure when you just need to throw some cores at something without any hope for reliability. Unless something has changed since I used python, it is essentially a wrapper on Fork. This means you need to deal with stuck/dead processes. I’ve used multiprocessing extensively and once you hit a certain amount of usage, even in a pool, you just get hangs and un…
There is a "fork" mode and a "spawn" mode. Fork (the default) tends to result in broken process pools as you say, spawn seems to work a lot better but the performance is worse.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#253Earlier quoted context omitted.
> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…
You have to manually set up shared memory with its own API that has its own limitations, right? I thought some seamless integration was a new feature, but AFAICT, transfers between multiprocesses still leads to things being pickled and copied. Am I wrong?
> Am I wrong?
Only partially. When you send things to a multiprocessing.Pool/concurrent.futures.ProcessPoolExecutor, they're pickled and copied. "Sending" happens when passing arguments to e.g. "multiprocessing.Pool.apply_async()", "multiprocessing.Queue.put()" or "concurrent.futures.ProcessPoolExecutor.submit()".
However, there are two other ways to share data into your multiprocessing processes:
1. Copy-on-write via fork(2). In this mode, globally-visible data structures in Python that were created before your Pool/ProcessPoolExecutor are made accessible to code in child processes for (nearly) free, with no pickling, and no copying unless they are mutated in the child process. Two caveats here, which I've discussed in other comments on this thread: mutation may occur via garbage collection even if you don't explicitly change fork-shared data in Python[1]; and fork(2) is not used by default in multiprocessing on MacOS or Windows[2].
2. Using explicit shared memory data structures provided by Multiprocessing[3][4]. These do not incur the overhead (in CPU or copied memory) that pickle-based IPC does, but they are not without complexity or cost.
Unfortunately, truly "seamless integration" is not really possible with multiprocessing, so users will have to use one or more of the above strategies according to their application needs.
1. https://news.ycombinator.com/item?id=36940118 2. https://news.ycombinator.com/item?id=36941791 3. https://docs.python.org/3/library/multiprocessing.html#share... 4. https://docs.python.org/3/library/multiprocessing.shared_mem...
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#254Earlier quoted context omitted.
You have to be much more careful about what you modify when using multithreading, so I'm not sure what you mean by that. A lot of people here mention that sharing data is much easier with multithreading, but doing this without races is not easy. You can't just use the values from difference threads like you would in normal code, you need to synchronize access with locks, which can be difficult to do correctly and can…
Multithreading is hard but once you have been doing it a while, it becomes easy and most importantly, it’s stable. When you have to deal with processes, there’s a lot of external factors out of your control because processes are much more visible and carry a lot of extra baggage. Hard multithreading problems are fun. Hard multi-process problems are just tedious.
As I understand it on Linux processes and threads are implemented in almost the same way, just that threads share memory. I've heard it said several times that the idea that processes are "heavier" is a bit of a myth. I guess they need to allocate heap space and threads don't. I'm not an expert, just mentioning because it sounded like you might be believing something which is at odds with what people say about processes and threads on Linux.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#255Earlier quoted context omitted.
It's the eternal pendulum: - take no risk, and people will blame the project for being static. - take risks, and people will blame the project for being reckless. E.G: - don't adopt a new feature, and your language is old, becoming irrelevant, and a wave of comments will tell you how they just can't use it for X because they don't have it. - break compat, and you will have a horde stating you don't care about users t…
World would need one more language which would have very barebone core something like very minimal go or python but strong metaprogramming features so you could expand language if you need.
LISP has been around since the 60s
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#256Earlier quoted context omitted.
If you have a non trivial application, multiprocessing just takes a lot of memory. Every child process that you create duplicates the parent memory. There are some interesting hacks like gc.freeze that exploits the copy on write feature of forks to reduce memory, but ultimately you can just create a few hundred of processes compared to thousands of threads because of memory consumption.
> If you have a non trivial application, multiprocessing just takes a lot of memory. Every child process that you create duplicates the parent memory. Not really, unless you want to alter it. The OS uses copy on write behind the scenes for forked processes, so will use the same memory locations already loaded until/if you modify that. So parent memory isn't really duplicated. As for any new memory allocated by each c…
The situation is a bit more complicated than this. While it's usually not the case that child processes always duplicate parent memory, that does happen on certain platforms (MacOS and Windows) on some Pythons. Additionally, the situation regarding unexpected page dirtying of copy-on-write memory is nuanced as well, which some of the sibling comments allude to.
I'll copy the tl;dr from another comment I've made nearby:
There are three main ways to share data into your multiprocessing processes:
1. By sending that data to them with IPC/pickling/copying, e.g. via "multiprocessing.Pool.apply_async()", "multiprocessing.Queue.put()" or "concurrent.futures.ProcessPoolExecutor.submit()".
2. Copy-on-write via fork(2). In this mode, globally-visible data structures in Python that were created before your Pool/ProcessPoolExecutor are made accessible to code in child processes for (nearly) free, with no pickling, and no copying unless they are mutated in the child process. Two caveats here, which I've discussed in other comments on this thread: mutation may occur via garbage collection even if you don't explicitly change fork-shared data in Python[1]; and fork(2) is not used by default in multiprocessing on MacOS or Windows[2].
3. Using explicit shared memory data structures provided by Multiprocessing[3][4].
1. https://news.ycombinator.com/item?id=36940118 2. https://news.ycombinator.com/item?id=36941791 3. https://docs.python.org/3/library/multiprocessing.html#share... 4. https://docs.python.org/3/library/multiprocessing.shared_mem...
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#257Earlier quoted context omitted.
> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…
I dont really partake in programming "wars", but the idea of launching a set of separate processes instead of separate threads to do a bunch of IOs has always seem to be weird to me. Yes, I have built software using Python. Yes, I have done things as you suggest. Now I use asyncio, since the syntax has matured and I finally understand coroutines, runners, tasks etc. Lets see where the GIL less Python takes us.
I'm confused. If you're doing a "bunch of IOs" then that's the situation where people use threads in Python, not processes. The argument for processes in Python is CPU-bound workloads.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#258Earlier quoted context omitted.
Multiprocessing is great. But then every process keeps its own copy of hundreds of gigabytes of stuff. May be okay, depending on how many processes you spawn. If the bulk of the data is immutable (or at least never mutated), it can be safely shared though, via shared memory.
> every process keeps its own copy of hundreds of gigabytes of stuff. May be okay, depending on how many processes you spawn That depends on how you're using multiprocessing. If you're using the "spawn" multiprocessing-start method (which was set to the default on MacOS a few years ago[1], unfortunately), then every process re-starts python from the beginning of your program and does indeed have its own copy of anyth…
Is what you're describing only true of the "Framework" Python build on MacOS? It sounds like that's the case from a quick read of the issue you linked. I would say that people should basically never use the "Framework" Python on MacOS. (There's some insanity IIRC where matplotlib wants you to use the Framework build? But that's matplotlib)
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#259Earlier quoted context omitted.
Which code is automatically going to run in threads? As you say, basically nobody uses Python threads. So even enabling no-gil, nothing is going to change because sequential code will still be sequential.
any existing async/await code.
async/await code already runs in threads, so that's not really a change.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#260Earlier quoted context omitted.
I would consider the following optimizations first before attempting to rewrite an HTTP API since you already did the hard part: 1. For multiples processes use `gunicorn` [1]. Runs your app across multiple processes without you having to touch your code much. It's the same as having the n instances of the same backend app where n being the number of CPU cores you're willing to throw at it. One backend process per cor…
These don't really apply to the parent commenter's scenario. 1) gunicorn or any solution with multiple processes is going to just multiply the RAM usage. Using 10-100GB of RAM per effective thread makes this sort of problem very RAM bound, to the point that it can be hard to find hardware or VM support. 2) This isn't I/O bound. 3) If your service is fundamentally just looking up data in a huge in-memory data store, a…
For #1, would copy on write help? Or does python store the counters on the objects?