What's up, Python? The GIL removed, a new compiler, optparse deprecated • Hacker News Board

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#241

Post by LexiMax » Mon, Jul 31, 2023, 12:02 PM UTC

acdha wrote: post #112

charrondev wrote: post #111

Earlier quoted context omitted.

I’m not familiar enough with the python transition to say much. I can think of a few things that the PHP developers did that helped make the transition easier: - multibyte aware string functions were implemented as a separate (and optional) extension with separately named functions (prefixed with mb) and there was a popular community polyfill from the Symfony project (and is for many new language functions). - Weird…

> multibyte aware string functions were implemented as a separate (and optional) extension with separately named functions (prefixed with mb) Python had a different take on this with some interesting psychology: you had a new string type which had to explicitly be converted (i.e. concatenating a Unicode string with a byte string causes an exception), which had a stark divide. Projects which had previously handled Uni…

There were valid reasons to be upset at Python 3's handling of Unicode.

- https://lucumr.pocoo.org/2014/5/12/everything-about-unicode/

- Discussion: https://news.ycombinator.com/item?id=7732572

- https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journ...

- Discussion: https://news.ycombinator.com/item?id=22036773

Chalking these complaints up to bad development practices is _precisely_ the reason why the Python 3 migration was handled so poorly. If this attitude is repeated for no-GIL Python, it will fail.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#242

Post by gjulianm » Mon, Jul 31, 2023, 12:04 PM UTC

zarzavat wrote: post #131

Jtsummers wrote: post #10

Earlier quoted context omitted.

That discussion was amusing. Removing the GIL opens up the possibility of actually getting a real performance benefit from multithreaded Python code. That's the value. Given every modern desktop and server is multicore (and increasingly getting to tens of cores if not hundreds), multithreading in Python unhampered by the GIL will be a useful thing. And no, multiprocessing is not a good alternative to multithreading.…

Python is not a language for writing fast code. Python is a relaxed language for things that don’t have to be fast. If you need something to be fast you are supposed to use a C extension and control it with Python - that’s been the dogma for as long as I can remember to avoid exactly this kind of pathological race to performance in a language that was never designed for it. By using Python you are already leaving a t…

It's always a tradeoff, but I'm surprised to see so many people say that just because Python isn't fast it shouldn't get multithreading.

Yes, by using Python we leave a lot of performance on the table. We also get a lot of dev performance, just because of the amount of libraries available, dynamic language features, quick development.. So it's not always a clear decision between Python or a compiled language.

> If you need something to be fast you are supposed to use a C extension and control it with Python

So what do we do with the case where we need to control the C extension from multiple threads? Because that's currently my problem. The C extension we developed do release the GIL, but because the Python code that does the calls to the extension can't be really multithreaded, the performance gain we get is minimal.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#243

Post by danpalmer » Mon, Jul 31, 2023, 12:04 PM UTC

iknownothow wrote: post #171

wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

I would consider the following optimizations first before attempting to rewrite an HTTP API since you already did the hard part: 1. For multiples processes use `gunicorn` [1]. Runs your app across multiple processes without you having to touch your code much. It's the same as having the n instances of the same backend app where n being the number of CPU cores you're willing to throw at it. One backend process per cor…

These don't really apply to the parent commenter's scenario.

1) gunicorn or any solution with multiple processes is going to just multiply the RAM usage. Using 10-100GB of RAM per effective thread makes this sort of problem very RAM bound, to the point that it can be hard to find hardware or VM support.

2) This isn't I/O bound.

3) If your service is fundamentally just looking up data in a huge in-memory data store, adding LRU caching around that is unlikely to make much of a difference because you're a) still doing a lookup in memory, just for the cache rather than the real data, and b) you're still subject to the GIL for those cache lookups.

I've also written services like this, we only loaded ~5GB of data, but it was sufficient to be difficult to manage in a few ways like this. The GIL-ectomy will probably have a significant impact on these sorts of use cases.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#244

Post by severino » Mon, Jul 31, 2023, 12:13 PM UTC

wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

May I ask why you didn't consider writing that quick implementation in Java in the first place?

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#245

Post by zbentley » Mon, Jul 31, 2023, 12:13 PM UTC

xmaayy wrote: post #228

iknownothow wrote: post #171

Earlier quoted context omitted.

I would consider the following optimizations first before attempting to rewrite an HTTP API since you already did the hard part: 1. For multiples processes use `gunicorn` [1]. Runs your app across multiple processes without you having to touch your code much. It's the same as having the n instances of the same backend app where n being the number of CPU cores you're willing to throw at it. One backend process per cor…

> 1. For multiples processes use `gunicorn` This will load up multiple processes like you say. OP loads a large dataset and gUnicorn would copy that dataset in each process. I have never figured out shared memory with gUnicorn.

> gUnicorn would copy that dataset in each process

Assuming you're on Linux/BSD/MacOS, sharing read-only memory is easy with Gunicorn (as opposed to actual POSIX shared memory, for which there are multiprocessing wrappers, but they're much harder to use).

To share memory in copy-on-write mode, add a call to load your dataset into something global (i.e. a global or class variable or an lru_cache of a free/class/static method) in gunicorn's "when_ready" config function[1].

This will load your dataset once on server start, before any processes are forked. After processes are forked, they'll gain access to that dataset in copy-on-write mode (this behavior is not specific to python/gunicorn; rather, it's a core behavior of fork(2)). If those processes do need to mutate the dataset, they'll only mutate their copy-on-write copies of it, so their mutations won't be visible to other parallel Gunicorn workers. In other words, if one request in a parallel=2 gunicorn mutates the dataset, a subsequent request has only a 50% likelihood of observing that mutation.

If you do need mutable shared memory, you could either check out databases/caches as other commenters have mentioned (Redislite[2] is a good way to embed Redis as a per-application cache into Python without having to run or configure a separate server at all; you can launch it in gunicorn's "when_ready" as well), or try true shared memory[3][4]

1. https://docs.gunicorn.org/en/stable/settings.html#when-ready 2. https://pypi.org/project/redislite/ 3. https://docs.python.org/3/library/multiprocessing.html#share... 4. https://docs.python.org/3/library/multiprocessing.shared_mem...

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#246

Post by woodrowbarlow » Mon, Jul 31, 2023, 12:16 PM UTC

gjulianm wrote: post #236

crabbone wrote: post #215

Earlier quoted context omitted.

> your average python dev can just ignore it if they want to. Oh, so naive... All the mutation code in Python which "worked" because Python didn't really have any real concurrency. Add to it -- there's no real plan about what to do with Python concurrency. Removing GIL is only one "half" of the problem, you need to give developers some sort of a framework to use to deal with concurrency. Python's threads are extremel…

Which code is automatically going to run in threads? As you say, basically nobody uses Python threads. So even enabling no-gil, nothing is going to change because sequential code will still be sequential.

any existing async/await code.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#247

Post by zbentley » Mon, Jul 31, 2023, 12:34 PM UTC

the8472 wrote: post #204

nwallin wrote: post #53

Earlier quoted context omitted.

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

Loading 100GB into RAM and then calling fork() is just painting a giant OOM Killer target on your back. It'll work until something breaks the CoWs or the parent gets restarted while some forks still linger or other fun things like that. Threads make it transparent to the OS that this memory really must be shared between compute tasks.

While that does sometimes happen, I find the risk to be overstated. Most simple "allocate a large, complex data structure (e.g. dict of vectors of dataclasses) before creating a multiprocessing.Pool/Process/concurrent.futures.ProcessPoolExecutor and then refer to parts of it in the executor's jobs" work that deals in GBs of data does not suffer from copy-on-write-induced OOM issues in my experience. If the data in the shared memory isn't mutated in python, the refcount mutations are rarely enough to dirty more than a fraction of a percent of pages (though there are pathological allocation/reference schemes where that's not true).

If you do have memory issues, calling 'gc.freeze()' right before creating your multiprocessing.Pool/Process/concurrent.futures.ProcessPoolExecutor is sufficient to mitigate refcount-related page dirtying in the vast majority of cases. In the small remaining minority of cases, 'gc.disable()' as suggested by the freeze docs[1] may help. If that still doesn't do it, or if your page-dirtying is due to actual mutations of data (not just refcounts), it may be time to reach for actual shared memory instead[2][3].

1. https://docs.python.org/3/library/gc.html#gc.freeze 2. https://docs.python.org/3/library/multiprocessing.html#share... 3. https://docs.python.org/3/library/multiprocessing.shared_mem...

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#248

Post by ptx » Mon, Jul 31, 2023, 12:38 PM UTC

crabbone wrote: post #213

Every release brings Python closer to becoming Java. Another ten or so years, and we'll have feature parity with Java 8 or something. Maybe it will even be as fast!

And conversely every release of Java brings it closer to becoming Python.

With Java 4 we got regular expressions.

With Java 5 we got varargs, string formatting, boxed numbers, syntax for looping over collections and imports of static methods.

With Java 7 we got Timsort, the sorting algorithm from Python.

With Java 8 we got first-class functions.

With Java 9 we got a REPL.

With Java 11 we got implicit compilation of source files so you can run them directly.

In more recent releases, we have previews of features corresponding to f-strings and ctypes.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#249

Post by __d » Mon, Jul 31, 2023, 12:43 PM UTC

pjmlp wrote: post #210

tsimionescu wrote: post #163

Earlier quoted context omitted.

To be fair, if you use CL in a similarly dynamic way as Python (don't compile anything, don't add any declarations etc) it won't be that much faster. You'll get some boost out of the stdlib stuff being compiled already, but otherwise it will incur similar performance penalties.

We can add Smalltalk, SELF, Dylan, JavaScript into the discussion then.

And maybe Strongtalk

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#250

Post by zbentley » Mon, Jul 31, 2023, 12:47 PM UTC

nine_k wrote: post #150

nwallin wrote: post #53

Earlier quoted context omitted.

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

Multiprocessing is great. But then every process keeps its own copy of hundreds of gigabytes of stuff. May be okay, depending on how many processes you spawn. If the bulk of the data is immutable (or at least never mutated), it can be safely shared though, via shared memory.

> every process keeps its own copy of hundreds of gigabytes of stuff. May be okay, depending on how many processes you spawn

That depends on how you're using multiprocessing. If you're using the "spawn" multiprocessing-start method (which was set to the default on MacOS a few years ago[1], unfortunately), then every process re-starts python from the beginning of your program and does indeed have its own copy of anything not explicitly shared.

However, the "fork" and "forkserver" start methods make everything available in python before your multiprocessing.Pool/Process/concurrent.futures.ProcessPoolExecutor was created accessible for "free" (really: via fork(2)'s copy-on-write semantics) in the child processes without any added memory overhead. "fork" is the default startup mode on everything other than MacOS/Windows[2].

I find that those differing defaults are responsible for a lot of FUD around memory management regarding multiprocessing (some of which can be found in these comments!); folks who are watching memory while using multiprocessing on MacOS or Windows observe massively different memory consumption behavior than folks on Linux/BSD (which includes folks validating in Docker on MacOS/Windows). There's an additional source of FUD among folks who used Python on MacOS before the default was changed from "fork" to "spawn" and who assume the prior behavior still exists when it does not.

This sometimes results in the humorously counterintuitive situation of someone testing some Python code in Docker on MacOS/Windows observing far better performance inside Docker (and its accompanying virtual machine) than they observe when running that same code natively directly on the host operating system.

If you're on MacOS (not Windows) and wish to use the "fork" or "forkserver" behaviors of multiprocessing for memory sharing, do "export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES" in your shell before starting Python (modifying os.environ or calling os.setenv() in Python will not work), and then call "multiprocessing.set_start_method("fork", force=True)" in your entry point. Per the linked GitHub issue below, this can occasionally cause issues, but in my experience it does so rarely if ever.

1. https://github.com/python/cpython/issues/77906

2. https://docs.python.org/3/library/multiprocessing.html#conte...