Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's up, Python? The GIL removed, a new compiler, optparse deprecated

bitecode.dev

Post reply on HN
Print view View original

221–230 of 302 posts

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#221

Post by empthought »

wiseowise wrote: post #175
empthought wrote: post #152

Earlier quoted context omitted.

Your trouble seems to involve not understanding how to set up signal handlers, which ProcessPoolExecutor handles for you and exposes via a BrokenProcessPool exception.

> Derived from BrokenExecutor (formerly RuntimeError), this exception class is raised when one of the workers of a ProcessPoolExecutor has terminated in a non-clean fashion (for example, if it was killed from the outside). What if it hangs?

That isn’t the scenario originally described, but there is a timeout parameter in future.result().

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#222

Post by quietbritishjim »

crabbone wrote: post #215
doctoboggan wrote: post #31

Earlier quoted context omitted.

The big and obvious difference is that all the GIL vs no-GIL stuff happens in the background and your average python dev can just ignore it if they want to. The interpreter will note if you have C extensions that don't opt in to no-GIL and then will give you the GIL version. This is _very_ different to the 2-to-3 transition where absolutely every single person, even those who couldn't care less, had to change their c…

> your average python dev can just ignore it if they want to. Oh, so naive... All the mutation code in Python which "worked" because Python didn't really have any real concurrency. Add to it -- there's no real plan about what to do with Python concurrency. Removing GIL is only one "half" of the problem, you need to give developers some sort of a framework to use to deal with concurrency. Python's threads are extremel…

> So, all synchronization requires dealing with locks, mutexes, condition variables...

As always, by far the best way to interact between threads is to use thread-safe queues (AKA message passing). Luckily, Python has one of those [1]. No complicated synchronisation needed.

[1] https://docs.python.org/3/library/queue.html

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#223

Post by silon42 »

nickcw wrote: post #74
cvnmalk wrote: post #28

Which, except for optparse, was all on the front page yesterday. So optparse is deprecated. More work I guess apart from auditing extensions for threading. Life is great in the Python treadmill.

I've tried to love argparse but it is so complicated. I always have to read the docs each time I use it. getopt has its own brutal simplicity.

Just last week I had to parse arguments again after a long time in a script of mine.

After some googling, getopt was chosen due to it's simplicity.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#224

Post by wanderingmind »

antupis wrote: post #176
BiteCode_dev wrote: post #169

Earlier quoted context omitted.

It's the eternal pendulum: - take no risk, and people will blame the project for being static. - take risks, and people will blame the project for being reckless. E.G: - don't adopt a new feature, and your language is old, becoming irrelevant, and a wave of comments will tell you how they just can't use it for X because they don't have it. - break compat, and you will have a horde stating you don't care about users t…

World would need one more language which would have very barebone core something like very minimal go or python but strong metaprogramming features so you could expand language if you need.

You are thinking mojo [1] that's claims full python compatibility but can be extended for static typing and high performance scenarios

[1] https://www.modular.com/mojo

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#225

Post by School-Cotton »

tsimionescu wrote: post #166
School-Cotton wrote: post #92

Earlier quoted context omitted.

> Why do you believe an obscure runtime concurrency detail It is not obscure. It will make it much more difficult to write native-code extensions which is IMO the whole point of Python.

The point of Python in your opinion is to write non-Python code?

Yes, like bash, Python is a language that exists mainly as glue for code written in other languages. Do you think we need to add multithreading to bash?

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#226

Post by School-Cotton »

BiteCode_dev wrote: post #198
schemescape wrote: post #51

The title says "GIL removed", but the article says "This means in the coming years, Python will have its GIL removed." I'm assuming the article is correct and the GIL has not been removed yet (but there is a plan to remove it in the future). If that's not the case, please correct me!

Yes. I tried to come up with something that would convey in a few words that the GIL was going to be removed for sure this time. But as a Frenchmen, I couldn't find better. "GIL will be removed" was the closest, but it's very long, and it sounds like all those times we had the promise it would be, but it never did. So the Prophetic perfect tense is the best compromise: it asserts near certainty, it's short, and worst…

> the Prophetic perfect tense

That is not really a thing in normal English. I had to look up what it even means, and it apparently exists only in the translation of a few passages of Biblical Hebrew (and now, apparently, the title of your post).

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#227

Post by mathisfun123 »

wanderingmind wrote: post #224
antupis wrote: post #176

Earlier quoted context omitted.

World would need one more language which would have very barebone core something like very minimal go or python but strong metaprogramming features so you could expand language if you need.

You are thinking mojo [1] that's claims full python compatibility but can be extended for static typing and high performance scenarios [1] https://www.modular.com/mojo

How many lines of mojo have you written?

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#228

Post by xmaayy »

iknownothow wrote: post #171
wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

I would consider the following optimizations first before attempting to rewrite an HTTP API since you already did the hard part: 1. For multiples processes use `gunicorn` [1]. Runs your app across multiple processes without you having to touch your code much. It's the same as having the n instances of the same backend app where n being the number of CPU cores you're willing to throw at it. One backend process per cor…

> 1. For multiples processes use `gunicorn`

This will load up multiple processes like you say. OP loads a large dataset and gUnicorn would copy that dataset in each process. I have never figured out shared memory with gUnicorn.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#229

Post by akvadrako »

jjoonathan wrote: post #81
RayVR wrote: post #71

Earlier quoted context omitted.

Multiprocessing is not a real solution, it’s a break-glass procedure when you just need to throw some cores at something without any hope for reliability. Unless something has changed since I used python, it is essentially a wrapper on Fork. This means you need to deal with stuck/dead processes. I’ve used multiprocessing extensively and once you hit a certain amount of usage, even in a pool, you just get hangs and un…

Yep, multiprocessing is a cope. If processes were a universal substitute for threads we wouldn't have threads. That reasoning only gets stronger when you apply python's heavy limitations, but it gets the most strength when you experience the awkwardness of multiprocessing firsthand.

There isn't much difference on Linux between threads and processes that share memory. Multiprocessing is fine, it's just slightly more isolated threads.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#230

Post by aragilar »

mort96 wrote: post #194
nwallin wrote: post #53

Earlier quoted context omitted.

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

I want to warn people against multiprocessing in python though. If you're thinking about parallelizing your Python process, chances are your Python code is CPU-bound. That's when you should stop and think, is Python really the right tool for this job? From experience, translating a Python program into C++ or Rust often gives a speed-up of around 100x, without introducing threads. Go probably has a similar level of sp…

The first thing you should do is profile the code (py-spy is my preferred option) and see if there are any obvious hotspots. Then I'd actually look at the code, and understand what the structure is. For example, are you making lots of unnecessary copies of data? Are you recomputing something expensive you can store (functools.cache is one line and can make things much faster at the cost of memory)?

Once you've done that, then you should be familiar enough the code to know which bits are worth using multiprocessing on (i.e. the large embarrassingly parallel bits), which if they are a significant part of your code should scale near linearly.

The other thing to check is which libraries are you using (and what are your dependencies using). numpy now includes openblas (though mkl may be faster for your usecase), but sometimes you can achieve large speedups though choosing a different library, or ensuring speedups are being built.

Post reply on HN