Earlier quoted context omitted.
They are not threaded at all.
Correct. Async stuff in Python is based on libuv like event loops similar to how Nodejs and others operate, not full threads.
Yeah, they have their own asyncio python runner
Python 3.13.0 Is Released
111–120 of 135 posts
Re: Python 3.13.0 Is Released
#112Good to get advanced notice, if I read all the way down, that they will silently completely change the behavior of multiprocessing in 3.14 (only on Unix/Linux, in case other people wonder what’s going on), which is going to break a bunch of programs I work with. I really like using Python, but I can’t keep using it when they just keep breaking things like this. Most people don’t read all the release notes.
> I really like using Python, but I can’t keep using it when they just keep breaking things like this. So much perl clutching. Just curious, since I guess you've made up your mind, what's your plan to migrate away? Or are you hoping maintainers see your comment and reconsider the road-map?
Rust. I've already re-written several of my decent sized research programs to Rust, and plan to finish converting what's left soon.
Rust isn't perfect (no language is), but they do seem to try much harder to not break backwards compatability.
Re: Python 3.13.0 Is Released
#113Earlier quoted context omitted.
I think should have a dig. While it’s not perfect, I know a few other people people who do “set up lots of data structures, including in libraries, then make use of the fact multiprocessing uses fork to duplicate them”. While fork always has sharp edges, it’s also long been clearly documented that’s the behavior on Linux.
I'm pretty sure that significantly more people were burned by fork being the default with no actual benefit to their code, whether because of the deadlocks etc that it triggers in multithreaded non-fork-aware code, or because their code wouldn't work correctly on other platform. Keeping it there as an option that one can explicitly enable for those few cases where it's actually useful and with full understanding of c…
I agree that fork was an awful default.
However, changing the default silently just means people's code is going to change behaviour between versions, or silently break if someone with an older version runs their code. At this point, it's probably better to just require people give an explicit choice (they can even make one of the choice names be 'default' or something, to make life easy for people who don't really care).
Re: Python 3.13.0 Is Released
#114Good to get advanced notice, if I read all the way down, that they will silently completely change the behavior of multiprocessing in 3.14 (only on Unix/Linux, in case other people wonder what’s going on), which is going to break a bunch of programs I work with. I really like using Python, but I can’t keep using it when they just keep breaking things like this. Most people don’t read all the release notes.
You are complaining about spawn()? both fork() and spawn() are just wrappers around clone() on most libc types anyway. spawn() was introduced to POSIX in the last century to address some of the problems with fork() especially related to multi threading, so I an curious how your code is so dependent on UTM, yet multi threading.
My code isn't dependant on multi-threading at all.
It use fork in Python multiprocess, because many packages can't be "pickled" (the standard way of copying data structures between processes), so instead my code looks like:
* Set up big complicated data-structures.
* Use fork to make a bunch of copies of my running program, and all my datastructures
* Use multiprocessing to make all those python programs talk to each other and share work, thereby using all my CPU cores.
Re: Python 3.13.0 Is Released
#115Earlier quoted context omitted.
For those interested in the REPL improvements: " Python now uses a new interactive shell by default, based on code from the PyPy project. When the user starts the REPL from an interactive terminal, the following new features are now supported: Multiline editing with history preservation. Direct support for REPL-specific commands like help, exit, and quit, without the need to call them as functions. Prompts and traceb…
No vi editing mode :-( > The new REPL will not be implementing inputrc support, and consequently there won't be a vi editing mode. https://github.com/python/cpython/issues/118840#issuecomment...
iPython is a much better repl anyway, use that, it has a vi mode
Re: Python 3.13.0 Is Released
#116Python versions 3.11, 3.12 and now 3.13 have contained far fewer additions to the language than earlier 3.x versions. Instead the newest releases have been focusing on implementation improvements - and in 3.13, the new REPL, experimental JIT & GIL-free options all sound great! The language itself is (more than) complex enough already - I hope this focus on implementation quality continues.
Removing GIL only increases complexity.
“Make things as simple as possible, but no simpler”, I for one am glad they’ll be letting us use modern CPUs much more easily instead of it being designed around 1998 cpus
Re: Python 3.13.0 Is Released
#117Earlier quoted context omitted.
It definitely does, but don't you think that it could be worth it if it makes multithreading usable for CPU-heavy tasks?
No. Python is orders of magnitude slower than even C# or Java. It’s doing hash table lookups per variable access. I would write a separate program to do the number crunching. Everyone must now pay the mental cost of multithreading for the chance that you might want to optimize something.
If you’re in a primarily python coding house, your argument won’t mean anything when you bring up you’ll have to rewrite millions of lines of code in C# or Java, you might as well ask them to liquidate the company and start fresh.
Re: Python 3.13.0 Is Released
#118Any rule of thumb when it comes to adopting Python releases? Is it usually best to wait for the first patch version before using in production?
n-1 is the rule I follow. So if asked today I'd look at 3.12.
So simple, yet so effective.
Old Systems Admins like me have been following this simple rule for decades. It's the easiest way at scale.
Re: Python 3.13.0 Is Released
#119Earlier quoted context omitted.
You are complaining about spawn()? both fork() and spawn() are just wrappers around clone() on most libc types anyway. spawn() was introduced to POSIX in the last century to address some of the problems with fork() especially related to multi threading, so I an curious how your code is so dependent on UTM, yet multi threading.
My code isn't dependant on multi-threading at all. It use fork in Python multiprocess, because many packages can't be "pickled" (the standard way of copying data structures between processes), so instead my code looks like: * Set up big complicated data-structures. * Use fork to make a bunch of copies of my running program, and all my datastructures * Use multiprocessing to make all those python programs talk to each…
'Threading' is an overload term. And while I didn't know, I was wondering if at the library level, the fact that posix_spawn() pauses the parent, while fork() doesn't, that is what you were leveraging.
The python multiprocessing module has been problematic for a while, as the platform abstractions are leaky and to be honest the POSIX version of spawn() was poorly implemented and mostly copied the limits of Windows.
I am sure that some of the recent deadlocks are due to this pull request as an example that calls out how risky this is.
https://github.com/python/cpython/pull/114279
Personally knowing the pain of fork() in the way you are using it, I have moved on.
But I would strongly encourage you to look into how clone() and the CLONE_VM and CLONE_VFORK options interact, document your use case and file an actionable issue against the multiprocessing module.
Go moved away from fork in 1.9 which may explain the issues with it better than the previous linked python discussion.
But looking at the git blame, all the 'fixes' have been about people trading known problems and focusing on the happy path.
My reply was intended for someone to address that tech debt and move forward with an intentional designed refactoring.
As I just focus on modern Linux, I avoid the internal submodule and just call clone() in a custom module or use python as glue to languages that have better concurrency.
Re: Python 3.13.0 Is Released
#120Earlier quoted context omitted.
What type checker do you recommend?
Beartype is incredible. It is soo fast I put it as decorator on all functions of all my projects. It's day and night compared to typeguard. Also the dev is... Completely out of this world
I don't get the point of a runtime type checker.
It adds a lot of noise with those decorators everywhere and you need to call each section of the code to get full coverage, meaning 100% test coverage.
At that point just use rust, or am I missing something?