Sometimes there is no bandaid big enough to cover up a fundamental design decision in a language. Stick to solving problems it was designed for instead of crapping it up. Python isn't the only language , unless it's the only language you know.
What's up, Python? The GIL removed, a new compiler, optparse deprecated
101–110 of 302 posts
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#102Earlier quoted context omitted.
> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…
Multiprocessing is not a real solution, it’s a break-glass procedure when you just need to throw some cores at something without any hope for reliability. Unless something has changed since I used python, it is essentially a wrapper on Fork. This means you need to deal with stuck/dead processes. I’ve used multiprocessing extensively and once you hit a certain amount of usage, even in a pool, you just get hangs and un…
Why would they get stuck/dead and why wouldn't that happen with threads which might be even worse as they're more tightly bound? At least with zombies or inactive processes you can detect and kill them externally - if needs be.
Haven't played with multiprocess at scale, so am genuinely interested.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#103Earlier quoted context omitted.
Similar experience. Even with multi process and threads python is slow, very slow. Java, Go and .NET all provide a very performant out of box experience.
3.11 and on should be comparable to Java for most use cases with multiprocessing (set up correctly of course)
How do you mean? 3.11 is something like 10-20% faster than earlier Python releases. Why should that make it comparable to Java? Typically Java is still several times faster than Python, and this is totally natural since Java performance benefits from static type declarations and the language is generally less dynamic than Python.
That said I still use Python for CPU intensive tasks since in my experience Numpy/Scipy/Numba etc does a good job speeding up the CPU intensive parts of Python code.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#104Earlier quoted context omitted.
Python is both an interpreter, and quite dynamic. Both of these lead to lower performance when compared to less dynamic, compiled solutions. All of Java, Go, and .NET are compiled and (much) less dynamic. This is absolutely an expected outcome.
"absolutely an expected outcome." Good day. Is it the right time to talk to you about Common Lisp?
Always a good time.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#105Earlier quoted context omitted.
Similar experience. Even with multi process and threads python is slow, very slow. Java, Go and .NET all provide a very performant out of box experience.
Python is both an interpreter, and quite dynamic. Both of these lead to lower performance when compared to less dynamic, compiled solutions. All of Java, Go, and .NET are compiled and (much) less dynamic. This is absolutely an expected outcome.
These days even elisp can be compiled. I think python need to be dragged kicking and screaming into cutting edge '80s dynamic compilation technology.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#106Still not encouraged by the no-GIL, "We don't want another Python 2->3 situation", yet very little proffered on how to avoid that scenario. More documentation on writing thread-safe code, suggested tooling to lint for race conditions (for whatever it is worth), discussions with popular C libraries, dedicated support channels for top tier packages, what about the enormous long-tail of abandoned extensions which still…
> what about the enormous long-tail of abandoned extensions which still work today, etc.
I mean there they're talking about keeping GIL in (and I imagine that will be the case for many many years) so those would still keep working. The fear is if some libraries just drop GIL-ful support, but there too I am hopeful for that not to be the case.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#107From reading the thread on HN the other day, it sounds like removing the GIL isn't really of much value. Maybe for somewhat obscure multithreading cases. Is that right?
That discussion was amusing. Removing the GIL opens up the possibility of actually getting a real performance benefit from multithreaded Python code. That's the value. Given every modern desktop and server is multicore (and increasingly getting to tens of cores if not hundreds), multithreading in Python unhampered by the GIL will be a useful thing. And no, multiprocessing is not a good alternative to multithreading.…
Heck, even my watch is dual-core.
Now i doubt I’ll be writing Python for it any time soon, but to call multithreading obscure is… really odd.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#108Earlier quoted context omitted.
Python is both an interpreter, and quite dynamic. Both of these lead to lower performance when compared to less dynamic, compiled solutions. All of Java, Go, and .NET are compiled and (much) less dynamic. This is absolutely an expected outcome.
Node is pretty performant for anything IO related, not compiled and reasonably dynamic.
I think it's worth the clarification that Javascript is usually JITed; (C)Python isn't.
And that CPython's I/O isn't really the problem: some of its async event loop implementations are fairly competitive with Node.
But still ... yes.
Javascript has benefited from two decades of intensive, well-funded work by the best people in the business, with clear focus on performance as a high priority goal. Not to take away from those who work on Python, but I think it's fair to say the effort has had orders of magnitude difference.
I don't have a deep enough understanding to say whether the nature of Python or Javascript makes one better suited for performance optimization than the other. Python is perhaps able to benefit from seeing what's been done with Javascript, although of course Javascript has stood on the shoulders of its own giants.
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#109Earlier quoted context omitted.
There seems to be some confusion here. The GIL is not an invariant of Python that makes your code thread safe. Python is not altering the deal. You can still use threads to write concurrent code in Python today, and you'll still run into all of the classic concurrency related bugs. People just mostly don't bother writing threaded code in Python today because it provides no performance benefit. That may change, and it…
You're right, the GIL certainly doesn't automatically make all code thread-safe. However, it does make some operations that would normally be problematic in a multithreaded context thread-safe. Such as: appending to a list, updating a dict, modifying object attributes, and others. https://docs.python.org/3/faq/library.html#what-kinds-of-glo...
The no-gil fork of Python makes the builtin collections thread-safe, so that will remain not an issue (see the "Collection thread-safety" section of the design doc https://docs.google.com/document/d/18CXhDb1ygxg-YXNBJNzfzZsD... )
Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated
#110Earlier quoted context omitted.
Python is both an interpreter, and quite dynamic. Both of these lead to lower performance when compared to less dynamic, compiled solutions. All of Java, Go, and .NET are compiled and (much) less dynamic. This is absolutely an expected outcome.
These days even elisp can be compiled. I think python need to be dragged kicking and screaming into cutting edge '80s dynamic compilation technology.
I'm sure skilled volunteers would be very welcome.
There are numerous active, moderately serious efforts to both optimize and/or JIT Python bytecode. I think AOT compilation is mostly out-of-scope for 100% compatibility, but again, there's lots of different efforts to compile either subset languages or subsets of programs.
"Kicking and screaming" suggests some reluctance to embrace this, but I think that's probably unfair: it's just hard.