Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python 3.11 vs 3.10 performance

github.com

Post reply on HN
Print view View original

331–340 of 460 posts

Re: Python 3.11 vs 3.10 performance

#331

Post by pclmulqdq »

metadat wrote: post #248
SiempreViernes wrote: post #247

Earlier quoted context omitted.

Pretty sure it is actually a statement that comes from the "python is a scripting language" school, and not because the huge horde of programmers that craves concurrency when they write a script to reformat a log file to csv keeps being put off by the python multiprocessing story.

Not sure I understand your point, can you clarify? Python is used across many different domains, being able to really take advantage of multiple cores would be a big deal. I'd really appreciate if python included concurrency or parallelism capabilities that didn't disappoint and frustrate me. If you've tried using the thread module, multiprocessing module, or async function coloring feature, you probably can relate.…

I'm not the person you responded to, but I think the gist of it is: what it is is not defined by how it is used. Python, at its core, is a scripting language, like awk and bash. The other uses don't change that.

Occasionally, a technology breaks out of its intended domain. Python is one of these - it plays host to lots of webservers, and even a filesystem (dropbox). Similarly, HTML is a text markup language, but that doesn't stop people from making video games using it.

The developers of the technology now have to make a philosophical decision about what that technology is. They can decide to re-optimize for the current use cases and risk losing the heart of their technology as tradeoffs get made, or they can decide to keep the old vision. All of the design choices flow down from the vision.

They have decided that Python is a scripting language. Personally, I agree (https://specbranch.com/posts/python-and-asm/). In turn, the people using the language for something other than its intended purpose have choices to make - including whether to abandon the technology.

If instead Python moves toward competing with languages like go, it is going to need to make a lot of different tradeoffs. Ditching the GIL comes with tradeoffs of making Python slower for those who just want to write scripts. Adding more rigorous scoping would make it easier to have confidence in a production server, but harder to tinker. Everything comes with tradeoffs, and decisions on those tradeoffs come from the values of the development team.

Right now, they value scripting.

Re: Python 3.11 vs 3.10 performance

#332

Post by Nimitz14 »

tempest_ wrote: post #306
tyingq wrote: post #250

Earlier quoted context omitted.

Getting rid of the GIL will also immediately expose all the not-thread-safe stuff that currently exists, so there's a couple of waves you would need before it would be broadly usable.

Cool, they should start now. As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Single threaded performance is not that useful while processors have been growing sideways for 10 years. I often look at elixir with jealousy.

As another python dev, it has basically never been a painpoint for me.

Your anecdote adds little.

Re: Python 3.11 vs 3.10 performance

#333

Post by IceWreck »

tempest_ wrote: post #306
tyingq wrote: post #250

Earlier quoted context omitted.

Getting rid of the GIL will also immediately expose all the not-thread-safe stuff that currently exists, so there's a couple of waves you would need before it would be broadly usable.

Cool, they should start now. As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Single threaded performance is not that useful while processors have been growing sideways for 10 years. I often look at elixir with jealousy.

Or maybe keep things the way they are.If you really need performance python is not the language you should be looking for.

Instead of breaking decades of code, maybe use a language like Go or Rust for performance instead.

Re: Python 3.11 vs 3.10 performance

#334

Post by metadat »

pclmulqdq wrote: post #331
metadat wrote: post #248

Earlier quoted context omitted.

Not sure I understand your point, can you clarify? Python is used across many different domains, being able to really take advantage of multiple cores would be a big deal. I'd really appreciate if python included concurrency or parallelism capabilities that didn't disappoint and frustrate me. If you've tried using the thread module, multiprocessing module, or async function coloring feature, you probably can relate.…

I'm not the person you responded to, but I think the gist of it is: what it is is not defined by how it is used. Python, at its core, is a scripting language, like awk and bash. The other uses don't change that. Occasionally, a technology breaks out of its intended domain. Python is one of these - it plays host to lots of webservers, and even a filesystem (dropbox). Similarly, HTML is a text markup language, but that…

Python has already become a lot more than a scripting language. To say today that scripting is it's core identity seems naive at best. Yes, it has roots but has object oriented and functional facets which do not exist in awk or bash.

Pandas, numpy, scipy, tensorflow. All of these go way beyond what is possible with a scripting language.

Since when is the runtime performance of a script a serious concern? Why is it a problem if this aspect gets slightly slower if it brings real concurrency support?

Re: Python 3.11 vs 3.10 performance

#335

Post by FridgeSeal »

pclmulqdq wrote: post #331
metadat wrote: post #248

Earlier quoted context omitted.

Not sure I understand your point, can you clarify? Python is used across many different domains, being able to really take advantage of multiple cores would be a big deal. I'd really appreciate if python included concurrency or parallelism capabilities that didn't disappoint and frustrate me. If you've tried using the thread module, multiprocessing module, or async function coloring feature, you probably can relate.…

I'm not the person you responded to, but I think the gist of it is: what it is is not defined by how it is used. Python, at its core, is a scripting language, like awk and bash. The other uses don't change that. Occasionally, a technology breaks out of its intended domain. Python is one of these - it plays host to lots of webservers, and even a filesystem (dropbox). Similarly, HTML is a text markup language, but that…

If we’re going to leave Python as a scripting language (fine by me), can we get the machine learning community to swap to something better suited?

It strikes me as a bit of a waste of resources to keep stapling engineering effort into the Python ML/data ecosystem when it’s basically a crippled language capable of either: mindlessly driving C binaries, or scripting simple tasks.

What other performance, feature and technique advancements are we leaving on the table because the only “viable” ecosystem is built on a fundamentally crippled language?

Re: Python 3.11 vs 3.10 performance

#336

Post by jolux »

galangalalgol wrote: post #329
ModernMech wrote: post #263

Earlier quoted context omitted.

> you should not bother to write fast Python, just delegate all heavy lifting to optimized C/c++ and the likes Certainly that's something you can do, but unfortunately for Python it opens the door for languages like Julia, which are trying to say that you can have your cake and eat it too.

I like Julia but its easy to write slow julia unless you keep the performance tips in mind. Arrays are horribly slow, tuples are much faster, but having tuples be a multiple of 128 bytes adds 10% or more to speed. I honestly don't understand how much slower arrays are. Its like 30x or similar.

That’s bizarre, are they implemented as linked lists or something? Why would arrays be that slow?

Re: Python 3.11 vs 3.10 performance

#337

Post by magic_hamster »

BerislavLopac wrote: post #229
antman wrote: post #208

There was always a denial of removing the Global Interpreter Lock because it would decrease single threaded Python speed for which most people din’t care. So I remember a guy recently came up with a patch that both removed GIL and also to make it easier for the core team to accept it he added also an equivalent number of optimizations. I hope this release was is not we got the optimizations but ignored the GIL part.…

The GIL removal by that guy reverted some of the improvement done by other optimisations, so the overall improvement was much smaller. And most people do care for single-threaded speed, because the vast majority of Python software is written as single-threaded.

A lot of python runs in celery and possibly even more python runs parallelized cuda code. Not sure at all the majority of python code is single threaded, especially for more serious projects.

Re: Python 3.11 vs 3.10 performance

#338

Post by dahart »

metadat wrote: post #237
BerislavLopac wrote: post #229

Earlier quoted context omitted.

The GIL removal by that guy reverted some of the improvement done by other optimisations, so the overall improvement was much smaller. And most people do care for single-threaded speed, because the vast majority of Python software is written as single-threaded.

> the vast majority of Python software is written as single-threaded. This is a self-fulfilling prophecy, as the GIL makes Python's (and Ruby's) concurrency story pretty rough compared to nearly all other widely used languages: C, C++, Java, Go, Rust, and even Javascript (as of late).

I agree with your point, but the vast majority of C, C++, Java, and Javacript code is also written as single-threaded. It’s fair to acknowledge that the primary use case of most languages is single threaded programming, and also that improving the ergonomics of concurrency in Python would be a huge boon.

Re: Python 3.11 vs 3.10 performance

#339

Post by ModernMech »

galangalalgol wrote: post #329
ModernMech wrote: post #263

Earlier quoted context omitted.

> you should not bother to write fast Python, just delegate all heavy lifting to optimized C/c++ and the likes Certainly that's something you can do, but unfortunately for Python it opens the door for languages like Julia, which are trying to say that you can have your cake and eat it too.

I like Julia but its easy to write slow julia unless you keep the performance tips in mind. Arrays are horribly slow, tuples are much faster, but having tuples be a multiple of 128 bytes adds 10% or more to speed. I honestly don't understand how much slower arrays are. Its like 30x or similar.

I'm not really saying Julia is going to eat Python's lunch, because I also don't think they've cracked the code exactly. But some language will eventually.

Re: Python 3.11 vs 3.10 performance

#340

Post by FridgeSeal »

superjan wrote: post #198

Yesterday, I watched Emery Berger’s “Python performance matters” Which is you should not bother to write fast Python, just delegate all heavy lifting to optimized C/c++ and the likes. His group has a Python profiler whose main goal is to point out which parts should be delegated. Of course optimized is better but the numeric microbenchmarks in this post are mostly examples of code that is better delegated to low over…

> just delegate all heavy lifting to optimized C/c++ and the likes

The more o used Python, and the more Python I saw written, the more I am convinced this is not a good or reasonable argument. A good chunk of the Python devs I’ve interacted with are mystified by the concept of virtual environments, regularly abuse global variables, struggle to read the docs, and structure their code poorly.

Telling these people “oh just write this section in C” will go nowhere. It’s asking them to figure out: installation, compilation, packaging, FFI, tool chains, and foot-guns of language(s) that are arguably more dangerous and whose operation varies from “not super straightforward” to “arcane” based on your dev and deployment environment and application needs.

Post reply on HN