Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python 3.11 vs 3.10 performance

github.com

Post reply on HN
Print view View original

391–400 of 460 posts

Re: Python 3.11 vs 3.10 performance

#391

Post by pjmlp »

Mehdi2277 wrote: post #354
IceWreck wrote: post #333

Earlier quoted context omitted.

Or maybe keep things the way they are.If you really need performance python is not the language you should be looking for. Instead of breaking decades of code, maybe use a language like Go or Rust for performance instead.

Python is also the dominant language for machine learning which does care for performance. The person who made recent nogil work is one of the core maintainers of key ML library. The standard workaround is ML libraries, the performance sensitive stuff is written in C/C++ (either manually or with cython) and then uses python bindings. But it would be much friendlier if we could just use python directly. It's also comm…

Nope, C++ and Fortran are.

The bindings available in Python can also be used from other languages.

Re: Python 3.11 vs 3.10 performance

#392

Post by pjmlp »

xwdv wrote: post #353
IceWreck wrote: post #333

Earlier quoted context omitted.

Or maybe keep things the way they are.If you really need performance python is not the language you should be looking for. Instead of breaking decades of code, maybe use a language like Go or Rust for performance instead.

I would have thought convincing people they’ll just have to use Go or Rust or Elixir would have been an easy sell around here. Turns out they just want a better Python.

It already exists, but they don't want to learn other languages.

Re: Python 3.11 vs 3.10 performance

#394

Post by pjmlp »

oofbey wrote: post #330
tempest_ wrote: post #306

Earlier quoted context omitted.

Cool, they should start now. As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Single threaded performance is not that useful while processors have been growing sideways for 10 years. I often look at elixir with jealousy.

On the flip side, if your workload can be parallelized across thousands of cores, python has about the best CUDA support anywhere.

That would be C++ and Fortran actually.

Re: Python 3.11 vs 3.10 performance

#395

Post by gpderetta »

FridgeSeal wrote: post #340
superjan wrote: post #198

Yesterday, I watched Emery Berger’s “Python performance matters” Which is you should not bother to write fast Python, just delegate all heavy lifting to optimized C/c++ and the likes. His group has a Python profiler whose main goal is to point out which parts should be delegated. Of course optimized is better but the numeric microbenchmarks in this post are mostly examples of code that is better delegated to low over…

> just delegate all heavy lifting to optimized C/c++ and the likes The more o used Python, and the more Python I saw written, the more I am convinced this is not a good or reasonable argument. A good chunk of the Python devs I’ve interacted with are mystified by the concept of virtual environments, regularly abuse global variables, struggle to read the docs, and structure their code poorly. Telling these people “oh j…

Very much yes, if you are fluent in both C++ and python, it is just easier to rewrite in the former than deal with FFI.

Unfortunately sometimes you have to provide a library for consumption to programmers that only speak python so you don't have many options.

Re: Python 3.11 vs 3.10 performance

#396

Post by pjmlp »

metadat wrote: post #334
pclmulqdq wrote: post #331

Earlier quoted context omitted.

I'm not the person you responded to, but I think the gist of it is: what it is is not defined by how it is used. Python, at its core, is a scripting language, like awk and bash. The other uses don't change that. Occasionally, a technology breaks out of its intended domain. Python is one of these - it plays host to lots of webservers, and even a filesystem (dropbox). Similarly, HTML is a text markup language, but that…

Python has already become a lot more than a scripting language. To say today that scripting is it's core identity seems naive at best. Yes, it has roots but has object oriented and functional facets which do not exist in awk or bash. Pandas, numpy, scipy, tensorflow. All of these go way beyond what is possible with a scripting language. Since when is the runtime performance of a script a serious concern? Why is it a…

Except most of them are bindings written in native languages, being scripted from Python.

Re: Python 3.11 vs 3.10 performance

#397

Post by gpderetta »

atwood22 wrote: post #79
gostsamo wrote: post #68

Earlier quoted context omitted.

A few people tried to excuse the slow python, but as far as I know the story, excuses are not necessary. Truth is that python was not meant to be fast, its source code was not meant to be fast, and its design was not optimized with the idea of being fast. Python was meant as a scrypting language that was easy to learn and work with on all levels and the issue of its slowness became important when it outgrew its role…

> Python was meant as a scrypting language that was easy to learn and work with on all levels. Being fast isn't contradictory with this goal. If anything, this is a lesson that so many developers forget. Things should be fast by default.

Or at the very least they should be designed in such a way that it optimizing them later is still possible.

Re: Python 3.11 vs 3.10 performance

#398

Post by pjmlp »

superjan wrote: post #198

Yesterday, I watched Emery Berger’s “Python performance matters” Which is you should not bother to write fast Python, just delegate all heavy lifting to optimized C/c++ and the likes. His group has a Python profiler whose main goal is to point out which parts should be delegated. Of course optimized is better but the numeric microbenchmarks in this post are mostly examples of code that is better delegated to low over…

My experience writing TCL native extensions made me realize, this kind of languages are only good for scripting purposes.

Anything where performance matters should be written in languages where JIT and AOT compilers come as standard options on the reference implementations.

Re: Python 3.11 vs 3.10 performance

#399

Post by pjmlp »

gpderetta wrote: post #395
FridgeSeal wrote: post #340

Earlier quoted context omitted.

> just delegate all heavy lifting to optimized C/c++ and the likes The more o used Python, and the more Python I saw written, the more I am convinced this is not a good or reasonable argument. A good chunk of the Python devs I’ve interacted with are mystified by the concept of virtual environments, regularly abuse global variables, struggle to read the docs, and structure their code poorly. Telling these people “oh j…

Very much yes, if you are fluent in both C++ and python, it is just easier to rewrite in the former than deal with FFI. Unfortunately sometimes you have to provide a library for consumption to programmers that only speak python so you don't have many options.

Plus something like C++20 and hot code reload (VC++ and Live++), make it quite easy to do so.

Re: Python 3.11 vs 3.10 performance

#400

Post by gpderetta »

otherme123 wrote: post #222
atwood22 wrote: post #151

Earlier quoted context omitted.

My point is that you can write fast code just as easily as you can write slow code. So engineers should write fast code when possible. Obviously you can spend a lot of time making things faster, but that doesn't mean you can't be fast by default.

Not true. Premature optimization is the root of all evil. You first write clean code, and then you profile and optimize. I refer you to the underlyings of dicts through the years ( https://www.youtube.com/watch?v=npw4s1QTmPg ) as an example of that optimization taking years of incremental changes. Once you see the current version it's easy to claim that you would have get to the current and best version in the first…

CPython and the Python design in general clearly show that writing clean code and optimizing later is significantly harder and take much more effort than keeping optimizations in mind from the start. It doesn't mean you need to write optimal code form day one, just that you need to be careful not to program yourself into a corner.
Post reply on HN