Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python performance myths and fairy tales

lwn.net

Post reply on HN
Print view View original

171–180 of 221 posts

Re: Python performance myths and fairy tales

#171

Post by IshKebab »

postexitus wrote: post #26
mrkeen wrote: post #17

I didn't read with 100% focus, but this lwn account of the talk seemed to confirm those myths instead of debunking.

A more careful reading of the article is required. The first myth is "Python is not slow" - it is debunked, it is slow. The second myth is ""it's just a glue language / you just need to rewrite the hot parts in C/C++" - it is debunked, just rewriting stuff in C/Rust does not help. The third myth is " Python is slow because it is interpreted" - it is debunked, it is not slow only because it is interpreted.

In fairness I wouldn't really call those "myths", just bad defences of Python's slowness. I don't think the people saying them really believe it - if it came to life or death. They just really like Python and are trying to avoid the cognitive dissonance of liking a really slow language.

Like, I wouldn't say it's a "myth" that Linux is easy to use.

Re: Python performance myths and fairy tales

#173

Post by kragen »

rightbyte wrote: post #170
kragen wrote: post #135

Earlier quoted context omitted.

Type checking and bounds checking aren't "real work" in the sense that, when somebody checks their bank account balance on your website or applies a sound effect to an audio track in their digital audio workstation, they don't think, "Oh good! The computer is going to do some type checking for me now!" Type checking and bounds checking may be good means to an end, but they are not the end, from the point of view of t…

> and the sound effect is only a means to the end of making your music sound less like Daft Punk or something What do you mean. Daft Punk is not daft punk. Why single them out :)

Well, originally I wrote "more like Daft Punk", but then I thought someone might think I was stereotyping musicians as being unoriginal and derivative, so I swung the other way.

Re: Python performance myths and fairy tales

#174

Post by akkad33 »

postexitus wrote: post #26
mrkeen wrote: post #17

I didn't read with 100% focus, but this lwn account of the talk seemed to confirm those myths instead of debunking.

A more careful reading of the article is required. The first myth is "Python is not slow" - it is debunked, it is slow. The second myth is ""it's just a glue language / you just need to rewrite the hot parts in C/C++" - it is debunked, just rewriting stuff in C/Rust does not help. The third myth is " Python is slow because it is interpreted" - it is debunked, it is not slow only because it is interpreted.

> The first myth is "Python is not slow" - it is debunked, it is slow

This is strange. Most people in programming community know python is slow. If it has any reputation, it's that it is quite slow

Re: Python performance myths and fairy tales

#175

Post by jononor »

zahlman wrote: post #117
ActorNightly wrote: post #103

Earlier quoted context omitted.

>how inefficient the boundary crossing is For 99.99% of the programs that people write, the modern M.2 NVME hard drives are plenty fast, and thats the laziest way to load data into a C extension or process. Then there is unix pipes which are sufficiently fast. Then there is shared memory, which basically involves no loading. As with Python, all depends on the setup.

The problem isn't loading the data, but marshalling it (i.e, transforming it into a data structure that makes sense for the faster language to operate on, and back again). Or if you don't transform (or the data is special-cased enough that no transformation makes sense) then the available optimizations become much more limited.

There are several datastructures for numeric data that do not need marshalling, and are suitable for very efficient interoperetion between Python and C/C++/Rust etc. Examples include array.array (in standard library), numpy.array, and PyArrow.

Re: Python performance myths and fairy tales

#176

Post by ehsantn »

The article highlights important challenges regarding Python performance optimization, particularly due to its highly dynamic nature. However, a practical solution involves viewing Python fundamentally as a Domain Specific Language (DSL) framework, rather than purely as a general-purpose interpreted language. DSLs can effectively be compiled into highly efficient machine code.

Examples such as Numba JIT for numerical computation, Bodo JIT/dataframes for data processing, and PyTorch for deep learning demonstrate this clearly. Python’s flexible syntax enables creating complex objects and their operators such as array and dataframe operations, which these compilers efficiently transform into code approaching C++-level performance. DSL operator implementations can also leverage lower-level languages such as C++ or Rust when necessary. Another important aspect not addressed in the article is parallelism, which DSL compilers typically handle quite effectively.

Given that data science and AI are major use cases for Python, compilers like Numba, Bodo, and PyTorch illustrate how many performance-critical scenarios can already be effectively addressed. Investing further in DSL compilers presents a practical pathway to enhancing Python’s performance and scalability across numerous domains, without compromising developer usability and productivity.

Disclaimer: I have previously worked on Numba and Bodo JIT.

Re: Python performance myths and fairy tales

#177

Post by echoangle »

ehsantn wrote: post #176

The article highlights important challenges regarding Python performance optimization, particularly due to its highly dynamic nature. However, a practical solution involves viewing Python fundamentally as a Domain Specific Language (DSL) framework, rather than purely as a general-purpose interpreted language. DSLs can effectively be compiled into highly efficient machine code. Examples such as Numba JIT for numerical…

Was this comment written by an LLM?

Re: Python performance myths and fairy tales

#178

Post by jononor »

01HNNWZ0MV43FF wrote: post #149
didip wrote: post #123

Earlier quoted context omitted.

These days it's "rewrite in Rust". Typically Python is just the entry and exit point (with a little bit of massaging), right? And then the overwhelming majority of the business logic is done in Rust/C++/Fortran, no?

With computer vision you end up wanting to read and write to huge buffers that aren't practical to serialize and are difficult to share. And even allocating and freeing multi-megabyte framebuffers at 60 FPS can put a little strain on the allocator, so you want to reuse them, which means you have to think about memory safety. That is probably why his demo was Sobel edge detection with Numpy. Sobel can run fast enough…

The "numpy" Sobel code is not that good, unfortunately - all the iteration is done in Python, so there is not much benefit from involving numpy. If one would use say scipy.convolve2d on a numpy.array, it would be much faster.

Re: Python performance myths and fairy tales

#179

Post by nomel »

nromiun wrote: post #9
quantumspandex wrote: post #7

So we are paying 99% of the performance just for the 1% of cases where it's nice to code in. Why do people think it's a good trade-off?

Because it's nice to code in. Not everything needs to scale or be fast. Personally I think it is more crazy that you would optimize 99% of the time just to need it for 1% of the time.

That’s why Python is the second best language for everything.

The amount of complexity you can code up in a short time, that most everyone can contribute to, is incredible.

Re: Python performance myths and fairy tales

#180

Post by ngrilly »

NeutralForest wrote: post #2

Cool article, I think a lot of those issues are not Python specific so it's a good overview of whatever others can learn from a now 30 years old language! I think we'll probably go down the JS/TS route where another compiler (Pypy or mypyc or something else) will work alongside CPython but I don't see Python4 happening.

I’m not sure I understand the reference to JS/TS: TS is only a type checker and has zero effect on runtime performance.
Post reply on HN