Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python 3.11 vs 3.10 performance

github.com

Post reply on HN
Print view View original

121–130 of 460 posts

Re: Python 3.11 vs 3.10 performance

#121

Post by dekhn »

Jasper_ wrote: post #61
uniqueuid wrote: post #2

These speedups are awesome, but of course one wonders why they haven't been a low-hanging fruit over the past 25 years. Having read about some of the changes [1], it seems like the python core committers preferred clean over fast implementations and have deviated from this mantra with 3.11. Now let's get a sane concurrency story (no multiprocessing / queue / pickle hacks) and suddenly it's a completely different lang…

> one wonders why they haven't been a low-hanging fruit over the past 25 years Because the core team just hasn't prioritized performance, and have actively resisted performance work, at least until now. The big reason has been about maintainership cost of such work, but often times plenty of VM engineers show up to assist the core team and they have always been pushed away. > Now let's get a sane concurrency story Yo…

Yup. Sad but true. I just wanted c++ style multithreading with unsafe shared memory but it's too late. I was impressed by the recent fork that addresses this in a very sophisticated way, but realistically gil cpython will keep finding speed ups to stave off the transition.

Re: Python 3.11 vs 3.10 performance

#122

Post by GrayShade »

winstonewert wrote: post #108
GrayShade wrote: post #102

Earlier quoted context omitted.

You should probably read the full context around that quote, I'm sick and tired of everyone repeating it mindlessly: https://softwareengineering.stackexchange.com/a/80092 > Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.

> You should probably read the full context around that quote, I'm sick and tired of everyone repeating it mindlessly: I'm confused. What do you think the context changes? At least as I read it, both the short form and full context convey the same idea.

That quote has been thrown around every time in order to justify writing inefficient code and never optimizing it. Python is 10-1000x slower than C, but sure, let's keep using it because premature optimization is the root of all evil, as Knuth said. People really love to ignore the "premature" word in that quote.

Instead, what he meant is that you should profile what part of the code is slow and focus on it first. Knuth didn't say you should be fine with 10-1000x slower code overall.

Re: Python 3.11 vs 3.10 performance

#123

Post by NelsonMinar »

sod wrote: post #14

I wish there was something like llvm for scripting languages. Imagine if python, php, javascript, dart or ruby would not interpret the code themself, but compile to an interpretable common language, where you could just plug in the fastest interpreter there is for the job.

The Java VM has been that in the past with projects like Jython being the bridge. But it's never worked out very well in practice.

Re: Python 3.11 vs 3.10 performance

#124

Post by coldtea »

Angostura wrote: post #70
mgaunard wrote: post #62

Earlier quoted context omitted.

first, there are lots of volunteers that want to mess with assembly language. second, CPython is just a C interpeter, there isn't much assembly if any. third, contributing to CPython is sufficiently high-profile you could easily land a 500k-job just by putting it on your CV. No, those are not the real reasons why it hasn't happened before.

Go on then - what are the real reasons, in your mind?

Unfriendly core team?

Re: Python 3.11 vs 3.10 performance

#125

Post by enragedcacti »

FartyMcFarter wrote: post #58
missblit wrote: post #18

Earlier quoted context omitted.

So we have: * Statically allocated ("frozen") core modules for fast imports * Avoid memory allocation for frames / faster frame creation * Inlined python functions are called in pure python without needing to jump through C * Optimizations that take advantage of speculative typing (Reminds me of Javascript JIT compilers -- though according to the FAQ Python isn't JIT yet) * Smaller memory usage for frames, objects, a…

> Inlined python functions are called in pure python without needing to jump through C Given that Python is interpreted, it's quite unclear what this could mean. Also, what does it mean to "call" an inlined function?? Isn't the point of inline functions that they don't get called at all?

It's a little confusing but I don't think they meant inlining in the traditional sense. its more like they inlined the C function wrapper around python functions.

> During a Python function call, Python will call an evaluating C function to interpret that function’s code. This effectively limits pure Python recursion to what’s safe for the C stack.

> In 3.11, when CPython detects Python code calling another Python function, it sets up a new frame, and “jumps” to the new code inside the new frame. This avoids calling the C interpreting function altogether.

> Most Python function calls now consume no C stack space. This speeds up most of such calls. In simple recursive functions like fibonacci or factorial, a 1.7x speedup was observed. This also means recursive functions can recurse significantly deeper (if the user increases the recursion limit). We measured a 1-3% improvement in pyperformance.

Re: Python 3.11 vs 3.10 performance

#126

Post by coldtea »

dragonwriter wrote: post #97
dekhn wrote: post #37

Earlier quoted context omitted.

Not really. There were a couple engineers working at Google on a project called unladen swallow which was extremely promising but it eventually got canceled. The developer who worked at Microsoft to make iron python I think that was his full-time project as well and it was definitely faster than cpython at the time

Those people were not being paid to speed up CPython, though, but mostly-source-compatible (but not at all native extension compatible) alternative interpreters.

That's because the core team wasn't friendly to them making changes in CPython.

Not because they deliberately only wanted to build an alternative interpreter.

Re: Python 3.11 vs 3.10 performance

#128

Post by coldtea »

BugsJustFindMe wrote: post #22
siekmanj wrote: post #6

Earlier quoted context omitted.

Python has actually had concurrency since about 2019: https://docs.python.org/3/library/asyncio.html . Having used it a few times, it seems fairly sane, but tbf my experience with concurrency in other languages is fairly limited. edit: ray https://github.com/ray-project/ray is also pretty easy to use and powerful for actual parallelism

I find asyncio to be horrendous, both because of the silliness of its demands on how you build your code and also because of its arbitrarily limited scope. Thread/ProcessPoolExecutor is personally much nicer to use and universally applicable...unless you need to accommodate Ctrl-C and then it's ugly again. But fixing _that_ stupid problem would have been a better expenditure of effort than asyncio.

>I find asyncio to be horrendous, both because of the silliness of its demands on how you build your code and also because of its arbitrarily limited scope.

Do you compare it to threads and pools, or judge it on its merits as an async framework (with you having experience of those that you think are done better elsewhere, e.g. in Javascript, C#, etc)?

Because both things you mention "demands on how you build your code" and "limited scope" are part of the course with async in most languages that aren't async-first.

Re: Python 3.11 vs 3.10 performance

#129

Post by DasIch »

winstonewert wrote: post #108
GrayShade wrote: post #102

Earlier quoted context omitted.

You should probably read the full context around that quote, I'm sick and tired of everyone repeating it mindlessly: https://softwareengineering.stackexchange.com/a/80092 > Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.

> You should probably read the full context around that quote, I'm sick and tired of everyone repeating it mindlessly: I'm confused. What do you think the context changes? At least as I read it, both the short form and full context convey the same idea.

I agree with you, the context changes nothing (and I upvoted you for this reason). However programming languages and infrastructure pieces like this are a bit special, in that optimizations here are almost never premature.

  * Some of the many applications relying on these pieces, could almost certainly use the speedup and for those it wouldn't be premature
  * The return of investment is massive due to the scale
  * There are tremendous productivity gains by increasing the performance baseline because that reduces the time people have to spend optimizing applications
This is very different from applications where you can probably define performance objectives and define much more clearly what is and isn't premature.

Re: Python 3.11 vs 3.10 performance

#130

Post by wheybags »

tasubotadas wrote: post #106
speedgoose wrote: post #28

Earlier quoted context omitted.

Python being interpreted is the main reason why it’s slow, but not the excuse to not compare it to similar and faster programming languages.

>Python being interpreted is the main reason why it’s slow Common Lisp and Java begs to disagree.

Jvm at least is jitted, not interpreted. Dunno anything about common lisp.
Post reply on HN