Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python 3.11 vs 3.10 performance

github.com

Post reply on HN
Print view View original

361–370 of 460 posts

Re: Python 3.11 vs 3.10 performance

#361

Post by shpx »

FridgeSeal wrote: post #340
superjan wrote: post #198

Yesterday, I watched Emery Berger’s “Python performance matters” Which is you should not bother to write fast Python, just delegate all heavy lifting to optimized C/c++ and the likes. His group has a Python profiler whose main goal is to point out which parts should be delegated. Of course optimized is better but the numeric microbenchmarks in this post are mostly examples of code that is better delegated to low over…

> just delegate all heavy lifting to optimized C/c++ and the likes The more o used Python, and the more Python I saw written, the more I am convinced this is not a good or reasonable argument. A good chunk of the Python devs I’ve interacted with are mystified by the concept of virtual environments, regularly abuse global variables, struggle to read the docs, and structure their code poorly. Telling these people “oh j…

The only "FFI" I've seen people actually use is writing a command line tool with C++ and calling it with the subprocess library.

Re: Python 3.11 vs 3.10 performance

#362

Post by urthor »

actuallyalys wrote: post #294
dec0dedab0de wrote: post #224

Earlier quoted context omitted.

This is the first time I ever heard someone complain about Python's error messages in the 10+ years I've been using it. Even people who just learned it, pick up reading tracebacks after about a day of practice. The only problem I ever see is if a library swallows too much, and gives a generic error, but that's not something the language can fix. I really hope they don't change things too much.

I don’t think they’re that bad, but I think the bar has (fortunately) become higher.

Exactly right.

The old Python stack trace, circa 2015, was the best in the world.

The new ones are even better.

The gap between Python and Rust, and the JVM languages/C/C++, is increasingly widening.

Stack trace is one of the areas Python and Rust are making the case for why they're the languages of the future.

Re: Python 3.11 vs 3.10 performance

#363

Post by int_19h »

dragonwriter wrote: post #356
int_19h wrote: post #351

Earlier quoted context omitted.

Not really; where would you put it inside the []?

In the example from the end of the earlier post: y for x in xs if (y:=x.ComputeSomething()).IsFoo and y.IsBar

I suppose that works, although readability is not great, to put it mildly.

Re: Python 3.11 vs 3.10 performance

#364

Post by texaslonghorn5 »

dragonwriter wrote: post #356
int_19h wrote: post #351

Earlier quoted context omitted.

Not really; where would you put it inside the []?

In the example from the end of the earlier post: y for x in xs if (y:=x.ComputeSomething()).IsFoo and y.IsBar

Much better would be

[(y:=x.ComputeSomething()) for x in xs if y.IsFoo and y.IsBar]

Re: Python 3.11 vs 3.10 performance

#366

Post by nine_k »

Galanwe wrote: post #349
tempest_ wrote: post #306

Earlier quoted context omitted.

Cool, they should start now. As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Single threaded performance is not that useful while processors have been growing sideways for 10 years. I often look at elixir with jealousy.

> As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Hmm, how is that so? As a python dev as well, I don't have much complaint with multiprocessing. The API is simple, it works OK, the overall paradigm is simple to grok, you can share transparently with pickle, etc.

Multiprocessing is fine, but the cost of starting another interpreter is pretty visible, so you need to create a pool, and it may not be an overall speedup if the run time is short.

It takes more careful planning than async in JS?, say, or goroutines.

Re: Python 3.11 vs 3.10 performance

#367

Post by ChadNauseam »

int_19h wrote: post #351
texaslonghorn5 wrote: post #270

Earlier quoted context omitted.

Does the new walrus := in python solve your problem?

Not really; where would you put it inside the []?

For another example of how to use the walrus operator, I use it multiple times in the code I linked a few comments up

Re: Python 3.11 vs 3.10 performance

#368

Post by lmm »

liuliu wrote: post #319
lmm wrote: post #304

Earlier quoted context omitted.

Because making it easy to write C/C++ extensions that work the way you expect (including for things like passing a Python callback to a C/C++ library) has always been a priority for Python in a way that it isn't for Golang?

Any C/C++ extensions that wants to enable more efficient Python has to learn GIL and how to manipulate that as well. Including not limited to: how to give up GIL (so that other Python code can progress); how to prepare your newly initiated threads to be Python GIL friendly etc. Personally, GIL is more surprising to me when interop with Python.

> Any C/C++ extensions that wants to enable more efficient Python has to learn GIL and how to manipulate that as well. Including not limited to: how to give up GIL (so that other Python code can progress); how to prepare your newly initiated threads to be Python GIL friendly etc.

Sure, but the easy cases are easy (in particular, you can usually do nothing and it will be correct but slow, which is much better than fast but incorrect) and the hard cases are possible.

> Personally, GIL is more surprising to me when interop with Python.

Any GCed language, including Go, will oblige you to integrate with its APIs if you want to handle complex cases correctly.

Re: Python 3.11 vs 3.10 performance

#369

Post by ghshephard »

pclmulqdq wrote: post #344
metadat wrote: post #334

Earlier quoted context omitted.

Python has already become a lot more than a scripting language. To say today that scripting is it's core identity seems naive at best. Yes, it has roots but has object oriented and functional facets which do not exist in awk or bash. Pandas, numpy, scipy, tensorflow. All of these go way beyond what is possible with a scripting language. Since when is the runtime performance of a script a serious concern? Why is it a…

Awk is a functional language with a surprising number of features. Bash, not so much. Developers really do care about runtime performance of scripting languages: They often wait for scripts to finish running before doing other work, and if the sum of the time it takes to write and execute a script is too long, they will look for an alternative. All of the libraries you have cited are from the ML and stats communities…

"Personally, I like that Python has kept the GIL so far because I would never run a 24/7 server in Python and I am happy to use it very frequently for single-threaded scripting tasks."

Just as a side-note - my prior gig used Python on both the Server and Data Collection industrial systems. It was very much a 24x7x365 must-never-go-down type of industrial application, and, particularly when we had a lot of data-sources, was very multi-processing. Was not unusual to see 32 processes working together (we used sqlite and kafka as our handoff and output of processes) running on our data collection appliances.

Our core data modelling engine would routinely spin up 500 worker pods to complete the work needed to be done in 1/500th of the time, but we would still see some of the long term historian runs take upwards of a week to complete (many hundreds of machines for multiple years with thousands of tags coming in every 5 seconds is just a lot of data to model).

I say this mostly to demonstrate that people and companies do use python in both large-processing intensive environments as well as industrial-must-never-stop-24x7 mission critical appliances.

I don't ever recall any of our engineers looking at Python as "a scripting language" - it was no different to them than Java, C#, Rust, Go, C++ or any other language.

Re: Python 3.11 vs 3.10 performance

#370

Post by Dangisshit »

IceWreck wrote: post #333
tempest_ wrote: post #306

Earlier quoted context omitted.

Cool, they should start now. As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Single threaded performance is not that useful while processors have been growing sideways for 10 years. I often look at elixir with jealousy.

Or maybe keep things the way they are.If you really need performance python is not the language you should be looking for. Instead of breaking decades of code, maybe use a language like Go or Rust for performance instead.

So you are essentially saying Python is obsolete. It's used for decade old code and for new code you should use go or rust.
Post reply on HN