Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python 3.11 vs 3.10 performance

github.com

Post reply on HN
Print view View original

351–360 of 460 posts

Re: Python 3.11 vs 3.10 performance

#351

Post by int_19h »

texaslonghorn5 wrote: post #270
int_19h wrote: post #266

Earlier quoted context omitted.

Yours is nice and readable. Parents' one is not, but it feels like the indentation is deliberately confusing. I'd lay it out like so: tags = list(set([ nel for subli in [ mel for subl in [ [[jel.split('/')[2:] for jel in el] for el in classified ] for mel in subl ] for nel in subli if nel ])) Still not very readable, tho. But that's largely due to Python's outputs-first sequence comprehension syntax being a mess that…

Does the new walrus := in python solve your problem?

Not really; where would you put it inside the []?

Re: Python 3.11 vs 3.10 performance

#352

Post by dragonwriter »

bornfreddy wrote: post #220
hn_saver wrote: post #103

Earlier quoted context omitted.

Threading IS concurrency. When you say "real" concurrency, you actually mean parallelism.

Not in CPython it isn't. Threading in CPython doesn't allow 2 threads to run concurrently (because of GIL). As GP correctly stated, you need multiprocessing (in CPython) for concurrency.

> Threading in CPython doesn't allow 2 threads to run concurrently (because of GIL)

It does allow threads to execute concurrently. It doesn't allow them to execute in parallel if they all are running Python code (if at least one is rubbing native code and has released the GIL, then those plus one that has not can run in parallel.)

Re: Python 3.11 vs 3.10 performance

#353

Post by xwdv »

IceWreck wrote: post #333
tempest_ wrote: post #306

Earlier quoted context omitted.

Cool, they should start now. As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Single threaded performance is not that useful while processors have been growing sideways for 10 years. I often look at elixir with jealousy.

Or maybe keep things the way they are.If you really need performance python is not the language you should be looking for. Instead of breaking decades of code, maybe use a language like Go or Rust for performance instead.

I would have thought convincing people they’ll just have to use Go or Rust or Elixir would have been an easy sell around here.

Turns out they just want a better Python.

Re: Python 3.11 vs 3.10 performance

#354

Post by Mehdi2277 »

IceWreck wrote: post #333
tempest_ wrote: post #306

Earlier quoted context omitted.

Cool, they should start now. As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Single threaded performance is not that useful while processors have been growing sideways for 10 years. I often look at elixir with jealousy.

Or maybe keep things the way they are.If you really need performance python is not the language you should be looking for. Instead of breaking decades of code, maybe use a language like Go or Rust for performance instead.

Python is also the dominant language for machine learning which does care for performance. The person who made recent nogil work is one of the core maintainers of key ML library. The standard workaround is ML libraries, the performance sensitive stuff is written in C/C++ (either manually or with cython) and then uses python bindings. But it would be much friendlier if we could just use python directly.

It's also commonly used language for numerical work in general. Most of the time numpy is enough and then occasionally you'll need something not already implemented and then have to do your own bindings.

Re: Python 3.11 vs 3.10 performance

#355

Post by Mehdi2277 »

pclmulqdq wrote: post #341
FridgeSeal wrote: post #335

Earlier quoted context omitted.

If we’re going to leave Python as a scripting language (fine by me), can we get the machine learning community to swap to something better suited? It strikes me as a bit of a waste of resources to keep stapling engineering effort into the Python ML/data ecosystem when it’s basically a crippled language capable of either: mindlessly driving C binaries, or scripting simple tasks. What other performance, feature and tec…

From what I can tell, the ML community is moving toward Julia. I don't think anyone predicted that they would end up locked into Python so heavily.

That's not been my experience much at all work or research wise. Tensorflow, pytorch, jax are still very dominant. I've worked at several companies and interviewed at several dozen for ML roles. They have 100% been python/c++ for ml. I'd be impressed if even 2% of ML engineers used Julia.

Re: Python 3.11 vs 3.10 performance

#356

Post by dragonwriter »

int_19h wrote: post #351
texaslonghorn5 wrote: post #270

Earlier quoted context omitted.

Does the new walrus := in python solve your problem?

Not really; where would you put it inside the []?

In the example from the end of the earlier post:

   y
   for x in xs
   if (y:=x.ComputeSomething()).IsFoo and y.IsBar

Re: Python 3.11 vs 3.10 performance

#357

Post by winstonewert »

dragonwriter wrote: post #327
winstonewert wrote: post #174

Earlier quoted context omitted.

I would note that the choice of programming language is a bit different. Projects are pretty much locked into that choice. You've got to decide upfront whether the trade off in a rapid prototyping language is good or not, not wait until you've written the project and then profile it.

> Projects are pretty much locked into that choice. But, they aren't. I mean, profile, identify critical components, and rewrite in C (or some other low-level language) for performance is a normal thing for most scripting languages. > You've got to decide upfront whether the trade off in a rapid prototyping language is good or not, not wait until you've written the project and then profile it. No, you absolutely don'…

Yes, it true, if you use Python you can rewrite portions in C to get improved performance. But my point was rather that you couldn't later decide you should have written the entire project in another language like Rust or C++ or Java or Go. You've got the make decision about your primary language up-front.

Or to look at it another way: Python with C extensions is effectively another language. You have to consider it as an option along with Pure Python, Rust, Go, C++, Java, FORTRAN, or what have you. Each language has different trade-offs in development time vs performance.

Re: Python 3.11 vs 3.10 performance

#358

Post by bbkane »

cercatrova wrote: post #322
bbkane wrote: post #321

Earlier quoted context omitted.

He's already using Go. Does Nim provide any "killer features" that Go doesn't?

Well, good generics and sane error handling, for two examples.

Go recently got pretty decent generics. I'm with you on the error handling though. At least it's explicit. Go also has a plethora of really useful libraries, and that along with good editor and build tooling is probably the real draw of the language for me.

Re: Python 3.11 vs 3.10 performance

#359

Post by dragonwriter »

int_19h wrote: post #266
ChadNauseam wrote: post #223

Earlier quoted context omitted.

Hold mine :D https://github.com/anchpop/genomics_viz/blob/master/genomics... That's one expression because it used to be part of a giant comprehension, but I moved it into a function for a bit more readability. I'm considering moving it back just for kicks though. My philosophy is: if you're only barely smart enough to code it, you aren't smart enough to debug it. Therefore, you should code at your limit, to force yo…

Yours is nice and readable. Parents' one is not, but it feels like the indentation is deliberately confusing. I'd lay it out like so: tags = list(set([ nel for subli in [ mel for subl in [ [[jel.split('/')[2:] for jel in el] for el in classified ] for mel in subl ] for nel in subli if nel ])) Still not very readable, tho. But that's largely due to Python's outputs-first sequence comprehension syntax being a mess that…

> But that's largely due to Python's outputs-first sequence comprehension syntax being a mess that doesn't scale at all.

Choices like wrapping a set constructor around a list comprehension rather than just using a set comprehension don't help, neither does using multiple nested no-transformation comprehensions laid out that way.

By hand, on mobile, I think this reduces to (assuming “from itertools import chain” in the file header):

   tags = list({
     nel
     for subli in chain.from_iterables(chain(
       [[jel.split('/')[2:] for jel in el] 
       for el in classified))
     for nel in subli
     if nel
   })

Re: Python 3.11 vs 3.10 performance

#360

Post by shepardrtc »

IceWreck wrote: post #333
tempest_ wrote: post #306

Earlier quoted context omitted.

Cool, they should start now. As a python dev, pythons multiprocess/multithreading story is one the largest pain points in the language. Single threaded performance is not that useful while processors have been growing sideways for 10 years. I often look at elixir with jealousy.

Or maybe keep things the way they are.If you really need performance python is not the language you should be looking for. Instead of breaking decades of code, maybe use a language like Go or Rust for performance instead.

> Instead of breaking decades of code

Pin your version.

Post reply on HN