Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python 3.11 vs 3.10 performance

github.com

Post reply on HN
Print view View original

291–300 of 460 posts

Re: Python 3.11 vs 3.10 performance

#291

Post by staticassertion »

wheelerof4te wrote: post #252
staticassertion wrote: post #90

Earlier quoted context omitted.

Also "what is equals?". That's why people will do: def fooer(i: str, strings: list[str]): push = str.append for s in strings: push(i, s) Apparently this helps the interpreter understand that `append` is not getting overwritten in the global scope elsewhere.

This is one of the most common optimizations in Python, simply because the lookup code for modules/class instances is horrible and slow.

It'd be nice if there were a tool you could run over your code that did this sort of thing. Even if it can technically break shit in absurd cases, I'd totally run it on my code, which isn't absurd.

Re: Python 3.11 vs 3.10 performance

#292

Post by dbrueck »

coliveira wrote: post #281
mixmastamyk wrote: post #242

Earlier quoted context omitted.

The expressions are dynamic, so they have to be evaluated every time. Python is excessively dynamic, so it can't (conventionally) be sped up as easily as many other languages unfortunately. Finally some folks are being paid and allowed to be working on it.

I don't buy this. There are many contexts in which a smart JIT compiler can detect that an expression cannot be modified. Especially since, due to the GIL, python code is mostly non-threaded. They just didn't spend enough time to do the hard work that people spent on Javascript.

> can detect that an expression cannot be modified

Can you give some examples? I use Python a lot and it's absurdly dynamic. I've sometimes wondered if there'd be some way to let me as the programmer say "dynamicness of this module is only 9 instead of the default 11" but I'm skeptical of a tool's ability to reliably deduce things. Here's just one example:

  a = 0
  for i in range(10):
      a += 1
      log('A is', a)
In most languages you could do static analysis to determine that 'a' is an int, and do all sorts of optimizations. In Python, you could safely deduce that 99.9999% of the time it's an int, but you couldn't guarantee it because technically the 'log' function could randomly be really subversive and reach into the calling scope and remap 'a' to point to some other int-like object that behaves differently.

Would "legitimate" code do that? Of course not, but it's technically legal Python and just one example of the crazy level of dynamic-ness, and a tool that automatically optimizes things would need to preserve the correctness of the program.

EDIT: tried to fix code formatting

Re: Python 3.11 vs 3.10 performance

#293

Post by icegreentea2 »

int_19h wrote: post #261
antman wrote: post #208

There was always a denial of removing the Global Interpreter Lock because it would decrease single threaded Python speed for which most people din’t care. So I remember a guy recently came up with a patch that both removed GIL and also to make it easier for the core team to accept it he added also an equivalent number of optimizations. I hope this release was is not we got the optimizations but ignored the GIL part.…

Removing GIL also breaks existing native packages, and would require wholesale migration across the entire ecosystem, on a scale not dissimilar to what we've seen with Python 3.

They believe that the current nogil approach can allow the vast majority of c-extension packages to adapt with a re-compile, or at worst relatively minor code changes (they thought it would be ~15 lines for numpy for example).

Since c-extension wheels are basically built for single python versions anyways, this is potentially manageable.

Re: Python 3.11 vs 3.10 performance

#294

Post by actuallyalys »

dec0dedab0de wrote: post #224
alexfromapex wrote: post #36

One of the biggest features I'm looking forward to is the more specific error messages. I can't tell you how much time I've wasted with cryptic errors that just point to a line that has a list comprehension or similar.

This is the first time I ever heard someone complain about Python's error messages in the 10+ years I've been using it. Even people who just learned it, pick up reading tracebacks after about a day of practice. The only problem I ever see is if a library swallows too much, and gives a generic error, but that's not something the language can fix. I really hope they don't change things too much.

I don’t think they’re that bad, but I think the bar has (fortunately) become higher.

Re: Python 3.11 vs 3.10 performance

#295

Post by metadat »

citrin_ru wrote: post #287
metadat wrote: post #237

Earlier quoted context omitted.

> the vast majority of Python software is written as single-threaded. This is a self-fulfilling prophecy, as the GIL makes Python's (and Ruby's) concurrency story pretty rough compared to nearly all other widely used languages: C, C++, Java, Go, Rust, and even Javascript (as of late).

IMHO majority of Python software doesn't use threads because it is easier to write single threaded code (for many reasons), not because of GIL.

In Golang you can spawn a green thread on any function call with a single keyword: `go'.

The ergonomics are such that it's not difficult to use.

Why can't or shouldn't we have a mechanism comparably fantastic and easy to use in Python?

Re: Python 3.11 vs 3.10 performance

#296

Post by dubbel »

simias wrote: post #52
uniqueuid wrote: post #2

These speedups are awesome, but of course one wonders why they haven't been a low-hanging fruit over the past 25 years. Having read about some of the changes [1], it seems like the python core committers preferred clean over fast implementations and have deviated from this mantra with 3.11. Now let's get a sane concurrency story (no multiprocessing / queue / pickle hacks) and suddenly it's a completely different lang…

I suspect that the amount of people and especially companies willing to spend time and money optimizing Python are fairly low. Think about it: if you have some Python application that's having performance issues you can either dig into a foreign codebase to see if you can find something to optimize (with no guarantee of result) and if you do get something done you'll have to get the patch upstream. And all that "only…

You are right, that's usually not how it works.

Instead, there are big companies, who are running let's say the majority of their workloads in python. It's working well, it doesn't need to be very performant, but together all of the workloads are representing a considerable portion of your compute spend.

At a certain scale it makes sense to employ experts who can for example optimize Python itself, or the Linux kernel, or your DBMS. Not because you need the performance improvement for any specific workload, but to shave off 2% of your total compute spend.

This isn't applicable to small or medium companies usually, but it can work out for bigger ones.

Re: Python 3.11 vs 3.10 performance

#297

Post by cercatrova »

int_19h wrote: post #268
cercatrova wrote: post #234

Somewhat related, had anyone used Nim? These days I've been using it for its Python like syntax year C like speed, especially if I just need to script something quickly.

The problem with any new language is libraries and tooling.

I use it for scripting mainly when I don't need Python specific libraries like pandas. The tooling is pretty good as well.

Re: Python 3.11 vs 3.10 performance

#298

Post by Twirrim »

metadat wrote: post #295
citrin_ru wrote: post #287

Earlier quoted context omitted.

IMHO majority of Python software doesn't use threads because it is easier to write single threaded code (for many reasons), not because of GIL.

In Golang you can spawn a green thread on any function call with a single keyword: `go'. The ergonomics are such that it's not difficult to use. Why can't or shouldn't we have a mechanism comparably fantastic and easy to use in Python?

https://docs.python.org/3/library/concurrent.futures.html sort of gives you that, syntax works with threads or processes.

Re: Python 3.11 vs 3.10 performance

#299

Post by metadat »

mgaunard wrote: post #272
coldtea wrote: post #124

Earlier quoted context omitted.

Unfriendly core team?

Why should major technology projects be "friendly" to random people? I think the word you're looking for is "politics".

Being friendly is inclusive and generally a winning strategy compared to others.

The underlying source of the Politics of Python and associated perceptions stems from the core team's culture of not being "friendly".

Re: Python 3.11 vs 3.10 performance

#300

Post by metadat »

metadat wrote: post #295
citrin_ru wrote: post #287

Earlier quoted context omitted.

IMHO majority of Python software doesn't use threads because it is easier to write single threaded code (for many reasons), not because of GIL.

In Golang you can spawn a green thread on any function call with a single keyword: `go'. The ergonomics are such that it's not difficult to use. Why can't or shouldn't we have a mechanism comparably fantastic and easy to use in Python?

@Twirrim How would you rate the user experience of the concurrent.futures package compared to the golang `go' keyword?

It is architecturally comparable to async Javascript programming, which imho is a shoehorn solution.

Post reply on HN