Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python performance myths and fairy tales

lwn.net

Post reply on HN
Print view View original

151–160 of 221 posts

Re: Python performance myths and fairy tales

#151

Post by ajross »

adsharma wrote: post #126
ajross wrote: post #101

Earlier quoted context omitted.

> These features have one thing in common: they're only useful for prototype-quality throwaway code, if at all. As a matter of practice: the python community disagrees strongly. And the python community ate the world. It's fine to have an opinion, but you're not going to change python.

That assumes python is one monolithic thing and everyone agrees what it is. True, the view you express here has strong support in the community and possibly in the steering committee. But there are differing ideas on what python is and why it's successful.

> That assumes python is one monolithic thing and everyone agrees what it is.

It's exactly the opposite! I'm saying that python is BIG AND DIVERSE and that attempts like SPy to invent a new (monolithic!) subset language that everyone should use instead are doomed, because it won't meet the needs of all the yahoos out there doing weird stuff the SPy authors didn't think was important.

It's fine to have "differing ideas on what python is", but if those ideas don't match those of all of the community, and not just what you think are the good parts, it's not really about what "python" is, is it?

Re: Python performance myths and fairy tales

#152

Post by CraigJPerry »

jerf wrote: post #48
btown wrote: post #16

I think an important bit of context here is that computers are very, very good at speculative happy-path execution. The examples in the article seem gloomy: how could a JIT possibly do all the checks to make sure the arguments aren’t funky before adding them together, in a way that’s meaningfully better than just running the interpreter? But in practice, a JIT can create code that does these checks, and modern proces…

That makes it so that in absolute terms, Python is not as slow as you might naively expect. But we don't measure programming language performance in absolute terms. We measure them in relative terms, generally against C. And while your Python code is speculating about how this Python object will be unboxed, where its methods are, how to unbox its parameters, what methods will be called on those, etc., compiled code i…

> And while your Python code is speculating about how this Python object will be unboxed

This is wrong i think? The GP is talking about JIT'd code.

Re: Python performance myths and fairy tales

#153

Post by actinium226 »

A lot of the examples he gives, like the numpy/calc function, are easily converted to C/C++/Rust. The article sort of dismisses this at the start, and that's fine if we want to focus on the speed of Python itself, but it seems like both the only solution and the obvious solution to many of the problems specified.

Re: Python performance myths and fairy tales

#154

Post by game_the0ry »

throwaway6041 wrote: post #70
game_the0ry wrote: post #61

I know I am going to get some hate for this from the "Python-stans" but..."python" and "performance" should never be associated with each other, and same for any scripting/interpreted programming language. Especially if it has a global interpreter lock. While performance (however you may mean that) is always a worthy goal, you may need to question your choice of language if you start hitting performance ceilings. As…

> the "Python-stans" I think the term "Pythonistas" is more widely used > you may need to question your choice of language if you start hitting performance ceilings. Developers should also question if a "fast" language like Rust is really needed, if implementing a feature takes longer than it would in Python. I don't like bloat in general, but sometimes it can be worth spinning up a few extra instances to get to mark…

> Developers should also question if a "fast" language like Rust is really needed...

Agreed.

Re: Python performance myths and fairy tales

#155

Post by olejorgenb »

zahlman wrote: post #116
Mithriil wrote: post #108

> His "sad truth" conclusion is that "Python cannot be super-fast" without breaking compatibility. A decent case of Python 4.0? > So, maybe, "a JIT compiler can solve all of your problems"; they can go a long way toward making Python, or any dynamic language, faster, Cuni said. But that leads to "a more subtle problem". He put up a slide with a trilemma triangle: a dynamic language, speed, or a simple implementation.…

> A decent case of Python 4.0? I think "Python 4.0" is going to have to be effectively a new language by a different team that simply happens to bear strong syntactic similarities. (And at least part of why that isn't already happening is that everyone keeps getting scared off by the scale of the task.) Thanks for the reminder that I never got around to checking out Julia.

Isn't that kinda what Mojo is?

Re: Python performance myths and fairy tales

#156

Post by zahlman »

hansvm wrote: post #150
zahlman wrote: post #147

Earlier quoted context omitted.

Care to show how you believe this can be achieved, from within Python?

import ctypes ten = 10 addr = id(ten) class PyLongObject(ctypes.Structure): _fields_ = [ ("ob_refcnt", ctypes.c_ssize_t), ("ob_type", ctypes.c_void_p), ("ob_size", ctypes.c_ssize_t), ("ob_digit", ctypes.c_uint32 * 1), ] long_obj = PyLongObject.from_address(addr) long_obj.ob_digit[0] = 3 assert 10 == 3 # using an auxiliary variable to prevent any inlining # done at the interpreter level before actually querying # the…

Okay, but this is going out of one's way to view the runtime itself as a C program and connect to it with the FFI. For that matter, the notion that the result of `id` (https://docs.python.org/3/library/functions.html#id) could sensibly be passed to `from_address` is an implementation detail. This is one reason the language suffers from not having a formal specification: it's unclear exactly how much of this madness alternative implementations like PyPy are expected to validate against. But I think people would agree that poking at the runtime's own memory cannot be expected to give deterministic results, and thus the implementation should in fact consider itself free to assume that isn't happening. (After all, we could take that further; e.g. what if we had another process do the dirty work?)

Re: Python performance myths and fairy tales

#157

Post by zahlman »

olejorgenb wrote: post #155
zahlman wrote: post #116

Earlier quoted context omitted.

> A decent case of Python 4.0? I think "Python 4.0" is going to have to be effectively a new language by a different team that simply happens to bear strong syntactic similarities. (And at least part of why that isn't already happening is that everyone keeps getting scared off by the scale of the task.) Thanks for the reminder that I never got around to checking out Julia.

Isn't that kinda what Mojo is?

I haven't tried it, but that matches my understanding, yeah.

Personally I'd be more interested in designing from scratch.

Re: Python performance myths and fairy tales

#158

Post by hansvm »

zahlman wrote: post #156
hansvm wrote: post #150

Earlier quoted context omitted.

import ctypes ten = 10 addr = id(ten) class PyLongObject(ctypes.Structure): _fields_ = [ ("ob_refcnt", ctypes.c_ssize_t), ("ob_type", ctypes.c_void_p), ("ob_size", ctypes.c_ssize_t), ("ob_digit", ctypes.c_uint32 * 1), ] long_obj = PyLongObject.from_address(addr) long_obj.ob_digit[0] = 3 assert 10 == 3 # using an auxiliary variable to prevent any inlining # done at the interpreter level before actually querying # the…

Okay, but this is going out of one's way to view the runtime itself as a C program and connect to it with the FFI. For that matter, the notion that the result of `id` ( https://docs.python.org/3/library/functions.html#id ) could sensibly be passed to `from_address` is an implementation detail. This is one reason the language suffers from not having a formal specification: it's unclear exactly how much of this madness…

Except, that sort of thing is important in places like gevent, pytest, and numba, and that functionality isn't easy to replace without a lot of additional language/stdlib work (no sane developer would reach for it if other APIs sufficed).

The absurd example of overwriting the literal `10` is "obviously" bad, but your assertion that the interpreter should be able to assume nobody is overwriting its memory isn't borne out in practice.

Re: Python performance myths and fairy tales

#159

Post by aeroevan »

01HNNWZ0MV43FF wrote: post #149
didip wrote: post #123

Earlier quoted context omitted.

These days it's "rewrite in Rust". Typically Python is just the entry and exit point (with a little bit of massaging), right? And then the overwhelming majority of the business logic is done in Rust/C++/Fortran, no?

With computer vision you end up wanting to read and write to huge buffers that aren't practical to serialize and are difficult to share. And even allocating and freeing multi-megabyte framebuffers at 60 FPS can put a little strain on the allocator, so you want to reuse them, which means you have to think about memory safety. That is probably why his demo was Sobel edge detection with Numpy. Sobel can run fast enough…

In the data science/engineering world apache arrow is the bridge between languages, so you don't actually need to serialize into language specific structures which is really nice

Re: Python performance myths and fairy tales

#160

Post by lkirk »

For me, in my use of Python as a data analysis language, it's not python's speed that is an annoyance or pain point, it's the concurrency story. Julia's built in concurrency primatives are much more ergonomic in my opinion.
Post reply on HN