Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Python fast – Adventures with mypyc

blog.meadsteve.dev

Post reply on HN
Print view View original

31–40 of 95 posts

Re: Making Python fast – Adventures with mypyc

#31

Post by an1sotropy »

chrisseaton wrote: post #29
zzzeek wrote: post #28

Earlier quoted context omitted.

> It just makes me sad that in a world with multiple high-performance JIT engines (including pypy, for Python itself), the standard Python version that most people use is an interpreter. I know it's largely due to compatibility reasons (C extensions being deeply intertwined with CPython's API). this is misleading, if one sees the phrase "interpreter" as that code is represented as syntax-derived trees or other datast…

Bytecode is just another data structure that you traverse at runtime to produce results. It's a postfix transformation of the AST. It's still an interpreter.

Well, ok, but then isn't a CPU is also just an interpreter, traversing the object code text of compiled code?

Re: Making Python fast – Adventures with mypyc

#32

Post by detaro »

an1sotropy wrote: post #30

I'm curious how to compare this with a PyPy FAQ: https://doc.pypy.org/en/latest/faq.html#would-type-annotatio... which describes a bit about why type hints aren't as helpful to optimize code under PyPy as one (including myself) might think. Can someone explain more about how mypyc is in a better position to produce better optimizations than pypy, or am I confused about this?

pypy argues that considering type annotations gives them less useful data than their existing tracing does, and thus pypy wouldn't be faster if it considered them. Something like mypyc by design has no chance of doing tracing, and thus has to work with annotations. (I also don't see where you get the claim from that that mypyc has better optimizations than pypy? But the two also follow different designs, so they might be good at different things)

Re: Making Python fast – Adventures with mypyc

#33

Post by chrisseaton »

an1sotropy wrote: post #31
chrisseaton wrote: post #29

Earlier quoted context omitted.

Bytecode is just another data structure that you traverse at runtime to produce results. It's a postfix transformation of the AST. It's still an interpreter.

Well, ok, but then isn't a CPU is also just an interpreter, traversing the object code text of compiled code?

We don't normally call hardware or firmware implementations an 'interpreter'.

Almost all execution techniques include some combination of compilation and interpretation. Even some ASTs include aspects of transformation to construct them from the source code, which we could call a compiler. Native compilers sometimes have to interpret metadata to do things like roll forward for deoptimisation.

But most people in the field would describe CPython firmly as an 'interpreter'.

Re: Making Python fast – Adventures with mypyc

#34

Post by mixmastamyk »

chrisseaton wrote: post #10
peterkelly wrote: post #9

mypyc is cool and all, but I can't help thinking about how Node just JITs everything automatically without the need for any special steps like this.

That's not Node - that's V8. And it's possible to do the same thing for Python - there's nothing magic about JavaScript compared to Python - it's just a lot of engineering work to do it, which is beyond what this project's scope is. PyPy does it, but not inside standard Python.

Python is a bit more dynamic than JS, which makes it uniquely hard to optimize. There is more improvement to be done however and is being done.

Re: Making Python fast – Adventures with mypyc

#35

Post by chrisseaton »

mixmastamyk wrote: post #34
chrisseaton wrote: post #10

Earlier quoted context omitted.

That's not Node - that's V8. And it's possible to do the same thing for Python - there's nothing magic about JavaScript compared to Python - it's just a lot of engineering work to do it, which is beyond what this project's scope is. PyPy does it, but not inside standard Python.

Python is a bit more dynamic than JS, which makes it uniquely hard to optimize. There is more improvement to be done however and is being done.

Right, but I think we know how to optimise all these things. It's all solved problems.

Re: Making Python fast – Adventures with mypyc

#36

Post by an1sotropy »

detaro wrote: post #32
an1sotropy wrote: post #30

I'm curious how to compare this with a PyPy FAQ: https://doc.pypy.org/en/latest/faq.html#would-type-annotatio... which describes a bit about why type hints aren't as helpful to optimize code under PyPy as one (including myself) might think. Can someone explain more about how mypyc is in a better position to produce better optimizations than pypy, or am I confused about this?

pypy argues that considering type annotations gives them less useful data than their existing tracing does, and thus pypy wouldn't be faster if it considered them. Something like mypyc by design has no chance of doing tracing, and thus has to work with annotations. (I also don't see where you get the claim from that that mypyc has better optimizations than pypy? But the two also follow different designs, so they migh…

sorry I didn't mean to claim that mypyc does have better optimizations, I meant to be asking if that was possible. My superficial read was: this post about mypyc goes from type hints to compiling to "faster", and then I remembered the pypy FAQ which says type hints didn't help with that.

But if mypyc has no runtime information to go on (which pypy does have), then certainly having some type information is better than having none.

Re: Making Python fast – Adventures with mypyc

#37

Post by mixmastamyk »

chrisseaton wrote: post #35
mixmastamyk wrote: post #34

Earlier quoted context omitted.

Python is a bit more dynamic than JS, which makes it uniquely hard to optimize. There is more improvement to be done however and is being done.

Right, but I think we know how to optimise all these things. It's all solved problems.

A few things are impossible without changing/subsetting the language. What I was trying to get at.

Re: Making Python fast – Adventures with mypyc

#38

Post by atoav »

kodablah wrote: post #27
atoav wrote: post #14

Anybody using Python and Rust should also check out maturin and pyo3. I run some (non public) Python modules created in Rust and both the performance and the testability is stellar.

We built the logic backing the Temporal Python SDK[0] in Rust and leverage PyO3 (and PyO3 Asyncio). Unfortunately Maturin didn't let us do some of the advanced things we needed to do for wheel creation (at the time, unsure now), so we use setuptools-rust with Poetry. 0 - https://github.com/temporalio/sdk-python

I had no issues with the standard maturin way of building wheels – but my requirements were not special at all. I also did this maybe 5 months ago, so maybe it has indeed gotten better, I cannot tell.

Re: Making Python fast – Adventures with mypyc

#39

Post by chrisseaton »

mixmastamyk wrote: post #37
chrisseaton wrote: post #35

Earlier quoted context omitted.

Right, but I think we know how to optimise all these things. It's all solved problems.

A few things are impossible without changing/subsetting the language. What I was trying to get at.

What things are you thinking of?

(Not trying to interrogate you or prove you wrong, but I've got an interest in optimising very difficult meta-programming patterns.)

Post reply on HN