Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Python fast – Adventures with mypyc

blog.meadsteve.dev

Post reply on HN
Print view View original

21–30 of 95 posts

Re: Making Python fast – Adventures with mypyc

#21

Post by bsenftner »

Worth mentioning Taichi, a high-performance parallel programming language embedded in Python. I've experimented with it a bit, and high-performance is very true. One can pretty much just write ordinary Python, plus enhancing existing Python is not that difficult either.

From their docs:

You can write computationally intensive tasks in Python while obeying a few extra rules imposed by Taichi to take advantage of the latter's high performance. Use decorators @ti.func and @ti.kernel as signals for Taichi to take over the implementation of the tasks, and Taichi's just-in-time (JIT) compiler would compile the decorated functions to machine code. All subsequent calls to them are executed on multi-CPU cores or GPUs. In a typical compute-intensive scenario (such as a numerical simulation), Taichi can lead to a 50x~100x speed up over native Python code.

Taichi's built-in ahead-of-time (AOT) system also allows you to export your code as binary/shader files, which can then be invoked in C/C++ and run without the Python environment.

https://www.taichi-lang.org/

Re: Making Python fast – Adventures with mypyc

#22

Post by intrepidhero »

I like the concept of using mypyc to leverage type hints to compile python. But I was pretty frustrated recently when I got bit by a bug in mypyc[1] while trying to use black. Especially since I wasn't using mypyc myself and so didn't realize it was even in my dependency tree. Beware adding "alpha" quality software as a dependency to your supposedly production ready tool.

[1] https://github.com/psf/black/issues/2846

Re: Making Python fast – Adventures with mypyc

#23

Post by meadsteve »

cinntaile wrote: post #20
4140tm wrote: post #15

Earlier quoted context omitted.

I just found out about Lagom from this blog post and it's exactly what I have been looking for. All other Python options I've seen feel too involved or leak too much into your code. Lagom seems to balance everything just right. Thank you!

Haha, I can imagine Steve is quite pleased with this comment. You should look up the meaning of the (Swedish) word lagom.

Thanks 4140tm and thanks cinntaile. I was very pleased. That was very much the intention of the name

Re: Making Python fast – Adventures with mypyc

#24

Post by meadsteve »

alanwreath wrote: post #18
bobsmooth wrote: post #17

What's with this fascination with making python fast? It's not supposed to be fast, it's supposed to be simple. If you want speed use a compiled language. Trying to make python fast is like trying to strap a turbocharger to a tricycle.

I agree with you -- but I also don't say no to free food. I mean regardless of whether mypy was going to make my code run faster I would have used it for the shear confidence it gives wrt to my code correctness. The fact that I can use that same code (untouched) to speed it up... that's just means I get to have my cake and eat it too :P

Yeah this is exactly it for me. I already had type annotations and ran mypy to help with correctness. And I tried this out because it felt like a nice thing to get for free.

Re: Making Python fast – Adventures with mypyc

#25

Post by meadsteve »

atoav wrote: post #14

Anybody using Python and Rust should also check out maturin and pyo3. I run some (non public) Python modules created in Rust and both the performance and the testability is stellar.

Yeah these are great approaches too. I'd actually considered a rewrite of the core in rust before I went with mypyc. But it was nice not to have to do a rewrite.

Re: Making Python fast – Adventures with mypyc

#26

Post by atoav »

meadsteve wrote: post #25
atoav wrote: post #14

Anybody using Python and Rust should also check out maturin and pyo3. I run some (non public) Python modules created in Rust and both the performance and the testability is stellar.

Yeah these are great approaches too. I'd actually considered a rewrite of the core in rust before I went with mypyc. But it was nice not to have to do a rewrite.

Totally understandable. More options are better anyways.

Re: Making Python fast – Adventures with mypyc

#27

Post by kodablah »

atoav wrote: post #14

Anybody using Python and Rust should also check out maturin and pyo3. I run some (non public) Python modules created in Rust and both the performance and the testability is stellar.

We built the logic backing the Temporal Python SDK[0] in Rust and leverage PyO3 (and PyO3 Asyncio). Unfortunately Maturin didn't let us do some of the advanced things we needed to do for wheel creation (at the time, unsure now), so we use setuptools-rust with Poetry.

0 - https://github.com/temporalio/sdk-python

Re: Making Python fast – Adventures with mypyc

#28

Post by zzzeek »

peterkelly wrote: post #11
chrisseaton wrote: post #10

Earlier quoted context omitted.

That's not Node - that's V8. And it's possible to do the same thing for Python - there's nothing magic about JavaScript compared to Python - it's just a lot of engineering work to do it, which is beyond what this project's scope is. PyPy does it, but not inside standard Python.

I'm well aware of V8 and pypy. I also really like Python as a language, especially with mypy. It just makes me sad that in a world with multiple high-performance JIT engines (including pypy, for Python itself), the standard Python version that most people use is an interpreter. I know it's largely due to compatibility reasons (C extensions being deeply intertwined with CPython's API). There is a really important (if…

> It just makes me sad that in a world with multiple high-performance JIT engines (including pypy, for Python itself), the standard Python version that most people use is an interpreter. I know it's largely due to compatibility reasons (C extensions being deeply intertwined with CPython's API).

this is misleading, if one sees the phrase "interpreter" as that code is represented as syntax-derived trees or other datastructures which are then traversed at runtime to produce results - someone correct me if I'm wrong but this would apply to well known interpreted languages like Perl 5. cPython is a bytecode interpreter, not conceptually unlike the Java VM before JITs were added. It just happens to compile scripts to bytecode on the fly.

Re: Making Python fast – Adventures with mypyc

#29

Post by chrisseaton »

zzzeek wrote: post #28
peterkelly wrote: post #11

Earlier quoted context omitted.

I'm well aware of V8 and pypy. I also really like Python as a language, especially with mypy. It just makes me sad that in a world with multiple high-performance JIT engines (including pypy, for Python itself), the standard Python version that most people use is an interpreter. I know it's largely due to compatibility reasons (C extensions being deeply intertwined with CPython's API). There is a really important (if…

> It just makes me sad that in a world with multiple high-performance JIT engines (including pypy, for Python itself), the standard Python version that most people use is an interpreter. I know it's largely due to compatibility reasons (C extensions being deeply intertwined with CPython's API). this is misleading, if one sees the phrase "interpreter" as that code is represented as syntax-derived trees or other datast…

Bytecode is just another data structure that you traverse at runtime to produce results. It's a postfix transformation of the AST. It's still an interpreter.

Re: Making Python fast – Adventures with mypyc

#30

Post by an1sotropy »

I'm curious how to compare this with a PyPy FAQ: https://doc.pypy.org/en/latest/faq.html#would-type-annotatio... which describes a bit about why type hints aren't as helpful to optimize code under PyPy as one (including myself) might think.

Can someone explain more about how mypyc is in a better position to produce better optimizations than pypy, or am I confused about this?

Post reply on HN