Live data from Hacker News

Making Python fast – Adventures with mypyc

blog.meadsteve.dev

11–20 of 95 posts

Re: Making Python fast – Adventures with mypyc

#11

mypyc is cool and all, but I can't help thinking about how Node just JITs everything automatically without the need for any special steps like this.

That's not Node - that's V8. And it's possible to do the same thing for Python - there's nothing magic about JavaScript compared to Python - it's just a lot of engineering work to do it, which is beyond what this project's scope is. PyPy does it, but not inside standard Python.

I'm well aware of V8 and pypy. I also really like Python as a language, especially with mypy.

It just makes me sad that in a world with multiple high-performance JIT engines (including pypy, for Python itself), the standard Python version that most people use is an interpreter. I know it's largely due to compatibility reasons (C extensions being deeply intertwined with CPython's API).

There is a really important (if not "magic") difference between JavaScript and Python. JS has always (well, since IE added support) been a language with multiple widely-used implementations in the wild, which has prevented the emergence of a third-party package ecosystem which is heavily tied to one particular implementation. Python on the other hand is for a large proportion of the userbase considered CPython, with alternate implementations being second class citizens, despite some truly impressive efforts on the latter.

The fact that packages written in JS are not tied to (or at least work best with) a single implementation is also what made it possible for developers of JS engines to experiment with different implementation approaches, including JIT. While I'm not intimately familiar with writing native extension modules for Node (having dabbled only a little), my understanding is the API surface is much narrower than Python, allowing for changes in the engine that avoid breaking APIs. But there is less need for native modules in JS, because of the presence of JIT in all major engines.

Re: Making Python fast – Adventures with mypyc

#12
post #5

I didn't know that you can compile individual modules with mypyc. That's very interesting since it allows a gradual adoption of the compiler, which really helps with big codebases. Do you know if there are any requirements for which modules can be compiled? E.g. can they be imported in other modules or do they have to be a leaf in the import tree/graph ?

Having read through the docs Mypyc has a concept of "native classes" and python classes, and it looks like you can use a "native" (compiles) class from regular python and vice-versa.

So my reading is that it should be pretty seamless.

Re: Making Python fast – Adventures with mypyc

#15

I recently experimented with using mypyc to make some of my python a little faster. I was pleasantly surprised with how well it worked for very little code change so I thought I'd share my experiences. The blog post wanders around a little because I had to add setuptools and wheel building as my project had previously skipped this.

I just found out about Lagom from this blog post and it's exactly what I have been looking for.

All other Python options I've seen feel too involved or leak too much into your code. Lagom seems to balance everything just right.

Thank you!

Re: Making Python fast – Adventures with mypyc

#16

mypyc is cool and all, but I can't help thinking about how Node just JITs everything automatically without the need for any special steps like this.

That's what Microsoft is paying Guido for, for the next versions of python.

I think that's not really the plan - they're talking about just basic template compilation, nothing like V8 https://github.com/markshannon/faster-cpython/blob/master/pl....

Re: Making Python fast – Adventures with mypyc

#18

What's with this fascination with making python fast? It's not supposed to be fast, it's supposed to be simple. If you want speed use a compiled language. Trying to make python fast is like trying to strap a turbocharger to a tricycle.

I agree with you -- but I also don't say no to free food.

I mean regardless of whether mypy was going to make my code run faster I would have used it for the shear confidence it gives wrt to my code correctness. The fact that I can use that same code (untouched) to speed it up... that's just means I get to have my cake and eat it too :P

Re: Making Python fast – Adventures with mypyc

#19
post #14

Anybody using Python and Rust should also check out maturin and pyo3. I run some (non public) Python modules created in Rust and both the performance and the testability is stellar.

I have the exact same experience. Both Maturin and PyO3 have been a game changer for the work that I have been doing lately. It works so seamlessly.

Re: Making Python fast – Adventures with mypyc

#20
post #15

I recently experimented with using mypyc to make some of my python a little faster. I was pleasantly surprised with how well it worked for very little code change so I thought I'd share my experiences. The blog post wanders around a little because I had to add setuptools and wheel building as my project had previously skipped this.

I just found out about Lagom from this blog post and it's exactly what I have been looking for. All other Python options I've seen feel too involved or leak too much into your code. Lagom seems to balance everything just right. Thank you!

Haha, I can imagine Steve is quite pleased with this comment. You should look up the meaning of the (Swedish) word lagom.
Post reply on HN