Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python-based compiler achieves orders-of-magnitude speedups

news.mit.edu

Post reply on HN
Print view View original

11–20 of 193 posts

Re: Python-based compiler achieves orders-of-magnitude speedups

#11

Post by hgomersall »

wageslave99 wrote: post #5

I have no idea about compilers, so bear with me with this question: Can't we have a faster compiler for a subset of Python? I mean AFAIK the hard part of Python is that the language allows dynamic overwriting of attributes (or something like that). Is that feature actually needed for projects like Django, FastAPI, numpy, etc? Maybe I'm wrong, but the main idea I'd like to ask is, can we make a compiler for a subset o…

That would be things like cython (https://cython.org/) and rpython (https://rpython.readthedocs.io/en/latest/).

Re: Python-based compiler achieves orders-of-magnitude speedups

#12

Post by LtWorf »

wageslave99 wrote: post #5

I have no idea about compilers, so bear with me with this question: Can't we have a faster compiler for a subset of Python? I mean AFAIK the hard part of Python is that the language allows dynamic overwriting of attributes (or something like that). Is that feature actually needed for projects like Django, FastAPI, numpy, etc? Maybe I'm wrong, but the main idea I'd like to ask is, can we make a compiler for a subset o…

> Is that feature actually needed for projects like Django, FastAPI, numpy, etc?

Yes.

FastAPI depends on really slow pydantic (disclaimer: I'm the author of the faster typedload).

All those dynamic typechecking modules rely on the dynamic nature of the language. The alternative would be to having to generate code at compile time instead.

pydantic is also in the process of being rewritten in rust to be not so slow any longer, and in the process it will become incompatible with anything else than cpython (the normal python runtime). Which in turns means fastapi won't be able to run on anything else (unless they decouple from pydantic… which probably won't be easy).

Re: Python-based compiler achieves orders-of-magnitude speedups

#13

Post by LtWorf »

KyeRussell wrote: post #9
wageslave99 wrote: post #5

I have no idea about compilers, so bear with me with this question: Can't we have a faster compiler for a subset of Python? I mean AFAIK the hard part of Python is that the language allows dynamic overwriting of attributes (or something like that). Is that feature actually needed for projects like Django, FastAPI, numpy, etc? Maybe I'm wrong, but the main idea I'd like to ask is, can we make a compiler for a subset o…

Django makes use of a whole lot of Python’s fancypants stuff. For this reason, for instance, mypy doesn’t do well on Django projects without a purpose-built plugin. But I still take your point.

pydantic also requires a mypy plugin… But it's just how it was designed. I designed typedload with mypy in mind, so it kinda works (except for some limitations in the type system that don't allow to express some things, as of now).

Re: Python-based compiler achieves orders-of-magnitude speedups

#14

Post by rustybolt »

CJefferson wrote: post #6
wageslave99 wrote: post #5

I have no idea about compilers, so bear with me with this question: Can't we have a faster compiler for a subset of Python? I mean AFAIK the hard part of Python is that the language allows dynamic overwriting of attributes (or something like that). Is that feature actually needed for projects like Django, FastAPI, numpy, etc? Maybe I'm wrong, but the main idea I'd like to ask is, can we make a compiler for a subset o…

The problem turns out to be that it's maintaining the C-API compatibility which is the main thing which makes it hard to make Python fast, not the other stuff -- Javascript has most of the nasty things Python does, and it's plenty fast on browsers. However, maintaining C-API compatibility means you need to set up lots of data structures exactly how the C API requires, and maintaining and updating those ends up losing…

Sorry for the ignorance, but what do you mean by C-API here?

Normally I'd say you mean the interface you use when you call native machine code from Python, but I don't see how this would slow things down.

Re: Python-based compiler achieves orders-of-magnitude speedups

#16

Post by radicalbyte »

rustybolt wrote: post #14
CJefferson wrote: post #6

Earlier quoted context omitted.

The problem turns out to be that it's maintaining the C-API compatibility which is the main thing which makes it hard to make Python fast, not the other stuff -- Javascript has most of the nasty things Python does, and it's plenty fast on browsers. However, maintaining C-API compatibility means you need to set up lots of data structures exactly how the C API requires, and maintaining and updating those ends up losing…

Sorry for the ignorance, but what do you mean by C-API here? Normally I'd say you mean the interface you use when you call native machine code from Python, but I don't see how this would slow things down.

The interface to the C language. It is what makes Python fast - you write the code which needs to be fast in optimised C and call into it with Python. The Python code is then basically just the glue.

Re: Python-based compiler achieves orders-of-magnitude speedups

#18

Post by dagw »

wageslave99 wrote: post #5

I have no idea about compilers, so bear with me with this question: Can't we have a faster compiler for a subset of Python? I mean AFAIK the hard part of Python is that the language allows dynamic overwriting of attributes (or something like that). Is that feature actually needed for projects like Django, FastAPI, numpy, etc? Maybe I'm wrong, but the main idea I'd like to ask is, can we make a compiler for a subset o…

Can't we have a faster compiler for a subset of Python?

Check out Pythran, that is exactly what they've done.

Re: Python-based compiler achieves orders-of-magnitude speedups

#19

Post by DeathArrow »

ar9av wrote: post #2

There are other python implementations like pypy which includes a JIT (Just In Time compiler). There are other jit which can run with official python (cpython) like numba (not all code can be optimized, but if you only need optimize your hot code path). You can use a superset language of python called cython that generate C code. It can be used to generate C bindings or fast python (for cpython) modules implemented i…

Can you use Django with those optimisations or are they good mainly for scientific computing?

Re: Python-based compiler achieves orders-of-magnitude speedups

#20

Post by pdpi »

rustybolt wrote: post #14
CJefferson wrote: post #6

Earlier quoted context omitted.

The problem turns out to be that it's maintaining the C-API compatibility which is the main thing which makes it hard to make Python fast, not the other stuff -- Javascript has most of the nasty things Python does, and it's plenty fast on browsers. However, maintaining C-API compatibility means you need to set up lots of data structures exactly how the C API requires, and maintaining and updating those ends up losing…

Sorry for the ignorance, but what do you mean by C-API here? Normally I'd say you mean the interface you use when you call native machine code from Python, but I don't see how this would slow things down.

That native code still needs to be able to interact with your Python objects somehow. You can't change the API around PyObject without forcing all C libraries to make changes on their side, and that API forces you to expose things a certain way.
Post reply on HN