Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python-based compiler achieves orders-of-magnitude speedups

news.mit.edu

Post reply on HN
Print view View original

141–150 of 193 posts

Re: Python-based compiler achieves orders-of-magnitude speedups

#141

Post by anakaine »

WhyCause wrote: post #135
plonk wrote: post #126

Earlier quoted context omitted.

Seconded, I deploy large packages with gigabytes of deep learning and GIS dependencies in single executables with Nuitka and it works very well. Also handles including data files into the executable if needed.

Out of curiosity, are the GIS dependencies the proprietary ones ( cough ESRI cough )?

Ha ing been down a similar path, this whole thing works so much better if you don't 'import arcpy'. Licencing issues aside, you've often got faster tools in shapely, fiona, geopandas, rasterio, xarray.

Re: Python-based compiler achieves orders-of-magnitude speedups

#142

Post by paro_nej »

dbrueck wrote: post #138
paro_nej wrote: post #61

> Faster than the speed of C One must note that this is impossible, unless you have chosen to handicap the C-implementations while benchmarking. Borderline unethical IMO to put forth such a claim.

Pretty much every JIT-enabled language has this as at least a theoretical advantage over C. So not impossible and therefore not unethical.

They aren't JIT. They're doing AoT AFAICT.

Re: Python-based compiler achieves orders-of-magnitude speedups

#143

Post by echelon »

brucethemoose2 wrote: post #106
rightbyte wrote: post #102

Earlier quoted context omitted.

Ye well if you remove the dynamic feutures of a dynamic language it gets fast. It would be really impressive of they can achieve those feutures with the sameish speed.

I dont necessarily need all that dynamism though, and would happily use a Python subset that removed some stuff (and forced type hinting) in exchange for better compilation. Yes there are already subsets like this, but its not as helpful if it isnt standard.

Starlark comes to mind, but that's probably too limited.

Re: Python-based compiler achieves orders-of-magnitude speedups

#144

Post by dbrueck »

paro_nej wrote: post #142
dbrueck wrote: post #138

Earlier quoted context omitted.

Pretty much every JIT-enabled language has this as at least a theoretical advantage over C. So not impossible and therefore not unethical.

They aren't JIT. They're doing AoT AFAICT.

I must have misunderstood what you were objecting to then, my bad. What claim are they making that is so impossible that it borders on being unethical?

I mentioned JIT because it seems to be based on a similar principle at least, that of optimizing things on the programmer's behalf by looking at the program's usage and not just by looking at how to speed up the code generally.

Re: Python-based compiler achieves orders-of-magnitude speedups

#145

Post by paro_nej »

dbrueck wrote: post #144
paro_nej wrote: post #142

Earlier quoted context omitted.

They aren't JIT. They're doing AoT AFAICT.

I must have misunderstood what you were objecting to then, my bad. What claim are they making that is so impossible that it borders on being unethical? I mentioned JIT because it seems to be based on a similar principle at least, that of optimizing things on the programmer's behalf by looking at the program's usage and not just by looking at how to speed up the code generally.

My claim is that there is no additional information that Python provides as opposed to C that would make it faster. And hence, the only conclusion I have is either they have supercharged their compiler for that particular benchmark OR they have chosen to handicap C as once can express the computation in C that emits the same assembly that they lowered to and hence my point on handicapping the C benchmark.

Re: Python-based compiler achieves orders-of-magnitude speedups

#146

Post by ptx »

sirwhinesalot wrote: post #85
adammarples wrote: post #83

Earlier quoted context omitted.

I took their fib example and ran it in mypyc too out of curiosity. I got speedups of ~10x rather than codon's 100x. Still pretty good, I like mypyc.

mypyc keeps Python's "BigIntegers", unicode string implementation, reference counting, and has little to no floating point-related optimizations yet. It prioritizes compatibility over overall performance, so I'm not surprised. I was also disappointed at how poor mypyc is at compiling across multiple files, but that they can fix at some point. The BigInteger "issue" pretty much makes something like Fibonacci a worst c…

Looks like they added support for unboxed floats yesterday (not sure about integers): https://github.com/python/mypy/commit/d05974b9b099ec755fd1c6...

Re: Python-based compiler achieves orders-of-magnitude speedups

#147

Post by afdbcreid »

jstarfish wrote: post #133
DeathArrow wrote: post #19

Earlier quoted context omitted.

Can you use Django with those optimisations or are they good mainly for scientific computing?

Pypy is great but I didn't find it very useful with Django. Quick, transactional HTTP exchanges (GET, POST, etc.) aren't really its thing-- there's no time for the compiler to get warmed up; the request is complete before pypy has gotten out of bed. But if you have to do really complex view rendering (graphs or something) where it would take cpython ~10s or more to process, then pypy will leave cpython in the dust.

I've never used PyPy, but shouldn't it warm after the Nth request even if you don't have loops?

Re: Python-based compiler achieves orders-of-magnitude speedups

#148

Post by afdbcreid »

ActorNightly wrote: post #119
netbioserror wrote: post #48

Preface: I don't just want to crap on Python here and sell Nim. I like Python, and still use it. But it still shocks me just how much money and manpower is thrown at trying to bikeshed and optimize and compile Python and its libraries, while the Nim compiler is essentially a community hobby project that has made the concept of a "compiled Python" a reality already. The orders of magnitude in scale difference, and the…

>I'm kind of starting to see what Guido is talking about when he says Python is a legacy language that's probably on its way out. Even in the interpreted world, languages like Janet and other newcomers are performing fascinating experiments, often doing more with less. Wow, what a way to mischaracterize what Guido said. His point was about languages evolving to be more abstract than Python or any of the ones you ment…

Rust is different because it is trying to answer real needs. It is not going to replace Python, and if you're seriously thinking about writing your code in Python you probably shouldn't be thinking of writing it in Rust. There will likely also be less Rust code than Python code. But it can replace C/C++, not completely and not in the near future, but it is possible.

Re: Python-based compiler achieves orders-of-magnitude speedups

#149

Post by yardshop »

BiteCode_dev wrote: post #121

Since this is highly incompatible with most python ecosystem right now, may I plug nuitka? https://nuitka.net/index.html It's a compiler for python code that can create stand alone executables, and up to 4 times the speed of the initial code. Best of all, it's extremely reliable, with a high level of support of event the tricky things like the scientic and gui stacks.

> high level of support of event the tricky things like the scientic and gui stacks

Could it compile an app that uses Pillow and AggDraw and ReportLab and OpenPyXL with a TKInter GUI into a standalone app I can give to a coworker? That would be extremely useful!

Re: Python-based compiler achieves orders-of-magnitude speedups

#150

Post by lozenge »

robertlagrant wrote: post #94
jstx1 wrote: post #25

Earlier quoted context omitted.

Have a look at Cinder - https://github.com/facebookincubator/cinder - it's Meta's performance oriented fork of CPython that they use to run Instagram (which is a big Django app).

I always wondered with Cinder why they didn't turbocharge PyPy development instead.

With a codebase of any significant size the priority is always to maintain compatibility while improving performance.

If you start with an incompatible, highly performant interpreter, the compatibility "distance" is difficult to measure and could create unknown performance cost. For example, PyPy doesn't support C modules due to the differing memory layout.

Post reply on HN