Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nuitka: An extremely compatible Python compiler

nuitka.net

Post reply on HN
Print view View original

71–80 of 88 posts

Re: Nuitka: An extremely compatible Python compiler

#71

Post by tylerhou »

eatonphil wrote: post #11

This project is super impressive. It makes me wonder why there's not a similarly mature JavaScript AOT compiler implementation for use in game development or other places where you want performance better than a VM and you're not allowed to JIT. A small recommendation for these release pages would be to link to the tag in whatever source host you're using! I just want to see the code.

The main slowdown in JavaScript is because it is inherently so dynamic — all function calls are "virtual," properties are string lookups, you need a lot of indirection (boxing) to store objects. Without actually running the program on some input, it is infeasible to determine what those indirections will resolve to. So static compilation is useless (or would produce instructions very similar to what an interpreter would execute). That's why we usually interpret the code (and JIT it to optimize places where the program is not actually dynamic).

Theoretically you could "precompile" JavaScript by running your code on some known input and caching the VM's state. Then you can "hydrate" your VM with that state. But this would only speed up the VM warmup (which is not that long) and requires significant browser/VM buy-in, so it is probably infeasible.

The only way to produce better performance in JavaScript is to restrict it to a less-dynamic subset (e.g. objects must have static keys) and add at least some static typing so that a compiler can resolve some references ahead of time. That's why asm.js was a thing. But asm.js has been superseded by WASM, which better in most regards.

Re: Nuitka: An extremely compatible Python compiler

#72

Post by int_19h »

nextaccountic wrote: post #60
adsharma wrote: post #52

Extreme compatibility has a cost. If you're willing to give up some compatibility, you could get faster code that can be checked statically by transpiling to one of the many great choices from a set of statically typed languages. This is the approach taken by py14, pyrs and their successor py2many, on which I've been working for some months.

But it was the lack of 100% compatibility that hampered pypy's adoption. Anyway what are your incompatibilities?

It seems to be fully statically typed, for starters:

https://github.com/adsharma/py2many/blob/main/doc/langspec.m...

But, given LuaJIT and V8, it's obviously possible to generate high-performance code without such restrictions.

As for PyPy adoption, it's hindered more by the lack of API compatibility with CPython native extensions.

Re: Nuitka: An extremely compatible Python compiler

#73

Post by emmelaich »

nextlevelwizard wrote: post #41
eatonphil wrote: post #24

Earlier quoted context omitted.

> Also is this actually faster than just python? I tried it out with some bad expensive looping using lists with python3.9 and it was 100ms slower (1.3sec vs 1.4sec). Mind sharing that code? I'm sure someone would like to be able to look into this.

I don't have it anymore, but it was something stupid like: import time start = time.time() list = [] for i in range(10000000): list.append(i) sum = 0 for item in list: sum = sum + i print(time.time() - start) I am not superduper python expert, but I know that arrays/lists are the stupidest thing you can use so that's why I did stupid things with them. So before someone posts more optimized version the code was suppos…

Maybe it was the size of the list.

Have you tried xrange?

Re: Nuitka: An extremely compatible Python compiler

#74

Post by vlovich123 »

tylerhou wrote: post #71
eatonphil wrote: post #11

This project is super impressive. It makes me wonder why there's not a similarly mature JavaScript AOT compiler implementation for use in game development or other places where you want performance better than a VM and you're not allowed to JIT. A small recommendation for these release pages would be to link to the tag in whatever source host you're using! I just want to see the code.

The main slowdown in JavaScript is because it is inherently so dynamic — all function calls are "virtual," properties are string lookups, you need a lot of indirection (boxing) to store objects. Without actually running the program on some input, it is infeasible to determine what those indirections will resolve to. So static compilation is useless (or would produce instructions very similar to what an interpreter wo…

Have you seen the work Fastly discussed about their AOT compiler? Seems like an interesting approach that maybe could be used to preprocess code into a more efficient form before running it in the cloud. The challenge is that JS engines are primarily focused on the browser and such an optimization opportunity isn’t interesting there - the majority of development is done by browser vendors.

Re: Nuitka: An extremely compatible Python compiler

#75

Post by kevin_thibedeau »

gnulinux wrote: post #61
kevin_thibedeau wrote: post #46

Earlier quoted context omitted.

You'd be stuck with float numerics everywhere because there's no fallback to recompile if an attempt to use integer fails.

compile both, dispatch dynamically in runtime.

It wouldn't be a "both" situation. Every number in a block of code may or may not be representable as an integer. You can't generate code for every possible combination.

Re: Nuitka: An extremely compatible Python compiler

#76

Post by adsharma »

nextaccountic wrote: post #60
adsharma wrote: post #52

Extreme compatibility has a cost. If you're willing to give up some compatibility, you could get faster code that can be checked statically by transpiling to one of the many great choices from a set of statically typed languages. This is the approach taken by py14, pyrs and their successor py2many, on which I've been working for some months.

But it was the lack of 100% compatibility that hampered pypy's adoption. Anyway what are your incompatibilities?

Doesn't use python's C API or runtime. So no metaclasses, monkey patching, eval(). All types must be known at compile time or it should be possible to infer them.

There are several challenges to solve - translating stdlib of one language to another (see plugins.py) and bridge semantic gaps (rust doesn't like mutable global state, python has them).

Re: Nuitka: An extremely compatible Python compiler

#77

Post by adsharma »

int_19h wrote: post #72
nextaccountic wrote: post #60

Earlier quoted context omitted.

But it was the lack of 100% compatibility that hampered pypy's adoption. Anyway what are your incompatibilities?

It seems to be fully statically typed, for starters: https://github.com/adsharma/py2many/blob/main/doc/langspec.m... But, given LuaJIT and V8, it's obviously possible to generate high-performance code without such restrictions. As for PyPy adoption, it's hindered more by the lack of API compatibility with CPython native extensions.

Yes - Julia also has a pretty good JIT too, but doesn't have a great story on AOT compiled binaries.

If you're finding it easy to write a quick command line tool in python, but end up rewriting it in another language for size/performance/self contained deployment, this may be a good fit.

Re: Nuitka: An extremely compatible Python compiler

#78

Post by cultofmetatron »

Eikon wrote: post #35
spenczar5 wrote: post #32

Earlier quoted context omitted.

What sort of web applications do you work on where Python interpreter speed is the limiting factor? Usually, those applications are constrained by network throughput and context switches and page faults.

This argument is so tiring. That's part of why websites are so slow despite having insane hardware at their disposal. At my company, for a web app we run in production, we strive to get every response out of our infrastructure in under a millisecond, everything above that except for a few select endpoints is considered as a bug. By using sensible technology choices, it's not even that hard to do. A RDMS like postgres…

same at my startup. I made the descision to use elixir because its so fast while still having the readability of ruby or python. end result saves us money on hardware. more importantly, the customers feel like they are using a responsive and fast application.

Re: Nuitka: An extremely compatible Python compiler

#79

Post by ficklepickle »

> Retry downloads without SSL if that fails, as some Python do not have working SSL. Fixed in 0.6.14.5 already.

Is this a good idea? If I understand correctly, if an HTTPS request fails, it falls back to HTTP. So an attacker could just block the HTTPS request, then MITM the HTTP request.

Re: Nuitka: An extremely compatible Python compiler

#80

Post by nextlevelwizard »

emmelaich wrote: post #73
nextlevelwizard wrote: post #41

Earlier quoted context omitted.

I don't have it anymore, but it was something stupid like: import time start = time.time() list = [] for i in range(10000000): list.append(i) sum = 0 for item in list: sum = sum + i print(time.time() - start) I am not superduper python expert, but I know that arrays/lists are the stupidest thing you can use so that's why I did stupid things with them. So before someone posts more optimized version the code was suppos…

Maybe it was the size of the list. Have you tried xrange?

No and I won’t. There is no reason why the compiled code shouldn’t optimize both loops away since they do nothing. And in any case it should at least be faster if it doesn’t optimize the code away.

And I am fairly sure xrange() is not in Python3

Post reply on HN