Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nuitka: An extremely compatible Python compiler

nuitka.net

Post reply on HN
Print view View original

81–88 of 88 posts

Re: Nuitka: An extremely compatible Python compiler

#81

Post by pjmlp »

mappu wrote: post #70
formerly_proven wrote: post #34

Earlier quoted context omitted.

Which probably works by stapling bytecode to a precompiled interpreter. A tried and true strategy, but not necessarily what people want when asking for AOT compilation.

It's only a few steps away from what Graal/Truffle do: unroll the interpreter loop over the bytecode, then keep re-running dead code elimination and inlining passes - you end up with something very close to optimized native output.

Specially since Graal/Truffle do include a JIT, which even has better optimization heuristics than Hotspot.

Re: Nuitka: An extremely compatible Python compiler

#82

Post by scbrg »

nextlevelwizard wrote: post #41
eatonphil wrote: post #24

Earlier quoted context omitted.

> Also is this actually faster than just python? I tried it out with some bad expensive looping using lists with python3.9 and it was 100ms slower (1.3sec vs 1.4sec). Mind sharing that code? I'm sure someone would like to be able to look into this.

I don't have it anymore, but it was something stupid like: import time start = time.time() list = [] for i in range(10000000): list.append(i) sum = 0 for item in list: sum = sum + i print(time.time() - start) I am not superduper python expert, but I know that arrays/lists are the stupidest thing you can use so that's why I did stupid things with them. So before someone posts more optimized version the code was suppos…

This is where Python being dynamic makes things like this hard. You have no idea what range does, as that name may point to any user defined object by the time the code is being executed. So no, I don't think it can "optimize both loops away".

Come to think of it, you don't particularly know what sum = sum + i is doing either. There could be an __radd__ implementation in whatever your custom range spits out that indexes the WWW, renders a Mandelbrot or whatever.

And yes, of course overloading range and __radd__ that way would also be considered stupid, but if the loops were optimized away, it wouldn't be a drop in replacement for Python.

Re: Nuitka: An extremely compatible Python compiler

#83

Post by nextaccountic »

adsharma wrote: post #76
nextaccountic wrote: post #60

Earlier quoted context omitted.

But it was the lack of 100% compatibility that hampered pypy's adoption. Anyway what are your incompatibilities?

Doesn't use python's C API or runtime. So no metaclasses, monkey patching, eval(). All types must be known at compile time or it should be possible to infer them. There are several challenges to solve - translating stdlib of one language to another (see plugins.py) and bridge semantic gaps (rust doesn't like mutable global state, python has them).

Does this mean that existing Python code - like dependencies on pypi won't be generally compatible?

Without metaclasses and dynamic types I think it's better to call the language something else, and say it's Python-inspired (like Elm is Haskell-inspired)

Re: Nuitka: An extremely compatible Python compiler

#84

Post by adsharma »

nextaccountic wrote: post #83
adsharma wrote: post #76

Earlier quoted context omitted.

Doesn't use python's C API or runtime. So no metaclasses, monkey patching, eval(). All types must be known at compile time or it should be possible to infer them. There are several challenges to solve - translating stdlib of one language to another (see plugins.py) and bridge semantic gaps (rust doesn't like mutable global state, python has them).

Does this mean that existing Python code - like dependencies on pypi won't be generally compatible? Without metaclasses and dynamic types I think it's better to call the language something else, and say it's Python-inspired (like Elm is Haskell-inspired)

Renaming the language in the future is still a possibility. For now, all the tests here:

https://github.com/adsharma/py2many/tree/main/tests/cases

are run with cpython interpreter and verified for compatibility.

Even when there is a desire to innovate (design by contract or pattern matching as an expression), hope to do so without breaking cpython (as long as you stick to the subset).

Re: Nuitka: An extremely compatible Python compiler

#85

Post by nextlevelwizard »

scbrg wrote: post #82
nextlevelwizard wrote: post #41

Earlier quoted context omitted.

I don't have it anymore, but it was something stupid like: import time start = time.time() list = [] for i in range(10000000): list.append(i) sum = 0 for item in list: sum = sum + i print(time.time() - start) I am not superduper python expert, but I know that arrays/lists are the stupidest thing you can use so that's why I did stupid things with them. So before someone posts more optimized version the code was suppos…

This is where Python being dynamic makes things like this hard. You have no idea what range does, as that name may point to any user defined object by the time the code is being executed. So no, I don't think it can "optimize both loops away". Come to think of it, you don't particularly know what sum = sum + i is doing either. There could be an __radd__ implementation in whatever your custom range spits out that inde…

the compiler can literally see that the `list` variable is only used to generate the `sum` variable and that the `sum` is never used.

Since the `sum` is never accessed by anyone it can be optimized away and now since the only accesser of `list` is gone it too can be optimized away.

Thus both loops most certainly CAN and SHOULD be removed from the final compiled code. Only reason Python can't do it is because it is not compiled, so it has to actually run through the code at runtime and see what happens.

This is basic static analysis and any other compiler for any other language would do this. This is literally why we have keywords like `volatile` in compiled languages.

I don't understand your first argument at all. Why wouldn't the compiler know what the standard `range()` function does? The second one only makes sense since we aren't type hinting, however that too could be reversed by analyzing the code.

If we can't trust the functions in standard library then what is the benefit of compiling the code?

Re: Nuitka: An extremely compatible Python compiler

#86

Post by scbrg »

nextlevelwizard wrote: post #85
scbrg wrote: post #82

Earlier quoted context omitted.

This is where Python being dynamic makes things like this hard. You have no idea what range does, as that name may point to any user defined object by the time the code is being executed. So no, I don't think it can "optimize both loops away". Come to think of it, you don't particularly know what sum = sum + i is doing either. There could be an __radd__ implementation in whatever your custom range spits out that inde…

the compiler can literally see that the `list` variable is only used to generate the `sum` variable and that the `sum` is never used. Since the `sum` is never accessed by anyone it can be optimized away and now since the only accesser of `list` is gone it too can be optimized away. Thus both loops most certainly CAN and SHOULD be removed from the final compiled code. Only reason Python can't do it is because it is no…

> I don't understand your first argument at all. Why wouldn't the compiler know what the standard `range()` function does? The second one only makes sense since we aren't type hinting, however that too could be reversed by analyzing the code.

I'm sure it knows what the standard range() function does. It just doesn't know that that's the standard range function you use when the code is being run. Any Python program is also a module. Any module can be imported. Any imported module's global namespace is unknown at the time you write the program.

Imagine I have your code, above, in a module called nextlevelwizard, and I write my own little progam as this:

    import builtins
    def myrange(n):
        cur = 0
        while cur
Now, suddenly, your loop has a side effect. Side effects cannot be "optimized away" - the whole point of programs are generally to produce some sort of side effect!

This is a silly example, of course, but the dynamic nature of Python means that it really is very hard to skip steps you assume are irrelevant.

Re: Nuitka: An extremely compatible Python compiler

#87

Post by nextlevelwizard »

scbrg wrote: post #86
nextlevelwizard wrote: post #85

Earlier quoted context omitted.

the compiler can literally see that the `list` variable is only used to generate the `sum` variable and that the `sum` is never used. Since the `sum` is never accessed by anyone it can be optimized away and now since the only accesser of `list` is gone it too can be optimized away. Thus both loops most certainly CAN and SHOULD be removed from the final compiled code. Only reason Python can't do it is because it is no…

> I don't understand your first argument at all. Why wouldn't the compiler know what the standard `range()` function does? The second one only makes sense since we aren't type hinting, however that too could be reversed by analyzing the code. I'm sure it knows what the standard range() function does. It just doesn't know that that's the standard range function you use when the code is being run. Any Python program is…

Expect the compiler should import whatever module you are using and analyze that module's code just like in any compiled language.

Even in the example you gave it could replace the loop with a bunch of print calls since the result (yields) do nothing.

Compilers are hard.

Post reply on HN