Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Codon: A high-performance Python-like compiler using LLVM

github.com

Post reply on HN
Print view View original

151–160 of 184 posts

Re: Codon: A high-performance Python-like compiler using LLVM

#151

Post by JonChesterfield »

darawk wrote: post #51
ot wrote: post #34

Can we change the title to say Python-like or something similar? Based on the comments so far, it seems that the detail that it compiles its own Python-inspired language, not actual Python, is lost on many. EDIT: A list of differences here: https://docs.exaloop.io/codon/general/differences The summary minimizes with "many Python programs will work with few if any modifications", but it actually looks like a substanti…

That list actually seems genuinely pretty minimal. Reading your comment I was expecting a long major list of changes, but it's only 3 things, most of which seem relatively unlikely to impact most programs, with the possible exception of dictionary sort order.

That list is minimal. Elsewhere there's the no heterogenous lists and no biginteger restrictions, and it looks like import doesn't work either. Presumably no heterogenous dictionaries either - so not only unordered, but also simply typed.

Re: Codon: A high-performance Python-like compiler using LLVM

#152

Post by pkkm »

mytherin wrote: post #117
jnxx wrote: post #91

Just out of curiosity: Why is it possible to compile Common Lisp Code (or Scheme, or Clojure) to high-performance native or jit-compiled code, but not Python? It is said that "Python is too dynamic", but is not everything in Lisp dynamic, too? And none of these languages is less powerful than Lisp, lack Unicode support, or whatever, so this can't be the reason.

It is possible to JIT compile Python just fine. There are projects like PyPy that have been doing this for a long time [1]. The reason these alternative projects never take off is because many of Python's most used libraries are written against CPython's C API. This API is giant, and exposes all of the nitty gritty implementation details of the CPython interpreter. As a result, changing anything significant about the…

The HPy project [0] looks like a promising way out of this.

[0] https://hpyproject.org/

Re: Codon: A high-performance Python-like compiler using LLVM

#153

Post by eddsh1994 »

joshmaker wrote: post #95
jnxx wrote: post #93

Earlier quoted context omitted.

I have been using Python since 25 years, and never needed that one.

In 25 years you’ve never once created a list with more than one type of object in it?

Huh, thinking about it I haven't in 9 years either

Re: Codon: A high-performance Python-like compiler using LLVM

#154

Post by arshajii »

Thanks a lot for all the comments and feedback! Wanted to add a couple points/clarifications:

- Codon is a completely standalone (from CPython) compiler that was started with the goal of statically compiling as much Python code as possible, particularly for scientific computing use cases. We're working on closing the gap further both in what we can statically compile, and by automatically falling back to CPython in cases we can't handle. Some of the examples brought up here are actually in the process of being supported via e.g. union types, which we just added (https://docs.exaloop.io/codon/general/releases).

- You can actually use any plain Python library in Codon (TensorFlow, matplotlib, etc.) — see https://docs.exaloop.io/codon/interoperability/python. The library code will run through Python though, and won't be compiled by Codon. (We are working on a Codon-native NumPy implementation with NumPy-specific compiler optimizations, and might do the same for other popular libraries.)

- We already use Codon and its compiler/DSL framework to build quite a few high-performance scientific DSLs. For example, Seq for bioinformatics (the original motivation for Codon), and others are coming out soon.

Hope you're able to give Codon a try and looking forward to further feedback and suggestions!

Re: Codon: A high-performance Python-like compiler using LLVM

#155

Post by cjohnson318 »

mrfox321 wrote: post #128
cjohnson318 wrote: post #96

Earlier quoted context omitted.

Maybe a list with ThingObject or None, but my lists are usually homogenous.

thats just homogenous Sequence[Optional[T]], though.

Yeah, that's a better way of putting it :) 90% of the time, I'm something like Sequence[T], but I'm sure I've used Sequence[Optional[T]] a couple of times. I mean, I could drive donuts in the Piggly Wiggly parking lot at 3a, I just don't, and the same for heterogenous lists.

Re: Codon: A high-performance Python-like compiler using LLVM

#157

Post by thayne »

bb88 wrote: post #142
mytherin wrote: post #117

Earlier quoted context omitted.

It is possible to JIT compile Python just fine. There are projects like PyPy that have been doing this for a long time [1]. The reason these alternative projects never take off is because many of Python's most used libraries are written against CPython's C API. This API is giant, and exposes all of the nitty gritty implementation details of the CPython interpreter. As a result, changing anything significant about the…

To be fair the 2 -> 3 upgrade path was terrible. And there wasn't a killer feature in 3 which was terrible. And the tooling around the upgrade was terrible. Basically the python devs completely botched it -- which was terrible. So one thing of golang that is nice is that go 1.19 compiler will compile go 1.1 just fine, and people can iterate from 1.1 to 1.19 in their own time -- or not if they choose not to. It would…

this hypothetical 3 -> 4 upgrade would run into a lot of the same issues.

Presumably the killer feature here is that it would be faster. Or at least have the potential to be faster because of less constraints on the c API. But for a lot of python applications, speed isn't all that important. After all, if it really needs to be fast, you probably aren't doing it in python (unless python is just a thin wrapper around a lot of c code like numpy).

And for changes to the C API, it would probably much, much harder, maybe even impossible to automate migrating libraries to the new API. The only way I could see this working well is if you had some kind of shim library between the old API and the new API, but that adds constraints on how much you can change the API, and might add additional overhead.

Re: Codon: A high-performance Python-like compiler using LLVM

#158

Post by travisoliphant »

amelius wrote: post #120
dr_zoidberg wrote: post #29

Earlier quoted context omitted.

I got a massive jump in performance when moving from Python 3.8 to 3.10 (over some function call optimizations I think, based on the project). And 3.11 got even better (up to 50% faster on special cases, and 10~15% on average) with respect to 3.10. Python 3.12 is already getting even more speedups and a there's a lot more down the road[0]. But Python core developers value keeping "not breaking anyones code" (Python 3…

Interesting. What is the status of the GIL these days?

There is a branch of the 3.9 release that removed the GIL created by Sam Gross that you can read about here: https://gavincyi.github.io/2022-10-03-does-sam-gross-nogil-c...

There is some work to bring it up to 3.12 and some resistance to merge it into 3.X because of the impact on extension modules (they all have to be recompiled and in some cases changed a bit).

If you are interested in it, reach out to Sam. He has done a pretty impressive piece of engineering work.

Re: Codon: A high-performance Python-like compiler using LLVM

#159

Post by agumonkey »

mrfox321 wrote: post #128
cjohnson318 wrote: post #96

Earlier quoted context omitted.

Maybe a list with ThingObject or None, but my lists are usually homogenous.

thats just homogenous Sequence[Optional[T]], though.

And everything is a subtype of Any. The usual 'static typing' of dynlangs.

Re: Codon: A high-performance Python-like compiler using LLVM

#160

Post by victoryhb »

arshajii wrote: post #154

Thanks a lot for all the comments and feedback! Wanted to add a couple points/clarifications: - Codon is a completely standalone (from CPython) compiler that was started with the goal of statically compiling as much Python code as possible, particularly for scientific computing use cases. We're working on closing the gap further both in what we can statically compile, and by automatically falling back to CPython in c…

Excellent job. I can already see this being much more flexible than Numba and much more elegant/easy to use than Cython. Please keep it coming:)
Post reply on HN